DuckDB v2.0 Adds Async I/O to Speed Up Queries Against Remote Storage

DuckDB is moving beyond its local-SSD roots. For most of its history the engine leaned on aggressive filter and projection pushdown to read only the data a query needed, and synchronous I/O was good enough because data sat on a fast local disk. But as DuckDB gets used for querying data lakes and now runs as a server via the Quack protocol, the common deployment has shifted to compute-storage splits like an EC2 instance reading Parquet from S3 in the same region. In that world, synchronous reads leave worker threads idle, blocked on network round-trips instead of decoding and aggregating, and queries fail to saturate available bandwidth.

The fix, landing on by default in v2.0 (slated for fall 2026 and already testable in v2.0.0-dev builds), is an asynchronous I/O pipeline for Parquet and uncompressed, seekable UTF-8 CSV, with native format and JSON support to follow. DuckDB splits threads into two pools: REGULAR workers that do the CPU-bound decoding and joins, and a much larger ASYNC pool (defaulting to 4x system threads, capped at 256) that exists to sit blocked on HTTP reads. A read-ahead queue schedules fetch tasks for upcoming row groups or CSV byte ranges before workers need them, so fetching and decoding overlap. Any worker looking for scan work tops up the queue itself, avoiding a dedicated producer thread; jobs use a shared countdown so the final fetch task unblocks decoding, and a parked scan can resume on any worker.

The main risk of prefetching is memory pressure: if the network outruns decoding, buffered data piles up and can trigger out-of-memory conditions. DuckDB addresses this with asynchronous memory governance that bounds how far read-ahead can run based on a memory budget rather than only a fixed slot count. The practical payoff is better bandwidth utilization for remote workloads, which is exactly where DuckDB’s usage has been heading.