Why DuckDB Is Fast, Part 1: Living In-Process and Reading Columns

DuckDB has grown from a 2019 CWI Amsterdam research project into one of the most widely deployed analytical databases around, turning up in notebooks, ETL jobs, CI runners, embedded SaaS analytics, and commercial products from MotherDuck, Hex, Fivetran, Rill, and others. Its appeal is partly ergonomic — a sub-20MB single binary with no dependencies that treats a directory of Parquet, CSV, or JSON files as a queryable database — and partly raw speed, where a single node routinely competes with clusters costing millions per year. This first installment of a three-part teardown traces a query from arrival to the point just before execution.

The biggest architectural win is that DuckDB is a library, not a server. Traditional warehouses like Snowflake or Postgres ship results over the network through row-by-row client protocols such as ODBC and JDBC, and a 2017 paper by DuckDB’s creators showed that this serialization step often costs more than computing the answer itself — gigabit links cap throughput around 125 MB/s, and per-value function calls balloon into hundreds of millions of tiny copies on large result sets. By loading directly into the host process, DuckDB skips the wire entirely. Through replacement scans it can query a pandas, NumPy, or Arrow object in place, frequently reading the caller’s existing buffers with zero copies when the physical layouts align (Arrow being the cleanest case).

Once SQL enters the engine, it runs the familiar parse–bind–plan–optimize pipeline. A fork of the Postgres parser produces an abstract syntax tree — explaining DuckDB’s familiar dialect — which the binder resolves against the catalog, attaching concrete tables, columns, types, and functions while catching unresolved names and type mismatches. The result is a fully typed tree the optimizer and physical planner can rewrite and map to executable operators. Actual query execution, along with vectorized processing and morsel-driven parallelism, is deferred to Part 2.