Clojure gets DuckDB: joining 1.4B rows on a laptop in 2.5 seconds

The tech.ml.dataset (TMD) library gives Clojure an in-memory, column-major engine for functional data science, but it hits a wall once datasets outgrow RAM — think 100GB CSVs with relational structure. The usual escape hatches are unappealing: sampling and filtering lose data, while Spark clusters add operational overhead, and bridging to Postgres over JDBC forces a slow, non-batched row-to-column conversion. The authors argue that with today’s large local disks and fast chips, none of that should be necessary.

Their answer is tmducken, a binding that wires DuckDB’s vectorized SQL engine into TMD. Early versions (2021) were limited — results came back all at once and had to fit in memory, and there was no fast append path — but DuckDB’s C interface has since gained batched inserts and streaming query results. The demo is the selling point: a 50GB CSV of 400 million transaction rows loads into DuckDB in under two minutes and compresses to 18GB with automatic indexing. A count returns in ~10ms, an inner join producing 1.4 billion rows finishes in 2.5 seconds, and a grouped color-by-month aggregation lands in about one second — all on a single laptop.

The deeper point is architectural. Because query results stream back in chunks, Clojure code can reduce over arbitrarily large result sets without exhausting memory, mixing SQL where it fits and TMD processing where it doesn’t. A zero-copy pathway (:reduce-type :zero-copy-imm) pushes memory use even lower when no chunk needs to escape the reducing function. For teams doing analytics, it’s a concrete case that embedded, single-node tooling now covers workloads that once seemed to demand a distributed cluster.