The Case Against Pandas: Most Data Fits on One Machine

The Python DataFrame library Pandas pushes users toward expensive distributed systems like Spark, Databricks, and Snowflake long before their data actually warrants that complexity. The argument, drawn from a Latency Conference talk, is that Pandas hits a performance cliff in the tens of gigabytes—memory pressure, slow computation, an awkward API—and the reflexive response is to jump to Big Data tooling. But there’s a wide gap between where Pandas fails and where distribution is genuinely necessary, roughly around 100GB, and that gap is now well covered by fast single-machine tools: Polars (a Rust-based DataFrame library) and DuckDB (essentially SQLite for analytics).

The claim that few workloads need distributed systems rests on Amazon’s 2024 paper analyzing its own Redshift fleet. Under reasonable assumptions—1KB rows and clusters reading 8GB/s—roughly 94.7% of tables hold under 100GB and about 86.9% of queries touch 80GB or less. Even with more pessimistic row-size assumptions, most tables stay in the low-terabyte range at worst. The takeaway: you almost certainly have a Medium Data problem, not a Big Data one, and the industry’s distributed-computing default is more marketed silver bullet than engineering necessity.

The technical difference comes down to execution models, illustrated with the 1 Billion Row Challenge. Pandas evaluates eagerly and sequentially, loading the entire dataset into memory before grouping and aggregating. Polars uses lazy evaluation: scan_csv builds a query plan that only executes on collect, letting it apply database-style optimizations like predicate pushdown and projection, stream data in chunks, and parallelize across threads. DuckDB goes further by treating analytics as plain SQL over an embedded columnar engine. Both borrow decades of database optimization that Pandas never adopted, closing the gap that once justified reaching for a cluster.