Polars 2.0 Makes the Streaming Engine the Default, Bets on Stricter, Fail-Fast Behavior
Polars is shipping its first 2.0 release candidate, and the headline change is that LazyFrame queries now run on the streaming engine by default when you call collect. The team estimates this yields roughly 5x better performance along with sizable memory reductions for typical workloads. The tradeoff is that streaming no longer guarantees observable row order for operations like joins, group-bys, and unpivots — the reason a major version bump was warranted. Users who depend on ordering can opt back in per-operation with maintain_order, or pin the old in-memory engine either process-wide via engine affinity or per-query.
The other theme is deliberate strictness: Polars now prefers to fail up front rather than silently produce wrong results deep into a pipeline. Concrete examples include is_in refusing lossy int-to-float coercion (which previously caused false positives above 2^53), horizontal concat raising on mismatched row counts instead of null-padding, and the removal of ambiguous casts in favor of dedicated parsers like str.to_date and cat.to. Removed methods and arguments now throw new typed exceptions (AttributeRemovedError, ArgumentRemovedError) that point to the replacement API. Notably, the team frames this fail-fast posture as increasingly valuable for AI-driven development, since agents can validate query structure early with collect_schema and iterate without materializing data.
The maintainers are framing 2.0 as intentionally ‘boring’ — better defaults and a cleaner API rather than a feature dump, since new features ship continuously in minor releases regardless of version. Much of the removed functionality has been deprecated for a while, so up-to-date pipelines should largely be unaffected. Looking ahead, they preview out-of-core streaming, a new IO-plugin design, a claimed fastest-in-class S3 reader, broader SQL coverage, a cost-based planner with join reordering, and the removal of mmap to make pipelines fully async. The RC is available via pip install polars==2.0rc1.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.