Polars 2.0: streaming engine by default and the promised “boring” upgrade

The team behind Polars, a data-processing library for Rust and Python, has unveiled the first release candidate of version 2.0. The final release is expected within the coming weeks. Ritchie Vink, the author of the announcement, stresses that the goal of the update is not to add flashy features but, on the contrary, to make the transition as “boring” as possible. The major version bump was needed to shed old architectural decisions and change the defaults to more sensible ones.

Streaming engine by default

The headline change: calling collect on a LazyFrame now runs on the streaming engine by default. The developers promise a noticeable reduction in memory usage and a speedup for most queries — on average roughly fivefold.

The new behavior comes with a caveat that is precisely why a major version bump was required: the streaming engine does not guarantee row order for operations like join, group_by, and unpivot. If order matters, it must be requested explicitly via maintain_order=True. For those who want to keep the old in-memory engine, there is an affinity setting: pl.Config.set_engine_affinity("in-memory"), or the engine="in-memory" parameter for an individual query.

Strictness instead of silent errors

The second direction: Polars has become stricter and now tries to fail as early as possible, rather than twenty minutes into a pipeline run. The authors note that this is especially valuable in the era of development with AI agents: they can check a query's structure in advance via collect_schema(), without materializing the data, and get feedback faster.

The examples of the new strictness speak for themselves. The is_in operation no longer performs lossy casts to a common type: previously, comparing an integer identifier with a float could silently round a value above 2^53 and produce a false match. Horizontal concatenation now checks length and no longer pads with null on the sly — if one of the sources comes up short on rows (for example, due to a failure in an upstream task), a ShapeError is raised. To pad, you must explicitly specify how="horizontal_extend". A host of ambiguous type coercions have also been removed: a string is now parsed into a date via .str.to_date() rather than through cast.

To keep the transition from being painful, meaningful errors have been added — the new AttributeRemovedError and ArgumentRemovedError exceptions tell you what to replace a deprecated method or parameter with.

What's next

Vink assures us that the 2.x branch will be significantly better than 1.x, but new features aren't tied to a major version bump — they are rolled out as they're ready. In the works, among other things: full out-of-core support for the streaming engine, a new design for IO plugins, the “fastest” reader for S3, expanded SQL support, a cost-based planner, and dropping mmap in favor of fully asynchronous pipelines.

You can try the candidate with the command pip install polars==2.0rc1.

Source: Hacker News