Polars 2.0 RC Makes Streaming the Default. Check Your Joins.

Ritchie Vink posted the Polars 2.0 release candidate on September 2. Install line: pip install polars==2.0rc1. Final 2.0 is “in the following weeks.” He wants the bump to feel boring. The reason it is a major version is not a feature dump. It is defaults. LazyFrame.collect() now runs on the streaming engine. Joins, group_by, and unpivot no longer promise row order unless you ask. A pile of silent casts now raise.

We published a Polars 2.0 API preview in April. Treat that as atmosphere. This RC is the thing you can pip-install. If your nightly CI still says 1.x, the interesting question is not “is Polars faster than pandas.” It is “which of my tests assumed the old collect().”

What actually changed in the default path

Streaming-as-default is the load-bearing change. Vink says they expect the streaming engine to be on the order of 5x faster in aggregate, with much lower memory on typical queries. That is the sales pitch. The contract change is the part that will page you.

Streaming does not guarantee row order for join, group_by, unpivot, and a few friends. If a downstream step assumes “the left table’s row 0 is still row 0,” 1.x hid that assumption. 2.0 will not. Opt in per operation with maintain_order=True (or maintain_order="left" on a join), or pin the old engine:

import polars as pl

lf = pl.LazyFrame({"k": [2, 1, 0], "v": ["a", "b", "c"]})
other = pl.LazyFrame({"k": [0, 1, 2], "r": ["x", "y", "z"]})

# 2.0: order of `k` may not match `lf`
lf.join(other, on="k", how="left").collect()

# keep left order
lf.join(other, on="k", how="left", maintain_order="left").collect()

# process-wide rollback
pl.Config.set_engine_affinity("in-memory")

# one-query rollback
lf.join(other, on="k", how="left").collect(engine="in-memory")

If you have tests that assert frame.equals(golden) including row order after a join, they are the migration. If you only check aggregates, you may notice nothing except a smaller RSS.

Finding those tests is a grep problem. Search for .join( followed later by .equals(, assert_frame_equal, or a snapshot file. Search for collect() without maintain_order on frames that then get head(1) or row(0) as “the first matching customer.” Time-series jobs that shift() after a join are another tell. None of this is theoretical. Streaming engines reorder because they pipeline partitions. That is the point.

A cheap smoke test: take a production LazyFrame, collect with default 2.0, collect with engine="in-memory", and compare on a hash of sorted rows plus a hash of unsorted rows. If sorted matches and unsorted does not, you had an order dependency. Decide whether the business cares. Invoice lines usually do not. “First click in the session” usually does.

The migration guide lives at docs.pola.rs/releases/upgrade/2/. Read it before you rewrite folklore in Slack.

Strictness that used to be a footgun

Vink’s second theme is fail-fast. Implicit behavior on type mismatches is opt-in. He also points at agents: collect_schema() resolves types without materializing rows, so a coding agent can see a schema error in milliseconds instead of after a 20-minute collect. That is a real API, not a vibe. If you are wiring Polars into an agent loop, call collect_schema() before collect().

Three examples from the RC post are worth reproducing in your own tests.

is_in will no longer quietly upcast to a lossy supertype. A JSON export that turned big integers into floats used to coerce an Int64 user id through float64, round it at 2^53, and match the wrong account. 2.0 raises InvalidOperationError and tells you is_in cannot check Int64 values in List(Float64). Cast on purpose.

Horizontal concat used to pad with nulls when heights differed. That is how a failed upstream day silently becomes a null fraud-flag column on day 5. 2.0 raises ShapeError in strict mode. If you wanted padding, you now pass how="horizontal_extend".

String-to-date via .cast(pl.Date) is gone. Use .str.to_date() / .str.to_datetime() with a format. Integer-to-enum .cast is gone. Use .cat.to(...) and .cat.physical(). One obvious parse path beats three.

Removed names get typed exceptions: AttributeRemovedError, ArgumentRemovedError. melt points you at unpivot with index / on. join_nulls points you at nulls_equal. If your pipeline still used 1.24-era kwargs and you ignored deprecation warnings, 2.0 is when they become errors. That is the deal of a major bump.

Why this landed next to a pandas horror story

The same week, a Python in Plain English piece by Saad Ahmed made the rounds: 47 logistics pipelines audited, 43 using pandas for everything, including jobs pandas should never have been asked to do. One daily job: 4 hours, 32 GB RAM, crash every other day. Rewrite: 80 lines of Polars plus the standard library, 8 minutes, 2 GB, no crash in three weeks. He claims $44,000 a year saved for that client. Treat the dollar figure as one freelancer’s invoice math. Treat the shape as familiar.

We already have a pandas vs Polars vs DuckDB chooser and a Polars lazy-evaluation pipeline piece. The RC is the moment lazy collect stops being an advanced trick. If you were on LazyFrame and calling collect in 1.x, you were already in the streaming neighborhood. In 2.0 you are in it unless you opt out. Eager pandas-shaped scripts that do read_csv then five copies then merge will not be saved by a version bump. They need a rewrite. Ahmed’s 43/47 ratio is the argument for doing that rewrite on the jobs that page people, not on the Jupyter notebook that built a chart once.

Library authors have a different problem: users show up with pandas, Polars, and sometimes cuDF. Narwhals is still the compatibility layer for “write once, run on several DataFrame backends.” 2.0 does not replace that. It does mean your Polars backend path now has a streaming default your pandas path will never grow. Document which engine you collect with.

A JSON-to-CSV microbenchmark from late August put Polars at 2.28 seconds total versus pandas at 9.45 seconds on the same pipeline, with Node streams in between. Tiny file, one task, vendor-shaped methodology. Directionally it matches every other “Polars wins at bulk transform, pandas wins at poking in a notebook” result we have been citing all year.

When not to stream: tiny frames, queries that need a stable row identity, and anything you are still debugging with explain(). In-memory collect is not deprecated. It is no longer the unnamed default. Set affinity in the process that serves notebooks if analysts will riot over shuffled joins. Leave streaming on in the batch jobs that were OOMing.

DuckDB is still the SQL door

Not every pipeline should become Polars expressions. Cláudio Tereso wrote on September 2 about pulling XLSX into SQL Server with DuckDB because he is a SQL person and pandas/Polars query syntax is a nightmare for him. duckdb.sql("select * from read_xlsx('orders.xlsx', sheet='Suppliers')"), join sheets as tables, then copy the relation to pandas only for to_sql. DuckDB still does not write SQL Server itself. It also does not read old .xls, only .xlsx.

That is a fine 2026 split. Polars 2.0 streaming for Python-native transforms. DuckDB SQL for people who think in joins. pandas as the last meter into a library that has not grown a Polars sink. The RC does not kill that split. It makes the Polars side less of a “remember to use lazy” tax.

Tereso’s Excel path is also a warning about file formats. DuckDB reads xlsx, not xls. Modern pandas dropped xls too. If a vendor still emails .xls, convert it before the job, do not keep a zombie converter in the pipeline. The WideWorldImporters example in that article is the right grain: join three sheets, filter quantity mismatches, group, then hand a small frame to SQLAlchemy. Do not load the whole workbook into pandas “because Excel.” That is how you get Ahmed’s 32 GB box.

If your warehouse already speaks Arrow, streaming Polars and DuckDB are closer than the brand war suggests. The RC just moved Polars’ default toward the engine DuckDB users already assumed they had.

What I would do this week

Pin the RC in a branch, not in prod. polars==2.0rc1 on a throwaway venv. Run the test suite. Sort failures into three buckets: row-order golden files, newly raised type errors, renamed kwargs.

Add maintain_order only where a test or a business rule needs it. Do not spray it on every join “to be safe.” That throws away the streaming default you just paid a major version for.

Turn on collect_schema() in any agent or codegen path. If you already validate frames with Pandera, keep it. Schema-at-plan-time and schema-at-data-time catch different lies. An agent that generates .cast(pl.Date) on a string column will now fail immediately in 2.0. That is a gift. Wire the new exception types into your retry prompt so the agent rewrites to .str.to_date() instead of looping on the same cast.

If a job is still pandas because “that is what the 2019 course used,” pick one crashing pipeline and rewrite it as LazyFrame. Measure wall time and peak RSS. If it does not move, you did not have a Polars problem. If it moves the way Ahmed’s did, you have a template: stop copying full frames, push filters into the scan, collect once.

Watch memory, not just time. Streaming’s pitch is that you do not need 32 GB to touch a file that is 8 GB on disk. If RSS does not drop, you probably collected too early or called an operation that still materializes. explain() before and after the RC is the debugging tool. The 5x number is an aggregate from Polars’ own benches. Your join-heavy graph may be 1.2x. Publish the number internally so nobody treats the blog post as a KPI.

Leave 1.x on the jobs that are quiet. Vink says 2.x will grow out-of-core streaming, a new IO plugin, a faster S3 reader, more SQL, a cost-based planner, join reordering, and dropping mmap so pipelines can be async end to end. None of that is in rc1. Do not migrate for a blog post about the future. Do migrate the jobs whose current collect() is the reason the box has 64 GB.

File issues on github.com/pola-rs/polars if a default bites a real query. That is what an RC is for.

A note on the April article

Our April 2.0 piece talked GPU via cuDF and a redesigned API as if the release had landed. The RC post is narrower and more honest: boring on purpose, streaming default, stricter errors, migration guide. If those two write-ups disagree, trust September 2 and the 2.0rc1 wheel. The Polars vs pandas migration guide is still the human-language map for teams leaving pandas. Pair it with Vink’s upgrade doc for teams already on 1.x.

Install the candidate. Read the join order. Then decide whether 2.0 is a week of test fixes or a month of “we depended on accidents.”