Polars 2.0 Collects on Streaming. Your Join Order Did Not

The Polars 2.0 release post is the switch. Calling collect on a LazyFrame now defaults to the streaming engine. Memory and performance get better on most queries, they say. The major version exists because streaming does not guarantee row order for operations that do not need it: join, group_by, unpivot, and friends. If you need the old observable order, you opt in with maintain_order. Out-of-core spill is on. It starts around 80% of RAM. Default disk budget is 64 GB. Sort, window functions, and many expressions can spill. Joins and group-bys are later.

The upgrade guide is the part that will bite a quiet pipeline. engine="auto" still looks friendly. For lazy queries it now resolves to streaming. Eager DataFrame work is unchanged. sink_ was already streaming. explain() and show_graph() only draw a streaming plan if you pass engine="streaming" yourself, so your diagrams can lie while production does not.

Andy’s Medium note dated October 13 walks the landmines: LazyFrame.profile() is gone, the DataFrame Interchange Protocol is gone (Seaborn’s fix is df.to_pandas()), lazy SQL is not validated when you build it, only at collect(). Call collect_schema() if you liked early errors.

We already covered the 2.0 RC making streaming the default. DuckDB still reading one month of Parquet is the other engine in the room. Python 3.15.0 after RC3 is the interpreter, not the frame. This week is the final default plus a CSV schema change that swaps columns if you were matching by position.

collect() is not the function you tested in 1.x

If your tests call collect() and then head(10), you were depending on incidental order. The release post says joins are easy to miss because the query does not look order-sensitive. The upgrade guide’s left-join example is the canonical break: left rows no longer come back in left order unless you pass maintain_order="left". Unpivot gets the same warning. Sort explicitly, or stop pretending head is a business rule.

A golden CSV you diff in CI is a contract on row order. Streaming will fail that contract without raising. That is worse than an exception. Andy’s suggested path is ugly and correct: pin 1.44.2, turn deprecations into errors, run the pipeline on a fixed input, save Parquet, upgrade, assert_frame_equal. A clean 1.44.2 run does not prove a clean 2.0, because the streaming default changes results, not call sites.

Eager frames are the escape hatch. If you already .collect() into a small frame and then do Python loops, you are not on the new engine for those loops. If the whole ETL is lazy until a single collect() at the end, you are.

sink_ writers were streaming already. Do not use a successful sink_parquet as proof that collect() order is stable. Different doors.

To restore the previous engine, follow the upgrade guide’s explicit engine argument. Do it per query you have not golden-tested, not as a global panic forever. The point of 2.0 is the new default. Pinning in-memory for everything is how you skip the release and keep the RAM bill.

80% RAM and 64 GB is a disk you have to own

Spill starts near 80% of RAM, and the release post says that may need tuning. Default disk budget is 64 GB. Sort and windows can use it now. Joins and group-bys are on the roadmap, which means a join-heavy job can still blow RAM in 2.0 while a sort spills politely.

64 GB is not a suggestion to leave on a laptop SSD that is already 90% full. It is a budget. If the job runs in a container with a 10 GB empty tmp, the default is a fiction. Set the spill path and the budget to values the box can survive, or you will debug a mysterious I/O stall that is actually a full volume.

Casual practitioners, Polars says, get more resilience on high-memory workloads. True if the operator knows spill happened. False if a dashboard just gets slower on Thursdays. Log engine, spill, and row counts. If you cannot tell whether a query spilled, you cannot tell whether 2.0 helped.

The release post’s benchmark aside: default Polars was fastest on all but one of their tests. At 192 threads they see constant overhead that hurts small queries. Limited to 32 cores, they say it is competitive or winning everywhere, cause diagnosed, fix hoped for in the next release. They published a repo. If you run tiny frames on a giant box, do not file “2.0 is slow” until you cap threads.

CSV schemas now believe the header

The upgrade guide’s scan_csv example is the silent swap. Before, a user schema {"b": String, "a": String} on a file headed a,b could attach types positionally and print the values under the wrong names. After, the schema matches by column name and the file’s column order is respected, so a,b with values A,B shows A under a and B under b. extra_columns and missing_columns exist. Literal default None behaves as "raise".

If you wrote a schema dict in “the order of the file” without using the names correctly, 2.0 will either raise or relabel. Raising is the gift. Relabeling is the incident. Open every schema= argument on read_csv / scan_csv and check names, not positions.

Avro read/write marked unstable, redundant strict=True on horizontal concat/union deprecated, cut() / qcut() deprecated for bin_ methods: those are the upgrade-guide extras. They fail at call sites. The streaming order change does not. Prioritize the silent one.

What disappeared: profile, interchange, early SQL errors

Andy: LazyFrame.profile() is gone. If your runbook starts with .profile(), rewrite it against whatever 2.0 documents for explain, or against OS tools. Do not cargo-cult a method that will AttributeError in production.

DataFrame Interchange Protocol is gone. Seaborn and anyone else who talked to Polars through that protocol now want to_pandas() at the boundary. That is a copy. Budget it. The pandas-versus-Polars-versus-DuckDB comparison that landed this week still describes Polars as lazy-plus-eager with a medium learning curve and a convert-at-the-ML-boundary story. 2.0 does not make Seaborn native.

Lazy SQL no longer validates when you assemble the string. Errors move to collect(). collect_schema() is the new early door: types and schema mismatches without materializing. The release post pushes this for agents as well as humans. If you generate SQL, call collect_schema() in CI. If you wait for collect(), you wait for the cluster.

Map is a new datatype in the release post. If you do not have Map columns, ignore it this week. If you do, read the docs before you assume JSON exploded the way it did in 1.x.

A migration hour that is not a rewrite

1. Inventory collect() on LazyFrames. Anything that head, write_csv for humans, or diffs against yesterday needs an explicit sort or maintain_order.
2. Run 1.44.2 with deprecations as errors. Fix call sites. Then golden Parquet.
3. Grep schema= on CSV. Names, not order.
4. Grep .profile( and interchange uses. Delete or replace.
5. Decide spill: disk path, 64 GB or less, container mounts.
6. Leave DuckDB jobs on DuckDB. 2.0 is not a conversion holiday. If SQL is the interface and spill-on-joins is the requirement today, DuckDB still owns that sentence until Polars finishes join OOC.
7. Cap threads on tiny frames if you are on a 192-thread box and the microbenchmark looks sad.

Do not upgrade a notebook and a production DAG in the same afternoon. The notebook will hide order bugs with a pretty plot. The DAG will ship a scrambled top-10.

Python 3.15’s data-science notes this week are a separate upgrade. Mixing a new interpreter and a new DataFrame default is how you spend the weekend attributing a NaN. One moving part.

Where DuckDB and pandas still sit

Techietory’s comparison this week still draws the old triangle. Pandas: eager Python DataFrame, mostly single-threaded, excellent ML ecosystem, not built for larger-than-RAM. Polars: expressions, lazy plus eager, native threads, streaming for many queries, convert at the ML boundary. DuckDB: SQL plus a Python relational API, vectorized, strong spilling, full cost-based optimizer. 2.0 moves Polars toward DuckDB’s “this might not fit in RAM” story for sorts and windows. It does not give you DuckDB’s join spill today. The release post is explicit that join and group-by OOC are later.

If your job is scan_parquet → filter → join → group_by → collect(engine="streaming"), you are in the example Techietory already printed, and you are in the order-risk zone the upgrade guide drew. Call .explain() before .collect() if you want to see predicate pushdown. Then decide whether the join needs maintain_order. If the consumer is a model that shuffles anyway, drop the flag. If the consumer is a finance CSV a human reads top to bottom, keep it.

Pandas 3 notes in the adjacent CodeCut 3.15 roundup are a different migration. Copy-on-write and Arrow-native frames will steal the weekend you thought you had for Polars. Do not stack them. Teams that “just pip install -U” both will not know which library reordered the rows.

Agents writing Polars is why the release post talks about collect_schema(). An agent that only learns from a successful collect() will happily emit a join that compiles, streams, and returns a permutation. Put collect_schema() in the tool wrapper. Put a golden Parquet in the eval. Otherwise you will ship the permutation with a commit message that says “upgrade polars.”

Thread cap is the small-data footnote. If your CI runs a 2,000-row fixture on a 192-vCPU runner and 2.0 looks slower than 1.44, that is the overhead the release post already admitted. Pin POLARS_MAX_THREADS=8 on that job before you file a regression. Production jobs that actually need the cores can keep them. Mixing the two in one environment variable is how the microbenchmark poisons the warehouse.

A second silent failure: explain() still showing an in-memory plan while collect() streams. The upgrade guide says you only get the streaming diagram when you pass the engine flag. Someone will paste an explain into Slack, declare the upgrade safe, and ship a permutation. Require the engine flag in the review checklist, or require the golden Parquet. One of those. Not a thumbs-up on a screenshot.

The RC article was the warning. The final is the default that will scramble a join you did not look at. maintain_order="left" is not nostalgia. It is the flag that makes 2.0 honest about what streaming stopped promising.