Pandas Does Not Need to Go Extinct. Your Spark Cluster Might

Eddie posted “Pandas Should Go Extinct” after a Latency Conference talk. GeekNews picked it up on September 12. The headline is doing what headlines do. The argument underneath is older and better: pandas’ eager, row-unfriendly habits push teams into distributed query engines before the data is big enough to deserve them.

We already ran the three-tool chooser and Polars 2.0 RC making streaming the default. This week is not another API tour. It is a specific claim from a talk, plus the off-ramp that Wes McKinney accidentally built when he left pandas for Arrow.

What the post actually argues

Eddie is not mad at pandas because it is unfashionable. He is mad because it makes medium data feel like big data. You load a CSV. Every step runs now. You keep a fatter frame than you need. Then someone says the laptop is the problem and you fill in a requisition for Spark.

His motivating example is NYC taxi parquet, on the order of 3GB, asking whether cash payments dropped during the pandemic years. That is not a lakehouse. That is a file you can keep on a disk. Polars and DuckDB can scan it lazily, project two columns, aggregate, and go home. Pandas can do the job too if you are careful. Careful is the part that does not survive a tutorial.

Polars and DuckDB speak Arrow natively. Pandas can, since 2.0 in April 2023, if you pass dtype_backend="pyarrow" when you build the frame. If you do not, you stay on the old BlockManager and you copy when you hand the frame over. The “free” interop people screenshot is free only after that flag.

DuckDB’s trick in the post is the obvious one that still surprises pandas shops: SQL over the file, and SQL over a Python object you already have in memory. .explain() is there if you want to see the scan. Polars has the same habit with lazy scan_csv / scan_parquet and explain. pandas will read the whole thing first and then groupby, which is why the laptop fans spin.

The 94 percent table

The Spanish write-up of the same talk, on Ecosistema Startup, repeats an Amazon figure: 94.68 percent of tables under 100GB. I have not re-run Amazon’s paper here. Treat the number as the talk’s exhibit, not as gospel. The shape is still right. Most internal “data platforms” are a pile of tables that would fit on a workstation if you stopped copying them into ten eager frames.

That is the extinction that would actually help. Not pandas the library. Spark-for-20GB. Kubernetes-for-a-join. A streaming job that exists because read_csv was the first function someone knew.

Polars 2.0 RC already moved the default toward streaming. If your team read our RC notes and then did nothing, Eddie’s post is the social version of the same nudge. DuckDB remains the SQL door for people who think in SELECT.

Comments that are more honest than the title

GeekNews translated the Arrow section and then the comment thread did the useful work. One note: pandas is still enough for a huge number of people, including the ones who would otherwise live in Excel. Another: moving a mid-size pipeline from pyarrow-backed pandas to DuckDB or Polars is not a rewrite. A third, from someone who actually teaches this, is that EDA still wants pandas because matplotlib and the muscle memory are there, and Polars classes do not fill.

Eddie himself said the real trigger was dropping pandas on an Excel ingest, 2 minutes to 2 seconds, with a TODO to rewrite the rest of the service still sitting in the repo. That is the adult version of “extinct.” You remove it from the hot path. You do not hold a funeral for df.head().

If your day is plotting a 200MB extract, pandas is not your villain. If your day is a 3GB parquet scan that keeps becoming a cluster, pandas is the tutorial you outgrew.

A migration that does not require a manifesto

Do not rewrite the warehouse this week. Do this in order.

One: measure the table. If it is under a few tens of GB and the queries are scans plus groupby, you are in laptop-or-big-workstation land. Write that down so the next architecture review cannot pretend otherwise.

Two: read with DuckDB or Polars. duckdb.read_parquet / scan_parquet. Project columns you need. Aggregate in the engine. Export to pandas only for the plotting layer, and export Arrow-backed if you must export.

Three: if the rest of the code is pandas, set dtype_backend="pyarrow" on new frames and on read_* where the option exists. We already wrote the memory piece. The flag is the whole trick. Skipping it is how people “try Arrow” and then say it did nothing.

Four: leave pandas where it is the interface. Vendor files, sklearn glue, a colleague’s notebook. Extinction rhetoric makes those people stop listening. Interop is how you get the win without a culture war.

Five: if you already tried Polars 2.0 RC, go re-read the streaming default notes before you copy Eddie’s taxi snippet as a new religion. Streaming does not fix a bad join key. It fixes the habit of pulling the whole frame up to look at it.

What I would not do

I would not ban pandas in a style guide. I would not convert every notebook on a Friday. I would not stand up Spark because a blog said a library should die. I would not ignore the fact that Arrow exists because the pandas author wanted a better memory format.

I would also not pretend DuckDB is “just SQL” with no cost. It is an engine. You can still write a cross join and cry. The difference is that the plan is visible and the scan can stay on disk.

Latency Conference talks are allowed to be sharp. Production calendars are not. If the post made your team defensive, skip the title and steal the taxi example. Time the same aggregation in pandas eager, Polars lazy, and DuckDB SQL. Put the three numbers in the PR. That is more persuasive than extinction.

Where this sits on this site

Chooser post for people who still need a map: pandas vs Polars vs DuckDB. RC post for people who are already on Polars. Arrow-backed pandas for people who cannot leave yet. This post is the weekly reminder that distributed compute is not a personality, and that a 3GB parquet file is allowed to stay a 3GB parquet file.

Pandas does not have to go extinct. The reflex to cluster-size a groupby does. Arrow is the boring way out. Use it.

Time the taxi file before you quote the talk

Eddie’s NYC taxi example is the right size because it is annoying, not because it is romantic. Three gigabytes of parquet is enough to make an eager laptop unhappy and not enough to justify a cluster ticket. Reproduce it on one machine.

Same question: cash share by month, 2019 through 2022. Same columns only. No df.head() tourism in the middle. Pandas: read_parquet with a column list if you can, then groupby. Polars: scan_parquet, group_by, collect with streaming if you are on the 2.0 RC path. DuckDB: read_parquet in SQL, GROUP BY, done.

Write down wall time, peak RSS, and whether you touched swap. If pandas is fine, congratulations, you did not need a talk. If pandas blows RAM and Polars does not, you have a slide for the next person who says “we should just Spark it.” If DuckDB wins because the SQL is shorter, that is also a result. Shorter is a feature when the alternative is a 40-line apply.

Do not paste a full notebook into production. Paste the numbers into the PR that would have created a cluster.

Teaching and EDA are allowed to be messy

The GeekNews comment about matplotlib is the one pandas fans should keep. Exploratory work is a conversation with a frame. Polars can do it. Many people will not, and shaming them does not make the warehouse faster. Keep pandas in the notebook kernel. Keep it off the scheduled job that reads the same 3GB every morning.

If you teach, stop opening with read_csv of an unbounded file. Open with a parquet scan and a projection. Students copy the first pattern they see. That pattern is how you get the Spark reflex in year two.

Apache Arrow as the quiet standard is the through-line. Pandas 2.0 support, Polars native, DuckDB native. The talk’s “extinct” line is a hook. The memory format is the plot. If you only remember one flag, remember dtype_backend="pyarrow". If you only remember one habit, remember not to cluster a table that still fits on a disk.

One more check before anyone files a platform ticket: count distinct tables in the job, not rows in the loudest one. A pipeline of twenty 2GB extracts is still not a lake. It is twenty files and a scheduler. DuckDB and Polars can do that on a box with RAM and disks. Spark can too, at a price you should be able to say out loud.