DuckLake Puts the Catalog in SQL. Iceberg Can Keep the JSON

Talk Python episode 562 is not a launch-week blog post. It is Pedro Holanda and Guillermo Sanchez Dionis walking through DuckLake: an open table format that keeps the data in Parquet and the metadata in a real SQL database. daily.dev’s write-up of the episode is the short version. The episode page is the long one.

We already said pandas does not need to go extinct and that Polars 2.0 RC made streaming the default. This is not another dataframe API. It is the catalog argument. Iceberg and Delta store table state in JSON and manifests. DuckLake stores it in tables you can SELECT.

If your “lakehouse” is 40 parquet files and a wish, you do not need Snowflake to hold the wish.

Three parts, one of which you already have

The architecture in the interview is storage, catalog, compute. Storage is object storage plus Parquet, the part everyone already copies into talks. Compute is DuckDB or anything else that can read the files. The argument is the middle: the catalog should be a database.

Pedro’s complaint about Iceberg, in the transcript chunks, is practical. Metadata lives in JSON, Avro, maybe another format. To read the table of contents you need several readers and a walk through files that exist only so you can find files. DuckLake puts that contents list in SQL. Anyone who can describe a schema can query the catalog the same way they query the data.

Catalog choices, as summarized on daily.dev: DuckDB for a quick start, Postgres for production, and a newer Quack client-server protocol when writers contend. The number they cite is the one to keep: about 200 transactions per second on Quack versus about 5 on Postgres for that workload. I have not re-run their bench. Treat 200 vs 5 as the podcast’s exhibit. The shape is still useful. A catalog that is “just Postgres” is not free under write storms.

Data inlining is the other design note. Tiny files are the classic lake failure. DuckLake can inline small data into the catalog so you are not paying a file create for a 20-row correction. That is a table-format trick, not a vibe.

Frozen DuckLake is the pattern worth stealing

The serverless pattern they describe is almost rude in its smallness. You keep an in-process DuckDB file as the catalog. Pull it locally. Write new Parquet to S3 through it. Upload the catalog file back to S3. Read-only clients attach to that file. No Postgres. No Quack. One big nightly load.

If your pipeline is already a nightly batch, that is the whole product. You do not need a lock service for a job that runs once. You need a catalog file you can copy, and Parquet that does not move.

This is the cousin of DuckDB as in-process analytics. Same engine, now holding the table list. If you already run DuckDB against files in a bucket, you have been doing a sloppy lake. DuckLake is the sloppy lake with a schema.

Do not turn a nightly load into a streaming platform because a keynote said “lakehouse.” Frozen is a compliment.

QuestDB’s version: register, do not rewrite

QuestDB’s blog is the first vendor note I trust this week because it is a warning, not a migration fantasy. They already tier cold partitions to a bucket in Hive-style paths:

fx_trades/year=2026/month=06/day=19/hour=08/data.parquet

DuckLake registers those files with ducklake_add_data_files. The Parquet is not rewritten. Registration is a reference. Then they tell you to turn compaction off:

CALL lake.set_option('auto_compact', false);

If you leave compaction on, DuckLake’s maintenance can rewrite or delete files that QuestDB still thinks it owns. That sentence is the article. Zero-copy is zero-copy until a helper job “helps.”

Types still bite. QuestDB has microsecond and nanosecond timestamps. DuckDB has no nanosecond-with-time-zone type, so a nanosecond column shows up as microseconds in the DuckDB view. The precision is still in the Parquet. Iceberg v3 or pyarrow can see it. DuckDB’s window is coarser. If you aggregate by nanosecond events in DuckDB, you are lying to yourself. Query the file with something that knows ns, or accept us.

Their example query is ordinary SQL over lake.fx_trades with a timestamp range. No hand-written DDL: read_parquet infers types, CREATE TABLE ... AS SELECT ... LIMIT 0 makes an empty table, then you add the files. That is the onboarding. If your team cannot read that script, they cannot operate a lakehouse of any brand.

What this is not

It is not a reason to delete Iceberg if you already have a working Glue catalog and Trino. Catalog conversion is how data teams spend a year. DuckLake is interesting if you are about to invent a third metadata layer because JSON manifests feel like a maze. It is also interesting if your lake is small enough that a DuckDB file is the catalog.

It is not a Spark replacement story. We already did the 94 percent of tables under 100GB. DuckLake does not make a 20GB join require Kubernetes. It also does not make a 20GB join require Iceberg.

It is not Polars. Polars will still scan Parquet. DuckDB will still scan Parquet. The catalog is the part that tells you which Parquet. The three-tool chooser still holds. Add a fourth question: where does the table list live?

Snowflake and Databricks, in Pedro’s aside, are a service on top of someone else’s disks, priced like a service. Fine if you want the service. If you wanted SQL over files you already paid to store, a catalog database is the missing object. The warehouse invoice is optional.

DuckLake 1.0 is described in the interview summary as production-ready. Production-ready in a podcast is a claim. Production-ready in your shop is a backup of the catalog file and a compaction switch you actually set.

A small Python check before you “migrate”

Install the extension. Attach a catalog. Register one partition. Run SELECT count(*). Diff it against the source system. Then stop. Do not register the whole bucket on day one.

If you use QuestDB cold storage, set auto_compact false before you do anything clever. If you use the frozen pattern, treat the DuckDB catalog file like a database backup, because it is one. If writers will collide during the day, the podcast’s own numbers say Postgres at 5 tx/s is the wrong default. Look at Quack, or stop writing during the day.

Column stores still hate single-row updates. Pedro’s products-and-prices example on the show is the old one: average price reads one column; a row store would drag quantity and name along. DuckDB is the analytics engine for that reason. DuckLake does not change the layout. It changes how you find the files that hold the columns.

A week of work, not a platform swap

Monday: pick one table you already understand. Not the 400-column monster. A facts table with a date and a grain you can count by hand.

Tuesday: dump one partition to Parquet if it is not there already. Hive-style paths if you can. QuestDB’s layout is a good picture even if you are not on QuestDB.

Wednesday: INSTALL ducklake; LOAD ducklake; Attach a local catalog. Create the empty table from read_parquet plus LIMIT 0. Add the files. SELECT count(*). If the count does not match the source, stop. You have a type problem or a glob problem. You do not have a lakehouse.

Thursday: if the files belong to another system, set auto_compact false and leave it false. Write that line in the runbook. Compaction is how zero-copy becomes a surprise rewrite.

Friday: if the job is nightly, try the frozen pattern. Catalog file in the bucket. One writer. Read-only attaches for the analysts who currently download CSVs. If two writers need the same catalog during the day, you are past the frozen pattern. Then the podcast’s 200 vs 5 matters, and you should read the Quack notes instead of pretending Postgres is “good enough” because it is familiar.

The Iceberg comparison will come up in Slack. Answer it without a conversion project. Iceberg is what you have if Trino and a cloud catalog already work. DuckLake is what you try if the catalog is a wiki page of prefixes. You can run both. Parquet does not care.

Pedro’s columnar primer on the show is worth repeating for the intern who only knows read_csv. Average price should not pick up quantity and name. DuckDB already knew that. DuckLake does not make DuckDB smarter. It makes the file list queryable, which is how you stop attaching the wrong year.

daily.dev also flags DuckDB 2.0 talk in the same interview. I am not writing a 2.0 preview from a podcast mention. When 2.0 ships, it will get its own piece. This piece is the catalog.

QuestDB’s timestamp warning is the one I would tape to the monitor. Nanoseconds in Parquet, microseconds in the DuckDB view, full precision still on disk. If your join key is a nanosecond event id dressed as a timestamp, DuckLake will look wrong and the file will be fine. Check types before you write a Slack message about “data loss.”

The Talk Python aside about Snowflake and Databricks charging for a service layer is not a boycott. It is a reminder that you already pay AWS for the bits. A SQL catalog is how you stop paying a second time to be told which bits you own. If you like the service, keep it. If you wanted a table list, you do not need a keynote. You need a catalog file you can copy, a compaction switch you can name, and a count query that matches the source. That is the week. Everything else is a platform conversation you can have after the count matches. If the count never matches, you do not have a catalog problem. You have a file problem, and Iceberg would have lied to you the same way. Count first. Argue later. Always. In that order too.

I would not rewrite a working Iceberg lake this week. I would use DuckLake for the lake that is currently a spreadsheet of S3 paths. SQL is a better spreadsheet. JSON manifests were never a good one.