PyCon JP 2026 · 広島 Hiroshima

Rediscovering
DataFrames

100× Analytics Without Leaving Pandas

再発見

sai·hak·ken — rediscovery

Auxten Wang · ClickHouse · Creator of chDB@auxten

Auxten Wang

@auxten · auxten.com

Technical Director @ ClickHouse

Creator of chDB — acquired by ClickHouse in 2024

Builds embedded databases & data tooling for Python

ex Shopee principal engineer

Everyone's first data tool

import pandas as pd
df = pd.read_csv("sales.csv")
df[df.amount > 100] \
.groupby("region").amount.sum()

In every tutorial, every notebook, every team

Muscle memory for millions of engineers

The DataFrame dialect LLMs write best

This API took 15 years
to learn everywhere.

The pandas API is an asset. The engine underneath is the problem.

限界

Then your data grew.

Memory
Error

a 10 GB DataFrame chokes a 16 GB laptop

1 / 16

cores doing the work while fifteen watch

> query

reading from S3 takes longer than the analysis itself

Your code didn't hit the ceiling. The execution model did.

The ceiling has three walls

EAGER

every step in the chain materializes a full intermediate copy

SINGLE-CORE

the GIL-era design: one thread computes, the rest idle

ALL-IN-RAM

the whole table decompresses into memory before row one is read

33 MB

Parquet on disk

→

1.5 GB

as a pandas DataFrame

×45 blow-up,
measured on my laptop.

Not fixable with another .apply() trick. It's the engine.

The usual exits all cost the same thing

Spark

a cluster to babysit, for one notebook

Polars

fast — but a new API for the whole team

DuckDB

great engine — but now you write SQL

Rewrite

months of migration, retraining, re-testing

非常口 hijōguchi — emergency exit

Every exit asks you to leave the API. What if you didn't have to?

Change one line

- import pandas as pd + import chdb.datastore as pd
df  = pd.read_parquet("events.parquet")
top = (df[df.status == 200]
        .groupby("user_id").amount.sum()
        .sort_values(ascending=False).head(10))

同じコード。
違うエンジン。
Same code.
Different engine.

The rest of the file doesn't know. 一行だけ — one line.

chDB: a rocket engine on a bicycle

# pip install chdb
import chdb
chdb.query("SELECT count()
FROM file('orders.parquet')")
# schema inferred · nothing loaded

The ClickHouse engine, in-process — like SQLite

No server, no config, no connection string

Columnar · vectorized · every core

Same engine that runs analytics at Anthropic, Cursor, Vercel, Cloudflare.

Layered, not mixed — お好み焼き o-ko-no-mi-ya-ki

コテ kote

ソース sōsu · sauce — your pandas calls

df[df.x > 1].groupby(…).sum() — untouched

麺 men · noodles — the lazy op chain

every call recorded, nothing executed

キャベツ kyabetsu · cabbage — one SQL plan

the whole chain compiles to a single query

生地 kiji · batter — the chDB engine

columnar · vectorized · all cores

鉄板 teppan · teppan — your Python process

everything cooks in-process, no server

len() · print() · for — the コテ kote flip: everything cooks at once.

The engine knocks down all three walls

A SQL COMPILER

an internal compiler rewrites your whole pandas chain into one SQL query for the ClickHouse engine

pandas → SQL → ClickHouse

EVERY CORE

pandas computes on 1 of 14 cores — the engine fills them all

STREAMING

top-10 of 10M rows keeps a 10-row heap — never the sorted table

RAM: 100% → 33%

…and each core is vectorized

one SIMD instruction processes a whole block of values

1 instruction →

per tick

that's the ×51.7 sort
and the 33% RAM.

Not a faster pandas — a different machine underneath.

Mirror of the three walls from slide 5, one by one. First wall: inside datastore sits a compiler — it records your pandas calls and rewrites as much of the chain as it can into a single SQL statement, handed to the embedded ClickHouse engine. Ten chained ops cost one execution, not ten copies; anything it cannot translate falls back to real pandas mid-pipeline, so you never lose. Single-core becomes every core: this laptop has 14; pandas computes on one, the engine's pipeline saturates all of them. All-in-RAM becomes streaming: data flows through in blocks — a top-10 sort keeps a 10-row heap instead of materializing 10 million sorted rows, which is exactly why sort is ×51.7 and why RAM sits at a third. And inside each core, execution is vectorized — SIMD processes blocks of values per instruction instead of one Python-object at a time. This is 15 years of ClickHouse engine engineering, compiled into your import line.

Where the 100× lives

chDB

×1.35

DuckDB

×1.40

Polars

×9.99

pandas

×71.18

Worst single query: pandas 94.3 s vs chDB 0.037 s — ×2,550

各駅停車 → のぞみ

ClickBench · DataFrame category · 43.77 GiB · same c6a.metal machine · relative runtime, lower is better

Same DataFrame workload. ×71 less waiting — before any tuning.

10M rows, 16 everyday ops — faster on 14 of 16

DataStore faster (14) pandas faster (2) thin bar + blue % = the RAM we use vs pandas (33% = ⅓) · linear scale
read parquet → answer · best of 2 · outputs identical 16/16 · peak RSS median: pandas 1,059 MB → DataStore 440 MB (42%)

Sort (single col)×51.733%

Sort (multi col)×39.034%

Mixed Filter+Sort×35.231%

Filter+Select+Sort×21.736%

Chain 5 filters×12.931%

Head / limit×6.847%

Filter+Sort+Head×6.647%

Multi-filter (4x)×4.937%

Filter+Sort+Select×4.241%

Filter (multi AND)×4.233%

Ultra-complex chain×4.036%

Complex pipeline×4.029%

Filter (single)×3.337%

GroupBy agg ×3×2.840%

Mask (replace)196%×8.5

Where (replace)199%×6.7

実測

thin blue bars:
⅓–½ the RAM
on every op we win.

Faster on 14 of 16 — bulk value-replace stays with pandas.

Measured on this laptop for this deck: 10M-row parquet, read-to-answer, best of 2 runs, outputs verified identical between engines on every op shown. Linear scale, no log tricks: fat bars are speed, thin blue bars are peak RAM (fresh process per op, ru_maxrss), labeled with the share of pandas' RAM we use — 33% means one third. Sorts are the blowout — top-k over 10M rows is ×35–×52 because the engine never materializes the full sorted frame. Filters and chains ×3–×13. The two red rows are one family: bulk value-replace (mask/where). pandas rewrites the column in place; we round-trip it through the engine — pandas wins those on speed and RAM, and the honest advice is: keep using pandas there. One clean concession, easy to say out loud. Full 20-op run in the repo (bench20_results.json) for anyone who checks.

Your data lives on many islands

pd.read_parquet("https://…/house_0.parquet") # the lake
pd.read_csv("crm_exports.tar :: users_*.csv") # archives
pd.read_sql("SELECT …", "postgresql://prod")  # live DBs
    

measured this week, this laptop

2,772,030 rows on S3 — counted in 3.3 s, no download
parquet ⋈ CSVs-inside-a-tar ⋈ groupby — one chain

実測

70 table functions · 72 input formats — counted from the engine

Like the Miyajima torii: a gateway standing in the water. The data never moves.

It reads what the query needs — not the file

house_0.parquet · 33 MB · 2.77M rows · 14 columns ▼ HTTP Range reads

type

price

date

town

street

…9 more columns

footer

■ fetched for avg(price) by type ■ skipped ■ footer = schema + stats, read first

3.3 s

len() on the remote file — footer + counts only, no download

2 / 14

columns fetched for the aggregation — ranged reads, no full scan

344 vs 975 MB

peak RAM, chDB streaming vs pandas on the same file

Ranged reads: you pay for the columns you touch — not the file.

Agent memory today: three databases

YOUR
AGENT

"what do I
remember?"

Vector DB

semantic recall — "which memories mean this?"

Metadata store

filters — "whose memory? which project?"

Recency cache

freshness — "what still matters today?"

3 systems · 3 hops · 3 bills

Every recall pays three round-trips — before the LLM starts.

One query does all three

SELECT text FROM memory
WHERE user_id = 'u1' -- metadata AND ts > now() - INTERVAL 30 DAY -- recency ORDER BY cosineDistance(emb, q_vec) -- vector LIMIT 3

top-3, measured — 10.6 ms

prefers dark mode           0.001
okonomiyaki: hiroshima-style  0.896
python 3.13 user            1.103

実測

One table — embeddings next to their rows

In-process — zero network hops

No servers — nothing to deploy or babysit

三本の矢 → 一本
sanbon-no-ya → ippon
three arrows, one query

One table, in-process. No servers.

The gap grows — with size, and with depth

DATA SIZE ↑ — same op

×0.5

×1.4

×28

×52.7

100K rows1M10M100M
ClickBench*

CHAIN DEPTH ↑ — 10M rows

×3.3

×4.9

×12.9

×21.7

×35.2

1 op4 filters5 chained+select
+sort+multi-
sort

実測

same laptop, shared linear scale · *100M point: published ClickBench DataFrame total (100M rows / 43.77 GiB), benchmark.clickhouse.com

Below ~1M rows keep pandas. Beyond it, the gap only widens.

Three takeaways

The pandas API is the asset. Fifteen years of muscle memory — keep it, swap the engine.

The gap grows with your data. More rows, deeper chains — the advantage compounds where pandas ends.

Your agent's memory is one query. Vector + metadata + recency — one table, in-process, no servers.

pandas is the language. chDB is the engine. You never had to leave.

ありがとうございました！

← arigatō gozaimashita

Thank you.

pip install chdb

github.com/chdb-io/chdb

clickhouse.com/docs/chdb

auxten.com — slides & verified demos

Q&A

Auxten Wang · ClickHouse · Creator of chDB@auxten

RediscoveringDataFrames

Auxten Wang

Everyone's first data tool

Then your data grew.

The ceiling has three walls

EAGER

SINGLE-CORE

ALL-IN-RAM

The usual exits all cost the same thing

Spark

Polars

DuckDB

Rewrite

Change one line

chDB: a rocket engine on a bicycle

Layered, not mixed — お好み焼き o-ko-no-mi-ya-ki

ソース sōsu · sauce — your pandas calls

麺 men · noodles — the lazy op chain

キャベツ kyabetsu · cabbage — one SQL plan

生地 kiji · batter — the chDB engine

鉄板 teppan · teppan — your Python process

The engine knocks down all three walls

A SQL COMPILER

EVERY CORE

STREAMING

Where the 100× lives

10M rows, 16 everyday ops — faster on 14 of 16

Your data lives on many islands

measured this week, this laptop

It reads what the query needs — not the file

Agent memory today: three databases

Vector DB

Metadata store

Recency cache

One query does all three

top-3, measured — 10.6 ms

The gap grows — with size, and with depth

Three takeaways

Thank you.

Rediscovering
DataFrames