Why AltDat Runs on Polars and DuckDB

When you send a file to AltDat, the work of reading, transforming, and querying it happens on two libraries we chose deliberately: Polars for dataframe operations and DuckDB for SQL. Pandas is the default choice for data work in Python, and it’s excellent — but for an API that has to process arbitrary uploads quickly and predictably, Polars and DuckDB are a better fit. Here’s why.

The constraints of a processing API

An API has different constraints than an analyst’s notebook:

• File sizes vary widely. A request might carry a few rows or a file near the tier limit. The engine has to stay responsive and memory-bounded across that range.
• Latency is user-facing. Every second of processing is a second the caller’s automation flow is waiting.
• Memory is shared. Many requests run on the same container, so a library that copies data freely is a liability.

These constraints pushed us toward tools designed for exactly this kind of workload.

Why Polars

Polars is a dataframe library written in Rust with Python bindings. Three of its design choices matter most for AltDat:

Lazy evaluation

Polars can build a query plan and optimize the whole chain before executing it, rather than materializing an intermediate result at every step:

# Polars optimizes this entire chain before execution
df = pl.scan_csv("large_file.csv")
result = (df
    .filter(pl.col("age") > 25)
    .group_by("department")
    .agg(pl.col("salary").mean())
    .collect()  # computation happens here, on the optimized plan
)

Columnar memory layout

Polars uses Apache Arrow’s columnar format, which is far more efficient for the scan-and-aggregate operations a conversion or transform API spends most of its time doing.

Built-in parallelism

Operations are automatically parallelized across CPU cores with no extra configuration — which keeps request latency down without us hand-tuning threads.

Polars publishes open benchmarks comparing itself against other dataframe libraries; the short version is that for the read-filter-aggregate patterns our endpoints use, it is consistently faster and lighter on memory than a Pandas-based equivalent. Rather than repeat numbers here, we’d point you to their published results — and to your own files, which are the only benchmark that really matters.

Why DuckDB for SQL

For the SQL endpoint (available on paid plans), we use DuckDB. It’s an in-process analytical SQL engine that shares Apache Arrow’s memory format with Polars, which means data moves between the two with zero-copy exchange:

# Zero-copy handoff between Polars and DuckDB
df = pl.scan_csv("data.csv")
result = duckdb.sql("""
    SELECT
        department,
        AVG(salary) OVER (PARTITION BY department) AS avg_salary
    FROM df
""").pl()

That combination lets us offer real SQL — window functions, joins, aggregates — directly against an uploaded file, without standing up a database or copying your data into one.

What this means for you

You don’t have to think about any of this. You upload a file, name a target format or describe a transform, and AltDat returns the result. The engine choices are ours to worry about — we picked Polars and DuckDB so that the API stays fast and predictable as your files grow.

Want to see it on your own data? Read the docs and make your first request.