How to query large CSV and Parquet files with DuckDB in a sandbox

On Runtime, an agent's SQL runs in a machine of its own, next to the data. Memory is the cheap part. Memory costs $0.0075 per GiB-hour. A 4 vCPU, 16 GiB sandbox therefore costs $0.22 an hour with all four CPUs busy, and less while the model decides on its next query. A bucket mounts as a folder, and the sandbox never holds the bucket's key. DuckDB 2.0.0 was the current release on 25 October 2026.

Upload files and run a query

Put the query in a script, run the script in the sandbox, and download the output:

TypeScriptimport { Sandbox } from "withruntime";const query = `import duckdbcon = duckdb.connect()con.execute("SET memory_limit = '3GB'")con.execute("SET temp_directory = '/workspace/duckdb-tmp'")con.execute("""    COPY (        SELECT customer_id, sum(amount) AS total        FROM read_csv('/workspace/data/*.csv')        GROUP BY customer_id        ORDER BY total DESC    ) TO '/workspace/out/totals.parquet' (FORMAT parquet)""")print(con.execute("SELECT count(*) FROM '/workspace/out/totals.parquet'").fetchone()[0], "customers")`;await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });await sbx.exec("pip install -q duckdb==2.0.0", { check: true, timeoutMs: 300_000 });await sbx.files.upload("./data", "/workspace/data");await sbx.files.write("/workspace/query.py", query);await sbx.files.mkdir("/workspace/out");await sbx.network.set({ internet: false }); // the query needs nothing from outsideconst run = await sbx.exec(["python3", "/workspace/query.py"], {  check: true,  timeoutMs: 1_200_000,});console.log(run.stdout);await sbx.files.download("/workspace/out/totals.parquet", "./totals.parquet");

Pythonfrom withruntime import SandboxQUERY = '''import duckdbcon = duckdb.connect()con.execute("SET memory_limit = '3GB'")con.execute("SET temp_directory = '/workspace/duckdb-tmp'")con.execute("""    COPY (        SELECT customer_id, sum(amount) AS total        FROM read_csv('/workspace/data/*.csv')        GROUP BY customer_id        ORDER BY total DESC    ) TO '/workspace/out/totals.parquet' (FORMAT parquet)""")print(con.execute("SELECT count(*) FROM '/workspace/out/totals.parquet'").fetchone()[0], "customers")'''with Sandbox.create(timeout_seconds=1800) as sbx:    sbx.exec("pip install -q duckdb==2.0.0", check=True, timeout_ms=300_000)    sbx.files.upload("./data", "/workspace/data")    sbx.files.write("/workspace/query.py", QUERY)    sbx.files.mkdir("/workspace/out")    sbx.network.set(internet=False)    print(sbx.exec(["python3", "/workspace/query.py"], check=True, timeout_ms=1_200_000).stdout)    sbx.files.download("/workspace/out/totals.parquet", "totals.parquet")

The CSV, Parquet and JSON readers are built into the Python package. In a test on 25 October 2026, duckdb_extensions() listed parquet and json as statically linked. So the query runs with the internet off. Reading https:// or s3:// URLs directly needs the httpfs extension. DuckDB downloads that extension the first time, so leave the network on for that.

Memory, disk and spilling

DuckDB processes data larger than memory by writing intermediate results to its temp directory. Two of its defaults decide how far that goes. Both follow from the sandbox's size:

SET memory_limit = '3GB' counts decimal gigabytes. DuckDB reports it back as 2.7 GiB. For a larger job, size the sandbox, not the setting:

• Memory: memoryMiB sets the sandbox's memory at create. By default, DuckDB uses 80% of it.
• Disk: diskMiB holds the input files, the output and anything DuckDB spills. Without credit, a sandbox can have up to 10 GiB.
• CPU: vcpu raises the thread count. You pay for the CPU the query actually uses.

Query a bucket without copying it

A bucket in S3, R2 or Google Cloud Storage can appear as a folder. Store the bucket's key once as a secret. The sandbox's mount client signs requests with a placeholder. The host then signs them again with the real key (mount your own bucket):

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });await sbx.mounts.add({  provider: "s3",  bucket: "acme-events",  region: "eu-west-1",  path: "/data",  secret: "EVENTS_BUCKET",  readOnly: true,});await sbx.exec("pip install -q duckdb==2.0.0", { check: true, timeoutMs: 300_000 });const run = await sbx.exec([  "python3",  "-c",  "import duckdb; print(duckdb.sql(\"SELECT event, count(*) FROM '/data/2026/09/*.parquet' GROUP BY event\"))",]);console.log(run.stdout);

readOnly means a query cannot change the bucket, whatever SQL the agent writes.

Let the model write the SQL

Pass the model's query to a small runner on standard input. Do not put it inside a shell command. The runner returns a bounded number of rows as JSON:

Pythonfrom withruntime import SandboxRUNNER = '''import json, sys, duckdbcursor = duckdb.connect().execute(sys.stdin.read())columns = [column[0] for column in cursor.description or []]print(json.dumps({"columns": columns, "rows": cursor.fetchmany(50)}, default=str))'''def run_sql(sbx, sql: str) -> str:    result = sbx.exec(["python3", "/workspace/run_sql.py"], stdin=sql, timeout_ms=300_000)    return result.stdout if result.exit_code == 0 else result.stderrwith Sandbox.create() as sbx:    sbx.exec("pip install -q duckdb==2.0.0", check=True, timeout_ms=300_000)    sbx.files.write("/workspace/run_sql.py", RUNNER)    print(run_sql(sbx, "SELECT 42 AS answer"))

A DuckDB error, such as Parser Error: syntax error at or near "SELEC", comes back on stderr with the position marked. The model needs that position to correct its query. Fifty rows keep the reply short. A query that needs the whole result should COPY it to a file that you download. The runner reads rows from the cursor execute returns, because DuckDB 2.0 relations no longer have fetchone() or fetchmany().

Start with DuckDB installed

A recipe image installs the package once, for every sandbox you create later:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();await runtime.images.build({ name: "duckdb", recipe: { pip: ["duckdb==2.0.0"] } });await using sbx = await runtime.sandboxes.create({ image: "duckdb", memoryMiB: 8192 });

The default image already includes pandas 2.1.4, so relation.df() works without installing it.

Related

• A data analysis agent that runs Python safely
• SQLite in a sandbox
• Run untrusted Python code safely
• Turn off sandbox internet

Sources

Checked 25 October 2026.

• DuckDB configuration: memory_limit (80% of RAM), temp_directory, max_temp_directory_size (90% of available disk), threads (the number of CPU cores)
• duckdb on PyPI: version 2.0.0, Python 3.11 or later
• A local test of duckdb 2.0.0 on 25 October 2026: duckdb_extensions() lists parquet, json and icu as statically linked and httpfs as not installed; SET memory_limit = '3GB' reads back as 2.7 GiB

Facts on this page were checked on 25 October 2026.