How to query large CSV and Parquet files with DuckDB in a sandbox

Install the duckdb Python package in the sandbox, upload the files or mount the bucket, run the SQL there and download the result.

On Runtime an agent's SQL runs in a machine of its own, next to the data, and memory is the cheap part. Memory costs $0.0075 per GiB-hour, so a 4 vCPU, 16 GiB sandbox costs $0.22 an hour with all four CPUs busy, and less while the model decides on its next query. A bucket mounts as a folder without the sandbox ever holding the bucket's key. DuckDB 1.5.5 was the current release on 25 September 2026.

Upload files and run a query

Put the query in a script, run it in the sandbox, and download what it wrote:

TypeScriptimport { Sandbox } from "withruntime";const query = `import duckdbcon = duckdb.connect()con.execute("SET memory_limit = '3GB'")con.execute("SET temp_directory = '/workspace/duckdb-tmp'")con.execute("""    COPY (        SELECT customer_id, sum(amount) AS total        FROM read_csv('/workspace/data/*.csv')        GROUP BY customer_id        ORDER BY total DESC    ) TO '/workspace/out/totals.parquet' (FORMAT parquet)""")print(con.sql("SELECT count(*) FROM '/workspace/out/totals.parquet'").fetchone()[0], "customers")`;await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });await sbx.exec("pip install -q duckdb==1.5.5", { check: true, timeoutMs: 300_000 });await sbx.files.upload("./data", "/workspace/data");await sbx.files.write("/workspace/query.py", query);await sbx.files.mkdir("/workspace/out");await sbx.network.set({ internet: false }); // the query needs nothing from outsideconst run = await sbx.exec(["python3", "/workspace/query.py"], {  check: true,  timeoutMs: 1_200_000,});console.log(run.stdout);await sbx.files.download("/workspace/out/totals.parquet", "./totals.parquet");

Pythonfrom withruntime import SandboxQUERY = '''import duckdbcon = duckdb.connect()con.execute("SET memory_limit = '3GB'")con.execute("SET temp_directory = '/workspace/duckdb-tmp'")con.execute("""    COPY (        SELECT customer_id, sum(amount) AS total        FROM read_csv('/workspace/data/*.csv')        GROUP BY customer_id        ORDER BY total DESC    ) TO '/workspace/out/totals.parquet' (FORMAT parquet)""")print(con.sql("SELECT count(*) FROM '/workspace/out/totals.parquet'").fetchone()[0], "customers")'''with Sandbox.create(timeout_seconds=1800) as sbx:    sbx.exec("pip install -q duckdb==1.5.5", check=True, timeout_ms=300_000)    sbx.files.upload("./data", "/workspace/data")    sbx.files.write("/workspace/query.py", QUERY)    sbx.files.mkdir("/workspace/out")    sbx.network.set(internet=False)    print(sbx.exec(["python3", "/workspace/query.py"], check=True, timeout_ms=1_200_000).stdout)    sbx.files.download("/workspace/out/totals.parquet", "totals.parquet")

The CSV, Parquet and JSON readers are built into the Python package: in a test on 25 September 2026, duckdb_extensions() listed parquet and json as statically linked, so the query runs with the internet off. Reading https:// or s3:// URLs directly needs the httpfs extension, which DuckDB downloads the first time, so leave the network on for that.

Memory, disk and spilling

DuckDB processes data larger than memory by writing intermediate results to its temp directory. Two of its defaults decide how far that goes, and both follow from the sandbox's size:

SET memory_limit = '3GB' counts decimal gigabytes: DuckDB reports it back as 2.7 GiB. For a larger job, size the sandbox rather than the setting:

• Memory: memoryMiB sets it at create; DuckDB takes 80% by default.
• Disk: diskMiB holds the input files, the output and anything DuckDB spills. The trial allows up to 10 GiB.
• CPU: vcpu raises the thread count, and you pay for the CPU the query actually uses.

Query a bucket without copying it

A bucket in S3, R2 or Google Cloud Storage can appear as a folder. Store the bucket's key once as a secret; the sandbox's mount client signs with a placeholder and the host signs again with the real key (mount your own bucket):

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });await sbx.mounts.add({  provider: "s3",  bucket: "acme-events",  region: "eu-west-1",  path: "/data",  secret: "EVENTS_BUCKET",  readOnly: true,});await sbx.exec("pip install -q duckdb==1.5.5", { check: true, timeoutMs: 300_000 });const run = await sbx.exec([  "python3",  "-c",  "import duckdb; print(duckdb.sql(\"SELECT event, count(*) FROM '/data/2026/09/*.parquet' GROUP BY event\"))",]);console.log(run.stdout);

readOnly means a query cannot change the bucket, whatever SQL the agent writes.

Let the model write the SQL

Pass the model's query on standard input to a small runner, not inside a shell command, and return a bounded number of rows as JSON:

Pythonfrom withruntime import SandboxRUNNER = '''import json, sys, duckdbrelation = duckdb.sql(sys.stdin.read())if relation is None:  # a statement with no result, such as CREATE TABLE    print(json.dumps({"columns": [], "rows": []}))else:    print(json.dumps({"columns": relation.columns, "rows": relation.fetchmany(50)}, default=str))'''def run_sql(sbx, sql: str) -> str:    result = sbx.exec(["python3", "/workspace/run_sql.py"], stdin=sql, timeout_ms=300_000)    return result.stdout if result.exit_code == 0 else result.stderrwith Sandbox.create() as sbx:    sbx.exec("pip install -q duckdb==1.5.5", check=True, timeout_ms=300_000)    sbx.files.write("/workspace/run_sql.py", RUNNER)    print(run_sql(sbx, "SELECT 42 AS answer"))

A DuckDB error, such as Parser Error: syntax error at or near "SELEC", comes back on stderr with the position marked, which is what the model needs to correct its query. Fifty rows keep the reply short; a query that needs the whole result should COPY it to a file you download.

Start with DuckDB installed

A recipe image installs the package once, for every later sandbox:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();await runtime.images.build({ name: "duckdb", recipe: { pip: ["duckdb==1.5.5"] } });await using sbx = await runtime.sandboxes.create({ image: "duckdb", memoryMiB: 8192 });

pandas 2.1.4 is in the default image already, so relation.df() works without installing it.

Related

• A data analysis agent that runs Python safely
• SQLite in a sandbox
• Run untrusted Python code safely
• Turn off sandbox internet

Sources

Checked 25 September 2026.

• DuckDB configuration: memory_limit (80% of RAM), temp_directory, max_temp_directory_size (90% of available disk), threads (the number of CPU cores)
• duckdb on PyPI: version 1.5.5, Python 3.10 or later
• A local test of duckdb 1.5.5 on 25 September 2026: duckdb_extensions() lists parquet, json and icu as statically linked and httpfs as not installed; SET memory_limit = '3GB' reads back as 2.7 GiB

Facts on this page were checked on 25 September 2026.