# How to query large CSV and Parquet files with DuckDB in a sandbox

Install the `duckdb` Python package in the sandbox, upload the files or mount the bucket, run the SQL there and download the result.

**On Runtime an agent's SQL runs in a machine of its own, next to the data,
and memory is the cheap part.** Memory costs $0.0075 per GiB-hour, so a
4 vCPU, 16 GiB sandbox costs $0.22 an hour with all four CPUs busy, and
less while the model decides on its next query. A bucket mounts as a folder
without the sandbox ever holding the bucket's key. DuckDB 1.5.5 was the current
release on 25 September 2026.

## Upload files and run a query

Put the query in a script, run it in the sandbox, and download what it wrote:

```ts check
import { Sandbox } from "withruntime";

const query = `
import duckdb
con = duckdb.connect()
con.execute("SET memory_limit = '3GB'")
con.execute("SET temp_directory = '/workspace/duckdb-tmp'")
con.execute("""
    COPY (
        SELECT customer_id, sum(amount) AS total
        FROM read_csv('/workspace/data/*.csv')
        GROUP BY customer_id
        ORDER BY total DESC
    ) TO '/workspace/out/totals.parquet' (FORMAT parquet)
""")
print(con.sql("SELECT count(*) FROM '/workspace/out/totals.parquet'").fetchone()[0], "customers")
`;

await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });
await sbx.exec("pip install -q duckdb==1.5.5", { check: true, timeoutMs: 300_000 });
await sbx.files.upload("./data", "/workspace/data");
await sbx.files.write("/workspace/query.py", query);
await sbx.files.mkdir("/workspace/out");

await sbx.network.set({ internet: false }); // the query needs nothing from outside
const run = await sbx.exec(["python3", "/workspace/query.py"], {
  check: true,
  timeoutMs: 1_200_000,
});
console.log(run.stdout);
await sbx.files.download("/workspace/out/totals.parquet", "./totals.parquet");
```

```python check
from withruntime import Sandbox

QUERY = '''
import duckdb
con = duckdb.connect()
con.execute("SET memory_limit = '3GB'")
con.execute("SET temp_directory = '/workspace/duckdb-tmp'")
con.execute("""
    COPY (
        SELECT customer_id, sum(amount) AS total
        FROM read_csv('/workspace/data/*.csv')
        GROUP BY customer_id
        ORDER BY total DESC
    ) TO '/workspace/out/totals.parquet' (FORMAT parquet)
""")
print(con.sql("SELECT count(*) FROM '/workspace/out/totals.parquet'").fetchone()[0], "customers")
'''

with Sandbox.create(timeout_seconds=1800) as sbx:
    sbx.exec("pip install -q duckdb==1.5.5", check=True, timeout_ms=300_000)
    sbx.files.upload("./data", "/workspace/data")
    sbx.files.write("/workspace/query.py", QUERY)
    sbx.files.mkdir("/workspace/out")

    sbx.network.set(internet=False)
    print(sbx.exec(["python3", "/workspace/query.py"], check=True, timeout_ms=1_200_000).stdout)
    sbx.files.download("/workspace/out/totals.parquet", "totals.parquet")
```

The CSV, Parquet and JSON readers are built into the Python package: in a
test on 25 September 2026, `duckdb_extensions()` listed `parquet` and `json`
as statically linked, so the query runs with the internet off. Reading
`https://` or `s3://` URLs directly needs the `httpfs` extension, which DuckDB
downloads the first time, so leave the network on for that.

## Memory, disk and spilling

DuckDB processes data larger than memory by writing intermediate results to
its temp directory. Two of its defaults decide how far that goes, and both
follow from the sandbox's size:

| DuckDB setting            | Default                 | In a default sandbox (4 GiB memory, 4 GiB disk) |
| ------------------------- | ----------------------- | ----------------------------------------------- |
| `memory_limit`            | 80% of RAM              | About 3.2 GiB                                   |
| `max_temp_directory_size` | 90% of available disk   | Most of the roughly 2.5 GiB free                |
| `threads`                 | The number of CPU cores | 2, one per vCPU                                 |

`SET memory_limit = '3GB'` counts decimal gigabytes: DuckDB reports it back
as 2.7 GiB. For a larger job, size the sandbox rather than the setting:

- **Memory:** `memoryMiB` sets it at create; DuckDB takes 80% by default.
- **Disk:** `diskMiB` holds the input files, the output and anything DuckDB
  spills. The trial allows up to 10 GiB.
- **CPU:** `vcpu` raises the thread count, and you pay for the CPU the query
  actually uses.

## Query a bucket without copying it

A bucket in S3, R2 or Google Cloud Storage can appear as a folder. Store the
bucket's key once as a secret; the sandbox's mount client signs with a
placeholder and the host signs again with the real key
([mount your own bucket](/docs/storage#mount-your-own-bucket)):

```ts check
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });
await sbx.mounts.add({
  provider: "s3",
  bucket: "acme-events",
  region: "eu-west-1",
  path: "/data",
  secret: "EVENTS_BUCKET",
  readOnly: true,
});
await sbx.exec("pip install -q duckdb==1.5.5", { check: true, timeoutMs: 300_000 });
const run = await sbx.exec([
  "python3",
  "-c",
  "import duckdb; print(duckdb.sql(\"SELECT event, count(*) FROM '/data/2026/09/*.parquet' GROUP BY event\"))",
]);
console.log(run.stdout);
```

`readOnly` means a query cannot change the bucket, whatever SQL the agent
writes.

## Let the model write the SQL

Pass the model's query on standard input to a small runner, not inside a shell
command, and return a bounded number of rows as JSON:

```python check
from withruntime import Sandbox

RUNNER = '''
import json, sys, duckdb
relation = duckdb.sql(sys.stdin.read())
if relation is None:  # a statement with no result, such as CREATE TABLE
    print(json.dumps({"columns": [], "rows": []}))
else:
    print(json.dumps({"columns": relation.columns, "rows": relation.fetchmany(50)}, default=str))
'''

def run_sql(sbx, sql: str) -> str:
    result = sbx.exec(["python3", "/workspace/run_sql.py"], stdin=sql, timeout_ms=300_000)
    return result.stdout if result.exit_code == 0 else result.stderr

with Sandbox.create() as sbx:
    sbx.exec("pip install -q duckdb==1.5.5", check=True, timeout_ms=300_000)
    sbx.files.write("/workspace/run_sql.py", RUNNER)
    print(run_sql(sbx, "SELECT 42 AS answer"))
```

A DuckDB error, such as `Parser Error: syntax error at or near "SELEC"`, comes
back on `stderr` with the position marked, which is what the model needs to
correct its query. Fifty rows keep the reply short; a query that needs the
whole result should `COPY` it to a file you download.

## Start with DuckDB installed

A recipe image installs the package once, for every later sandbox:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime();
await runtime.images.build({ name: "duckdb", recipe: { pip: ["duckdb==1.5.5"] } });
await using sbx = await runtime.sandboxes.create({ image: "duckdb", memoryMiB: 8192 });
```

`pandas` 2.1.4 is in the default image already, so `relation.df()` works
without installing it.

## Related

- [A data analysis agent that runs Python safely](/use-cases/data-analysis-agent)
- [SQLite in a sandbox](/integrations/sqlite)
- [Run untrusted Python code safely](/languages/python)
- [Turn off sandbox internet](/how-to/turn-off-sandbox-internet)

## Sources

Checked 25 September 2026.

- [DuckDB configuration](https://duckdb.org/docs/current/configuration/overview.html):
  `memory_limit` (80% of RAM), `temp_directory`, `max_temp_directory_size`
  (90% of available disk), `threads` (the number of CPU cores)
- [duckdb on PyPI](https://pypi.org/project/duckdb/): version 1.5.5, Python
  3.10 or later
- A local test of duckdb 1.5.5 on 25 September 2026: `duckdb_extensions()`
  lists `parquet`, `json` and `icu` as statically linked and `httpfs` as not
  installed; `SET memory_limit = '3GB'` reads back as 2.7 GiB

Facts on this page were checked on 25 September 2026.