On Runtime grading 1,500 submissions costs $0.29. Each submission gets its own Firecracker microVM, so one student's code cannot read another's, reach your network or outlive its time limit. Runtime bills the CPU the tests use, $0.025 per vCPU-hour, and memory at $0.0075 per GiB-hour. The calculation is at the end of this page.
The short answer
One sandbox per submission. Copy the student's files and your hidden tests in, turn off the network, run the tests, and keep the result.
TypeScriptimport { Sandbox } from "withruntime";export async function grade(studentDir: string, testsDir: string) { await using sbx = await Sandbox.create({ vcpu: 1, memoryMiB: 2048, network: { internet: false }, timeoutSeconds: 300, onLeaseEnd: "stop", }); await sbx.files.upload(studentDir, "/workspace/submission"); await sbx.files.upload(testsDir, "/workspace/tests"); const run = await sbx.exec( ["python3", "-m", "unittest", "discover", "-s", "/workspace/tests", "-t", "/workspace"], { cwd: "/workspace/submission", timeoutMs: 60_000 }, ); return { passed: run.exitCode === 0 && !run.timedOut, timedOut: run.timedOut, output: run.stderr.slice(-4000), // unittest reports on stderr };}Pythonfrom withruntime import Sandboxdef grade(student_dir: str, tests_dir: str) -> dict: with Sandbox.create( vcpu=1, memory_mib=2048, network={"internet": False}, timeout_seconds=300, on_lease_end="stop", ) as sbx: sbx.files.upload(student_dir, "/workspace/submission") sbx.files.upload(tests_dir, "/workspace/tests") run = sbx.exec( ["python3", "-m", "unittest", "discover", "-s", "/workspace/tests", "-t", "/workspace"], cwd="/workspace/submission", timeout_ms=60_000, ) return { "passed": run.exit_code == 0 and not run.timed_out, "timed_out": run.timed_out, "output": run.stderr[-4000:], # unittest reports on stderr }Use your own test runner in place of unittest: pytest, Jest, JUnit or a
make check. The command is an array, so a file name a student chose cannot
inject shell syntax. The sandbox is deleted when the block ends, even after an
error.
What each setting protects against
| What a submission might do | What stops it | Setting |
|---|---|---|
| Read another student's work | Every submission is its own microVM with its own disk | One sandbox per submission |
| Fetch the answer or post the tests | No outbound connection at all, root included | network: { internet: false } |
| Loop forever | The command returns timedOut: true with its output so far |
timeoutMs |
| Keep the machine running | The lease ends and the sandbox stops | timeoutSeconds, onLeaseEnd |
| Fork-bomb or fill memory | Only its own sandbox slows; CPU and memory limits are enforced on the host | vcpu, memoryMiB |
| Print a gigabyte | At most 64 KiB of stdout and 64 KiB of stderr per result | Automatic, with truncation flags |
| Break out of the kernel | A hardware virtualization boundary, not a shared kernel | Every sandbox |
A container shares its host's kernel; a microVM gives the code a kernel of its own (microVM vs container).
Other languages
The default image has Python 3.12, Node.js 24, Bun, gcc, g++ and make, so C, C++, JavaScript and TypeScript assignments grade without setup. For Java, Go, Rust or a course's own toolchain, build an image once:
TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();await runtime.images.build({ name: "cs101-grader", recipe: { apt: ["openjdk-21-jdk-headless"], pip: ["pytest", "hypothesis"] },});await using sbx = await runtime.sandboxes.create({ image: "cs101-grader", network: { internet: false },});console.log((await sbx.exec("java -version")).stderr);Pythonfrom withruntime import Runtimeruntime = Runtime()runtime.images.build(name="cs101-grader", recipe={"apt": ["openjdk-21-jdk-headless"], "pip": ["pytest", "hypothesis"]})with runtime.sandboxes.create(image="cs101-grader", network={"internet": False}) as sbx: print(sbx.exec("java -version").stderr)Each build of a name is its next version, so pin a term's grader with
cs101-grader@3 and every regrade uses exactly the same tools
(custom images).
Grade a whole class at once
A paid account runs 100 sandboxes at the same time after its first top-up. A create past that limit waits for room instead of failing:
TypeScriptimport { Runtime } from "withruntime";const submissions = ["alice", "bob", "carol"];const runtime = new Runtime({ waitForCapacityMs: 600_000 });const scores = await Promise.all( submissions.map(async (student) => { await using sbx = await runtime.sandboxes.create({ labels: { course: "cs101", assignment: "hw3", student }, network: { internet: false }, timeoutSeconds: 300, onLeaseEnd: "stop", }); await sbx.files.write("/workspace/main.py", "print(sum([1, 2, 3]))\n"); const run = await sbx.exec(["python3", "main.py"], { timeoutMs: 30_000 }); return { student, correct: run.stdout.trim() === "6" }; }),);console.log(scores);Labels let you find every sandbox of one assignment afterwards. Give grading its own API key with a daily spending limit, which only a person can set. That way, a bug in the grading script cannot add charges (security).
What matters for an autograder
| Need | How Runtime covers it |
|---|---|
| Isolation per submission | A Firecracker microVM with its own kernel |
| No cheating over the network | Internet off, enforced on the host |
| Fair time limits | timeoutMs per command; the result says it timed out |
| Same tools every regrade | Versioned images, pinned with name@version |
| Deadline-night bursts | 100 at the same time on a paid account; creates past the limit wait for room |
| Detailed feedback | stdout, stderr and exit code per command; files read back with files.read |
| A capped budget | A daily spending limit per key; maxTotalCostMicros per sandbox |
What it costs
Take a course of 300 students with five assignments: 1,500 submissions. Each keeps a 1 vCPU, 2 GiB sandbox running for 30 seconds, upload and start included, and its tests use 10 CPU-seconds:
- CPU
- 1,500 × 10 s / 3,600 × $0.025
- $0.10
- Memory
- 1,500 × 30 s / 3,600 × 2 GiB × $0.0075
- $0.19
- Total
- $0.29
That is about two hundredths of a cent a submission. A new account gets 100 free hours of a 1 GB machine in its first 30 days with no card, and an account without credit runs four sandboxes at the same time, enough to try a real assignment (pricing).
Start
Terminalprintf '%s\n' 'print(sum([125, 250, 375]))' > invoice.pynpx withruntime sandboxes run -- python3 - < invoice.pyThe first run prints a link to approve in your browser. Then use the JavaScript or Python SDK as above.
Related: run untrusted LLM code safely, turn off sandbox internet, what a microVM is, sandbox cost calculator.
Facts on this page were checked on 26 October 2026.