Runtime

How to run Hugging Face Transformers models on CPU in a sandbox

Install PyTorch's CPU wheel and transformers, cache the model in an image, and run the pipeline with the internet off.

On Runtime you pay for the CPU the model uses, and the model files can be baked in once. Inference that keeps 4 vCPUs busy costs $0.10 an hour for the CPU, plus $0.0075 per GiB-hour of memory, and a sandbox made from an image with the model cached starts with nothing to download. A gated model's token stays out of the sandbox as a Runtime secret. transformers 5.17.0 and PyTorch 2.14.0 were current on 25 September 2026.

Install the CPU build of PyTorch

PyTorch's default wheel on PyPI is built for NVIDIA GPUs and pulls in CUDA libraries. Its CPU index serves a much smaller wheel, which is the one to use here, as the Transformers install guide shows:

Wheel for Python 3.12 on x86-64 Linux, torch 2.14.0 Size
torch from PyPI, before its nvidia-* dependencies 555 MB
torch from download.pytorch.org/whl/cpu 196 MB

In a test on 25 September 2026, a virtual environment with the CPU wheel and transformers 5.17.0 took 1.2 GB of disk. The default 4 GiB sandbox has about 2.5 GiB free, so ask for diskMiB: 8192 before adding a model.

Run a pipeline

TypeScriptimport { Sandbox } from "withruntime";const classify = `from transformers import pipelineclf = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")for row in clf(["The sandbox ran out of disk.", "The release shipped on time."]):    print(row["label"], round(row["score"], 4))`;await using sbx = await Sandbox.create({  vcpu: 2,  memoryMiB: 4096,  diskMiB: 8192,  timeoutSeconds: 1800,});const slow = { check: true, timeoutMs: 900_000 } as const;await sbx.exec(  "pip install -q --no-cache-dir torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu",  slow,);await sbx.exec("pip install -q --no-cache-dir transformers==5.17.0", slow);await sbx.files.write("/workspace/classify.py", classify);const run = await sbx.exec(["python3", "/workspace/classify.py"], slow);console.log(run.stdout); // NEGATIVE 0.9998 for the first sentence
Pythonfrom withruntime import SandboxCLASSIFY = """from transformers import pipelineclf = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")for row in clf(["The sandbox ran out of disk.", "The release shipped on time."]):    print(row["label"], round(row["score"], 4))"""with Sandbox.create(vcpu=2, memory_mib=4096, disk_mib=8192, timeout_seconds=1800) as sbx:    sbx.exec("pip install -q --no-cache-dir torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu",             check=True, timeout_ms=900_000)    sbx.exec("pip install -q --no-cache-dir transformers==5.17.0", check=True, timeout_ms=900_000)    sbx.files.write("/workspace/classify.py", CLASSIFY)    print(sbx.exec(["python3", "/workspace/classify.py"], check=True, timeout_ms=900_000).stdout)

The first run downloads the model from the Hugging Face Hub. For this model the pipeline fetched 256 MB, the safetensors weights and tokenizer, not the TensorFlow and Rust copies the repository also holds.

Bake the model into an image

Download the model during the image build, set HF_HOME so the cache sits in the image, and every sandbox from it starts ready:

TypeScriptimport { Runtime } from "withruntime";const model = "distilbert/distilbert-base-uncased-finetuned-sst-2-english";const runtime = new Runtime();await runtime.images.build(  {    name: "sst2-cpu",    recipe: {      env: { HF_HOME: "/opt/hf" },      commands: [        "pip install --no-cache-dir torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu",        "pip install --no-cache-dir transformers==5.17.0",        `python3 -c "from transformers import pipeline; pipeline('text-classification', model='${model}')"`,        "chown -R 1000:1000 /opt/hf",      ],    },    build: { diskMiB: 8192 },  },  { onLog: (line) => console.log(line.text) },);await using sbx = await runtime.sandboxes.create({  image: "sst2-cpu",  network: { internet: false },});const run = await sbx.exec(  [    "python3",    "-c",    `from transformers import pipeline; print(pipeline('text-classification', model='${model}')('fine'))`,  ],  { env: { HF_HUB_OFFLINE: "1" }, timeoutMs: 300_000 },);console.log(run.stdout);
  • HF_HOME: where huggingface_hub keeps its cache and token. Pointing it at /opt/hf in the recipe's env puts the downloaded model in the image.
  • HF_HUB_OFFLINE=1: no calls to the Hub; files come only from the cache. With the internet off as well, the model cannot fetch code or weights at run time, and in a test on 25 September 2026 the cached pipeline ran offline.
  • The build machine: its scratch disk is 4 GiB by default; build.diskMiB raises it, up to 32 GiB. An image can be up to 20 GiB.

A stored image is charged at $0.08 per GB per 30-day month on its whole file, so a 2 GB image costs $0.16 a month to keep. Building it is free.

Gated and private models

Models such as Llama and Gemma need an access token. Store it as a secret for huggingface.co; every sandbox then has HF_TOKEN holding a placeholder, which huggingface_hub sends as it would the real token, and the host's proxy swaps in the value on its way to the Hub:

Pythonimport osfrom withruntime import Runtimeruntime = Runtime()runtime.secrets.set("HF_TOKEN", value=os.environ["HF_TOKEN"], hosts=["huggingface.co"])

HF_TOKEN is the variable huggingface_hub reads for the Hub's access token. Accept the model's licence on its Hub page first, with the account that owns the token.

How big a model fits

Weights in 32-bit floats take 4 bytes per parameter, so memory, not the CPU, decides what loads:

Model size Weights in float32 Sandbox to ask for
66 M (DistilBERT) about 0.27 GB The default 4 GiB
0.5 B about 2 GB memoryMiB: 8192
1.5 B about 6 GB memoryMiB: 16384, paid

Runtime runs on CPUs only, with no GPUs, so the models that suit a sandbox are encoders, embedding models, classifiers and small generators. For more throughput, raise vcpu: CPU is billed on what the model actually uses, so a batch that finishes sooner on more vCPUs costs about the same CPU.

Sources

Checked 25 September 2026.

Facts on this page were checked on 25 September 2026.

Claim 50 hours free