# How to run Hugging Face Transformers models on CPU in a sandbox

Install PyTorch's CPU wheel and `transformers`, cache the model in an image, and run the pipeline with the internet off.

**On Runtime you pay for the CPU the model uses, and the model files can be
baked in once.** Inference that keeps 4 vCPUs busy costs $0.10 an hour for
the CPU, plus $0.0075 per GiB-hour of memory, and a sandbox made from an image
with the model cached starts with nothing to download. A gated model's token
stays out of the sandbox as a Runtime secret. transformers 5.17.0 and PyTorch
2.14.0 were current on 25 September 2026.

## Install the CPU build of PyTorch

PyTorch's default wheel on PyPI is built for NVIDIA GPUs and pulls in CUDA
libraries. Its CPU index serves a much smaller wheel, which is the one to use
here, as the Transformers install guide shows:

| Wheel for Python 3.12 on x86-64 Linux, torch 2.14.0   | Size   |
| ----------------------------------------------------- | ------ |
| `torch` from PyPI, before its `nvidia-*` dependencies | 555 MB |
| `torch` from `download.pytorch.org/whl/cpu`           | 196 MB |

In a test on 25 September 2026, a virtual environment with the CPU wheel and
transformers 5.17.0 took 1.2 GB of disk. The default 4 GiB sandbox has about
2.5 GiB free, so ask for `diskMiB: 8192` before adding a model.

## Run a pipeline

```ts check
import { Sandbox } from "withruntime";

const classify = `
from transformers import pipeline
clf = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
for row in clf(["The sandbox ran out of disk.", "The release shipped on time."]):
    print(row["label"], round(row["score"], 4))
`;

await using sbx = await Sandbox.create({
  vcpu: 2,
  memoryMiB: 4096,
  diskMiB: 8192,
  timeoutSeconds: 1800,
});
const slow = { check: true, timeoutMs: 900_000 } as const;
await sbx.exec(
  "pip install -q --no-cache-dir torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu",
  slow,
);
await sbx.exec("pip install -q --no-cache-dir transformers==5.17.0", slow);

await sbx.files.write("/workspace/classify.py", classify);
const run = await sbx.exec(["python3", "/workspace/classify.py"], slow);
console.log(run.stdout); // NEGATIVE 0.9998 for the first sentence
```

```python check
from withruntime import Sandbox

CLASSIFY = """
from transformers import pipeline
clf = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
for row in clf(["The sandbox ran out of disk.", "The release shipped on time."]):
    print(row["label"], round(row["score"], 4))
"""

with Sandbox.create(vcpu=2, memory_mib=4096, disk_mib=8192, timeout_seconds=1800) as sbx:
    sbx.exec("pip install -q --no-cache-dir torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu",
             check=True, timeout_ms=900_000)
    sbx.exec("pip install -q --no-cache-dir transformers==5.17.0", check=True, timeout_ms=900_000)
    sbx.files.write("/workspace/classify.py", CLASSIFY)
    print(sbx.exec(["python3", "/workspace/classify.py"], check=True, timeout_ms=900_000).stdout)
```

The first run downloads the model from the Hugging Face Hub. For this model
the pipeline fetched 256 MB, the safetensors weights and tokenizer, not the
TensorFlow and Rust copies the repository also holds.

## Bake the model into an image

Download the model during the image build, set `HF_HOME` so the cache sits in
the image, and every sandbox from it starts ready:

```ts check
import { Runtime } from "withruntime";

const model = "distilbert/distilbert-base-uncased-finetuned-sst-2-english";
const runtime = new Runtime();
await runtime.images.build(
  {
    name: "sst2-cpu",
    recipe: {
      env: { HF_HOME: "/opt/hf" },
      commands: [
        "pip install --no-cache-dir torch==2.14.0 --index-url https://download.pytorch.org/whl/cpu",
        "pip install --no-cache-dir transformers==5.17.0",
        `python3 -c "from transformers import pipeline; pipeline('text-classification', model='${model}')"`,
        "chown -R 1000:1000 /opt/hf",
      ],
    },
    build: { diskMiB: 8192 },
  },
  { onLog: (line) => console.log(line.text) },
);

await using sbx = await runtime.sandboxes.create({
  image: "sst2-cpu",
  network: { internet: false },
});
const run = await sbx.exec(
  [
    "python3",
    "-c",
    `from transformers import pipeline; print(pipeline('text-classification', model='${model}')('fine'))`,
  ],
  { env: { HF_HUB_OFFLINE: "1" }, timeoutMs: 300_000 },
);
console.log(run.stdout);
```

- **`HF_HOME`:** where `huggingface_hub` keeps its cache and token. Pointing
  it at `/opt/hf` in the recipe's `env` puts the downloaded model in the image.
- **`HF_HUB_OFFLINE=1`:** no calls to the Hub; files come only from the cache.
  With the internet off as well, the model cannot fetch code or weights at run
  time, and in a test on 25 September 2026 the cached pipeline ran offline.
- **The build machine:** its scratch disk is 4 GiB by default; `build.diskMiB`
  raises it, up to 32 GiB. An image can be up to 20 GiB.

A stored image is charged at $0.08 per GB per 30-day month on its whole file,
so a 2 GB image costs $0.16 a month to keep. Building it is free.

## Gated and private models

Models such as Llama and Gemma need an access token. Store it as a secret for
`huggingface.co`; every sandbox then has `HF_TOKEN` holding a placeholder,
which `huggingface_hub` sends as it would the real token, and the host's proxy
swaps in the value on its way to the Hub:

```python check
import os
from withruntime import Runtime

runtime = Runtime()
runtime.secrets.set("HF_TOKEN", value=os.environ["HF_TOKEN"], hosts=["huggingface.co"])
```

`HF_TOKEN` is the variable `huggingface_hub` reads for the Hub's access token.
Accept the model's licence on its Hub page first, with the account that owns
the token.

## How big a model fits

Weights in 32-bit floats take 4 bytes per parameter, so memory, not the CPU,
decides what loads:

| Model size        | Weights in float32 | Sandbox to ask for       |
| ----------------- | ------------------ | ------------------------ |
| 66 M (DistilBERT) | about 0.27 GB      | The default 4 GiB        |
| 0.5 B             | about 2 GB         | `memoryMiB: 8192`        |
| 1.5 B             | about 6 GB         | `memoryMiB: 16384`, paid |

Runtime runs on CPUs only, with no GPUs, so the models that suit a sandbox are
encoders, embedding models, classifiers and small generators. For more
throughput, raise `vcpu`: CPU is billed on what the model actually uses, so a
batch that finishes sooner on more vCPUs costs about the same CPU.

## Related

- [Run a Gradio demo in a sandbox](/integrations/gradio)
- [Run untrusted Python code safely](/languages/python)
- [Turn off sandbox internet](/how-to/turn-off-sandbox-internet)
- [Custom images](/docs/images)
- [Secrets sandboxes never see](/docs/security#secrets-sandboxes-never-see)

## Sources

Checked 25 September 2026.

- [Transformers installation](https://huggingface.co/docs/transformers/installation):
  the CPU-only install from `https://download.pytorch.org/whl/cpu`, Python
  3.10 and later, PyTorch 2.5 and later
- [PyTorch CPU wheel index](https://download.pytorch.org/whl/cpu/torch/):
  `torch-2.14.0+cpu-cp312-cp312-manylinux_2_28_x86_64.whl`, 196,253,793 bytes;
  [torch 2.14.0 on PyPI](https://pypi.org/project/torch/2.14.0/): the cp312
  x86-64 wheel is 554,620,488 bytes and requires `nvidia-*` packages on Linux
- [huggingface_hub environment variables](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables):
  `HF_HOME`, `HF_TOKEN`, `HF_HUB_OFFLINE`
- [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english):
  `model.safetensors`, 267,832,558 bytes
- [transformers on PyPI](https://pypi.org/project/transformers/): version 5.17.0
- A local test on 25 September 2026: torch 2.14.0+cpu and transformers 5.17.0
  in a 1.2 GB virtual environment; the pipeline cached 256 MB, labelled
  "The sandbox ran out of disk." NEGATIVE (0.9998), and ran again with
  `HF_HUB_OFFLINE=1`

Facts on this page were checked on 25 September 2026.