Build a GitHub issue-to-pull-request agent that handles review

Label an issue and a webhook starts an agent in that issue's own sandbox, which opens a draft pull request and wakes for each review.

On Runtime (withruntime.com) the issue's sandbox pauses between review rounds and has run its next command 153 ms after the wake request, on Runtime's servers, with the checkout, the installed packages and the agent's notes exactly as it left them. Opening the first pull request is the demo. The product is everything after it: a reviewer asks for a change on Tuesday, CI fails on Wednesday, someone comments on Thursday, and the PR merges on Friday. This post builds the service that handles that whole life: the events it listens to, a webhook receiver you can run as written, the worker that drives the sandbox, and the cost of one issue from label to merge.

What events does an issue agent have to handle?

Six, and each one maps to a single thing the sandbox does.

The last row is the one people learn the hard way. The agent's own comment mentions the agent, its own push triggers CI, and a service that reacts to everything ends up talking to itself all night.

Why keep one sandbox per issue for the whole pull request?

Because each review round needs the same working tree the last one left, and rebuilding it costs more than keeping it. A fresh machine per round means a new clone, a new npm install or pip install, cold test caches and an agent that has lost its scratch notes. That is minutes per round before the agent reads the review.

A paused sandbox keeps all of it, memory included, and costs storage only while the reviewer is busy elsewhere. The sandbox's name is derived from the repository and issue number, so getOrCreate finds it on every event, and any request wakes it (pause and resume). A paused sandbox is kept as long as your account has credit, so a pull request that sits for weeks wakes the same sandbox; one deleted meanwhile is replaced by a fresh sandbox, which checks out the branch the earlier one pushed.

How do you receive GitHub's webhooks safely?

Check the signature against the raw body, skip deliveries you have already seen, ignore the agent's own actions, and answer fast. GitHub signs each delivery with HMAC-SHA256 in X-Hub-Signature-256, names it with a unique X-GitHub-Delivery id, may deliver the same one again, and gives up on a receiver that takes longer than ten seconds. So the receiver only verifies and queues; a worker does the slow part. This receiver runs as written and replays seven deliveries against itself:

TypeScriptimport { createHmac, timingSafeEqual } from "node:crypto";const SECRET = "the-webhook-secret-from-your-github-app";const BOT = "acme-agent[bot]";const seen = new Set<string>(); // keep delivery ids in your database in productionconst prToIssue = new Map([[51, 42]]); // saved when the agent opens pull request #51type Payload = {  action?: string;  sender?: { login: string };  label?: { name: string };  issue?: { number: number; pull_request?: object };  comment?: { body: string };  review?: { state: string; body: string | null };  pull_request?: { head: { ref: string } };  check_suite?: { conclusion: string | null; head_branch: string | null };};type Job = { kind: "start" | "revise" | "fix-checks" | "cleanup"; issue: number; note: string };function signed(body: string, signature: string | null): boolean {  const expected = Buffer.from(`sha256=${createHmac("sha256", SECRET).update(body).digest("hex")}`);  const given = Buffer.from(signature ?? "");  return given.length === expected.length && timingSafeEqual(given, expected);}const issueOf = (branch: string | null | undefined) =>  Number(/^agent\/issue-(\d+)$/.exec(branch ?? "")?.[1] ?? 0);function route(event: string, p: Payload): Job | string {  if (p.sender?.login === BOT) return "ignored: the agent's own action"; // no loops  if (event === "issues" && p.action === "labeled" && p.label?.name === "agent")    return { kind: "start", issue: p.issue!.number, note: "" };  if (event === "pull_request_review" && p.review?.state === "changes_requested")    return { kind: "revise", issue: issueOf(p.pull_request?.head.ref), note: p.review.body ?? "" };  if (    event === "issue_comment" &&    p.action === "created" &&    p.comment?.body.includes("@acme-agent")  )    return {      kind: "revise",      issue: prToIssue.get(p.issue!.number) ?? p.issue!.number,      note: p.comment.body,    };  if (    event === "check_suite" &&    p.check_suite?.conclusion === "failure" &&    issueOf(p.check_suite.head_branch)  )    return { kind: "fix-checks", issue: issueOf(p.check_suite.head_branch), note: "" };  if (event === "pull_request" && p.action === "closed" && issueOf(p.pull_request?.head.ref))    return { kind: "cleanup", issue: issueOf(p.pull_request?.head.ref), note: "" };  return "ignored: not for the agent";}function receive(delivery: string, event: string, body: string, signature: string | null): string {  if (!signed(body, signature)) return "401 bad signature";  if (seen.has(delivery)) return "200 duplicate delivery, skipped";  seen.add(delivery);  const job = route(event, JSON.parse(body));  return typeof job === "string" ? `200 ${job}` : `202 queued ${job.kind} for issue #${job.issue}`;}const sign = (body: string) => `sha256=${createHmac("sha256", SECRET).update(body).digest("hex")}`;const tests: [string, string, object, boolean?][] = [  [    "d1",    "issues",    {      action: "labeled",      label: { name: "agent" },      issue: { number: 42 },      sender: { login: "maya" },    },  ],  [    "d1",    "issues",    {      action: "labeled",      label: { name: "agent" },      issue: { number: 42 },      sender: { login: "maya" },    },  ],  [    "d2",    "issue_comment",    {      action: "created",      issue: { number: 42 },      comment: { body: "Done, @acme-agent opened a PR" },      sender: { login: BOT },    },  ],  [    "d3",    "issue_comment",    {      action: "created",      issue: { number: 51, pull_request: {} },      comment: { body: "@acme-agent add a test" },      sender: { login: "maya" },    },  ],  [    "d4",    "pull_request_review",    {      review: { state: "changes_requested", body: "Handle empty input" },      pull_request: { head: { ref: "agent/issue-42" } },      sender: { login: "maya" },    },  ],  [    "d5",    "check_suite",    {      action: "completed",      check_suite: { conclusion: "failure", head_branch: "agent/issue-42" },      sender: { login: "github-actions[bot]" },    },  ],  [    "d6",    "pull_request",    {      action: "closed",      pull_request: { head: { ref: "agent/issue-42" } },      sender: { login: "maya" },    },  ],  ["d7", "issues", { action: "labeled", label: { name: "agent" }, issue: { number: 7 } }, false],];for (const [delivery, event, payload, valid = true] of tests) {  const body = JSON.stringify(payload);  console.log(    `${delivery} ${event}: ${receive(delivery, event, body, valid ? sign(body) : "sha256=0")}`,  );}

Pythonimport hashlibimport hmacimport jsonimport reSECRET = b"the-webhook-secret-from-your-github-app"BOT = "acme-agent[bot]"seen = set()  # keep delivery ids in your database in productionpr_to_issue = {51: 42}  # saved when the agent opens pull request #51def signed(body: bytes, signature: str | None) -> bool:    expected = "sha256=" + hmac.new(SECRET, body, hashlib.sha256).hexdigest()    return hmac.compare_digest(expected, signature or "")def issue_of(branch):    m = re.fullmatch(r"agent/issue-(\d+)", branch or "")    return int(m.group(1)) if m else 0def route(event, p):    if p.get("sender", {}).get("login") == BOT:        return "ignored: the agent's own action"  # no loops    if event == "issues" and p.get("action") == "labeled" and p.get("label", {}).get("name") == "agent":        return {"kind": "start", "issue": p["issue"]["number"], "note": ""}    if event == "pull_request_review" and p.get("review", {}).get("state") == "changes_requested":        return {"kind": "revise", "issue": issue_of(p["pull_request"]["head"]["ref"]),                "note": p["review"].get("body") or ""}    if event == "issue_comment" and p.get("action") == "created" and "@acme-agent" in p["comment"]["body"]:        n = p["issue"]["number"]        return {"kind": "revise", "issue": pr_to_issue.get(n, n), "note": p["comment"]["body"]}    suite = p.get("check_suite") or {}    if event == "check_suite" and suite.get("conclusion") == "failure" and issue_of(suite.get("head_branch")):        return {"kind": "fix-checks", "issue": issue_of(suite["head_branch"]), "note": ""}    ref = (p.get("pull_request") or {}).get("head", {}).get("ref")    if event == "pull_request" and p.get("action") == "closed" and issue_of(ref):        return {"kind": "cleanup", "issue": issue_of(ref), "note": ""}    return "ignored: not for the agent"def receive(delivery, event, body: bytes, signature):    if not signed(body, signature):        return "401 bad signature"    if delivery in seen:        return "200 duplicate delivery, skipped"    seen.add(delivery)    job = route(event, json.loads(body))    return f"200 {job}" if isinstance(job, str) else f"202 queued {job['kind']} for issue #{job['issue']}"def sign(body: bytes) -> str:    return "sha256=" + hmac.new(SECRET, body, hashlib.sha256).hexdigest()labeled = {"action": "labeled", "label": {"name": "agent"}, "issue": {"number": 42}, "sender": {"login": "maya"}}tests = [    ("d1", "issues", labeled, True),    ("d1", "issues", labeled, True),    ("d2", "issue_comment", {"action": "created", "issue": {"number": 42},                             "comment": {"body": "Done, @acme-agent opened a PR"}, "sender": {"login": BOT}}, True),    ("d3", "issue_comment", {"action": "created", "issue": {"number": 51, "pull_request": {}},                             "comment": {"body": "@acme-agent add a test"}, "sender": {"login": "maya"}}, True),    ("d4", "pull_request_review", {"review": {"state": "changes_requested", "body": "Handle empty input"},                                   "pull_request": {"head": {"ref": "agent/issue-42"}}, "sender": {"login": "maya"}}, True),    ("d5", "check_suite", {"action": "completed", "check_suite": {"conclusion": "failure", "head_branch": "agent/issue-42"},                           "sender": {"login": "github-actions[bot]"}}, True),    ("d6", "pull_request", {"action": "closed", "pull_request": {"head": {"ref": "agent/issue-42"}},                            "sender": {"login": "maya"}}, True),    ("d7", "issues", {**labeled, "issue": {"number": 7}}, False),]for delivery, event, payload, valid in tests:    body = json.dumps(payload).encode()    print(f"{delivery} {event}: {receive(delivery, event, body, sign(body) if valid else 'sha256=0')}")

It prints the same seven lines in both languages:

Textd1 issues: 202 queued start for issue #42d1 issues: 200 duplicate delivery, skippedd2 issue_comment: 200 ignored: the agent's own actiond3 issue_comment: 202 queued revise for issue #42d4 pull_request_review: 202 queued revise for issue #42d5 check_suite: 202 queued fix-checks for issue #42d6 pull_request: 202 queued cleanup for issue #42d7 issues: 401 bad signature

Three details are easy to miss. The comparison is constant-time, so the signature cannot be guessed byte by byte. A comment on the pull request arrives as an issue_comment with the pull request's number, so the service keeps a map from pull request to issue. And the branch name, agent/issue-42, is how check and close events find their issue again.

What does the worker do for each job?

It finds or creates the issue's sandbox, writes the task, runs the agent, runs the tests, pushes, and reports back on GitHub. Then it pauses the sandbox until the next event. The model's key and the token for git push are Runtime secrets, set up once as in the overnight Claude Code post, so the sandbox holds placeholders. Your service keeps its own GitHub token for opening the pull request and commenting:

TypeScriptimport { Runtime, Sandbox } from "withruntime";const REPO = "acme/app";const runtime = new Runtime();const headers = {  authorization: `Bearer ${process.env.GITHUB_TOKEN}`,  accept: "application/vnd.github+json",};async function github(path: string, body?: object) {  const res = await fetch(`https://api.github.com/repos/${REPO}${path}`, {    method: body ? "POST" : "GET",    headers,    body: body ? JSON.stringify(body) : undefined,  });  if (!res.ok) throw new Error(`GitHub ${path}: ${res.status} ${await res.text()}`);  return res.json();}type Job = { kind: "start" | "revise" | "fix-checks" | "cleanup"; issue: number; note: string };export async function work(job: Job): Promise<number | undefined> {  const name = `${REPO.replace("/", "-")}-issue-${job.issue}`;  if (job.kind === "cleanup") {    for await (const sbx of await runtime.sandboxes.list({ name })) await sbx.delete();    return;  }  const branch = `agent/issue-${job.issue}`;  const sbx = await Sandbox.getOrCreate(name, {    vcpu: 2,    memoryMiB: 4096,    labels: { repo: REPO.replace("/", "-"), issue: String(job.issue) },  });  const app = { cwd: "/workspace/app" };  if (!sbx.info.reused) {    await sbx.exec(["git", "clone", `https://github.com/${REPO}.git`, "/workspace/app"], {      check: true,    });    // Picks up the branch if an earlier sandbox for this issue already pushed it.    await sbx.exec(`git switch ${branch} || git switch -c ${branch}`, { ...app, check: true });  }  const issue = await github(`/issues/${job.issue}`);  const why = {    start: "Resolve the issue.",    revise: "Address this review:",    "fix-checks": "CI failed. Read the failing check and fix it.",  };  await sbx.files.write(    "/workspace/TASK.md",    `# ${issue.title}\n\n${issue.body ?? ""}\n\n## Now\n\n${why[job.kind]}\n\n${job.note}\n`,  );  await sbx.exec(process.env.AGENT_COMMAND!, { ...app, timeoutMs: 1_800_000 }); // a round's ceiling  if (await sbx.files.exists("/workspace/QUESTION.md")) {    const question = await sbx.files.readText("/workspace/QUESTION.md");    await sbx.exec("rm /workspace/QUESTION.md");    await github(`/issues/${job.issue}/comments`, { body: `Before I go on: ${question}` });    await sbx.pause();    return;  }  const tests = await sbx.exec("npm test 2>&1 | tail -20", { ...app, timeoutMs: 600_000 });  const commit = `git add -A && git -c user.name="Acme Agent" -c user.email=agent@acme.dev commit -qm ${JSON.stringify(`#${job.issue}: ${job.kind}`)} && git push -q -u origin ${branch}`;  const pushed = await sbx.exec(commit, app);  if (pushed.exitCode === 0 && job.kind === "start") {    const pr = await github("/pulls", {      title: issue.title,      head: branch,      base: "main",      draft: true,      body: `Closes #${job.issue}\n\nTests:\n\n\`\`\`\n${tests.stdout}\n\`\`\``,    });    await sbx.pause();    return pr.number; // save pull request -> issue for comments on the pull request  }  const said = pushed.exitCode === 0 ? "Pushed a new commit." : "No change made this round.";  await github(`/issues/${job.issue}/comments`, {    body: `${said}\n\n\`\`\`\n${tests.stdout}\n\`\`\``,  });  await sbx.pause();}

Pythonimport jsonimport osimport urllib.requestfrom withruntime import Runtime, SandboxREPO = "acme/app"runtime = Runtime()HEADERS = {"authorization": f"Bearer {os.environ['GITHUB_TOKEN']}",           "accept": "application/vnd.github+json", "content-type": "application/json"}WHY = {"start": "Resolve the issue.", "revise": "Address this review:",       "fix-checks": "CI failed. Read the failing check and fix it."}def github(path, body=None):    req = urllib.request.Request(f"https://api.github.com/repos/{REPO}{path}", headers=HEADERS,                                 data=json.dumps(body).encode() if body else None,                                 method="POST" if body else "GET")    with urllib.request.urlopen(req, timeout=30) as res:        return json.load(res)def work(job):    name = f"{REPO.replace('/', '-')}-issue-{job['issue']}"    if job["kind"] == "cleanup":        for sbx in runtime.sandboxes.list(name=name):            sbx.delete()        return None    branch = f"agent/issue-{job['issue']}"    sbx = Sandbox.get_or_create(name, vcpu=2, memory_mib=4096,                                labels={"repo": REPO.replace("/", "-"), "issue": str(job["issue"])})    app = "/workspace/app"    if not sbx.info.get("reused"):        sbx.exec(["git", "clone", f"https://github.com/{REPO}.git", app], check=True)        # Picks up the branch if an earlier sandbox for this issue already pushed it.        sbx.exec(f"git switch {branch} || git switch -c {branch}", cwd=app, check=True)    issue = github(f"/issues/{job['issue']}")    sbx.files.write("/workspace/TASK.md", f"# {issue['title']}\n\n{issue.get('body') or ''}\n\n"                    f"## Now\n\n{WHY[job['kind']]}\n\n{job['note']}\n")    sbx.exec(os.environ["AGENT_COMMAND"], cwd=app, timeout_ms=1_800_000)  # a round's ceiling    if sbx.files.exists("/workspace/QUESTION.md"):        question = sbx.files.read_text("/workspace/QUESTION.md")        sbx.exec("rm /workspace/QUESTION.md")        github(f"/issues/{job['issue']}/comments", {"body": f"Before I go on: {question}"})        sbx.pause()        return None    tests = sbx.exec("npm test 2>&1 | tail -20", cwd=app, timeout_ms=600_000)    message = json.dumps(f"#{job['issue']}: {job['kind']}")    pushed = sbx.exec(f'git add -A && git -c user.name="Acme Agent" -c user.email=agent@acme.dev '                      f"commit -qm {message} && git push -q -u origin {branch}", cwd=app)    if pushed.exit_code == 0 and job["kind"] == "start":        pr = github("/pulls", {"title": issue["title"], "head": branch, "base": "main", "draft": True,                               "body": f"Closes #{job['issue']}\n\nTests:\n\n```\n{tests.stdout}\n```"})        sbx.pause()        return pr["number"]  # save pull request -> issue for comments on the pull request    said = "Pushed a new commit." if pushed.exit_code == 0 else "No change made this round."    github(f"/issues/{job['issue']}/comments", {"body": f"{said}\n\n```\n{tests.stdout}\n```"})    sbx.pause()    return None

AGENT_COMMAND is whatever coding agent you run headless, reading /workspace/TASK.md: Claude Code, Codex or your own loop (coding agent sandbox). The worker's tests are a report for the reviewer, not a gate; your normal CI still runs on the pull request (CI for agent pull requests).

What should the agent be told?

The issue, the round's instruction, and a way out. Put these three lines in the agent's standing instructions:

• Stay on the issue. Change only what it needs; do not edit CI configuration, lock files or tests you were not asked to touch.
• Prove it. Run the tests before finishing, and add one that fails without your change.
• Ask instead of guessing. If the issue is ambiguous, write the question to /workspace/QUESTION.md and stop.

The last one is what makes the agent safe to point at real issues. An agent that guesses opens a confident pull request for the wrong reading; one that asks gets a one-line answer from the issue's author, and the comment wakes the sandbox to continue.

How do you keep it from doing damage?

By making a pull request the only thing it can produce. Protect the default branch so every change needs an approved review; the agent's token can push its own branches and nothing lands without a person. Open pull requests as drafts. Cap each round with the agent command's timeout, cap the number of rounds per issue in your service (five is plenty), and set a daily spending limit on the account so a runaway day has a ceiling. The sandbox's network rules can be narrowed to GitHub, your package registry and the model API.

What does one issue cost?

Take one issue from label to merge: a first round of 20 minutes, three review rounds of 8 minutes each, all on 2 vCPU and 4 GiB with the agent's tools averaging half a vCPU, and two days paused in between at 1.5 GB:

TextCPU:     44 min × 0.5 vCPU × $0.025 / 60          = $0.0092Memory:  44 min × 4 GiB × $0.0075 / 60            = $0.0220Paused:  1.5 GB × $0.08 × 2 days / 30 days    = $0.0080Total:                                          $0.0392

The model's tokens for those 44 minutes will cost many times more. A hundred issues a month is about $3.92 in sandboxes (pricing).

In short

• An issue agent is an event handler: label, review, comment, failed check and close, each mapped to one sandbox action.
• Keep one sandbox per issue for the pull request's life, paused between rounds, so each review starts where the last one stopped.
• Verify GitHub's signature, skip repeated deliveries, ignore the agent's own events and answer within ten seconds.
• Let the agent ask a question instead of guessing, and make a reviewed pull request the only thing it can produce.
• One issue with three review rounds costs about $0.0392 in sandbox time.

Run it on Runtime

Runtime costs 42% to 88% less than fourteen other sandbox providers for an agent that mostly waits on a model (compare costs). A Runtime sandbox named after the issue is found again by getOrCreate, pauses between rounds for storage only, and runs its next command 153 ms after the wake request on Runtime's servers. Secrets keep the GitHub and model keys out of the machine. Start with 100 free hours, no card: sign in or read get started.