Put an AI Reviewer on Every Pull Request: A Webhook Build with OpenCode and Railway

The bottleneck has moved. Coding agents generate pull requests faster than any human team can read them, and the review queue is where AI-assisted development backs up. One agent can open a PR every few minutes; a reviewer still takes twenty minutes to do the diff justice. The result is merged-with-a-glance or hours of unpaid night work, and both ship the interesting bugs.

The fix is a second pair of machine eyes that never sleeps and never skims: a webhook service that runs a headless coding agent against every PR in your repo, asks it for the three to five findings that actually matter, and posts them as a real GitHub review - without ever writing code itself. This guide builds that service end to end.

The receiving end is a small webhook service on Railway, and the reviewer is OpenCode in headless mode - the same opencode run pattern from our cron automation guide, pointed at the opposite job of the issue-to-PR webhook: that build turns labeled issues into pull requests, this one reviews them. We run a version of this shape for parts of this site; the mechanics below are the portable core. Seven steps, under an hour, every step ending in something you can run.

Official Sources#

Step 1: Install OpenCode and pick the review model#

Prerequisites: a GitHub repo you own (a small one with real code is ideal), a free Railway account (new accounts get a one-time $5 trial grant per Railway's free trial docs, as of 2026-09-25), and an API key for an LLM provider.

Install OpenCode with the official one-liner from the OpenCode docs, authenticate a provider (opencode auth login), then prove the capability the whole pattern depends on - one task, no TUI, clean exit:

curl -fsSL https://opencode.ai/install | bash
opencode auth login
opencode run --model opencode/deepseek-v4-flash "print the current directory tree, two levels deep"

If that prints a tree and returns, you have a reviewer. On model choice, the rule from our agent fleet economics post applies: escalate the judge, not the worker. A review is bounded and cheap to gate, so budget is the right default and a stronger model is one environment variable away.

The budget sweet spot for this job: DeepSeek's Flash line at $0.30 per million input tokens and $1.20 per million output tokens at peak hours, half that off-peak, per the DeepSeek pricing page as of 2026-09-25. Note the naming churn: DeepSeek retired the original V4 Flash on 2026-09-10, and deepseek-flash (plus the legacy deepseek-v4-flash alias OpenCode still accepts) now routes to the newer V4.1 Flash at the reduced price, per the change log as of 2026-09-10. The opencode/deepseek-v4-flash id from our earlier post keeps working - it is an alias, and it is the cheap default in the runner below. OpenCode reads provider API keys from the environment, so the same setup runs on a server with no interactive login.

What you have now: a proven headless agent command. One review costs about one to two cents at peak prices, so the gate below can afford to run on every PR.

Step 2: The webhook receiver for pull_request events#

The receiver is deliberately boring: standard library only, two routes - verify the request is really from GitHub, check the event type from the X-GitHub-Event header, acknowledge instantly, hand the payload to the runner. The GitHub docs are explicit about two rules this file honors: subscribe to the minimum number of events, and answer within 10 seconds with a 2XX or GitHub terminates the delivery. Both are built in - one subscribed event (pull_request), and a 202 sent before any heavy work starts.

// server.js
const http = require("node:http");
const crypto = require("node:crypto");
const { spawn } = require("node:child_process");

const SECRET = process.env.GITHUB_WEBHOOK_SECRET;
const ACTIONS = ["opened", "synchronize", "ready_for_review"];

function signatureMatches(rawBody, header) {
  if (!header || !SECRET) return false;
  const expected =
    "sha256=" +
    crypto.createHmac("sha256", SECRET).update(rawBody).digest("hex");
  const a = Buffer.from(expected);
  const b = Buffer.from(header);
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

http
  .createServer((req, res) => {
    if (req.method === "GET" && req.url === "/healthz") {
      res.writeHead(200);
      return res.end("ok");
    }
    if (req.method !== "POST" || req.url !== "/webhook") {
      res.writeHead(404);
      return res.end();
    }

    const chunks = [];
    req.on("data", (c) => chunks.push(c));
    req.on("end", () => {
      const raw = Buffer.concat(chunks).toString();
      if (!signatureMatches(raw, req.headers["x-hub-signature-256"])) {
        res.writeHead(401);
        return res.end("bad signature");
      }

      const event = JSON.parse(raw);
      const pr = event.pull_request;
      const relevant =
        req.headers["x-github-event"] === "pull_request" &&
        ACTIONS.includes(event.action) &&
        pr &&
        pr.draft !== true;

      // GitHub expects a 2xx within 10 seconds. Acknowledge now,
      // run the review in the background.
      res.writeHead(202);
      res.end("accepted");

      if (!relevant) return;

      const child = spawn("bash", ["review.sh"], {
        detached: true,
        stdio: "ignore",
        env: {
          ...process.env,
          REPO: event.repository.full_name,
          PR_NUMBER: String(pr.number),
          HEAD_SHA: pr.head.sha,
          BASE_REF: pr.base.ref,
        },
      });
      child.unref();
    });
  })
  .listen(process.env.PORT || 3000);

Three details are the security story:

• Signature first. GitHub signs every delivery with your secret as HMAC-SHA256 in X-Hub-Signature-256; the timing-safe comparison over the raw body is the documented validation pattern and stops random internet traffic spending your tokens.
• Filter on header and payload. Checking X-GitHub-Event before touching the body, and filtering actions to opened, synchronize (new commits pushed), and ready_for_review (a draft marked ready), keeps the bot quiet outside the moments that matter - the action list comes straight from the webhook events docs.
• Drafts never get reviewed. The payload's pull_request.draft flag (part of the pull request object in the Pulls REST API) makes the filter one line, and no review lands until someone says the PR is ready.

What you have now: a receiver that can only ever do nothing or spawn a review. Test it locally: PORT=3000 GITHUB_WEBHOOK_SECRET=test node server.js.

Step 3: The review runner#

One script, six moves: fresh clone, checkout the PR branch, write the diff to a file, one bounded agent run, validate the output, post the review. Note what is missing on purpose: no git push anywhere. This bot comments; it does not write code.

#!/bin/bash
# review.sh - runs from the webhook receiver
set -eu
: "${REPO:?}"; : "${PR_NUMBER:?}"; : "${GITHUB_TOKEN:?}"; : "${HEAD_SHA:?}"

export GH_TOKEN="$GITHUB_TOKEN"
gh auth setup-git

WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
cd "$WORK"

# A blobless clone gets full history for the merge-base diff
# without downloading every file's contents up front.
git clone --filter=blob:none "https://github.com/${REPO}.git" .
gh pr checkout "$PR_NUMBER"

# Already reviewed this exact commit? Quiet exit. The stamp lives on
# GitHub itself, so it survives service restarts with no extra storage.
if gh api "repos/${REPO}/pulls/${PR_NUMBER}/reviews" \
     --jq '.[] | select(.body | contains("reviewed-sha: '"$HEAD_SHA"'"))' \
     | grep -q .; then
  exit 0
fi

git fetch origin "${BASE_REF:-main}"
git diff "origin/${BASE_REF:-main}...HEAD" > diff.txt
test "$(wc -l < diff.txt)" -gt 1500 && echo "diff too large" && exit 0

PROMPT="You are a senior engineer doing a pull request review. \
The file diff.txt in this directory is the complete diff between the \
base branch and the head of PR #${PR_NUMBER} in ${REPO}. \
Read the diff, then inspect the actual files in this repo where a \
finding needs verification. Report exactly 3 to 5 findings, most \
severe first, skipping anything that is style preference. For each \
finding, one line of severity (blocker | should-fix | nit), the file \
path and line from the diff, one sentence describing the problem, and \
one concrete suggested fix. If nothing is wrong, write 'No findings.' \
Write the report to review.md in this directory. End the file with a \
line: reviewed-sha: ${HEAD_SHA}"

timeout 900 opencode run --model "${AGENT_MODEL:-opencode/deepseek-v4-flash}" \
  --file diff.txt "$PROMPT"

# Quiet no-op exits: most reviews should find little, and silence
# is the signal that nothing changed.
if [ ! -s review.md ] || [ "$(wc -l < review.md)" -gt 200 ]; then
  exit 0
fi
grep -q "reviewed-sha: ${HEAD_SHA}" review.md || \
  printf "\nreviewed-sha: %s\n" "$HEAD_SHA" >> review.md

gh pr review "$PR_NUMBER" --comment --body-file review.md

The pipeline details:

• gh pr checkout fetches the PR head and checks out its branch in the fresh clone, per the gh CLI manual.
• The diff is data, not prompt glue. It lands in diff.txt and is attached with OpenCode's --file flag (CLI docs), and head.sha and base.ref arrive via environment variables - a PR title full of shell metacharacters is data, never a command.
• The review is posted with gh pr review --comment --body-file review.md, the documented form of the create-review endpoint, whose event field accepts APPROVE, REQUEST_CHANGES, or COMMENT (REST docs). COMMENT is the honest default: the bot has opinions, not vetoes.

Runnable check: run the script once by hand against a test PR: REPO=you/test PR_NUMBER=1 GITHUB_TOKEN=... HEAD_SHA=<head-sha> BASE_REF=main ./review.sh. Watch it clone, review, and post. A reviewer you have never watched succeed is not ready to run unattended.

Step 4: Dedupe and the safety rails#

The failure mode of an always-on reviewer is not one bad comment, it is a bot repeating itself on every push. The rails that matter:

• The stamp is the dedupe. Every review body carries reviewed-sha: <HEAD_SHA>, and the runner checks for it before each run - GitHub itself is the state store, so a restart cannot double-post. A mkdir-based lock on the PR number guards the rare double-delivery race.
• timeout on the agent. A review loop at 3am dies at the bound you set; 900 seconds covers real reviews, and the run happens in the background after the 202, so it never contends with Railway's edge timeout (5 minutes with no data transferred, 15 minutes if data keeps flowing, per Railway's specs as of 2026-09-25).
• Quiet no-op exits and a size gate. Most changes should produce nothing worth saying - if the bot comments on every PR, its prompt is padded - and a diff over 1500 lines is almost certainly noise: skip it instead of burning tokens on a megadiff.
• A comment-only token. A fine-grained personal access token scoped to Contents Read, Pull requests Read and write, and Metadata Read on this one repo: it can post reviews and read code, nothing else. Compromised, the blast radius is one repo and one revocable token.
• Spend caps. The bounded run and budget model keep a review near a cent, but the $400 overnight bill is the canonical failure mode - set provider spending alerts before you enable the webhook.

What you have now: a reviewer that is safe to point at real traffic, by construction.

Step 5: Containerize and deploy to Railway#

Three files become a service. The Dockerfile is short because everything is a package: OpenCode installs from npm as opencode-ai, and gh and git come from apt.

FROM node:22-slim

RUN apt-get update && apt-get install -y git gh && rm -rf /var/lib/apt/lists/*
RUN npm install -g opencode-ai

WORKDIR /app
COPY server.js review.sh ./
RUN chmod +x review.sh

CMD ["node", "server.js"]

Push server.js, review.sh, and the Dockerfile to a new GitHub repo. In Railway, create a new project and deploy from that repo - Railway detects the Dockerfile automatically, per the Services docs. Then add these variables in the service's Variables tab:

The RAW editor accepts a whole .env, and values can be sealed so they are never visible again (Railway docs). The agent inside the container reads the same env vars OpenCode reads locally, so no interactive login ever happens on the server.

Last, expose it: Settings → Networking → Public Networking → Generate Domain. Railway provisions a *.railway.app domain with automatic SSL (Public Networking docs), satisfying GitHub's requirement to deliver webhooks over HTTPS - and a health check pointed at /healthz gives you uptime monitoring for free.

Runnable check: curl https://<your-service>.up.railway.app/healthz returns ok.

Step 6: Wire GitHub to the endpoint#

In your target repository: Settings → Webhooks → Add webhook:

• Payload URL: https://<your-service>.up.railway.app/webhook
• Content type: application/json
• Secret: the same random string from Step 5
• Which events: "Let me select individual events", and subscribe only to Pull requests

Save it. GitHub immediately sends a ping event - the Recent Deliveries tab shows both sides, and your service logs show the 202.

Runnable check: the delivery shows a 200 or 202 and no review is spawned (a ping is not a pull_request event, and the receiver's event-type check drops it).

Step 7: The first PR that reviews itself#

Open a real PR on the repo - a small change with one planted issue in it, like an unawaited promise that swallows an error. Then push a second commit to the branch. Watch: the synchronize event fires, the signature checks, the receiver answers 202 in milliseconds, a clone happens, OpenCode reads the diff and the code behind it, and a review lands on the PR naming the problem and a fix.

From here, compounding is a prompt change away: add "check against CONTRIBUTING.md", or use the bot as first-pass triage for agent-authored PRs (the policy half lives in our governance post, the mechanism here). If a repo gets noisy, gate the receiver on a review label the way the issue-to-PR build gates on agent.

What you have now: a repo where every pull request gets a fresh pair of machine eyes in under two minutes, for about a cent per review. The shape that makes it safe is repeated at every step here: verify, acknowledge fast, run one bounded job in a clean checkout, ship the result as a reviewable artifact - the harnesses post says it plainly, the script around the agent is the product.

FAQ#

Does the bot review draft PRs?#

No. The receiver checks the payload's pull_request.draft flag and drops the event - the only triggers are opened, synchronize (new commits on the branch), and ready_for_review, per the pull_request webhook events.

How does it avoid re-reviewing the same commit?#

Every posted review carries a reviewed-sha: <head-sha> stamp. Before each run the runner lists existing reviews via the Pulls reviews REST endpoint and skips when the stamp matches the current head. New commits change the SHA, so only new revisions get reviewed.

Can the bot request changes or approve?#

Yes, with one variable: the create-review endpoint accepts APPROVE, REQUEST_CHANGES, or COMMENT, and gh pr review exposes the same three via --approve, --request-changes, and --comment. Start with --comment: a machine veto on every PR trains your team to filter it out; a machine opinion does not.

How much does this cost to run?#

On Railway's Hobby plan ($5 per month with $5 of included resource usage, per Railway's plans page as of 2026-09-25), a service this small usually stays inside the included usage. Reviews run about one to two cents each on the DeepSeek Flash tier ($0.30 in / $1.20 out per million tokens at peak, half off-peak, per the pricing page as of 2026-09-25), so even a busy repo stays under a few dollars a month - as long as the timeout and spend alerts contain the loops.

Will this review agent ever commit code?#

It cannot. The runner contains no git push, and the token is scoped to read code and write reviews only. Reviewer agents hold opinions; humans hold the merge button.

Some links to tools above are referral links - see our affiliate disclosure.

Sources#

Last updated: September 25, 2026

Continue Reading#

• Put an AI Agent Behind a Webhook: Turn GitHub Issues into Pull Requests - the sibling build that opens agent PRs; this bot reviews exactly those PRs
• Put an AI Agent on a Cron Job - the same headless opencode run pattern on a schedule
• Why AI Code Review Is Becoming the Bottleneck - the analysis half: what gates fix the review queue
• AI Coding Agents Are Piling Up Review Queues - the queue mechanics and the receipt reviewers should ask for
• Long-Running Agents Need Harnesses - why the script around the agent matters
• The $400 Overnight Bill - agent FinOps, learned the hard way