Shipping OpenAI Symphony in Prod: A Real-World Guide

Why Symphony Matters

When OpenAI open-sourced Symphony, they buried the headline number under a wall of Elixir docs: a 500% increase in PR throughput inside their own infra teams. That is not a "we shipped a chatbot" stat. That is "we replaced a junior engineering pod with a fleet of Codex agents and the kanban moves itself." The thesis Symphony forces you to internalize is manage work, not agents. You stop thinking about prompts and start thinking about tickets, queues, and merge gates.

I forked the repo on a Saturday night, pointed it at a live Linear board for one of my smaller DD products, and watched it land four PRs before I finished my coffee. This post is the unredacted ops story — what setup actually looks like on Apple Silicon, what the runtime feels like in production, and where it breaks.

If you want the visual demo first, the DevDigest YouTube channel has the screen-recording walkthrough where Linear ticket goes in and PR comes out in under three minutes.

Why Elixir / BEAM Was the Right Pick

Before the setup walkthrough, this matters: Symphony runs on the BEAM. That sounds like an aesthetic choice until you watch six Codex agents melt down concurrently and the supervisor tree just… restarts the failing one. No restart loop in your Node process. No half-dead Python worker holding a file lock. The actor model is what lets a single laptop juggle a real fleet without becoming a babysitting job.

If you have ever tried to orchestrate agents with a Python asyncio gather and a Postgres queue, you already know why this matters. Symphony's choice to put each agent run in its own supervised process means a runaway Codex session crashes itself, not the orchestrator.

Setup: From git clone to First Run

The README is honest about engineering preview status, but it underplays how rough Erlang/OTP install is on Apple Silicon. Here is the actual sequence that worked on M3 Max, macOS 15:

# 1. Erlang via asdf (brew install erlang takes ~30min and frequently fails)
brew install asdf
asdf plugin add erlang
KERL_CONFIGURE_OPTIONS="--without-javac --with-ssl=$(brew --prefix openssl@3)" \
  asdf install erlang 27.1.2
asdf install elixir 1.17.3-otp-27

# 2. Clone and bootstrap
git clone https://github.com/openai/symphony.git
cd symphony
mix deps.get
mix ecto.setup

Three env vars do real work:

export OPENAI_API_KEY=sk-...
export LINEAR_API_KEY=lin_api_...
export SYMPHONY_WORKSPACE_ROOT=$HOME/symphony-runs

SYMPHONY_WORKSPACE_ROOT is the one most people miss. Symphony clones target repos into ephemeral subdirectories under that root. If you do not set it, it picks /tmp and macOS nukes it on reboot mid-run. Ask me how I know.

For Codex auth, Symphony uses your existing codex CLI session, so codex login once on the host and Symphony inherits it. The Linear API key needs read and write on issues — read only is not enough because Symphony writes status comments back to the ticket.

Anatomy of a Run

Once you boot Symphony with mix phx.server and open the dashboard at localhost:4000, you connect a Linear team. Symphony pulls open issues with the label agent (configurable) and queues them. Each run looks like this:

1. Plan phase — Codex reads the ticket, the linked repo, and produces a plan in a sandboxed worktree.
2. Implement phase — Codex edits files, runs tests, iterates. The whole thing happens inside a per-run git worktree under SYMPHONY_WORKSPACE_ROOT.
3. Review phase — Symphony pushes a branch and opens a PR with the ticket linked.
4. Status phase — Symphony comments back on the Linear ticket with the PR URL and moves it to In Review.

The isolation boundary is a git worktree plus a process-level chroot of sorts. The agent sees the repo, your shell, your test runner. It does not see your other repos, your home dir, your secrets file. That is the whole point. If you want true container-level isolation, you can wire SYMPHONY_RUN_COMMAND to wrap each run in a Docker invocation — there is a stub for this in lib/symphony/runner.ex but it is not wired up by default.

Production Hardening

Out of the box Symphony will happily spawn 20 concurrent Codex runs and burn $40 in 15 minutes. Three knobs matter:

# config/runtime.exs
config :symphony, Symphony.Orchestrator,
  max_concurrent_runs: 3,
  max_run_duration_ms: 20 * 60 * 1000,
  max_cost_per_run_usd: 2.50

max_cost_per_run_usd is the one I would not ship without. Symphony tracks token spend per run via the OpenAI API response headers and will kill the run if it exceeds the cap. I had a single ticket on a gnarly refactor consume $11 in tokens before I added this. Now nothing crosses $2.50 without my review.

For observability, Symphony emits OTel spans for every phase. I pipe them into DD Traces so I can see exactly where each run spent its tokens — turns out 60% of cost on most tickets is the plan phase re-reading the same files. Caching prompts for the implement phase cut my bill in half.

The other thing nobody documents: Symphony's kill_run/1 is a soft kill. It signals the agent process. If Codex is mid-API-call, the call completes and bills you. If you actually want hard-kill semantics, patch lib/symphony/runner.ex to Process.exit(pid, :kill) instead of the graceful path.

Where Symphony Breaks Down

This is engineering preview, not GA, and it shows in three places:

No auth roles. Anyone with the dashboard URL can trigger runs and spend your tokens. Put it behind a VPN or Cloudflare Access. There is no built-in user model.

No native multi-repo. Each Linear team maps to one repo. If your ticket touches two repos, Symphony picks the first and the agent fakes the rest. I hit this on a frontend/backend coordinated change and had to manually split the ticket.

No retry queue. When a run fails (rate limit, transient git error, flaky test), Symphony marks the ticket failed and stops. There is no exponential-backoff retry. I built a 30-line GenServer wrapper that catches :run_failed events and reschedules with a backoff. OpenAI will probably ship this soon but until then, expect to write it.

OpenAI's own README says "do not run this in production yet." Take them seriously. This is a fork-and-run-on-your-laptop tool today, not a multi-tenant SaaS.

My Ship-It Verdict

For a solo dev or a small team where one person is the platform owner, Symphony is the most leveraged thing I have run this year. Six Codex agents working a backlog feels like running a junior team without standups. But it is your responsibility to keep the wheels on — cost caps, observability, kill switches.

If you are at a 50-person eng org, wait for the productized version. The auth and multi-repo gaps are real.

If you want a lighter-weight orchestration pattern that does not require Erlang, check out DD Orchestrator — same management-not-agents thesis, simpler runtime, less throughput. The right pick depends on how much volume you are actually pushing through.

What I would steal for any agent stack regardless of which orchestrator you pick: per-run cost caps, OTel-everywhere, sandboxed git worktrees, and the discipline to label tickets agents are allowed to touch. Those four habits alone are worth the whole exercise.

The 500% PR uplift number is real, but only if you do the ops work. Symphony hands you the chassis. You still have to drive.