Fix Daily Rig Decomposition Benchmark timeout by replacing the LLM coordinator with a workflow root

[Copilot]

The Daily Rig Decomposition Benchmark job exceeded its 55-minute limit. In run 31380038557 the agent step ran 48m18s: the benchmark's root was an LLM agent that had to re-emit the entire report as structured JSON, failed after ~13 minutes with Agent decomposition-bench-coordinator returned invalid JSON: No JSON value found., and the outer agent then spent ~25 more minutes rewriting and re-running the program before the job was killed.

All changes are in .github/workflows/daily-rig-decomposition-bench.md and its recompiled .lock.yml.

Replace the coordinator agent with workflow()

decomposition-bench-coordinator existed only to call a tool and echo its result back through the model. Its output schema (BenchReport) included the full generated program source plus every attempt's typecheck/execute output — thousands of tokens the model had to reproduce verbatim, which is where structured-output parsing broke.

// before: the whole report round-trips through a small model
const benchmarkCoordinator = agent({
  name: "decomposition-bench-coordinator",
  tools: [verifyProgram, runDecompositionBenchmark],
  instructions: "Call run_decomposition_benchmark exactly once and return its result unchanged.",
  output: BenchReport,
});

// after: deterministic TypeScript owns orchestration; the launcher returns the report directly
const decompositionBenchmark = workflow({
  meta: { name: "decomposition-bench", phases: ["Pick task", "Single call", "Decompose", "Grade"], /* ... */ },
  body: async ({ phase, log }) => { /* same sequence, no model in the loop */ },
});

BenchReport, Attempt, and the verify_program tool are now unused and removed; gradeSolutions reuses the existing Grading schema instead of an inline duplicate.

Bound the wall clock

Previously each subprocess got a flat 10 minutes with no global budget, so 2 attempts plus the single-call solve and large-model grading could overrun 55 minutes even with no failures.

• BENCH_DEADLINE — 25-minute budget for the whole benchmark, leaving the outer agent time to file the issue.
• runRigEntry(source, flags, timeoutMs) takes an explicit timeout clamped to the remaining budget, and returns immediately once exhausted.
• Typecheck 2 min / execute 5 min, and the retry loop breaks rather than starting an attempt it cannot finish.

Bound the outer agent

Added prose instructing the agent to run the program exactly once and report the failure with captured stderr, rather than rewriting and re-running it — the behavior that consumed the second half of the timed-out run.

Notably, this does not bump timeout-minutes to 65 as the failure issue suggested; that would have masked the structured-output failure while still risking an overrun.

• Fixes [aw] Daily Rig Decomposition Benchmark timed out #397

[pelikhan]

@copilot do not inline the rig program, instead provide a natural language spec + tell the agent to turn it into a rig then run it.

[Copilot]

De-inlined in 8fbb700. The body is now a spec in three steps: write the program to /tmp/gh-aw/agent/bench.ts and typecheck it via cat bench.ts | node rig.ts --typecheck, run it exactly once, then report — including the program the agent wrote in the issue body.

The behavior is described in prose (pick task → single call → decomposition with up to 2 verify/repair attempts → grade), with a "Hard requirements" section carrying the timeout fixes forward so the generated program inherits them: the root must be a workflow() rather than a coordinator agent relaying the report, and the program must impose its own ~25-minute deadline with subprocess timeouts clamped to the remaining budget. The agent is also pointed at SKILL.md first so it writes against the current API instead of guessing.