ruby: Replay worker history on retries

[dpashk-figma]

Describe

• Preserve process-local artifacts when a distributed Minitest worker is retried.
• Add opt-in worker-history retry selection while keeping failure-only selection as the default.
• Fail closed when the worker's durable reservation history or suite-chunk metadata cannot reconstruct the replay set, or when a cutoff stops replay before completion.

Implementation design

A Redis worker already records every reservation it receives. The existing retry path intersects those reservations with the build's failed-test set. That is correct for rerunning failures, but it cannot reconstruct process-local output such as SimpleCov data: tests that passed before a worker was terminated are omitted from the retry.

This change makes retry selection explicit:

queue.retry_queue(selection: :failed_tests)   # existing default
queue.retry_queue(selection: :worker_history) # opt-in replay

Minitest exposes the same choice at the CLI:

--retry-selection failed-tests
--retry-selection worker-history

failed-tests preserves the current behavior and ordering. worker-history reads the retried worker's existing build-scoped reservation list, restores chronological order, expands suite-bin-packing chunk reservations through their existing Redis metadata, and deduplicates test IDs while preserving first-reservation order.

flowchart LR
  A["Buildkite retries worker N"] --> B["Read worker N reservation history"]
  B --> C["Expand chunks and deduplicate test IDs"]
  C --> D["Create the existing local Redis::Retry queue"]
  D --> E["Replay worker N's complete reserved test set"]

Loading

The replay uses the existing Redis::Retry static queue. There is no recovery wrapper, second shared-queue phase, or recovery manifest. Static queues honor shutdown, and worker-history retry queues additionally require complete exhaustion: shutdown or a circuit breaker that stops replay early fails the retry rather than accepting a partial process-local artifact.

Worker-history selection fails closed when:

• the stable worker ID has no reservation history;
• a reserved suite chunk has missing or malformed metadata;
• a reserved suite chunk contains no test IDs; or
• shutdown or another cutoff stops replay before every selected test runs.

Centralized reporting caveat

A worker-history retry does not rejoin the shared Redis queue. In the intended single-worker failure case, surviving workers continue draining shared work while the restarted worker reconstructs its own process-local output. If the full fleet is restarted before the shared queue is exhausted, no retry may remain to claim undistributed work; centralized reporting will then fail the build. This correlated-failure limitation is deliberate and documented: it may require rerunning the build, but it cannot publish an incomplete artifact as a successful run.

Fixes: https://app.asana.com/1/10497086658021/project/1217018759338815/task/1217018759338827

Test Plan

• Focused Redis worker tests cover default failure ordering, chronological worker-history replay, stable deduplication, suite-chunk expansion, fail-closed metadata errors, and incomplete replay detection.
• Static queue tests cover signal-driven shutdown during replay.
• Minitest subprocess tests cover a successful first run followed by a complete worker-history retry, missing-history failure, and a circuit-breaker cutoff that fails an incomplete replay.
• bundle exec rake build succeeds.
• GitHub's supported Ruby 3.0-3.3 matrix exercises the complete Ruby suite.

[yfang-figma]

LGTM. Thanks!