Skip to content

test(firestore): wait out the emulator's Listen-stream backoff - #791

Open
tyler-reitz wants to merge 1 commit into
FirebaseExtended:mainfrom
tyler-reitz:fix/emulator-backoff-timeouts
Open

test(firestore): wait out the emulator's Listen-stream backoff#791
tyler-reitz wants to merge 1 commit into
FirebaseExtended:mainfrom
tyler-reitz:fix/emulator-backoff-timeouts

Conversation

@tyler-reitz

Copy link
Copy Markdown
Contributor

Why

test/firestore.test.tsx flakes at roughly 18% per run (11 of 60 iterations measured 2026-08-11). The Firestore emulator intermittently corrupts a Listen frame, grpc-js reads four body bytes as a length prefix, and the SDK answers the resulting RESOURCE_EXHAUSTED with a 60 second maximum backoff before self-healing.

This is an unresolved upstream emulator bug, firebase-tools#8654. Nothing to pin, so this survives it rather than preventing it.

What

Four waitFor calls in the two tests that read a nonexistent document get a 120s budget, and those two tests get a 150s vitest timeout. They are the only tests needing a live server round trip, so the only ones that can die; the rest are served from the local cache. waitFor polls at 50ms, so the larger ceiling costs nothing on a healthy run: 80-114ms per test with the change over five runs, 79-105ms without over three.

120s rather than 60s because a third party on the same upstream bug found 60s insufficient at least once and observed a self-heal at ~35s under 120s.

What is not verified

That this rescues a real occurrence. The desync has never reproduced on a dev machine (0 stalls in five fresh-emulator runs here). What is verified is the budget itself: a simulated 65s stall passes under it and fails under the default.

The evidence is CI after this lands: desyncs should keep appearing in logs while the flake stops. That is also the signal for closing #776.

The scoped cost: a genuine regression in these two tests now takes 120s to report instead of 1s. That is confined to the two tests this bug can kill.

Refs #776

test/firestore.test.tsx flakes in CI at roughly 18% per run (11 of 60
iterations, 2026-08-11). The Firestore emulator intermittently corrupts a
Listen frame, so grpc-js reads four body bytes as a length prefix and reports
RESOURCE_EXHAUSTED with an absurd size. The SDK special-cases that code with
backoff.resetToMax(), parking the stream on a 60 second maximum backoff, then
self-heals.

This is an unresolved upstream emulator bug, firebase/firebase-tools#8654.
There is no fixed version to pin, so the goal is to survive it rather than
prevent it.

Only the tests that read a document which does not exist need a live server
round trip, so they are the only ones that can die; everything else is served
from the local cache. Those tests currently abandon the stream after one
second, so they fail while the SDK is still recovering. Raising both ceilings
lets them wait the backoff out. waitFor polls every 50ms, so a larger budget
costs nothing when the stream is healthy: measured at 80-114ms per test with
the change across five runs, against 79-105ms without it across three.

The budget is 120s rather than 60s because a third party hitting the same
upstream bug found a 60s ceiling insufficient at least once, and observed a
test self-heal at ~35s under a 120s ceiling.

Verified: a simulated 65 second stall survives the new budget while failing
under the default, and both typechecks are clean. Not verified, and it cannot
be locally: that this rescues a real occurrence. The desync has never
reproduced on a dev machine, so the evidence is CI once this lands, where
desyncs should keep appearing while the flake stops.

Refs FirebaseExtended#776
@tyler-reitz

Copy link
Copy Markdown
Contributor Author

Got a real occurrence in CI while this sat waiting, so the "not verified" caveat in the description is now partly answerable.

The test workflow has run nine times on this PR (18 job samples). Three attempts hit the desync; one of those landed on a vulnerable test and was rescued rather than failing:

GrpcConnection RPC 'Listen' stream error. Code: 8 RESOURCE_EXHAUSTED:
  Received message larger than max (1650553701 vs 4194304)
Using maximum backoff delay to prevent overloading the backend.
 ✓ test/firestore.test.tsx (15 tests) 12818ms
       ✓ returns undefined if document does not exist  9795ms

That test normally runs in ~90ms. Under the current ceilings it would have failed at 1000ms, or been killed by vitest at 5000ms. Recovery took about 10 seconds rather than the full 60, which matches the ~35s another project reported on the upstream issue and suggests the 120s budget has headroom.

18 of 18 jobs passed. On its own that is only p ≈ 0.03 against the 18.3% baseline flake rate, so the rescued stall is the persuasive part rather than the pass count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant