test(firestore): wait out the emulator's Listen-stream backoff - #791
test(firestore): wait out the emulator's Listen-stream backoff#791tyler-reitz wants to merge 1 commit into
Conversation
test/firestore.test.tsx flakes in CI at roughly 18% per run (11 of 60 iterations, 2026-08-11). The Firestore emulator intermittently corrupts a Listen frame, so grpc-js reads four body bytes as a length prefix and reports RESOURCE_EXHAUSTED with an absurd size. The SDK special-cases that code with backoff.resetToMax(), parking the stream on a 60 second maximum backoff, then self-heals. This is an unresolved upstream emulator bug, firebase/firebase-tools#8654. There is no fixed version to pin, so the goal is to survive it rather than prevent it. Only the tests that read a document which does not exist need a live server round trip, so they are the only ones that can die; everything else is served from the local cache. Those tests currently abandon the stream after one second, so they fail while the SDK is still recovering. Raising both ceilings lets them wait the backoff out. waitFor polls every 50ms, so a larger budget costs nothing when the stream is healthy: measured at 80-114ms per test with the change across five runs, against 79-105ms without it across three. The budget is 120s rather than 60s because a third party hitting the same upstream bug found a 60s ceiling insufficient at least once, and observed a test self-heal at ~35s under a 120s ceiling. Verified: a simulated 65 second stall survives the new budget while failing under the default, and both typechecks are clean. Not verified, and it cannot be locally: that this rescues a real occurrence. The desync has never reproduced on a dev machine, so the evidence is CI once this lands, where desyncs should keep appearing while the flake stops. Refs FirebaseExtended#776
|
Got a real occurrence in CI while this sat waiting, so the "not verified" caveat in the description is now partly answerable. The test workflow has run nine times on this PR (18 job samples). Three attempts hit the desync; one of those landed on a vulnerable test and was rescued rather than failing: That test normally runs in ~90ms. Under the current ceilings it would have failed at 1000ms, or been killed by vitest at 5000ms. Recovery took about 10 seconds rather than the full 60, which matches the ~35s another project reported on the upstream issue and suggests the 120s budget has headroom. 18 of 18 jobs passed. On its own that is only p ≈ 0.03 against the 18.3% baseline flake rate, so the rescued stall is the persuasive part rather than the pass count. |
Why
test/firestore.test.tsxflakes at roughly 18% per run (11 of 60 iterations measured 2026-08-11). The Firestore emulator intermittently corrupts a Listen frame, grpc-js reads four body bytes as a length prefix, and the SDK answers the resultingRESOURCE_EXHAUSTEDwith a 60 second maximum backoff before self-healing.This is an unresolved upstream emulator bug, firebase-tools#8654. Nothing to pin, so this survives it rather than preventing it.
What
Four
waitForcalls in the two tests that read a nonexistent document get a 120s budget, and those two tests get a 150s vitest timeout. They are the only tests needing a live server round trip, so the only ones that can die; the rest are served from the local cache.waitForpolls at 50ms, so the larger ceiling costs nothing on a healthy run: 80-114ms per test with the change over five runs, 79-105ms without over three.120s rather than 60s because a third party on the same upstream bug found 60s insufficient at least once and observed a self-heal at ~35s under 120s.
What is not verified
That this rescues a real occurrence. The desync has never reproduced on a dev machine (0 stalls in five fresh-emulator runs here). What is verified is the budget itself: a simulated 65s stall passes under it and fails under the default.
The evidence is CI after this lands: desyncs should keep appearing in logs while the flake stops. That is also the signal for closing #776.
The scoped cost: a genuine regression in these two tests now takes 120s to report instead of 1s. That is confined to the two tests this bug can kill.
Refs #776