feat(ingester): Add owned series tracking to prevent false throttling… - #7758
Open
anant10 wants to merge 1 commit into
Open
feat(ingester): Add owned series tracking to prevent false throttling…#7758anant10 wants to merge 1 commit into
anant10 wants to merge 1 commit into
Conversation
dosubot
Bot
added
component/ingester
component/ring
type/feature
type/performance
labels
… during ring changes When ingesters scale up, the per-ingester local series limit drops immediately but stale series data remains in TSDB head for up to 2 hours. This causes PreCreation() to incorrectly reject new writes. This PR introduces owned series tracking in ActiveSeries: - Each series stores its ring token (computed via TokenForLabels) - Ownership is evaluated against current ring state on each update cycle - When ring changes, unowned series are excluded from limit enforcement Two feature flags for safe rollout: - owned_series_metrics_enabled: enables cortex_ingester_owned_series metric - owned_series_limit_enforcement_enabled: switches PreCreation to use owned count for both per-user and instance-level max_series limits Key design decisions: - Zone-local ownership check via SearchToken + instance token map lookup - Ring state stored behind atomic.Pointer[ringState] for lock-free reads on the hot push path - instanceOwnedCount recalculated every ~1 min (not incremental) to avoid drift from edge cases - Startup fallback: when instanceOwnedCount==0, uses instanceSeriesCount Code consolidation: - FNV hash functions consolidated into pkg/util/fnv.go (single source) - Sharding functions moved to pkg/ring/token.go (eliminates duplication between distributor and ring packages) Production validation: tested with 5M active series, scale-up 9->18 ingesters showed owned_series=984K vs memory_series=1.8M (813K stale series correctly excluded). Zero throttle errors. Fixes cortexproject#7509 Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
anant10
force-pushed
the
feat/owned-series-tracking
branch
from
August 11, 2026 17:28
97d8157 to
305bc32
Compare
|
|
||
| // If ring tokens are loaded, check ownership before creating. | ||
| // This prevents tracking series we don't own (e.g., stale distributor routes). | ||
| if len(ringTokens) > 0 && !isOwnedByInstance(key, ringTokens, instanceTokens) { |
Contributor
There was a problem hiding this comment.
we should maintain active behaviour the same. we should make the check impact on the owned metric which is the new one
| // This test just validates ActiveSeries itself works correctly. | ||
| } | ||
|
|
||
| func BenchmarkActiveSeries_UpdateSeries(b *testing.B) { |
Contributor
There was a problem hiding this comment.
Did we replace tests removing benchmark to add new test?
We should still have benchmark and would be nice to run to see how it perform with owned active
| // if the responsible token belongs to this instance. | ||
| func isOwnedByInstance(key uint32, ringTokens []uint32, instanceTokens map[uint32]struct{}) bool { | ||
| i := ring.SearchToken(ringTokens, key) | ||
| _, found := instanceTokens[ringTokens[i]] |
Contributor
There was a problem hiding this comment.
lets add a check here to make sure ringTokens is not empty otherwise it will crash.
| * [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446 | ||
| * [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481 | ||
| * [FEATURE] Ingester: Add owned series tracking to prevent false customer throttling during ingester scale-up and ring resharding. When enabled, the ingester tracks which series it currently owns according to the ring and uses that count (instead of total in-memory series) for limit enforcement. Eliminates a up-to-2-hour window of incorrect throttling after any ring change. Controlled by `-ingester.owned-series-metrics-enabled` (metric emission) and `-ingester.owned-series-limit-enforcement-enabled` (limit enforcement). #7509 | ||
| * [ENHANCEMENT] Ring: Consolidate sharding functions (`TokenForLabels`, `ShardByMetricName`, etc.) into `pkg/ring/token.go` for reuse by both distributor and ingester. Export `SearchToken`. #7509 |
Contributor
There was a problem hiding this comment.
We dont need this entries. Just one for feature is enough
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Adds per-ingester series ownership tracking to prevent false throttling during ingester scale-up and ring resharding.
The problem: When ingesters scale up, the per-ingester local series limit drops immediately (recalculated based on new ingester count), but stale series data remains in the TSDB head for up to 2 hours until head compaction.
PreCreation()usesHead().NumSeries()for limit checks, so it incorrectly rejects new writes during this window — the ingester appears over its new lower limit, but many of those series have been resharded to other ingesters and will be cleaned up at next compaction.The solution: Track which series each ingester actually owns according to the ring, and use that count for limit enforcement. The owned count drops immediately when the ring changes (within 1 minute), eliminating the 2-hour dependency on head compaction.
How it works:
ActiveSeriesupdateActiveSeriescycle), if the ring changed, re-scan all entries and remove series whose token no longer maps to this ingesterPreCreation()usesactiveSeries.Owned()instead ofHead().NumSeries()for the limit checkDesign decisions:
-ingester.owned-series-metrics-enabled: enablescortex_ingester_owned_seriesmetric emission only (no enforcement change)-ingester.owned-series-limit-enforcement-enabled: switches limit enforcement to use owned count (requires first flag)SearchToken(zoneTokens, key)→ is responsible token in this instance's set?atomic.Pointer[ringState]— zero lock contentionmax_series:instanceOwnedCountrecalculated every ~1 min (not incremental, avoids drift). Startup fallback toinstanceSeriesCountwhen count is 0pkg/util/fnv.go, sharding functions →pkg/ring/token.go(eliminates duplication between distributor and ring)Validation:
Tested in a multi-zone deployment under sustained write load. After scale-up:
memory_serieson old ingesters remained high (stale data in head)owned_serieson old ingesters dropped proportionally to ring redistributionNew configuration flags (experimental):
yaml
Which issue(s) this PR fixes: Fixes #7509