Repository navigation
Conversation
On a 10-leaf lab with 1000 mac-vrfs per leaf the server hung: renders copied whole state subtrees under the stream locks, getters read tables of tens of thousands of entries to use a handful, and SAMPLE subscriptions re-sent more than the server could apply - which then aged out good data and raised false incidents. Streaming and state - Select before materializing: reports and dashboards copy only the paths they read (select_materialized, select_many); faster JSON copy and module-prefix stripping. - Stream on change what has no counters to speak of: subinterfaces with their ARP/ND, MAC tables, BGP RIBs, VXLAN VNIs and destinations, ES destination VTEPs. - Read less: VXLAN and ES reports read the leaves they use; line cards are read leaf by leaf rather than with their forwarding tables; the IRB report asks for the irb bindings alone. - Do not age entries out while a stream's notifications pile up; the queue shows as 'backlog' in /api/status. Lookups by key - get_rib looks up only the next-hops and groups its routes use, by key; attribute sets likewise, batched, or the table read once past 10000 keys. - lookup()/query() serve keyed reads from the streamed tree when it holds them, else with one cached Get, and are never subscribed. BGP RIB reports - Asked by the route list's YANG keys, matched by the node in the Get, with the fabric's values offered in a drop-down; 'List all' streams the whole RIB. Used (or best) paths by default. CLI --key/--all, MCP keys. - Received/advertised routes need a peer and a family and ask the node for that peer's routes only. Dashboards - Overview and topology answer within 2s with what has streamed, saying how far the rest is; their reports are read first at start-up; the browser polls without piling requests up. Old readings and recordings still replay, narrowed from the wider paths they were taken with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Telemetry Engine card: gNMI paths across nodes, how many stream, are pending or polled, and the worst node's backlog, amber past the limit. Clicking it opens Server Activity. - Server Activity: per-node streaming rates (leaves and notifications per second, Gets per minute) with sparklines, backlog against its limit, paths by how they are served, a fabric-wide rate chart with hover. The streams count what they apply; /api/status carries the counters, whether a node is resyncing, and the backlog limit. - Checks: a board of one card per check, passing and skipped ones too, with its status, a strip of the nodes it read coloured by the worst found there, and its findings; the table stays a view away. check_results() summarises each check alongside the checks report's rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evpn_service_mismatch compared a missing vxlan-interface as VNI "?", so one node with a gap in its VXLAN state raised an error per service on every node carrying it - over 10,000 false incidents on a 1000-service fabric. An unknown VNI is now left out of the comparison, and a node read without the vxlan-interfaces its services use gets one warning. Route-targets are still compared. The services lens follows suit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Full-table BGP RIB views stream partial tables as each node finishes its first read, with ready/total counts, instead of waiting for the slowest node. The page shows Stop loading / Resume; closing a view cancels queued node activations no other viewer or background caller needs. First reads get cheaper too: - concurrent clients share each node's activation, on a bounded pool separate from the render workers - bootstrap reuses the discovery response unless a related update arrived in between - lookups of up to 10,000 keys fetch only those keys rather than subscribing to the whole attribute table; larger ones read the table once for both discovery and bootstrap - attribute sets a route references but the tree lacks are verified with a TTL-cached fallback Get - insert strips module prefixes once instead of at every level Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Incidents and the overview read the same watcher reading. health() fell back to a full reading of its own whenever an inventory filter was set or the latest reading was older than three watch intervals - on a large fabric, a second reading competing with the watcher's for the same nodes. Now: - the latest reading answers however old it is - a filter narrows that reading to its nodes and re-runs the checks, cached until the next reading, with no gNMI - before the first reading, health waits for the watcher instead of starting its own; only without a watcher, or after its first reading failed, is the fabric read here Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every first read shared the activation pool bounded to four workers, so opening a report like the IPv4 RIB while a full BGP RIB load or the watcher's first reading was activating queued it for minutes behind reads it has nothing to do with. Full-table RIB reads keep the bounded pool; every other first read gets workers of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resync re-reads the paths a node had when it started into a fresh tree and swaps it in. A path bootstrapped meanwhile - a report opened during the resync - went into the tree being replaced, and the swap dropped its state until the subscription happened to re-send it, which an ON_CHANGE path never does. On the 18-leaf lab a fabric-wide EVPN view came up with 869 of a leaf's ~4,370 routes, and earlier a leaf's VXLAN and ES state vanished for minutes, raising thousands of false service-mismatch incidents. Such paths are now carried over into the fresh tree before the swap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on an 18-leaf lab with 250 fabric-wide BDs, a fabric-wide EVPN type-2 view (~88k rows) took 231s to open and, left open, re-rendered every node back to back and pushed 36 MB per change. Now it opens in 97s, settles to re-rendering one node in 1-3s, and sends ~1.8 MB. - Re-render only the nodes that changed: each node's rows are kept with the reads they came from, and a table re-render runs the getter only for nodes whose paths changed. - Encode and send per node: a table is built from per-node parts that keep their JSON encoding, and an SSE stream sends a "patch" with the changed nodes' rows after the first full table. The browser rebuilds the table from the rows it already has. - Keep BGP attribute sets by index: a set never changes under its index (changed attributes move a route to a new index; verified on SR Linux 26.7), so a found set is kept for 10 minutes and its age never makes a table stale. An absent set is still retried after the Get TTL. - Give BGP RIB paths their own Subscribe RPC, so opening another report no longer replaces the RPC carrying a full RIB and makes it re-send every route. Each RPC's initial sync only sweeps its own paths. Debug logging now says which read made a render stale and which paths a notification deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.