Dev

docs/live-data.md

@@ -1,6 +1,6 @@
 # How the Live Data Works
 
-`fcli server` serves the same reports as the CLI as a live web UI, kept continuously up-to-date by gNMI **subscriptions** rather than periodic polling. Every node in the inventory receives a single `Subscribe` RPC carrying the paths needed by the open reports. Whenever data changes on a device, updated tables are rendered and pushed to the browser via Server-Sent Events (SSE).
+`fcli server` serves the same reports as the CLI as a live web UI, kept continuously up-to-date by gNMI **subscriptions** rather than periodic polling. Every node in the inventory receives a `Subscribe` RPC carrying the paths needed by the open reports, and a second one for BGP RIB paths once a RIB view streams. Whenever data changes on a device, updated tables are rendered and pushed to the browser via Server-Sent Events (SSE).
 
 ## Telemetry architecture
 
@@ -19,7 +19,11 @@ flowchart TD
 When a report is opened for the first time, the server executes its getter once against a recording proxy of the gNMI connection. This captures the exact set of gNMI paths the report reads. As a result, subscription paths never have to be manually declared or maintained separately from report getters.
 
 ### 2. State bootstrapping & shape pinning
-Each discovered path is bootstrapped with a standard gNMI `Get`. This seeds an in-memory state tree for the node and pins down the response structure the report getter expects.
+Each discovered path seeds an in-memory state tree for the node and pins down the response structure the report getter expects. The server reuses the discovery response for bootstrap, unless a related update arrived during discovery. Predeclared paths and later resynchronizations use a fresh gNMI `Get`. Small lookups fetch only the referenced keys in one batched Get, shared with the first render through the fallback cache. Above 10,000 keys, discovery reads the lookup table once and reuses that response for bootstrap and subscription. Concurrent clients share each node's activation.
+
+When a BGP RIB view lists all routes, the browser receives partial tables as individual nodes finish their first reads. It shows how many nodes have answered and keeps the loading indicator until every selected node has finished or reported an error. A node's routes arrive together, after its routes and attributes have been read; this is progress across nodes, not pagination of a single node's Get. At most four nodes run these full-table reads concurrently to limit competition for CPU and memory. Other reports' first reads have workers of their own, so a small report such as the IPv4 RIB never queues behind a RIB download. Activation has separate workers so it cannot queue completed nodes' renders behind it. The one-shot report API and saved readings still wait for the complete result.
+
+BGP RIB reports show **Stop loading** while a query is running. It closes that browser's stream and preserves the rows already displayed; **Resume** continues loading. For full-table reads, queued node activations are cancelled when no other viewer or background request needs them. A discovery Get already in flight may finish, then cancellation stops the next discovery read. Existing shared subscriptions and reads used by other viewers continue. Leaving the report also releases its unfinished activation work.
 
 ### 3. STREAM subscriptions: ON_CHANGE and SAMPLE
 A gNMI `Subscribe` in STREAM mode keeps the node's state tree current. Report getters execute directly against this local tree instead of querying the physical device, rendering tables with zero device round-trips.
@@ -36,10 +40,18 @@ Because SR Linux streams entire subtrees when subscribing to a branch, multiple
 ### 4. Fallback polling & cache resynchronization
 Paths that cannot be streamed fall back to short-TTL `Get` operations. Additionally, every node is fully re-read every `--resync` seconds (default: 300s) to guarantee that missed deletions cannot leave stale rows behind. Re-sync sweeps are executed round-robin across nodes rather than simultaneously.
 
+BGP routes reference separate attribute sets for fields such as RT, AS-path and communities. If a referenced set is missing from the streamed tree, the server checks the device with a batched fallback `Get`. An absent answer expires after the fallback TTL; newly streamed attributes take precedence immediately. This lets missing attributes recover without requiring a full report reload or adding subscriptions for individual attribute IDs.
+
+Small BGP reports subscribe to their routes without subscribing to the shared attribute table, which also contains attributes for other address families. Their attribute lookups reuse an existing table subscription when available, or a Get of just the referenced indexes. An attribute set does not change under its index: when a route's attributes change, the route moves to a new index and the old one is removed (verified on SR Linux 26.7). A set found by a Get is therefore kept for 10 minutes and its age never makes a table stale; a route update naming a new index re-renders the view and fetches only that index. Before this, a large EVPN view re-read every referenced set of every node each 30 seconds, ~8,000 per node at ~1 ms each, and never stopped re-rendering.
+
 Whatever arrives while a node is being re-read is recorded and applied again, in order, to the re-read tree once it is swapped in. SAMPLE would re-send it on the next tick, but ON_CHANGE never would. When one path covers another, such as `route-table/ipv4-unicast` and its `route/ipv4-prefix` keys, only the wider path's `Get` is written to the tree. A `Get` does not say which leaves key a list, so the narrower answer would otherwise replace every entry with its keys alone.
 
 In between sweeps, list entries that a SAMPLE subscription ceases to send are aged out after a few sample intervals. This aging is measured relative to the last update timestamp received by that subtree rather than wall-clock time, preventing a node with a lagging telemetry stream from spuriously clearing out data.
 
+Rendered reports retain the revisions of the paths they read on their selected nodes, and so does each node's share of a table: a re-render reuses the rows of every node whose paths did not change and runs the getter only for the nodes that did. Heartbeats, unrelated counters, and updates on other nodes leave those tables cached. Relevant updates, deletes, resynchronization, and connection replacement invalidate them; fallback reads also carry their original Get-cache expiry, so rendering a table does not extend its data's lifetime. Revisions group entries by schema path to keep bookkeeping bounded even with large route tables: changing one route can invalidate other queries into the same route list.
+
+Clients sharing a cached table also share its JSON encoding and SSE fingerprint. Encoding runs in a worker thread, and its bytes are released with the cached table. Each node's rows are encoded once and reused by every later table that keeps them, so a change on one node re-encodes that node alone. After the first full table, a stream sends a change as a `patch` event carrying only the rows of the nodes that changed, plus the order of all of them; the browser rebuilds the table from the rows it already has. A patch is only sent while the columns stay the same; otherwise the full table is. Cards, graphs, and check summaries count as visible changes; render timings alone do not trigger an SSE update.
+
 ### 5. Narrow path subscriptions
 Subscriptions stream everything under their path (configuration and state alike). `fcli` subscribes strictly to the specific branches read by the getter. For example, subscribing to `/network-instance[name=*]` entirely would stream all route tables and the entire BGP RIB—on a spine router, this represents massive payload volume capable of delaying the telemetry stream. Instead, `fcli` selectively subscribes only to types, interfaces, overlays, and BGP instances.
 
@@ -115,7 +127,7 @@ SR Linux enforces a concurrent session limit per gRPC server:
 This budget is shared across all gRPC clients connecting to the switch. Every active RPC, including long-lived streaming `Subscribe` RPCs, counts against this limit.
 
 To prevent exhausting session limits:
-* **One session per node**: `fcli server` multiplexes all open reports into a **single `Subscribe` RPC** per node, with at most one auxiliary `Get` in flight at any time.
+* **Two sessions per node at most**: `fcli server` multiplexes all open reports into one `Subscribe` RPC per node, plus one for BGP RIB paths while a RIB view streams, with at most one auxiliary `Get` in flight at any time. gNMI cannot add paths to a running subscription, so opening a report replaces the RPC that carries its paths, and that RPC's initial sync re-sends everything on it. Kept apart, opening any other report no longer re-streams a full RIB, and opening a RIB view does not re-send every interface counter.
 * **Batched RPC restarts**: Because gNMI does not support dynamically adding paths to an existing subscription, adding new paths requires restarting the `Subscribe` RPC. These restarts are debounced and batched, so navigating a dashboard with multiple reports triggers a single re-subscription rather than one per widget.
 * **Idle path pruning**: Paths that have not been requested by any active report for longer than `--idle-timeout` (default: 900s) are automatically pruned from the subscription.
 

nornir_srl/checks.py

@@ -435,14 +435,16 @@ def check_mtu_outlier(state: FabricState) -> List[Finding]:
 # --------------------------------------------------------------------------- #
 
 
-def service_facts(vni: str, instances: Sequence[Any]) -> Dict[str, Any]:
+def service_facts(vni: Optional[str], instances: Sequence[Any]) -> Dict[str, Any]:
     """What one node thinks a service looks like, by the name of each fact.
 
     The route-targets are one fact per bgp-vpn instance: a gateway carries a
     second instance for its WAN side, with route-targets the leaves never
-    see, and that is not the two of them disagreeing.
+    see, and that is not the two of them disagreeing. A *vni* of ``None`` is
+    one that could not be read, which leaves the VNI out rather than holding
+    an unknown against every other node's.
     """
-    facts: Dict[str, Any] = {"VNI": vni}
+    facts: Dict[str, Any] = {} if vni is None else {"VNI": vni}
     for inst in instances:
         which = f" of bgp-instance {inst.id}" if inst.id != 1 else ""
         facts[f"import route-target{which}"] = _rt_set(inst.import_rts)
@@ -580,11 +582,17 @@ def check_evpn_service_mismatch(state: FabricState) -> List[Finding]:
     # and which nodes carry it as a gateway, with a WAN-side instance too.
     services: Dict[str, Dict[str, Dict[str, Any]]] = {}
     gateways: Dict[str, Set[str]] = {}
+    # vxlan-interfaces a service names that the node's VXLAN state does not
+    # have. Their VNI is unknown, not different: held against every other
+    # node, one gap in a node's state is an error per service per node.
+    unknown: Dict[str, List[str]] = {}
     for node, instance in state.items("ni"):
         if _text(instance.type) not in ("mac-vrf", "ip-vrf"):
             continue
+        missing = [overlay for overlay in instance.overlays if (node, overlay) not in vnis]
+        unknown.setdefault(node, []).extend(missing)
         services.setdefault(instance.name, {})[node] = service_facts(
-            ", ".join(str(vnis.get((node, overlay), "?")) for overlay in instance.overlays),
+            None if missing else ", ".join(str(vnis[(node, overlay)]) for overlay in instance.overlays),
             instance.instances,
         )
         if len(instance.instances) > 1:
@@ -608,6 +616,22 @@ def disagree(name: str, nodes: Sequence[str], facts_of: Callable[[str], Dict[str
                 )
 
     findings: List[Finding] = []
+    # A node whose VXLAN state could not be read at all is a collection
+    # finding already; one read without these interfaces is said once here.
+    read = state.reports.get("vxlan") or {}
+    for node, missing in sorted(unknown.items()):
+        if not missing or node not in read:
+            continue
+        shown = ", ".join(missing[:5]) + (f" and {len(missing) - 5} more" if len(missing) > 5 else "")
+        findings.append(
+            Finding(
+                check="evpn_service_mismatch",
+                severity=WARNING,
+                node=node,
+                subject="VNI",
+                detail=f"no VXLAN state for {len(missing)} vxlan-interface(s) its services use ({shown}); their VNI was not compared",
+            )
+        )
     for name, by_node in sorted(services.items()):
         wan = gateways.get(name, set())
         for domain in underlay_domains(list(by_node), system, hosts, wan):
@@ -1371,6 +1395,67 @@ def collect_fabric_state(
     return _collect(target, reports)
 
 
+#: What a check came to, as the Checks page shows it, worst first.
+CHECK_STATUSES = ("failed", "error", "warning", "skipped", "pass")
+
+
+def check_results(state: FabricState, findings: Sequence[Finding]) -> List[Dict[str, Any]]:
+    """Each check, whether it ran, and what it found where - the passing ones too.
+
+    *findings* is what :func:`run_checks` made of *state*. A check is
+    ``skipped`` when none of the reports it reads were collected, ``failed``
+    when it raised, and otherwise as bad as its worst finding, or ``pass``.
+    ``nodes`` holds every node the check had something to read on, with the
+    worst it found there - ``pass`` where it found nothing - so a passing node
+    shows as one rather than as missing. The nodes the checks could not read
+    at all are a check of their own, ``collection``.
+    """
+    by_check: Dict[str, List[Finding]] = {}
+    for finding in findings:
+        by_check.setdefault(finding.check, []).append(finding)
+
+    def summary(name: str, title: str, requires: Tuple[str, ...], ran: bool, checked: Set[str]) -> Dict[str, Any]:
+        found = by_check.get(name, [])
+        failed = any(f.node == "-" and f.subject == "check failed" for f in found)
+        errors = sum(1 for f in found if f.severity == ERROR)
+        warnings = sum(1 for f in found if f.severity == WARNING)
+        nodes: Dict[str, str] = {node: "pass" for node in checked}
+        for f in found:
+            if f.node == "-":
+                continue
+            worst = nodes.get(f.node, "pass")
+            if worst == "pass" or (worst == WARNING and f.severity == ERROR):
+                nodes[f.node] = f.severity
+        status = (
+            "failed" if failed
+            else "skipped" if not ran
+            else ERROR if errors
+            else WARNING if warnings
+            else "pass"
+        )
+        return {
+            "name": name,
+            "title": title,
+            "requires": list(requires),
+            "status": status,
+            "errors": errors,
+            "warnings": warnings,
+            "nodes": dict(sorted(nodes.items())),
+            "findings": [f.as_row() for f in found],
+        }
+
+    results = []
+    for check in CHECKS:
+        collected = {node for report in check.requires for node in (state.reports.get(report) or {})}
+        ran = not check.requires or bool(collected)
+        results.append(summary(check.name, check.title, check.requires, ran, collected if ran else set()))
+    unread = {node for (_report, node) in state.errors}
+    read = {node for payloads in state.reports.values() for node in payloads}
+    results.append(summary("collection", "Nodes the checks could not read", (), True, read | unread))
+    results.sort(key=lambda r: (CHECK_STATUSES.index(r["status"]), -(r["errors"] + r["warnings"]), r["title"]))
+    return results
+
+
 def run_checks(
     state: FabricState, only: Optional[Sequence[str]] = None
 ) -> List[Finding]:

nornir_srl/cli.py

@@ -993,9 +993,28 @@ def bgp_rib(
         help="Include all path attributes (communities, SoO, D-PATH, tunnel-encap, "
         "status). Automatically enabled for non-table output (json/yaml/csv).",
     ),
+    all_routes: bool = typer.Option(
+        False,
+        "--all",
+        help="Every path received, not only the routes the node uses.",
+    ),
+    key: Optional[List[str]] = typer.Option(
+        None,
+        "--key",
+        "-k",
+        help="Look routes up by a key of the route list, as key=value - e.g. "
+        "mac-address=1A:A4:02:FF:00:01 or esi=00:00:00:00:01:* - in the gNMI Get "
+        "itself. Repeat for several keys.",
+    ),
     field_filter: Optional[List[str]] = FIELD_FILTER,
 ) -> None:
-    """Displays BGP RIB"""
+    """Displays BGP RIB (the used routes, unless --all)"""
+    keys: Dict[str, str] = {}
+    for item in key or []:
+        name, sep, value = item.partition("=")
+        if not sep or not name.strip() or not value.strip():
+            raise typer.BadParameter(f"'{item}' is not key=value", param_hint="--key")
+        keys[name.strip()] = value.strip()
     family = BGP_RIB_ROUTE_FAM_ALIASES.get(route_fam.lower(), route_fam)
     run_report(
         ctx,
@@ -1004,6 +1023,8 @@ def bgp_rib(
         title=f"BGP RIB ({family})",
         route_fam=route_fam,
         route_type=route_type,
+        paths="all" if all_routes else "used",
+        keys=keys,
         # Structured output has room for every attribute, so always include them.
         detail=detail or ctx.obj["output"] != OutputFormat.TABLE,
     )

nornir_srl/connections/health.py

@@ -282,8 +282,11 @@ def get_components(self) -> Dict[str, Any]:
         components: List[Component] = []
         paths = [p for p in _COMPONENT_PATHS if p[0] != "fabric" or "chassis" in self._features()]
         for kind, path, key in paths:
-            payload = _payload(self._get_optional(path), "platform")
-            for item in _dicts(_container(payload).get(kind)):
+            if kind in _CARD_LEAVES_ONLY:
+                items = self._card_leaves(kind, path, key)
+            else:
+                items = _dicts(_container(_payload(self._get_optional(path), "platform")).get(kind))
+            for item in items:
                 components.append(
                     Component(
                         kind=kind,
@@ -296,6 +299,20 @@ def get_components(self) -> Dict[str, Any]:
                 )
         return {"components": components}
 
+    def _card_leaves(self, kind: str, path: str, key: str) -> List[Dict[str, Any]]:
+        """The leaves of each card a component is made of, card by card.
+
+        Not the card's subtree: a line card's holds the forwarding tables,
+        a hundred thousand leaves on an EVPN fabric of a thousand services,
+        which the server would otherwise be sent again every sample.
+        """
+        cards: Dict[str, Dict[str, Any]] = {}
+        for leaf in _CARD_LEAVES:
+            payload = _payload(self._get_optional(f"{path}/{leaf}", datatype="all"), "platform")
+            for item in _dicts(_container(payload).get(kind)):
+                cards.setdefault(_text(item.get(key)), {}).update(item)
+        return list(cards.values())
+
     def get_transceivers(self) -> Dict[str, Any]:
         resp = self._get_optional("/interface[name=*]/transceiver")
         found = []
@@ -308,6 +325,12 @@ def get_transceivers(self) -> Dict[str, Any]:
         return {"transceivers": found}
 
 
+#: What a component is read as: whether it is up and healthy, and what it is.
+_CARD_LEAVES: Tuple[str, ...] = ("oper-state", "healthz", "type", "serial-number")
+
+#: The components read leaf by leaf rather than whole; see _card_leaves.
+_CARD_LEAVES_ONLY = frozenset({"linecard"})
+
 #: Hardware lists, with the key each is indexed by. ``fabric`` is asked only
 #: of a modular chassis; see :meth:`HealthMixin.get_components`.
 _COMPONENT_PATHS: Tuple[Tuple[str, str, str], ...] = (