Add outcome-aware evaluation loop for Cisco Firepower response - #14997
Closed
Ahmed Hassan (AAH20) wants to merge 7 commits into
Closed
Ahmed Hassan (AAH20) wants to merge 7 commits into
Ahmed Hassan (AAH20) wants to merge 7 commits into
Conversation
Encode ML-only escalate vs signature/corroborated Gate/Prove dispositions so BlockIP playbooks do not treat SnortML GID 411 as classic signature true positives.
SnortML / GID 411 must not be treated as a signature true positive: NetworkGroup auto-contain cancels, Teams HITL warns and still requires analyst confirmation. Co-authored-by: Cursor <cursoragent@cursor.com>
One solution version for Microsoft review: ASCII-only detections, stricter GID 411 matching, and colon-free Logic Apps action names. Co-authored-by: Cursor <cursoragent@cursor.com>
aka.ms/deploytoazuregovbutton -> aka.ms/deploytoazuregovernbutton, per review comment.
Ran Create-Azure-Sentinel-Solution V3 against Solutions/Cisco Firepower EStreamer/Data per review request, so mainTemplate.json and createUiDefinition.json reflect the 3.0.3 analytic rules and Gate/Prove BlockIP playbook changes. Version bumped 3.0.3 -> 3.0.4 (local mode, patch), matching ReleaseNotes.md.
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds outcome-aware evaluation signals for Cisco Firepower response workflows, keeping ML-only evidence outside the auto-containment path while enabling measurable quality/drift tracking.
Changes:
- Introduces a versioned
[FirepowerOutcome:v1]incident-comment contract and updates BlockIP playbooks to emit structured outcomes (denials, HITL decisions, success/fail reasons). - Adds a new “Cisco Firepower Detection and Response Quality” workbook plus new analytic-rule templates (ML-only, signature-high, corroboration, signal-mix drift).
- Bumps solution packaging/versioning to 3.1.0 and wires workbook/analytics into solution templates & UI.
Reviewed changes
Copilot reviewed 19 out of 22 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| Workbooks/WorkbooksMetadata.json | Registers the new Cisco Firepower quality workbook in global workbook metadata. |
| Solutions/Cisco Firepower EStreamer/Workbooks/CiscoFirepowerDetectionResponseQuality.json | Adds the new workbook template with KQL panels for signal composition/trends/quality. |
| Solutions/Cisco Firepower EStreamer/ReleaseNotes.md | Documents new 3.1.0 release contents. |
| Solutions/Cisco Firepower EStreamer/Playbooks/readme.md | Updates playbook guidance and references the outcome contract; tweaks deploy buttons. |
| Solutions/Cisco Firepower EStreamer/Playbooks/CiscoFirepowerConnector/readme.md | Updates deploy button markup for Azure Gov. |
| Solutions/Cisco Firepower EStreamer/Playbooks/CiscoFirepower-BlockIP-Teams/readme.md | Documents HITL Gate/Prove warning + outcome recording; updates deploy button. |
| Solutions/Cisco Firepower EStreamer/Playbooks/CiscoFirepower-BlockIP-Teams/azuredeploy.json | Implements Gate/Prove ML-only warning and structured outcome comments; bumps playbook template version. |
| Solutions/Cisco Firepower EStreamer/Playbooks/CiscoFirepower-BlockIP-NetworkGroup/readme.md | Documents ML-only deny/cancel behavior + outcome record; updates deploy button. |
| Solutions/Cisco Firepower EStreamer/Playbooks/CiscoFirepower-BlockIP-NetworkGroup/azuredeploy.json | Implements Gate/Prove ML-only deny + terminate; adds structured outcomes; bumps template version. |
| Solutions/Cisco Firepower EStreamer/Playbooks/CiscoFirepower-BlockFQDN-NetworkGroup/readme.md | Updates deploy button markup for Azure Gov. |
| Solutions/Cisco Firepower EStreamer/Package/testParameters.json | Adds a parameter for the new workbook display name. |
| Solutions/Cisco Firepower EStreamer/Package/mainTemplate.json | Bumps solution to 3.1.0; packages new workbook + analytic rules into contentTemplates. |
| Solutions/Cisco Firepower EStreamer/Package/createUiDefinition.json | Updates solution install UX to include workbook + analytics sections. |
| Solutions/Cisco Firepower EStreamer/Evaluation/FirepowerOutcome-v1.md | Adds the outcome contract spec plus invariants and deterministic evaluation cases. |
| Solutions/Cisco Firepower EStreamer/Data/Solution_Cisco Firepower EStreamer.json | Adds workbook + analytic rules to solution manifest; bumps version. |
| Solutions/Cisco Firepower EStreamer/Analytic Rules/CiscoFirepower-SnortML-GID411-MLOnly.yaml | Adds ML-only (GID 411) analytic rule template. |
| Solutions/Cisco Firepower EStreamer/Analytic Rules/CiscoFirepower-IDS-Signature-HighPriority.yaml | Adds signature-high priority analytic rule template. |
| Solutions/Cisco Firepower EStreamer/Analytic Rules/CiscoFirepower-Signature-And-ML-Corroboration.yaml | Adds dual-signal corroboration analytic rule template. |
| Solutions/Cisco Firepower EStreamer/Analytic Rules/CiscoFirepower-Signal-Mix-Drift.yaml | Adds hourly signal-mix drift analytic rule template. |
Suppressed comments (1)
Solutions/Cisco Firepower EStreamer/Playbooks/CiscoFirepower-BlockIP-NetworkGroup/azuredeploy.json:1
- The ML-only Gate/Prove detection uses substring
contains(..., 'gid 411')(and a few variants) with exclusions only for4110. This can misclassify unrelated GIDs like4111,4112, etc. (which still contain the substringgid 411) and incorrectly deny auto-containment or emit ML-only outcomes. A more robust approach is to extract the numeric GID via a regex match and compare it as a number (or enforce non-digit boundaries around 411), then apply corroboration checks on separate tokens.
{
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| [](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FSolutions%2FCisco%2520Firepower%2520EStreamer%2FPlaybooks%2FCustomConnector%2Fazuredeploy.json) | ||
| [](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FSolutions%2FCisco%2520Firepower%2520EStreamer%2FPlaybooks%2FCustomConnector%2Fazuredeploy.json) | ||
| [](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FSolutions%2FCisco%2520Firepower%2520EStreamer%2FPlaybooks%2FCustomConnector%2Fazuredeploy.json) |
|
|
||
| [](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FSolutions%2FCisco%2520Firepower%2520EStreamer%2FPlaybooks%2FCustomConnector%2Fazuredeploy-gateway.json) | ||
| [](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FSolutions%2FCisco%2520Firepower%2520EStreamer%2FPlaybooks%2FCustomConnector%2Fazuredeploy-gateway.json) | ||
| [](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FSolutions%2FCisco%2520Firepower%2520EStreamer%2FPlaybooks%2FCustomConnector%2Fazuredeploy-gateway.json) |
Comment on lines
+818
to
+819
| "not": { | ||
| "or": [ |
Comment on lines
+839
to
+860
| "contains": [ | ||
| "@variables('DualSignalContext')", | ||
| "gid 4110" | ||
| ] | ||
| }, | ||
| { | ||
| "contains": [ | ||
| "@variables('DualSignalContext')", | ||
| "gid:4110" | ||
| ] | ||
| }, | ||
| { | ||
| "contains": [ | ||
| "@variables('DualSignalContext')", | ||
| "gid=4110" | ||
| ] | ||
| }, | ||
| { | ||
| "contains": [ | ||
| "@variables('DualSignalContext')", | ||
| "generator id 4110" | ||
| ] |
| "apiVersion": "2022-01-01-preview", | ||
| "name": "[concat(parameters('workspace'),'/Microsoft.SecurityInsights/',concat('Workbook-', last(split(variables('workbookId1'),'/'))))]", | ||
| "properties": { | ||
| "description": "@{workbookKey=CiscoFirepowerDetectionResponseQuality; logoFileName=cisco-logo-72px.svg; description=Evaluates Cisco Firepower detection signal composition and drift while preserving the safety boundary between SnortML GID 411, classic signatures, and corroborated evidence.; dataTypesDependencies=System.Object[]; dataConnectorsDependencies=System.Object[]; previewImagesFileNames=System.Object[]; version=1.0.0; title=Cisco Firepower Detection and Response Quality; templateRelativePath=CiscoFirepowerDetectionResponseQuality.json; subtitle=Outcome-aware signal quality and drift evaluation; provider=Cisco; support=; author=; source=; categories=}.description", |
| "name": "dataconnectors2-text", | ||
| "type": "Microsoft.Common.TextBlock", | ||
| "options": { | ||
| "text": "This Solution installs the data connector for Cisco Firepower EStreamer. You can get Cisco Firepower EStreamer CommonSecurityLog data in your Microsoft Sentinel workspace. After installing the solution, configure and enable this data connector by following guidance in Manage solution view." |
Comment on lines
+22
to
+41
| let FirepowerEvents = materialize( | ||
| CommonSecurityLog | ||
| | where TimeGenerated >= ago(8d) | ||
| | where DeviceVendor =~ "Cisco" | ||
| | where DeviceProduct has_any ("Firepower", "Secure Firewall", "FTD", "NGFW") | ||
| | extend Combined = strcat(tostring(Message), " ", tostring(AdditionalExtensions), " ", tostring(Activity), " ", tostring(DeviceEventClassID), " ", tostring(column_ifexists("FlexString1", "")), " ", tostring(column_ifexists("FlexString2", "")), " ", tostring(column_ifexists("DeviceCustomString1", "")), " ", tostring(column_ifexists("DeviceCustomString2", "")), " ", tostring(column_ifexists("DeviceCustomString3", ""))) | ||
| | extend ParsedGid = toint(extract(@"(?i)(?:gid|generator[\s_-]?id)[\s:=]*(\d+)", 1, Combined)) | ||
| | extend IsMlOnly = ParsedGid == 411 or Combined has "is_ml_only" | ||
| ); | ||
| let Recent = FirepowerEvents | ||
| | where TimeGenerated >= ago(1h) | ||
| | summarize RecentTotal=count(), RecentMl=countif(IsMlOnly) | ||
| | extend RecentRatio=iff(RecentTotal == 0, 0.0, todouble(RecentMl) / RecentTotal); | ||
| let Baseline = FirepowerEvents | ||
| | where TimeGenerated between (ago(8d) .. ago(1d)) | ||
| | summarize BaselineTotal=count(), BaselineMl=countif(IsMlOnly) | ||
| | extend BaselineRatio=iff(BaselineTotal == 0, 0.0, todouble(BaselineMl) / BaselineTotal); | ||
| Recent | ||
| | extend JoinKey=1 | ||
| | join kind=inner (Baseline | extend JoinKey=1) on JoinKey |
Collaborator
|
Hi Ahmed Hassan (@AAH20), |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dependency
Draft follow-up to #14925. Do not merge until #14925 is merged; this stacked PR currently includes its parent commits and will collapse afterward.
What changed
Safety boundary
The agentic layer is deliberately outside the enforcement path. It may propose reviewed changes, but cannot edit production analytics, policies, or Firepower objects. ML-only evidence cannot enter automatic containment.
Validation
Local .NET 3.1/8 validators were unavailable in the workstation runtime; KQL was instead validated against the live authenticated workspace. TruffleHog was not installed.