From 2d8bc080be892490ccd80dc4753725962b9f9d0f Mon Sep 17 00:00:00 2001 From: "Robert J. Good" Date: Sat, 3 Oct 2026 12:05:05 -0700 Subject: [PATCH 1/2] repeatability in UI --- README.md | 22 +++ docs/governance/playbook-workflow-types.md | 53 +++++ .../features/feature-playbook-essay.md | 25 ++- .../feature-playbook-sql-statistics.md | 29 ++- .../features/feature-playbook-taxonomy.md | 26 ++- .../Playbooks/Essay/EvaluateEssayCommand.cs | 4 +- .../Essay/RunEssayPlaybookCommand.cs | 39 +++- .../Persistence/PlaybookExecutionPersister.cs | 2 + .../PlaybookReplaySourceResolver.cs | 45 +++++ .../SavePlaybookExecutionCommand.cs | 4 + .../Playbooks/PlaybookExecutionResultDto.cs | 27 ++- .../RunSqlStatisticsPlaybookCommand.cs | 39 +++- .../SqlStatisticsClassificationRunner.cs | 12 +- .../Taxonomy/ClassifyTaxonomyCommand.cs | 4 +- .../Taxonomy/RunTaxonomyPlaybookCommand.cs | 38 +++- .../Playbooks/PlaybookExecutionEntity.cs | 23 +++ .../PlaybookWorkflowGraphExecutor.cs | 99 +++++++++- .../Playbooks/Essay/EssayEvaluationRunner.cs | 7 +- .../Taxonomy/TaxonomyClassificationRunner.cs | 7 +- ...lCreate-AgentFrameworkContext.Designer.cs} | 8 +- ...48_InitialCreate-AgentFrameworkContext.cs} | 2 + .../AgentFrameworkContextModelSnapshot.cs | 6 + .../PlaybookExecutionsConfig.cs | 6 + .../Components/PlaybookReplayControls.razor | 63 ++++++ .../Components/PlaybookResultPanel.razor | 16 +- .../Components/PlaybookStepToggle.razor | 43 +++-- .../Components/PlaybookStepToggle.razor.css | 18 ++ .../Playbooks/EssayPlaybookPage.razor | 31 ++- .../Models/PlaybookExecutionResultModel.cs | 7 + .../Playbooks/Models/PlaybookReplayMode.cs | 50 +++++ .../Playbooks/Services/PlaybookService.cs | 50 ++++- .../Playbooks/SqlStatisticsPlaybookPage.razor | 31 ++- .../Playbooks/TaxonomyPlaybookPage.razor | 31 ++- .../Clients/BackendApiClient.g.cs | 48 +++++ .../Playbooks/PlaybookReplayTests.cs | 182 ++++++++++++++++++ 35 files changed, 1025 insertions(+), 72 deletions(-) create mode 100644 src/Core.Application/Playbooks/Persistence/PlaybookReplaySourceResolver.cs rename src/Infrastructure.SqlServer/Migrations/{20261003162033_InitialCreate-AgentFrameworkContext.Designer.cs => 20261003172848_InitialCreate-AgentFrameworkContext.Designer.cs} (98%) rename src/Infrastructure.SqlServer/Migrations/{20261003162033_InitialCreate-AgentFrameworkContext.cs => 20261003172848_InitialCreate-AgentFrameworkContext.cs} (99%) create mode 100644 src/Presentation.Web/Features/Playbooks/Components/PlaybookReplayControls.razor create mode 100644 src/Presentation.Web/Features/Playbooks/Components/PlaybookStepToggle.razor.css create mode 100644 src/Presentation.Web/Features/Playbooks/Models/PlaybookReplayMode.cs create mode 100644 src/Tests.Integration/Playbooks/PlaybookReplayTests.cs diff --git a/README.md b/README.md index 34c9da2..ba4b83e 100644 --- a/README.md +++ b/README.md @@ -409,6 +409,28 @@ The Playbook *concept* and a Playbook *run* are persisted as two separate kinds See `docs/governance/playbook-workflow-types.md` ("Persistence: Playbook Catalog vs. Playbook Execution") for the full entity model. +### Repeatability: Rerun / Recall / Replay + +Every `Run*PlaybookCommand` accepts a `PlaybookReplayMode` (from +`Goodtocode.Agents.Playbook.Execution`) plus an optional `SourceExecutionId`: + +- **Rerun** (default) — a fresh execution: Collect, Evaluate, and Record all run against new + input. This is the only mode available on a Playbook's first execution. +- **Recall** — re-renders the prior execution's Finding with no new Collect or Evaluate work at + all, even if the live data source has since changed. Only Record re-runs, against the recalled + Finding. +- **Replay** — reuses the prior execution's Evidence (skips Collect) but re-runs Evaluate and + Record, to verify the governed result reproduces exactly from the same Evidence. + +Each `Rerun` persists the stage's typed Evidence/Finding as JSON +(`PlaybookExecutionEntity.EvidenceJson`/`FindingJson`) so a later Recall/Replay request can +reconstruct them. The 3 Blazor Playbook pages expose "Rerun", "Recall", and "Replay" controls +(with inline explanations of what each does) next to the latest execution's result; selecting +one re-renders the same result panel in place (no before/after comparison view) — see +`docs/governance/playbook-workflow-types.md` ("Repeatability: Rerun / Recall / Replay") for the +full mechanism, including how the SQL Statistics (`PlaybookExecutor`) and Taxonomy/Essay (MAF +graph) workflows each implement it. + These three Playbooks are exposed both as REST endpoints (`api/v{version}/my/playbooks/{sql-statistics,taxonomy,essay}` to run, `api/v{version}/playbooks` to manage the catalog) and as Blazor pages under `/playbooks/*` in Presentation.Web, in addition to being proven through `Tests.Integration` — see each feature diff --git a/docs/governance/playbook-workflow-types.md b/docs/governance/playbook-workflow-types.md index 3a4f1ed..96f3ade 100644 --- a/docs/governance/playbook-workflow-types.md +++ b/docs/governance/playbook-workflow-types.md @@ -123,6 +123,59 @@ answers: "what does this Playbook's Evaluate step always do?" (`PlaybookStepEnti versus "what did this Playbook's Evaluate step produce the last time someone ran it?" (`PlaybookExecutionEntity.EvaluateOutput`). +## Repeatability: Rerun / Recall / Replay + +The Repeatability pillar of governance is made directly actionable through +`Goodtocode.Agents.Playbook.Execution.PlaybookReplayMode`, which every `Run*PlaybookCommand` +(`RunSqlStatisticsPlaybookCommand`, `RunTaxonomyPlaybookCommand`, `RunEssayPlaybookCommand`) +accepts alongside an optional `SourceExecutionId`: + +| Mode | Collect | Evaluate | Record | Use case | +|------|---------|----------|--------|----------| +| `Rerun` (default) | Runs | Runs | Runs | Normal execution; also the only mode on a Playbook's first run. | +| `Recall` | Skipped | Skipped | Re-renders the prior Finding | "Show me exactly what happened last time," including after live data has since changed. | +| `Replay` | Skipped (reuses prior Evidence) | Re-runs | Runs | "Prove this Evaluate/Record logic reproduces the governed result from the same Evidence," without re-collecting from a (possibly now-different) live source. | + +**How prior evidence/finding is persisted and resolved.** Every `Rerun` persists the stage's +typed `TEvidence` and `TFinding` as `PlaybookExecutionEntity.EvidenceJson`/`FindingJson` +(`System.Text.Json` serialized), in addition to the existing plain-string +`CollectOutput`/`EvaluateOutput`. When a request arrives with `ReplayMode` of `Recall` or +`Replay`, the handler calls the shared `PlaybookReplaySourceResolver` +(`Core.Application/Playbooks/Persistence`) to look up the prior `PlaybookExecutionEntity` — by +`SourceExecutionId` if supplied, otherwise the caller's most recent execution of that Playbook — +deserializes `EvidenceJson`/`FindingJson` back into `TEvidence`/`TFinding`, and constructs a +`PlaybookReplayContext` to pass to the runner. A prior execution that +predates this feature (no `EvidenceJson`/`FindingJson` persisted) cannot be recalled or replayed +and raises a `CustomConflictException`. + +**Two different replay mechanisms, same contract.** The three example workflows reach +Recall/Replay through two different code paths, both driven by the same +`PlaybookReplayContext`: + +- **SQL Statistics** (no MAF) calls the package's own replay-aware + `PlaybookExecutor<...>.ExecuteAsync(definition, input, replayContext, ct, recorder)` overload + directly — the package itself understands how to skip Collect/Evaluate against a supplied + replay context. +- **Taxonomy and Essay** (MAF 3-node graph) use a repo-owned + `PlaybookWorkflowGraphExecutor<...>.ExecuteAsync` that branches on `replayContext?.Mode` before + touching the MAF graph at all: `Rerun` builds and runs the full `WorkflowBuilder` graph as + before; `Recall`/`Replay` bypass the MAF graph entirely and call only the remaining stage + tool(s) directly (`Recall` → `IRecordStepTool<,>.RecordAsync` against the prior Finding; + `Replay` → `IEvaluateStepTool<,>.EvaluateAsync` then `IRecordStepTool<,>.RecordAsync` against + the prior Evidence), while still invoking the same `IPlaybookStepActivityRecorder` calls the + graph nodes would have made, so governance capture parity is preserved even though the MAF + graph itself never runs for those two modes. + +**UI surface.** Each of the 3 Playbook pages +(`Presentation.Web/Features/Playbooks/*PlaybookPage.razor`) shows "Rerun", "Recall", and "Replay" +controls (`PlaybookReplayControls.razor`) alongside the latest execution's result panel, each with +an inline explanation of what it does so the distinction is clear without reading this document. +Selecting any control re-invokes the same page's run command with the corresponding `ReplayMode` +and the latest execution's id as `SourceExecutionId` (ignored server-side for `Rerun`), using the +latest execution's own `CollectInput` rather than requiring the user to retype it. The result +panel re-renders in place with the rerun/recalled/replayed result — there is no before/after +comparison view by design, to keep the UI simple for a quick-start template. + ## Unified Collect / Evaluate / Record Shape All three example workflows deliberately follow one common input/criteria/output pattern, so a diff --git a/docs/product/features/feature-playbook-essay.md b/docs/product/features/feature-playbook-essay.md index 2e355b0..5be4814 100644 --- a/docs/product/features/feature-playbook-essay.md +++ b/docs/product/features/feature-playbook-essay.md @@ -169,14 +169,31 @@ current governance gap (every score is still captured, attributed, and auditable ## UI Changes The `EssayPlaybookPage` (`Presentation.Web/Features/Playbooks`) runs this Playbook from an essay -text input, then renders the shared `PlaybookResultPanel`: a Collect/Evaluate/Record toggle, a -summary card, and a step-detail panel. The step-detail panel fetches this Playbook's catalog -entry (`GetPlaybookByKeyAsync("essay")`) to show each step's persisted `Description` and +text input, then renders the shared `PlaybookResultPanel`: "Rerun"/"Recall"/"Replay" +repeatability controls (`PlaybookReplayControls`, each with an inline explanation of what it +does), a Collect/Evaluate/Record toggle, a summary card, and a +step-detail panel. The step-detail panel fetches this Playbook's catalog entry +(`GetPlaybookByKeyAsync("essay")`) to show each step's persisted `Description` and `ActionDefinition` (the rubric-scoring prompt) alongside that run's dynamic input/output. +## Repeatability (Recall / Replay) +Since Collect and Record are deterministic but Evaluate is agentic, Replay is the most useful +mode here: it re-runs the rubric-scoring prompt against the exact same collected essay evidence, +proving the Evaluate prompt reproduces an equivalent scorecard without needing the (free) +deterministic Collect stage to run again. Recall skips both Collect and Evaluate, re-rendering +the prior scorecard at zero cost. Like Taxonomy, `EssayEvaluationRunner` passes the +`PlaybookReplayContext` through to the repo-owned +`PlaybookWorkflowGraphExecutor<...>`, which bypasses the MAF 3-node graph for Recall/Replay and +calls only the remaining stage tool(s) directly against the recalled Evidence/Finding, while +still emitting the same governance activity-recorder calls the graph nodes would have made. See +`docs/governance/playbook-workflow-types.md` ("Repeatability: Rerun / Recall / Replay") for the +full mechanism shared across all three Playbooks. + ## API Changes - `POST api/v{version}/my/playbooks/essay` (`RunMyEssayPlaybookCommand`) executes the Playbook - and persists a `PlaybookExecutionEntity` for the run. + and persists a `PlaybookExecutionEntity` for the run. Accepts an optional `ReplayMode` + (`Rerun`/`Recall`/`Replay`, default `Rerun`) and `SourceExecutionId` to recall or replay a prior + execution instead of collecting fresh input. - The catalog itself is queryable/creatable/updatable/deletable via the shared, unsecured `api/v{version}/playbooks` CRUD endpoints (`PlaybookCatalogEndpoints`): list, get by id, get by key, create, update general info, update one CER step, delete. diff --git a/docs/product/features/feature-playbook-sql-statistics.md b/docs/product/features/feature-playbook-sql-statistics.md index 78e5fd8..df29faf 100644 --- a/docs/product/features/feature-playbook-sql-statistics.md +++ b/docs/product/features/feature-playbook-sql-statistics.md @@ -134,15 +134,29 @@ are inclusive upper bounds. ## UI Changes The `SqlStatisticsPlaybookPage` (`Presentation.Web/Features/Playbooks`) runs this Playbook from a -database-name input, then renders the shared `PlaybookResultPanel`: a Collect/Evaluate/Record -toggle, a summary card, and a step-detail panel. The step-detail panel fetches this Playbook's -catalog entry (`GetPlaybookByKeyAsync("sql-statistics")`) to show each step's persisted -`Description` and `ActionDefinition` (the actual statistics query) alongside that run's dynamic -input/output. +database-name input, then renders the shared `PlaybookResultPanel`: "Rerun"/"Recall"/"Replay" +repeatability controls (`PlaybookReplayControls`, each with an inline explanation of what it +does), a Collect/Evaluate/Record toggle, a summary card, and a +step-detail panel. The step-detail panel fetches this Playbook's catalog entry +(`GetPlaybookByKeyAsync("sql-statistics")`) to show each step's persisted `Description` and +`ActionDefinition` (the actual statistics query) alongside that run's dynamic input/output. + +## Repeatability (Recall / Replay) +Because every stage of this Playbook is deterministic, Recall and Replay are low-stakes by +design — re-evaluating or re-recording the same Evidence always reproduces the same Finding. This +Playbook exercises the package's own replay path directly: `SqlStatisticsClassificationRunner` +calls `PlaybookExecutor<...>.ExecuteAsync(definition, input, replayContext, ct, recorder)` when a +`PlaybookReplayContext` is supplied (built +from the prior run's persisted `EvidenceJson`/`FindingJson`), rather than going through a +repo-owned graph executor like the other two Playbooks. See +`docs/governance/playbook-workflow-types.md` ("Repeatability: Rerun / Recall / Replay") for the +full mechanism shared across all three Playbooks. ## API Changes - `POST api/v{version}/my/playbooks/sql-statistics` (`RunMySqlStatisticsPlaybookCommand`) executes - the Playbook and persists a `PlaybookExecutionEntity` for the run. + the Playbook and persists a `PlaybookExecutionEntity` for the run. Accepts an optional + `ReplayMode` (`Rerun`/`Recall`/`Replay`, default `Rerun`) and `SourceExecutionId` to recall or + replay a prior execution instead of collecting fresh input. - The catalog itself is queryable/creatable/updatable/deletable via the shared, unsecured `api/v{version}/playbooks` CRUD endpoints (`PlaybookCatalogEndpoints`): list, get by id, get by key, create, update general info, update one CER step, delete. @@ -173,8 +187,7 @@ size metrics are operational metadata, not regulated or personal data. versioned bands before relying on this in production. ## Out Of Scope -- Exposing this Playbook via a chat tool, REST endpoint, or UI page. -- Persisting results by default (persistence is opt-in via `SavePlaybookMaterializationCommand`). +- Exposing this Playbook via a chat tool. - Any LLM-driven reasoning — that is intentionally the role of the other two example Playbooks. ## Definition Of Done diff --git a/docs/product/features/feature-playbook-taxonomy.md b/docs/product/features/feature-playbook-taxonomy.md index 3f74c17..4e7284b 100644 --- a/docs/product/features/feature-playbook-taxonomy.md +++ b/docs/product/features/feature-playbook-taxonomy.md @@ -151,15 +151,33 @@ already enforced; term-extraction *quality* is not yet separately evaluated). ## UI Changes The `TaxonomyPlaybookPage` (`Presentation.Web/Features/Playbooks`) runs this Playbook from a -free-text input, then renders the shared `PlaybookResultPanel`: a Collect/Evaluate/Record toggle, -a summary card, and a step-detail panel. The step-detail panel fetches this Playbook's catalog -entry (`GetPlaybookByKeyAsync("taxonomy")`) to show each step's persisted `Description` and +free-text input, then renders the shared `PlaybookResultPanel`: "Rerun"/"Recall"/"Replay" +repeatability controls (`PlaybookReplayControls`, each with an inline explanation of what it +does), a Collect/Evaluate/Record toggle, a summary card, and a +step-detail panel. The step-detail panel fetches this Playbook's catalog entry +(`GetPlaybookByKeyAsync("taxonomy")`) to show each step's persisted `Description` and `ActionDefinition` (the extraction/classification prompts) alongside that run's dynamic input/output. +## Repeatability (Recall / Replay) +Because every stage of this Playbook is agentic, Recall and Replay matter most here: Recall lets +a user see the exact prior Finding without spending any tokens, and Replay proves the +Evaluate/Record prompts reproduce the same classification from the same (recalled) extracted +terms, without re-running the more token-heavy Collect stage. Unlike SQL Statistics, this +Playbook's `TaxonomyClassificationRunner` does not call the package's `PlaybookExecutor` +directly — it passes the `PlaybookReplayContext` through to the +repo-owned `PlaybookWorkflowGraphExecutor<...>`, which bypasses the MAF 3-node graph entirely for +Recall/Replay and instead calls only the remaining stage tool(s) +(`IRecordStepTool<,>`/`IEvaluateStepTool<,>`, resolved to `TaxonomyRecordAgentTool`/ +`TaxonomyEvaluateAgentTool`) directly against the recalled Evidence/Finding, while still emitting +the same governance activity-recorder calls the graph nodes would have made. See `docs/governance/playbook-workflow-types.md` ("Repeatability: Rerun / Recall / Replay") for +the full mechanism shared across all three Playbooks. + ## API Changes - `POST api/v{version}/my/playbooks/taxonomy` (`RunMyTaxonomyPlaybookCommand`) executes the - Playbook and persists a `PlaybookExecutionEntity` for the run. + Playbook and persists a `PlaybookExecutionEntity` for the run. Accepts an optional `ReplayMode` + (`Rerun`/`Recall`/`Replay`, default `Rerun`) and `SourceExecutionId` to recall or replay a prior + execution instead of collecting fresh input. - The catalog itself is queryable/creatable/updatable/deletable via the shared, unsecured `api/v{version}/playbooks` CRUD endpoints (`PlaybookCatalogEndpoints`): list, get by id, get by key, create, update general info, update one CER step, delete. diff --git a/src/Core.Application/Playbooks/Essay/EvaluateEssayCommand.cs b/src/Core.Application/Playbooks/Essay/EvaluateEssayCommand.cs index d81578b..4f42860 100644 --- a/src/Core.Application/Playbooks/Essay/EvaluateEssayCommand.cs +++ b/src/Core.Application/Playbooks/Essay/EvaluateEssayCommand.cs @@ -15,7 +15,9 @@ namespace Goodtocode.AgentFramework.Core.Application.Playbooks.Essay; public interface IEssayEvaluationRunner { Task> EvaluateAsync( - string essayText, CancellationToken cancellationToken); + string essayText, + CancellationToken cancellationToken, + PlaybookReplayContext? replayContext = null); } /// diff --git a/src/Core.Application/Playbooks/Essay/RunEssayPlaybookCommand.cs b/src/Core.Application/Playbooks/Essay/RunEssayPlaybookCommand.cs index b6d87bb..4c05675 100644 --- a/src/Core.Application/Playbooks/Essay/RunEssayPlaybookCommand.cs +++ b/src/Core.Application/Playbooks/Essay/RunEssayPlaybookCommand.cs @@ -1,3 +1,6 @@ +using Goodtocode.AgentFramework.Core.Application.Playbooks.Persistence; +using Goodtocode.Agents.Playbook.Execution; + namespace Goodtocode.AgentFramework.Core.Application.Playbooks.Essay; /// @@ -5,15 +8,21 @@ namespace Goodtocode.AgentFramework.Core.Application.Playbooks.Essay; /// raw essay text (no chat-message dependency), projecting the result into the shared /// shape used by every example playbook's "run" endpoint. /// Distinct from , which resolves essay text from an existing -/// chat message. +/// chat message. selects the repeatability behavior: Rerun +/// (default) executes fresh against ; Recall and Replay +/// ignore and instead rehydrate the prior execution identified by +/// (or the user's latest execution, if omitted). /// -public sealed class RunEssayPlaybookCommand : IRequest +public sealed class RunEssayPlaybookCommand : UserScopedRequest, IRequest { public required string EssayText { get; init; } + public PlaybookReplayMode ReplayMode { get; init; } = PlaybookReplayMode.Rerun; + public string? SourceExecutionId { get; init; } } public sealed class RunEssayPlaybookCommandHandler( ISender sender, + IAgentFrameworkContext context, IEssayEvaluationRunner runner, EssayStageSummarySelector summarySelector) : IRequestHandler @@ -22,9 +31,29 @@ public async Task Handle(RunEssayPlaybookCommand req { ArgumentNullException.ThrowIfNull(request); - var result = await runner.EvaluateAsync(request.EssayText, cancellationToken); - var dto = PlaybookExecutionResultDtoFactory.CreateFrom(request.EssayText, result, summarySelector); - await Persistence.PlaybookExecutionPersister.SaveAsync(sender, dto, cancellationToken); + string essayText; + PlaybookReplayContext? replayContext = null; + + if (request.ReplayMode == PlaybookReplayMode.Rerun) + { + essayText = request.EssayText; + } + else + { + var source = await PlaybookReplaySourceResolver.ResolveAsync( + context, "essay", request.UserContext.OwnerId, request.UserContext.TenantId, + request.SourceExecutionId, cancellationToken); + + essayText = source.CollectInput; + var priorEvidence = System.Text.Json.JsonSerializer.Deserialize(source.EvidenceJson!)!; + var priorFinding = System.Text.Json.JsonSerializer.Deserialize(source.FindingJson!)!; + replayContext = new PlaybookReplayContext( + request.ReplayMode, source.Id.ToString(), priorEvidence, priorFinding); + } + + var result = await runner.EvaluateAsync(essayText, cancellationToken, replayContext); + var dto = PlaybookExecutionResultDtoFactory.CreateFrom(essayText, result, summarySelector); + dto.ExecutionId = await Persistence.PlaybookExecutionPersister.SaveAsync(sender, dto, cancellationToken); return dto; } } diff --git a/src/Core.Application/Playbooks/Persistence/PlaybookExecutionPersister.cs b/src/Core.Application/Playbooks/Persistence/PlaybookExecutionPersister.cs index 316dcd3..37ceb4e 100644 --- a/src/Core.Application/Playbooks/Persistence/PlaybookExecutionPersister.cs +++ b/src/Core.Application/Playbooks/Persistence/PlaybookExecutionPersister.cs @@ -19,6 +19,8 @@ public static Task SaveAsync(ISender sender, PlaybookExecutionResultDto dt CollectOutput = dto.CollectSummary, EvaluateOutput = dto.EvaluateSummary, RecordOutput = dto.RecordSummary, + EvidenceJson = dto.EvidenceJson, + FindingJson = dto.FindingJson, StartedUtc = dto.StartedUtc, CompletedUtc = dto.CompletedUtc }, cancellationToken); diff --git a/src/Core.Application/Playbooks/Persistence/PlaybookReplaySourceResolver.cs b/src/Core.Application/Playbooks/Persistence/PlaybookReplaySourceResolver.cs new file mode 100644 index 0000000..0dee432 --- /dev/null +++ b/src/Core.Application/Playbooks/Persistence/PlaybookReplaySourceResolver.cs @@ -0,0 +1,45 @@ +using Goodtocode.AgentFramework.Core.Application.Common.Exceptions; +using Goodtocode.AgentFramework.Core.Domain.Playbooks; + +namespace Goodtocode.AgentFramework.Core.Application.Playbooks.Persistence; + +/// +/// Resolves the prior a Recall or Replay request should run +/// against: either a specific execution by id, or - when no id is supplied - the current user's +/// most recently completed execution of that Playbook. Shared by every example playbook's +/// Run*PlaybookCommandHandler so "recall/replay the latest run" needs no per-playbook lookup code. +/// +public static class PlaybookReplaySourceResolver +{ + public static async Task ResolveAsync( + IAgentFrameworkContext context, + string playbookKey, + Guid ownerId, + Guid tenantId, + string? sourceExecutionId, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(context); + + var query = context.PlaybookExecutions + .Where(x => x.PlaybookKey == playbookKey && x.OwnerId == ownerId && x.TenantId == tenantId); + + var entity = !string.IsNullOrWhiteSpace(sourceExecutionId) && Guid.TryParse(sourceExecutionId, out var id) + ? await query.FirstOrDefaultAsync(x => x.Id == id, cancellationToken) + : await query.OrderByDescending(x => x.CompletedUtc).FirstOrDefaultAsync(cancellationToken); + + if (entity is null) + { + throw new CustomNotFoundException( + $"No prior execution of playbook '{playbookKey}' was found to recall or replay."); + } + + if (string.IsNullOrWhiteSpace(entity.EvidenceJson) || string.IsNullOrWhiteSpace(entity.FindingJson)) + { + throw new CustomConflictException( + $"Execution '{entity.Id}' of playbook '{playbookKey}' does not have recorded evidence/finding and cannot be recalled or replayed."); + } + + return entity; + } +} diff --git a/src/Core.Application/Playbooks/Persistence/SavePlaybookExecutionCommand.cs b/src/Core.Application/Playbooks/Persistence/SavePlaybookExecutionCommand.cs index 566f886..cf089c8 100644 --- a/src/Core.Application/Playbooks/Persistence/SavePlaybookExecutionCommand.cs +++ b/src/Core.Application/Playbooks/Persistence/SavePlaybookExecutionCommand.cs @@ -20,6 +20,8 @@ public sealed class SavePlaybookExecutionCommand : UserScopedRequest, IRequest Handle(SavePlaybookExecutionCommand request, Cancellatio collectOutput: request.CollectOutput, evaluateOutput: request.EvaluateOutput, recordOutput: request.RecordOutput, + evidenceJson: request.EvidenceJson, + findingJson: request.FindingJson, startedUtc: request.StartedUtc, completedUtc: request.CompletedUtc); diff --git a/src/Core.Application/Playbooks/PlaybookExecutionResultDto.cs b/src/Core.Application/Playbooks/PlaybookExecutionResultDto.cs index 4b5e4ef..8ff4bb4 100644 --- a/src/Core.Application/Playbooks/PlaybookExecutionResultDto.cs +++ b/src/Core.Application/Playbooks/PlaybookExecutionResultDto.cs @@ -13,6 +13,12 @@ namespace Goodtocode.AgentFramework.Core.Application.Playbooks; /// public sealed class PlaybookExecutionResultDto { + /// + /// The persisted 's own id. Used as + /// the SourceExecutionId of a later Recall or Replay request against this execution. + /// + public Guid ExecutionId { get; set; } + public string PlaybookKey { get; set; } = string.Empty; public string Version { get; set; } = string.Empty; public DateTimeOffset StartedUtc { get; set; } @@ -40,6 +46,18 @@ public sealed class PlaybookExecutionResultDto /// Human-readable summary of the Record stage's typed materialization. /// public string RecordSummary { get; set; } = string.Empty; + + /// + /// JSON-serialized Collect-stage typed evidence, carried only so this execution can later be + /// used as the source of a Replay request; never rendered directly by the UI. + /// + public string? EvidenceJson { get; set; } + + /// + /// JSON-serialized Evaluate-stage typed finding, carried only so this execution can later be + /// used as the source of a Recall request; never rendered directly by the UI. + /// + public string? FindingJson { get; set; } } /// @@ -62,6 +80,7 @@ public static PlaybookExecutionResultDto CreateFrom(Core.Domain.Playbooks.Playbo return new PlaybookExecutionResultDto { + ExecutionId = entity.Id, PlaybookKey = entity.PlaybookKey, Version = entity.PlaybookVersion, StartedUtc = entity.StartedUtc, @@ -71,7 +90,9 @@ public static PlaybookExecutionResultDto CreateFrom(Core.Domain.Playbooks.Playbo CollectInput = entity.CollectInput, CollectSummary = entity.CollectOutput, EvaluateSummary = entity.EvaluateOutput, - RecordSummary = entity.RecordOutput + RecordSummary = entity.RecordOutput, + EvidenceJson = entity.EvidenceJson, + FindingJson = entity.FindingJson }; } @@ -94,7 +115,9 @@ public static PlaybookExecutionResultDto CreateFrom /// API-facing command for running the SQL Statistics Classification example playbook end to end /// and projecting the result into the shared shape used -/// by every example playbook's "run" endpoint. +/// by every example playbook's "run" endpoint. selects the repeatability +/// behavior: Rerun (default) executes fresh against ; Recall +/// and Replay ignore and instead rehydrate the prior execution +/// identified by (or the user's latest execution, if omitted). /// -public sealed class RunSqlStatisticsPlaybookCommand : IRequest +public sealed class RunSqlStatisticsPlaybookCommand : UserScopedRequest, IRequest { public required string DatabaseName { get; init; } + public PlaybookReplayMode ReplayMode { get; init; } = PlaybookReplayMode.Rerun; + public string? SourceExecutionId { get; init; } } public sealed class RunSqlStatisticsPlaybookCommandHandler( ISender sender, + IAgentFrameworkContext context, ISqlStatisticsClassificationRunner runner, SqlStatisticsStageSummarySelector summarySelector) : IRequestHandler @@ -20,9 +29,29 @@ public async Task Handle(RunSqlStatisticsPlaybookCom { ArgumentNullException.ThrowIfNull(request); - var result = await runner.ClassifyAsync(request.DatabaseName, cancellationToken); - var dto = PlaybookExecutionResultDtoFactory.CreateFrom(request.DatabaseName, result, summarySelector); - await Persistence.PlaybookExecutionPersister.SaveAsync(sender, dto, cancellationToken); + string databaseName; + PlaybookReplayContext? replayContext = null; + + if (request.ReplayMode == PlaybookReplayMode.Rerun) + { + databaseName = request.DatabaseName; + } + else + { + var source = await PlaybookReplaySourceResolver.ResolveAsync( + context, "sql-statistics", request.UserContext.OwnerId, request.UserContext.TenantId, + request.SourceExecutionId, cancellationToken); + + databaseName = source.CollectInput; + var priorEvidence = System.Text.Json.JsonSerializer.Deserialize(source.EvidenceJson!)!; + var priorFinding = System.Text.Json.JsonSerializer.Deserialize(source.FindingJson!)!; + replayContext = new PlaybookReplayContext( + request.ReplayMode, source.Id.ToString(), priorEvidence, priorFinding); + } + + var result = await runner.ClassifyAsync(databaseName, cancellationToken, replayContext); + var dto = PlaybookExecutionResultDtoFactory.CreateFrom(databaseName, result, summarySelector); + dto.ExecutionId = await Persistence.PlaybookExecutionPersister.SaveAsync(sender, dto, cancellationToken); return dto; } } diff --git a/src/Core.Application/Playbooks/SqlStatistics/SqlStatisticsClassificationRunner.cs b/src/Core.Application/Playbooks/SqlStatistics/SqlStatisticsClassificationRunner.cs index e9a7447..7be1582 100644 --- a/src/Core.Application/Playbooks/SqlStatistics/SqlStatisticsClassificationRunner.cs +++ b/src/Core.Application/Playbooks/SqlStatistics/SqlStatisticsClassificationRunner.cs @@ -13,7 +13,9 @@ namespace Goodtocode.AgentFramework.Core.Application.Playbooks.SqlStatistics; public interface ISqlStatisticsClassificationRunner { Task> ClassifyAsync( - string databaseName, CancellationToken cancellationToken); + string databaseName, + CancellationToken cancellationToken, + PlaybookReplayContext? replayContext = null); } public sealed class SqlStatisticsClassificationRunner( @@ -23,10 +25,14 @@ public sealed class SqlStatisticsClassificationRunner( : ISqlStatisticsClassificationRunner { public Task> ClassifyAsync( - string databaseName, CancellationToken cancellationToken) + string databaseName, + CancellationToken cancellationToken, + PlaybookReplayContext? replayContext = null) { ArgumentException.ThrowIfNullOrWhiteSpace(databaseName); - return executor.ExecuteAsync(definition, databaseName, cancellationToken, activityRecorder: recorder); + return replayContext is null + ? executor.ExecuteAsync(definition, databaseName, cancellationToken, activityRecorder: recorder) + : executor.ExecuteAsync(definition, databaseName, replayContext, cancellationToken, activityRecorder: recorder); } } diff --git a/src/Core.Application/Playbooks/Taxonomy/ClassifyTaxonomyCommand.cs b/src/Core.Application/Playbooks/Taxonomy/ClassifyTaxonomyCommand.cs index 1ca56d9..df8b517 100644 --- a/src/Core.Application/Playbooks/Taxonomy/ClassifyTaxonomyCommand.cs +++ b/src/Core.Application/Playbooks/Taxonomy/ClassifyTaxonomyCommand.cs @@ -12,7 +12,9 @@ namespace Goodtocode.AgentFramework.Core.Application.Playbooks.Taxonomy; public interface ITaxonomyClassificationRunner { Task> ClassifyAsync( - string text, CancellationToken cancellationToken); + string text, + CancellationToken cancellationToken, + PlaybookReplayContext? replayContext = null); } public sealed class ClassifyTaxonomyCommand : IRequest> diff --git a/src/Core.Application/Playbooks/Taxonomy/RunTaxonomyPlaybookCommand.cs b/src/Core.Application/Playbooks/Taxonomy/RunTaxonomyPlaybookCommand.cs index ecbeb89..d39515d 100644 --- a/src/Core.Application/Playbooks/Taxonomy/RunTaxonomyPlaybookCommand.cs +++ b/src/Core.Application/Playbooks/Taxonomy/RunTaxonomyPlaybookCommand.cs @@ -1,3 +1,6 @@ +using Goodtocode.AgentFramework.Core.Application.Playbooks.Persistence; +using Goodtocode.Agents.Playbook.Execution; + namespace Goodtocode.AgentFramework.Core.Application.Playbooks.Taxonomy; /// @@ -5,14 +8,21 @@ namespace Goodtocode.AgentFramework.Core.Application.Playbooks.Taxonomy; /// to end and projecting the result into the shared shape /// used by every example playbook's "run" endpoint. Distinct from , /// which returns the fully typed PlaybookExecutionResult for callers that need it. +/// selects the repeatability behavior: Rerun (default) executes +/// fresh against ; Recall and Replay ignore and +/// instead rehydrate the prior execution identified by (or the +/// user's latest execution, if omitted). /// -public sealed class RunTaxonomyPlaybookCommand : IRequest +public sealed class RunTaxonomyPlaybookCommand : UserScopedRequest, IRequest { public required string Text { get; init; } + public PlaybookReplayMode ReplayMode { get; init; } = PlaybookReplayMode.Rerun; + public string? SourceExecutionId { get; init; } } public sealed class RunTaxonomyPlaybookCommandHandler( ISender sender, + IAgentFrameworkContext context, ITaxonomyClassificationRunner runner, TaxonomyStageSummarySelector summarySelector) : IRequestHandler @@ -21,9 +31,29 @@ public async Task Handle(RunTaxonomyPlaybookCommand { ArgumentNullException.ThrowIfNull(request); - var result = await runner.ClassifyAsync(request.Text, cancellationToken); - var dto = PlaybookExecutionResultDtoFactory.CreateFrom(request.Text, result, summarySelector); - await Persistence.PlaybookExecutionPersister.SaveAsync(sender, dto, cancellationToken); + string text; + PlaybookReplayContext? replayContext = null; + + if (request.ReplayMode == PlaybookReplayMode.Rerun) + { + text = request.Text; + } + else + { + var source = await PlaybookReplaySourceResolver.ResolveAsync( + context, "taxonomy", request.UserContext.OwnerId, request.UserContext.TenantId, + request.SourceExecutionId, cancellationToken); + + text = source.CollectInput; + var priorEvidence = System.Text.Json.JsonSerializer.Deserialize(source.EvidenceJson!)!; + var priorFinding = System.Text.Json.JsonSerializer.Deserialize(source.FindingJson!)!; + replayContext = new PlaybookReplayContext( + request.ReplayMode, source.Id.ToString(), priorEvidence, priorFinding); + } + + var result = await runner.ClassifyAsync(text, cancellationToken, replayContext); + var dto = PlaybookExecutionResultDtoFactory.CreateFrom(text, result, summarySelector); + dto.ExecutionId = await Persistence.PlaybookExecutionPersister.SaveAsync(sender, dto, cancellationToken); return dto; } } diff --git a/src/Core.Domain/Playbooks/PlaybookExecutionEntity.cs b/src/Core.Domain/Playbooks/PlaybookExecutionEntity.cs index d6b2148..6ea152e 100644 --- a/src/Core.Domain/Playbooks/PlaybookExecutionEntity.cs +++ b/src/Core.Domain/Playbooks/PlaybookExecutionEntity.cs @@ -20,6 +20,21 @@ public class PlaybookExecutionEntity : SecuredEntity public string CollectOutput { get; private set; } = string.Empty; public string EvaluateOutput { get; private set; } = string.Empty; public string RecordOutput { get; private set; } = string.Empty; + + /// + /// JSON-serialized Collect-stage typed evidence (the playbook's own TEvidence), + /// persisted alongside the human-readable summary so a later + /// execution can Replay (re-run Evaluate+Record) against this exact evidence. + /// + public string? EvidenceJson { get; private set; } + + /// + /// JSON-serialized Evaluate-stage typed finding (the playbook's own TFinding), + /// persisted alongside the human-readable summary so a later + /// execution can Recall (re-run Record only) against this exact finding. + /// + public string? FindingJson { get; private set; } + public DateTimeOffset StartedUtc { get; private set; } public DateTimeOffset CompletedUtc { get; private set; } public virtual PlaybookEntity? Playbook { get; private set; } @@ -44,6 +59,8 @@ private PlaybookExecutionEntity( string collectOutput, string evaluateOutput, string recordOutput, + string? evidenceJson, + string? findingJson, DateTimeOffset startedUtc, DateTimeOffset completedUtc) : base(id: id, partitionKey: tenantId.ToString(), rowKey: canonicalKey, @@ -60,6 +77,8 @@ private PlaybookExecutionEntity( CollectOutput = collectOutput; EvaluateOutput = evaluateOutput; RecordOutput = recordOutput; + EvidenceJson = evidenceJson; + FindingJson = findingJson; StartedUtc = startedUtc; CompletedUtc = completedUtc; } @@ -77,6 +96,8 @@ public static PlaybookExecutionEntity Create( string collectOutput, string evaluateOutput, string recordOutput, + string? evidenceJson, + string? findingJson, DateTimeOffset startedUtc, DateTimeOffset completedUtc) { @@ -98,6 +119,8 @@ public static PlaybookExecutionEntity Create( collectOutput: collectOutput, evaluateOutput: evaluateOutput, recordOutput: recordOutput, + evidenceJson: evidenceJson, + findingJson: findingJson, startedUtc: startedUtc, completedUtc: completedUtc); } diff --git a/src/Infrastructure.AgentFramework/Execution/PlaybookWorkflowGraphExecutor.cs b/src/Infrastructure.AgentFramework/Execution/PlaybookWorkflowGraphExecutor.cs index 9529e68..73df5d9 100644 --- a/src/Infrastructure.AgentFramework/Execution/PlaybookWorkflowGraphExecutor.cs +++ b/src/Infrastructure.AgentFramework/Execution/PlaybookWorkflowGraphExecutor.cs @@ -14,7 +14,11 @@ namespace Goodtocode.AgentFramework.Infrastructure.AgentFramework.Execution; /// fully agentic workflow - mirroring the pattern already proven in crucible-web's /// PipelineWorkflowStepExecutor. Each node calls the shared /// after its -/// stage completes, so governance capture is identical regardless of orchestration model. +/// stage completes, so governance capture is identical regardless of orchestration model. When an +/// optional is supplied, the full graph is +/// bypassed in favor of direct, in-order tool calls for just the stages that mode requires - +/// Record only for Recall, Evaluate+Record for Replay - mirroring the same repeatability pillar +/// the package's own PlaybookExecutor implements for the non-MAF SQL Statistics playbook. /// public sealed class PlaybookWorkflowGraphExecutor { @@ -25,14 +29,39 @@ public async Task IRecordStepTool recordTool, TCollectInput input, CancellationToken cancellationToken = default, - IPlaybookStepActivityRecorder? activityRecorder = null) + IPlaybookStepActivityRecorder? activityRecorder = null, + PlaybookReplayContext? replayContext = null) { ArgumentNullException.ThrowIfNull(collectTool); ArgumentNullException.ThrowIfNull(evaluateTool); ArgumentNullException.ThrowIfNull(recordTool); + replayContext?.Validate(); + var startedUtc = DateTimeOffset.UtcNow; + return replayContext?.Mode switch + { + PlaybookReplayMode.Recall => await ExecuteRecallAsync(identity, recordTool, replayContext, activityRecorder, startedUtc, cancellationToken).ConfigureAwait(false), + PlaybookReplayMode.Replay => await ExecuteReplayAsync(identity, evaluateTool, recordTool, replayContext, activityRecorder, startedUtc, cancellationToken).ConfigureAwait(false), + _ => await ExecuteRerunAsync(identity, collectTool, evaluateTool, recordTool, input, activityRecorder, startedUtc, cancellationToken).ConfigureAwait(false) + }; + } + + /// + /// Fresh execution: runs the full Collect->Evaluate->Record Microsoft Agent Framework + /// workflow graph exactly as before. A first-time execution is always a Rerun. + /// + private async Task> ExecuteRerunAsync( + PlaybookIdentity identity, + ICollectStepTool collectTool, + IEvaluateStepTool evaluateTool, + IRecordStepTool recordTool, + TCollectInput input, + IPlaybookStepActivityRecorder? activityRecorder, + DateTimeOffset startedUtc, + CancellationToken cancellationToken) + { var collectNode = new CollectNode(identity, collectTool, activityRecorder, cancellationToken); var evaluateNode = new EvaluateNode(identity, evaluateTool, activityRecorder, cancellationToken); var recordNode = new RecordNode(identity, recordTool, activityRecorder, cancellationToken); @@ -55,8 +84,8 @@ public async Task identity.Version, startedUtc, DateTimeOffset.UtcNow, - PlaybookReplayMode.Replay, - identity.ExecutionId); + PlaybookReplayMode.Rerun, + SourceExecutionId: null!); return new PlaybookExecutionResult( collectNode.Evidence!, @@ -70,6 +99,68 @@ public async Task $"Playbook workflow '{identity.PlaybookKey}' did not yield a Record-stage output."); } + /// + /// Recall: rehydrates the prior execution's evidence and finding and runs only the Record + /// stage, bypassing the Collect/Evaluate nodes entirely so materialization can be re-rendered + /// from the recalled finding without any new tool calls. + /// + private static async Task> ExecuteRecallAsync( + PlaybookIdentity identity, + IRecordStepTool recordTool, + PlaybookReplayContext replayContext, + IPlaybookStepActivityRecorder? activityRecorder, + DateTimeOffset startedUtc, + CancellationToken cancellationToken) + { + var materialization = await recordTool.RecordAsync(replayContext.PriorFinding, cancellationToken).ConfigureAwait(false); + + if (activityRecorder is not null) + { + await activityRecorder.OnRecordedAsync(identity, materialization, recordTool.ToolName, cancellationToken).ConfigureAwait(false); + } + + var metadata = new PlaybookExecutionMetadata( + identity.PlaybookKey, identity.Version, startedUtc, DateTimeOffset.UtcNow, + PlaybookReplayMode.Recall, replayContext.SourceExecutionId); + + return new PlaybookExecutionResult( + replayContext.PriorEvidence, replayContext.PriorFinding, materialization, metadata); + } + + /// + /// Replay: reuses the prior execution's Collect-stage evidence (skipping Collect) and re-runs + /// Evaluate and Record against it, to verify exact reproduction of a governed result. + /// + private static async Task> ExecuteReplayAsync( + PlaybookIdentity identity, + IEvaluateStepTool evaluateTool, + IRecordStepTool recordTool, + PlaybookReplayContext replayContext, + IPlaybookStepActivityRecorder? activityRecorder, + DateTimeOffset startedUtc, + CancellationToken cancellationToken) + { + var evidence = replayContext.PriorEvidence; + + var finding = await evaluateTool.EvaluateAsync(evidence, cancellationToken).ConfigureAwait(false); + if (activityRecorder is not null) + { + await activityRecorder.OnEvaluatedAsync(identity, finding, evaluateTool.ToolName, cancellationToken).ConfigureAwait(false); + } + + var materialization = await recordTool.RecordAsync(finding, cancellationToken).ConfigureAwait(false); + if (activityRecorder is not null) + { + await activityRecorder.OnRecordedAsync(identity, materialization, recordTool.ToolName, cancellationToken).ConfigureAwait(false); + } + + var metadata = new PlaybookExecutionMetadata( + identity.PlaybookKey, identity.Version, startedUtc, DateTimeOffset.UtcNow, + PlaybookReplayMode.Replay, replayContext.SourceExecutionId); + + return new PlaybookExecutionResult(evidence, finding, materialization, metadata); + } + private sealed class CollectNode( PlaybookIdentity identity, ICollectStepTool tool, diff --git a/src/Infrastructure.AgentFramework/Playbooks/Essay/EssayEvaluationRunner.cs b/src/Infrastructure.AgentFramework/Playbooks/Essay/EssayEvaluationRunner.cs index cd40823..f9062d6 100644 --- a/src/Infrastructure.AgentFramework/Playbooks/Essay/EssayEvaluationRunner.cs +++ b/src/Infrastructure.AgentFramework/Playbooks/Essay/EssayEvaluationRunner.cs @@ -21,7 +21,9 @@ public sealed class EssayEvaluationRunner( private readonly EssayGovernanceActivityRecorder _recorder = recorder; public Task> EvaluateAsync( - string essayText, CancellationToken cancellationToken) + string essayText, + CancellationToken cancellationToken, + PlaybookReplayContext? replayContext = null) { ArgumentException.ThrowIfNullOrWhiteSpace(essayText); @@ -34,6 +36,7 @@ public Task> ClassifyAsync( - string text, CancellationToken cancellationToken) + string text, + CancellationToken cancellationToken, + PlaybookReplayContext? replayContext = null) { ArgumentException.ThrowIfNullOrWhiteSpace(text); @@ -34,6 +36,7 @@ public Task @@ -518,6 +518,12 @@ protected override void BuildTargetModel(ModelBuilder modelBuilder) .IsRequired() .HasColumnType("NVARCHAR(MAX)"); + b.Property("EvidenceJson") + .HasColumnType("NVARCHAR(MAX)"); + + b.Property("FindingJson") + .HasColumnType("NVARCHAR(MAX)"); + b.Property("ModifiedBy") .HasColumnType("uniqueidentifier"); diff --git a/src/Infrastructure.SqlServer/Migrations/20261003162033_InitialCreate-AgentFrameworkContext.cs b/src/Infrastructure.SqlServer/Migrations/20261003172848_InitialCreate-AgentFrameworkContext.cs similarity index 99% rename from src/Infrastructure.SqlServer/Migrations/20261003162033_InitialCreate-AgentFrameworkContext.cs rename to src/Infrastructure.SqlServer/Migrations/20261003172848_InitialCreate-AgentFrameworkContext.cs index 2b2b8c6..183ec7e 100644 --- a/src/Infrastructure.SqlServer/Migrations/20261003162033_InitialCreate-AgentFrameworkContext.cs +++ b/src/Infrastructure.SqlServer/Migrations/20261003172848_InitialCreate-AgentFrameworkContext.cs @@ -230,6 +230,8 @@ protected override void Up(MigrationBuilder migrationBuilder) CollectOutput = table.Column(type: "NVARCHAR(MAX)", nullable: false), EvaluateOutput = table.Column(type: "NVARCHAR(MAX)", nullable: false), RecordOutput = table.Column(type: "NVARCHAR(MAX)", nullable: false), + EvidenceJson = table.Column(type: "NVARCHAR(MAX)", nullable: true), + FindingJson = table.Column(type: "NVARCHAR(MAX)", nullable: true), StartedUtc = table.Column(type: "datetimeoffset", nullable: false), CompletedUtc = table.Column(type: "datetimeoffset", nullable: false), RowKey = table.Column(type: "nvarchar(max)", nullable: false), diff --git a/src/Infrastructure.SqlServer/Migrations/AgentFrameworkContextModelSnapshot.cs b/src/Infrastructure.SqlServer/Migrations/AgentFrameworkContextModelSnapshot.cs index d092a68..64a80e9 100644 --- a/src/Infrastructure.SqlServer/Migrations/AgentFrameworkContextModelSnapshot.cs +++ b/src/Infrastructure.SqlServer/Migrations/AgentFrameworkContextModelSnapshot.cs @@ -515,6 +515,12 @@ protected override void BuildModel(ModelBuilder modelBuilder) .IsRequired() .HasColumnType("NVARCHAR(MAX)"); + b.Property("EvidenceJson") + .HasColumnType("NVARCHAR(MAX)"); + + b.Property("FindingJson") + .HasColumnType("NVARCHAR(MAX)"); + b.Property("ModifiedBy") .HasColumnType("uniqueidentifier"); diff --git a/src/Infrastructure.SqlServer/Persistence/Configurations/PlaybookExecutionsConfig.cs b/src/Infrastructure.SqlServer/Persistence/Configurations/PlaybookExecutionsConfig.cs index fd26220..efd75dc 100644 --- a/src/Infrastructure.SqlServer/Persistence/Configurations/PlaybookExecutionsConfig.cs +++ b/src/Infrastructure.SqlServer/Persistence/Configurations/PlaybookExecutionsConfig.cs @@ -50,6 +50,12 @@ public void Configure(EntityTypeBuilder builder) .HasColumnType(ColumnTypes.NvarcharMax) .IsRequired(); + builder.Property(x => x.EvidenceJson) + .HasColumnType(ColumnTypes.NvarcharMax); + + builder.Property(x => x.FindingJson) + .HasColumnType(ColumnTypes.NvarcharMax); + builder.Property(x => x.OwnerId) .HasColumnType(ColumnTypes.Uniqueidentifier) .IsRequired(); diff --git a/src/Presentation.Web/Features/Playbooks/Components/PlaybookReplayControls.razor b/src/Presentation.Web/Features/Playbooks/Components/PlaybookReplayControls.razor new file mode 100644 index 0000000..0b1dd2c --- /dev/null +++ b/src/Presentation.Web/Features/Playbooks/Components/PlaybookReplayControls.razor @@ -0,0 +1,63 @@ +@using Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Models + + + + Repeatability + + Immediately re-execute this Playbook's latest completed run (@Result.ExecutionId) + using one of the three governed repeatability modes below, without retyping input. + + + + Rerun + + + Recall + + + Replay + + + + + Rerun: run Collect, Evaluate, and Record again from scratch, + using the same input as the latest run. Use this to see whether the result + changes against current live data. + + + Recall: re-render the already-recorded finding with no new + Collect or Evaluate work at all. Use this to view exactly what happened last + time, at zero cost, even if live data has since changed. + + + Replay: reuse the recorded evidence (skip Collect) and re-run + Evaluate and Record against it. Use this to verify the Evaluate/Record logic + reproduces the same governed result from the same evidence. + + + + + +@code { + /// + /// Lets the user immediately re-execute the latest completed Playbook execution using any of + /// the three repeatability modes the package's own PlaybookReplayMode enum exposes: + /// Rerun (fresh Collect/Evaluate/Record against the same input), Recall (re-render the + /// recorded finding with no new work), and Replay (reuse recorded evidence, re-run + /// Evaluate/Record). Shown only once a exists, since all three modes + /// operate against a prior execution's recorded input/evidence/finding. + /// + [Parameter, EditorRequired] public PlaybookExecutionResultModel Result { get; set; } = null!; + + [Parameter] public bool IsBusy { get; set; } + + [Parameter] public EventCallback OnReplay { get; set; } + + private Task TriggerAsync(PlaybookReplayMode mode) => OnReplay.InvokeAsync(mode); +} diff --git a/src/Presentation.Web/Features/Playbooks/Components/PlaybookResultPanel.razor b/src/Presentation.Web/Features/Playbooks/Components/PlaybookResultPanel.razor index 072bfa5..9a92b82 100644 --- a/src/Presentation.Web/Features/Playbooks/Components/PlaybookResultPanel.razor +++ b/src/Presentation.Web/Features/Playbooks/Components/PlaybookResultPanel.razor @@ -1,6 +1,7 @@ @using Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Models + @@ -8,7 +9,8 @@ @code { /// - /// Lays out one Playbook page's post-run section in the order the user requested: the 3 + /// Lays out one Playbook page's post-run section in the order the user requested: the + /// repeatability controls (re-execute the latest run via Recall/Replay), then the 3 /// Collect/Evaluate/Record toggle buttons, then the playbook summary (including the final /// Record-stage result), then the detail for whichever step is selected. Shared by every /// example playbook page so the layout never drifts between them. @@ -21,5 +23,17 @@ /// [Parameter] public PlaybookCatalogModel? Catalog { get; set; } + /// + /// True while a run or replay request is in flight, so the repeatability controls can disable + /// themselves to prevent duplicate submissions. + /// + [Parameter] public bool IsBusy { get; set; } + + /// + /// Raised when the user picks Recall or Replay from ; the + /// hosting page re-executes against with the chosen mode. + /// + [Parameter] public EventCallback OnReplay { get; set; } + private PlaybookCerStep SelectedStep { get; set; } = PlaybookCerStep.Collect; } diff --git a/src/Presentation.Web/Features/Playbooks/Components/PlaybookStepToggle.razor b/src/Presentation.Web/Features/Playbooks/Components/PlaybookStepToggle.razor index b3f08e4..6b7539b 100644 --- a/src/Presentation.Web/Features/Playbooks/Components/PlaybookStepToggle.razor +++ b/src/Presentation.Web/Features/Playbooks/Components/PlaybookStepToggle.razor @@ -1,16 +1,25 @@ @using Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Models - - - Collect - - - Evaluate - - - Record - - +
+ @foreach (var step in Steps) + { + var isSelected = step == SelectedStep; + + } +
@code { /// @@ -20,8 +29,16 @@ [Parameter] public PlaybookCerStep SelectedStep { get; set; } = PlaybookCerStep.Collect; [Parameter] public EventCallback SelectedStepChanged { get; set; } - private ButtonAppearance AppearanceFor(PlaybookCerStep step) => - step == SelectedStep ? ButtonAppearance.Primary : ButtonAppearance.Outline; + // Rendered as a tab strip (flat buttons under an accent-colored underline indicator) rather + // than filled/outlined buttons, so it reads as "select a view" rather than "run a command" - + // distinguishing it at a glance from the repeatability controls' Rerun/Recall/Replay buttons, + // which do execute a command. + private static readonly PlaybookCerStep[] Steps = + [ + PlaybookCerStep.Collect, + PlaybookCerStep.Evaluate, + PlaybookCerStep.Record + ]; private async Task SelectAsync(PlaybookCerStep step) { diff --git a/src/Presentation.Web/Features/Playbooks/Components/PlaybookStepToggle.razor.css b/src/Presentation.Web/Features/Playbooks/Components/PlaybookStepToggle.razor.css new file mode 100644 index 0000000..df90b8b --- /dev/null +++ b/src/Presentation.Web/Features/Playbooks/Components/PlaybookStepToggle.razor.css @@ -0,0 +1,18 @@ +.step-toggle { + display: flex; + gap: 4px; + border-bottom: 1px solid var(--neutral-stroke-divider-rest); +} + +.step-toggle-tab { + /* The bottom border carries the selection indicator; the 2px thickness matches the + indicator so the selected tab's accent-colored border sits flush over the shared + divider line instead of shifting layout when selection changes. */ + border-bottom: 2px solid transparent; + margin-bottom: -1px; + padding-bottom: 2px; +} + +.step-toggle-tab-selected { + border-bottom-color: var(--accent-fill-rest); +} diff --git a/src/Presentation.Web/Features/Playbooks/EssayPlaybookPage.razor b/src/Presentation.Web/Features/Playbooks/EssayPlaybookPage.razor index ece11e1..7750697 100644 --- a/src/Presentation.Web/Features/Playbooks/EssayPlaybookPage.razor +++ b/src/Presentation.Web/Features/Playbooks/EssayPlaybookPage.razor @@ -34,7 +34,7 @@ @if (result is not null) { - + } @@ -44,6 +44,7 @@ private PlaybookCatalogModel? catalog; private PlaybookExecutionResultModel? result; private string? errorMessage; + private bool isBusy; protected override async Task OnInitializedAsync() { @@ -56,6 +57,7 @@ private async Task RunAsync(string essayText) { errorMessage = null; + isBusy = true; try { result = await PlaybookService.RunEssayAsync(essayText); @@ -64,5 +66,32 @@ { errorMessage = $"The Essay Playbook could not be run: {ex.Message}"; } + finally + { + isBusy = false; + } + } + + private async Task ReplayAsync(PlaybookReplayMode mode) + { + if (result is null) + { + return; + } + + errorMessage = null; + isBusy = true; + try + { + result = await PlaybookService.RunEssayAsync(result.CollectInput, mode, result.ExecutionId); + } + catch (Exception ex) + { + errorMessage = $"The Essay Playbook could not {mode.ToVerbPhrase()}: {ex.Message}"; + } + finally + { + isBusy = false; + } } } diff --git a/src/Presentation.Web/Features/Playbooks/Models/PlaybookExecutionResultModel.cs b/src/Presentation.Web/Features/Playbooks/Models/PlaybookExecutionResultModel.cs index 5a6f582..049e024 100644 --- a/src/Presentation.Web/Features/Playbooks/Models/PlaybookExecutionResultModel.cs +++ b/src/Presentation.Web/Features/Playbooks/Models/PlaybookExecutionResultModel.cs @@ -8,6 +8,12 @@ namespace Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Models; /// public class PlaybookExecutionResultModel { + /// + /// The persisted execution's own id - supplied as SourceExecutionId on a later Recall + /// or Replay request against this execution. + /// + public Guid ExecutionId { get; set; } + public string PlaybookKey { get; set; } = string.Empty; public string Version { get; set; } = string.Empty; public DateTimeOffset StartedUtc { get; set; } @@ -25,6 +31,7 @@ public static PlaybookExecutionResultModel Create(PlaybookExecutionResultDto dto return new PlaybookExecutionResultModel { + ExecutionId = dto.ExecutionId, PlaybookKey = dto.PlaybookKey, Version = dto.Version, StartedUtc = dto.StartedUtc, diff --git a/src/Presentation.Web/Features/Playbooks/Models/PlaybookReplayMode.cs b/src/Presentation.Web/Features/Playbooks/Models/PlaybookReplayMode.cs new file mode 100644 index 0000000..fd8bd10 --- /dev/null +++ b/src/Presentation.Web/Features/Playbooks/Models/PlaybookReplayMode.cs @@ -0,0 +1,50 @@ +namespace Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Models; + +/// +/// UI-facing mirror of Goodtocode.Agents.Playbook.Execution.PlaybookReplayMode: which of +/// the three repeatability behaviors a re-execution should perform. Defined here - rather than +/// reusing the NSwag-generated client enum directly - because NJsonSchema cannot infer member +/// names from the backend's numeric enum values, so the generated type's members are unusable +/// placeholders (_0/_1/_2); this enum is cast to/from that generated type by +/// ordinal value in , mirroring the same pattern already +/// used for . +/// +public enum PlaybookReplayMode +{ + /// + /// Fresh execution: run Collect, Evaluate, and Record. + /// + Rerun = 0, + + /// + /// Rehydrate a prior execution's evidence and finding without running Collect or Evaluate; + /// Record still runs so materialization can be re-rendered from the recalled finding. + /// + Recall = 1, + + /// + /// Reuse prior evidence (skip Collect) and re-run Evaluate and Record against it, to verify + /// exact reproduction of a governed result. + /// + Replay = 2 +} + +/// +/// Display helpers for , kept separate from the enum so error +/// messages read as natural phrases (e.g. "could not be rerun") instead of mechanically +/// appending "-ed" to the member name. +/// +public static class PlaybookReplayModeExtensions +{ + /// + /// Returns the verb phrase describing this mode's action, for use after "could not" in an + /// error message (e.g. "could not {ToVerbPhrase()}"). + /// + public static string ToVerbPhrase(this PlaybookReplayMode mode) => mode switch + { + PlaybookReplayMode.Rerun => "be rerun", + PlaybookReplayMode.Recall => "be recalled", + PlaybookReplayMode.Replay => "be replayed", + _ => "be executed" + }; +} diff --git a/src/Presentation.Web/Features/Playbooks/Services/PlaybookService.cs b/src/Presentation.Web/Features/Playbooks/Services/PlaybookService.cs index 1187fa2..32ca56e 100644 --- a/src/Presentation.Web/Features/Playbooks/Services/PlaybookService.cs +++ b/src/Presentation.Web/Features/Playbooks/Services/PlaybookService.cs @@ -1,5 +1,7 @@ using Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Models; using Goodtocode.AgentFramework.Presentation.Web.Infrastructure.Http; +using ClientReplayMode = Goodtocode.AgentFramework.Api.Clients.PlaybookReplayMode; +using UiReplayMode = Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Models.PlaybookReplayMode; namespace Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Services; @@ -11,9 +13,21 @@ namespace Goodtocode.AgentFramework.Presentation.Web.Features.Playbooks.Services /// public interface IPlaybookService { - Task RunSqlStatisticsAsync(string databaseName); - Task RunTaxonomyAsync(string sourceText); - Task RunEssayAsync(string essayText); + /// + /// Runs the SQL Statistics playbook. selects the repeatability + /// behavior ( by default); for Recall/Replay, + /// is ignored server-side in favor of the resolved prior + /// execution's own Collect-stage input, and identifies + /// which prior execution to recall/replay (the user's latest, if omitted). + /// + Task RunSqlStatisticsAsync( + string databaseName, UiReplayMode replayMode = UiReplayMode.Rerun, Guid? sourceExecutionId = null); + + Task RunTaxonomyAsync( + string sourceText, UiReplayMode replayMode = UiReplayMode.Rerun, Guid? sourceExecutionId = null); + + Task RunEssayAsync( + string essayText, UiReplayMode replayMode = UiReplayMode.Rerun, Guid? sourceExecutionId = null); /// /// Fetches the persisted catalog definition (Name, Description, and the 3 CER steps' @@ -32,25 +46,43 @@ public class PlaybookService(BackendApiClient client) : ApiService, IPlaybookSer { private readonly BackendApiClient _apiClient = client; - public async Task RunSqlStatisticsAsync(string databaseName) + public async Task RunSqlStatisticsAsync( + string databaseName, UiReplayMode replayMode = UiReplayMode.Rerun, Guid? sourceExecutionId = null) { - var command = new RunSqlStatisticsPlaybookCommand { DatabaseName = databaseName }; + var command = new RunSqlStatisticsPlaybookCommand + { + DatabaseName = databaseName, + ReplayMode = (ClientReplayMode)(int)replayMode, + SourceExecutionId = sourceExecutionId?.ToString() + }; var response = await HandleApiException(() => _apiClient.RunMySqlStatisticsPlaybookAsync(command)); return PlaybookExecutionResultModel.Create(response); } - public async Task RunTaxonomyAsync(string sourceText) + public async Task RunTaxonomyAsync( + string sourceText, UiReplayMode replayMode = UiReplayMode.Rerun, Guid? sourceExecutionId = null) { - var command = new RunTaxonomyPlaybookCommand { Text = sourceText }; + var command = new RunTaxonomyPlaybookCommand + { + Text = sourceText, + ReplayMode = (ClientReplayMode)(int)replayMode, + SourceExecutionId = sourceExecutionId?.ToString() + }; var response = await HandleApiException(() => _apiClient.RunMyTaxonomyPlaybookAsync(command)); return PlaybookExecutionResultModel.Create(response); } - public async Task RunEssayAsync(string essayText) + public async Task RunEssayAsync( + string essayText, UiReplayMode replayMode = UiReplayMode.Rerun, Guid? sourceExecutionId = null) { - var command = new RunEssayPlaybookCommand { EssayText = essayText }; + var command = new RunEssayPlaybookCommand + { + EssayText = essayText, + ReplayMode = (ClientReplayMode)(int)replayMode, + SourceExecutionId = sourceExecutionId?.ToString() + }; var response = await HandleApiException(() => _apiClient.RunMyEssayPlaybookAsync(command)); return PlaybookExecutionResultModel.Create(response); diff --git a/src/Presentation.Web/Features/Playbooks/SqlStatisticsPlaybookPage.razor b/src/Presentation.Web/Features/Playbooks/SqlStatisticsPlaybookPage.razor index ef15978..942a16b 100644 --- a/src/Presentation.Web/Features/Playbooks/SqlStatisticsPlaybookPage.razor +++ b/src/Presentation.Web/Features/Playbooks/SqlStatisticsPlaybookPage.razor @@ -34,7 +34,7 @@ @if (result is not null) { - + } @@ -44,6 +44,7 @@ private PlaybookCatalogModel? catalog; private PlaybookExecutionResultModel? result; private string? errorMessage; + private bool isBusy; protected override async Task OnInitializedAsync() { @@ -56,6 +57,7 @@ private async Task RunAsync(string databaseName) { errorMessage = null; + isBusy = true; try { result = await PlaybookService.RunSqlStatisticsAsync(databaseName); @@ -64,5 +66,32 @@ { errorMessage = $"The SQL Statistics Playbook could not be run: {ex.Message}"; } + finally + { + isBusy = false; + } + } + + private async Task ReplayAsync(PlaybookReplayMode mode) + { + if (result is null) + { + return; + } + + errorMessage = null; + isBusy = true; + try + { + result = await PlaybookService.RunSqlStatisticsAsync(result.CollectInput, mode, result.ExecutionId); + } + catch (Exception ex) + { + errorMessage = $"The SQL Statistics Playbook could not {mode.ToVerbPhrase()}: {ex.Message}"; + } + finally + { + isBusy = false; + } } } diff --git a/src/Presentation.Web/Features/Playbooks/TaxonomyPlaybookPage.razor b/src/Presentation.Web/Features/Playbooks/TaxonomyPlaybookPage.razor index c6e85e8..0ed4d02 100644 --- a/src/Presentation.Web/Features/Playbooks/TaxonomyPlaybookPage.razor +++ b/src/Presentation.Web/Features/Playbooks/TaxonomyPlaybookPage.razor @@ -34,7 +34,7 @@ @if (result is not null) { - + } @@ -44,6 +44,7 @@ private PlaybookCatalogModel? catalog; private PlaybookExecutionResultModel? result; private string? errorMessage; + private bool isBusy; protected override async Task OnInitializedAsync() { @@ -56,6 +57,7 @@ private async Task RunAsync(string sourceText) { errorMessage = null; + isBusy = true; try { result = await PlaybookService.RunTaxonomyAsync(sourceText); @@ -64,5 +66,32 @@ { errorMessage = $"The Taxonomy Playbook could not be run: {ex.Message}"; } + finally + { + isBusy = false; + } + } + + private async Task ReplayAsync(PlaybookReplayMode mode) + { + if (result is null) + { + return; + } + + errorMessage = null; + isBusy = true; + try + { + result = await PlaybookService.RunTaxonomyAsync(result.CollectInput, mode, result.ExecutionId); + } + catch (Exception ex) + { + errorMessage = $"The Taxonomy Playbook could not {mode.ToVerbPhrase()}: {ex.Message}"; + } + finally + { + isBusy = false; + } } } diff --git a/src/Presentation.Web/Infrastructure/Clients/BackendApiClient.g.cs b/src/Presentation.Web/Infrastructure/Clients/BackendApiClient.g.cs index 97b7704..fb5d46c 100644 --- a/src/Presentation.Web/Infrastructure/Clients/BackendApiClient.g.cs +++ b/src/Presentation.Web/Infrastructure/Clients/BackendApiClient.g.cs @@ -2796,6 +2796,9 @@ public partial class PlaybookDto public partial class PlaybookExecutionResultDto { + [System.Text.Json.Serialization.JsonPropertyName("executionId")] + public System.Guid ExecutionId { get; set; } + [System.Text.Json.Serialization.JsonPropertyName("playbookKey")] public string PlaybookKey { get; set; } @@ -2826,6 +2829,24 @@ public partial class PlaybookExecutionResultDto [System.Text.Json.Serialization.JsonPropertyName("recordSummary")] public string RecordSummary { get; set; } + [System.Text.Json.Serialization.JsonPropertyName("evidenceJson")] + public string EvidenceJson { get; set; } + + [System.Text.Json.Serialization.JsonPropertyName("findingJson")] + public string FindingJson { get; set; } + + } + + [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.6.3.0 (NJsonSchema v11.5.2.0 (Newtonsoft.Json v13.0.0.0))")] + public enum PlaybookReplayMode + { + + _0 = 0, + + _1 = 1, + + _2 = 2, + } [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.6.3.0 (NJsonSchema v11.5.2.0 (Newtonsoft.Json v13.0.0.0))")] @@ -2886,27 +2907,54 @@ public enum PlaybookStepType public partial class RunEssayPlaybookCommand { + [System.Text.Json.Serialization.JsonPropertyName("userContext")] + public IUserContext UserContext { get; set; } + [System.Text.Json.Serialization.JsonPropertyName("essayText")] public string EssayText { get; set; } + [System.Text.Json.Serialization.JsonPropertyName("replayMode")] + public PlaybookReplayMode ReplayMode { get; set; } + + [System.Text.Json.Serialization.JsonPropertyName("sourceExecutionId")] + public string SourceExecutionId { get; set; } + } [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.6.3.0 (NJsonSchema v11.5.2.0 (Newtonsoft.Json v13.0.0.0))")] public partial class RunSqlStatisticsPlaybookCommand { + [System.Text.Json.Serialization.JsonPropertyName("userContext")] + public IUserContext UserContext { get; set; } + [System.Text.Json.Serialization.JsonPropertyName("databaseName")] public string DatabaseName { get; set; } + [System.Text.Json.Serialization.JsonPropertyName("replayMode")] + public PlaybookReplayMode ReplayMode { get; set; } + + [System.Text.Json.Serialization.JsonPropertyName("sourceExecutionId")] + public string SourceExecutionId { get; set; } + } [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.6.3.0 (NJsonSchema v11.5.2.0 (Newtonsoft.Json v13.0.0.0))")] public partial class RunTaxonomyPlaybookCommand { + [System.Text.Json.Serialization.JsonPropertyName("userContext")] + public IUserContext UserContext { get; set; } + [System.Text.Json.Serialization.JsonPropertyName("text")] public string Text { get; set; } + [System.Text.Json.Serialization.JsonPropertyName("replayMode")] + public PlaybookReplayMode ReplayMode { get; set; } + + [System.Text.Json.Serialization.JsonPropertyName("sourceExecutionId")] + public string SourceExecutionId { get; set; } + } [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.6.3.0 (NJsonSchema v11.5.2.0 (Newtonsoft.Json v13.0.0.0))")] diff --git a/src/Tests.Integration/Playbooks/PlaybookReplayTests.cs b/src/Tests.Integration/Playbooks/PlaybookReplayTests.cs new file mode 100644 index 0000000..a8115d9 --- /dev/null +++ b/src/Tests.Integration/Playbooks/PlaybookReplayTests.cs @@ -0,0 +1,182 @@ +using Goodtocode.AgentFramework.Core.Application.Playbooks; +using Goodtocode.AgentFramework.Core.Application.Playbooks.SqlStatistics; +using Goodtocode.AgentFramework.Core.Application.Playbooks.Taxonomy; +using Goodtocode.AgentFramework.Core.Domain.Playbooks; +using Goodtocode.Agents.Playbook.Execution; + +namespace Goodtocode.AgentFramework.Tests.Integration.Playbooks; + +/// +/// Verifies the repeatability governance pillar end to end through the real Run*PlaybookCommand +/// pipeline: a Rerun persists the typed evidence/finding needed for later Recall/Replay requests, +/// Recall re-renders the prior finding without any new Collect/Evaluate work, and Replay reuses +/// the prior evidence while re-running Evaluate and Record. Covers both the deterministic SQL +/// Statistics playbook (the package's own PlaybookExecutor replay overload) and the +/// agentic Taxonomy playbook (the repo-owned ), +/// since each uses a different replay mechanism. +/// +[TestClass] +public sealed class PlaybookReplayTests : TestBase +{ + private async Task SeedSqlStatisticsPlaybookAsync() + { + await Sender.Send(new CreatePlaybookCommand + { + Key = "sql-statistics", + Name = "SQL Statistics Classification", + Description = "Deterministic CER workflow for test purposes.", + WorkflowType = "Deterministic", + Version = "v1", + Collect = new PlaybookStepInput + { + Name = "Collect database statistics", + Description = "Runs a deterministic T-SQL statistics query.", + ActionFormat = PlaybookActionFormat.SqlQuery, + ActionDefinition = "SELECT 1;" + }, + Evaluate = new PlaybookStepInput + { + Name = "Classify database size", + Description = "Classifies the collected SizeGb against named size bands.", + ActionFormat = PlaybookActionFormat.Rubric, + ActionDefinition = "Small: 0-20 GB; Medium: 20-100 GB; Large: 100+ GB." + }, + Record = new PlaybookStepInput + { + Name = "Record classification", + Description = "Projects the classification into a display summary.", + ActionFormat = PlaybookActionFormat.Template, + ActionDefinition = "{DatabaseName} is {Classification} at {SizeGb} GB." + } + }, CancellationToken.None); + } + + private async Task SeedTaxonomyPlaybookAsync() + { + await Sender.Send(new CreatePlaybookCommand + { + Key = "taxonomy", + Name = "Taxonomy Extraction and Classification", + Description = "Agentic CER workflow for test purposes.", + WorkflowType = "Agentic", + Version = "v1", + Collect = new PlaybookStepInput + { + Name = "Extract taxonomy terms", + Description = "Asks the model to extract terms.", + ActionFormat = PlaybookActionFormat.Prompt, + ActionDefinition = "Extract terms from {input}." + }, + Evaluate = new PlaybookStepInput + { + Name = "Classify into taxonomy category", + Description = "Asks the model to classify the terms.", + ActionFormat = PlaybookActionFormat.Prompt, + ActionDefinition = "Classify {terms}." + }, + Record = new PlaybookStepInput + { + Name = "Record classification", + Description = "Projects the finding into a summary.", + ActionFormat = PlaybookActionFormat.Template, + ActionDefinition = "Classified as {Category}." + } + }, CancellationToken.None); + } + + [TestMethod] + public async Task RerunPersistsEvidenceAndFindingForLaterReplay() + { + await SeedSqlStatisticsPlaybookAsync(); + sqlDatabaseStatisticsProvider.SizeGb = 50; + + var dto = await Sender.Send(new RunSqlStatisticsPlaybookCommand { DatabaseName = "Contoso" }, CancellationToken.None); + + dto.ReplayMode.ShouldBe(nameof(PlaybookReplayMode.Rerun)); + dto.ExecutionId.ShouldNotBeEmpty(); + string.IsNullOrWhiteSpace(dto.EvidenceJson).ShouldBeFalse(); + string.IsNullOrWhiteSpace(dto.FindingJson).ShouldBeFalse(); + + var persisted = await context.PlaybookExecutions.FindAsync(dto.ExecutionId); + persisted.ShouldNotBeNull(); + string.IsNullOrWhiteSpace(persisted!.EvidenceJson).ShouldBeFalse(); + string.IsNullOrWhiteSpace(persisted.FindingJson).ShouldBeFalse(); + } + + [TestMethod] + public async Task RecallReusesPriorEvidenceAndFindingWithoutRecollecting() + { + await SeedSqlStatisticsPlaybookAsync(); + sqlDatabaseStatisticsProvider.SizeGb = 50; + + var original = await Sender.Send(new RunSqlStatisticsPlaybookCommand { DatabaseName = "Contoso" }, CancellationToken.None); + + // Change the live database size; Recall must still report the original Medium finding + // since it never re-collects or re-evaluates. + sqlDatabaseStatisticsProvider.SizeGb = 500; + + var recalled = await Sender.Send(new RunSqlStatisticsPlaybookCommand + { + DatabaseName = "ignored-for-recall", + ReplayMode = PlaybookReplayMode.Recall, + SourceExecutionId = original.ExecutionId.ToString() + }, CancellationToken.None); + + recalled.ReplayMode.ShouldBe(nameof(PlaybookReplayMode.Recall)); + recalled.SourceExecutionId.ShouldBe(original.ExecutionId.ToString()); + recalled.CollectInput.ShouldBe("Contoso"); + recalled.EvaluateSummary.ShouldBe(original.EvaluateSummary); + } + + [TestMethod] + public async Task ReplayReusesPriorEvidenceButReEvaluates() + { + await SeedSqlStatisticsPlaybookAsync(); + sqlDatabaseStatisticsProvider.SizeGb = 50; + + var original = await Sender.Send(new RunSqlStatisticsPlaybookCommand { DatabaseName = "Contoso" }, CancellationToken.None); + + // Change the live database size; Replay must still classify against the original 50 GB + // evidence (Medium) since it skips Collect, even though the live size is now Large. + sqlDatabaseStatisticsProvider.SizeGb = 500; + + var replayed = await Sender.Send(new RunSqlStatisticsPlaybookCommand + { + DatabaseName = "ignored-for-replay", + ReplayMode = PlaybookReplayMode.Replay, + SourceExecutionId = original.ExecutionId.ToString() + }, CancellationToken.None); + + replayed.ReplayMode.ShouldBe(nameof(PlaybookReplayMode.Replay)); + replayed.SourceExecutionId.ShouldBe(original.ExecutionId.ToString()); + replayed.CollectInput.ShouldBe("Contoso"); + replayed.EvaluateSummary.ShouldBe(original.EvaluateSummary); + } + + [TestMethod] + public async Task TaxonomyRecallSkipsCollectAndEvaluateAgentCalls() + { + await SeedTaxonomyPlaybookAsync(); + + agent.QueuedResponseTexts.Enqueue("""{"terms": ["budget", "forecast"]}"""); + agent.QueuedResponseTexts.Enqueue("""{"category": "Finance", "confidence": 0.9, "rationale": "Discusses budgeting."}"""); + agent.QueuedResponseTexts.Enqueue("""{"summary": "Classified as Finance with high confidence."}"""); + + var original = await Sender.Send(new RunTaxonomyPlaybookCommand { Text = "Quarterly budget forecast" }, CancellationToken.None); + + // Only the Record stage's agent call is queued; Recall must not invoke Collect or + // Evaluate, so no extra queued responses are needed for those stages. + agent.QueuedResponseTexts.Enqueue("""{"summary": "Recalled: Classified as Finance with high confidence."}"""); + + var recalled = await Sender.Send(new RunTaxonomyPlaybookCommand + { + Text = "ignored-for-recall", + ReplayMode = PlaybookReplayMode.Recall, + SourceExecutionId = original.ExecutionId.ToString() + }, CancellationToken.None); + + recalled.ReplayMode.ShouldBe(nameof(PlaybookReplayMode.Recall)); + recalled.EvaluateSummary.ShouldBe(original.EvaluateSummary); + agent.QueuedResponseTexts.Count.ShouldBe(0); + } +} From 34088c89b942c44d4a0e3ccc95e05bd984d1af5a Mon Sep 17 00:00:00 2001 From: "Robert J. Good" Date: Sat, 3 Oct 2026 12:31:36 -0700 Subject: [PATCH 2/2] height on textarea. Width broken --- .github/copilot-instructions.md | 2 +- .../Playbooks/Components/PlaybookRunForm.razor | 8 +++++--- src/Presentation.Web/Presentation.Web.csproj | 11 +++++++---- 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index f1d6e6c..c7af2b1 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -59,5 +59,5 @@ ## SEO and Component Usage - Prefer Blazor components over raw HTML to enhance maintainability and reusability. - Implement SEO and GEO optimization support by using semantic headings (H1, H2) while minimizing direct HTML usage. -- Prioritize root-cause fixes over brute-force styling overrides and adhere to global Blazor/Fluent UI conventions. +- Prioritize root-cause fixes over brute-force styling overrides and adhere to global Blazor/Fluent UI conventions. Verify behavior against current component rendering. - Use Blazor/Fluent component properties over CSS for layout/styling because CSS is harder and more brittle to maintain. \ No newline at end of file diff --git a/src/Presentation.Web/Features/Playbooks/Components/PlaybookRunForm.razor b/src/Presentation.Web/Features/Playbooks/Components/PlaybookRunForm.razor index b874aab..0097521 100644 --- a/src/Presentation.Web/Features/Playbooks/Components/PlaybookRunForm.razor +++ b/src/Presentation.Web/Features/Playbooks/Components/PlaybookRunForm.razor @@ -4,8 +4,9 @@ @if (Multiline) { - OnSubmit { get; set; } private sealed class PlaybookInputModel diff --git a/src/Presentation.Web/Presentation.Web.csproj b/src/Presentation.Web/Presentation.Web.csproj index f719c5d..15c42bb 100644 --- a/src/Presentation.Web/Presentation.Web.csproj +++ b/src/Presentation.Web/Presentation.Web.csproj @@ -10,6 +10,13 @@ c2325466-9b0d-494a-984a-11ad18457d3f + + + + + + + @@ -26,8 +33,4 @@ - - - -