Skip to content

Implement external integration dispatch requests on EF Core - #5765

Merged
abparticular merged 8 commits into
masterfrom
ef-external-integration
Aug 28, 2026
Merged

Implement external integration dispatch requests on EF Core#5765
abparticular merged 8 commits into
masterfrom
ef-external-integration

Conversation

@abparticular

Copy link
Copy Markdown
Contributor

Implements IExternalIntegrationRequestsDataStore for the SQL Server and PostgreSQL persisters, replacing the no-op stub. This is ServiceControl's outbox for publishing NServiceBus events to external subscribers on domain events (message failed, custom check failed, etc.): StoreDispatchRequest persists a batch, a background loop wakes up, hands dispatched contexts to the subscribed callback, and deletes the rows once the callback succeeds.

Storage. New ExternalIntegrationDispatchRequests table (auto-increment Id for FIFO ordering). DispatchContext's concrete type is only known at runtime and lives in ServiceControl.csproj, which the EF Core project doesn't reference, so there's no way to type this at compile time - a new DispatchContextSerializer stores it as a versionless assembly-qualified type name alongside a System.Text.Json blob, and resolves it back via Type.GetType(...) on read. New migrations for both SQL Server and PostgreSQL.

Notification. SQL has no equivalent of RavenDB's server-push Changes API, so the dispatch loop is woken by an in-process signal (SemaphoreSlim) on every StoreDispatchRequest, with a 30s PeriodicTimer as a safety net in case a signal is ever missed. This assumes single-process-per-role deployment, same as every other EF Core data store.

Batch dispatch. RavenDB does select → callback → delete inside one document session. There's no relational equivalent, so this is three separate phases, each its own DbContext scope (same shape as RetentionSweeper.Sweep), so no DB transaction is held open across the network call to publish on the bus. Delete only happens if the callback returns without throwing — matching RavenDB's actual behavior (verified by reading TryDispatchEventBatch: it has no try/catch around the callback, so a throw skips the deletes too). This is at-least-once with redelivery on callback failure, not at-most-once.

Settings. Added ExternalIntegrationsDispatchingBatchSize (default 100) to EFPersisterSettings, read from the same config key the RavenDB backend already exposes, so switching backends doesn't silently change behavior.

Testing. New shared cross-backend tests (ExternalIntegrationRequestsDataStoreTests, run against RavenDB/SqlServer/PostgreSql) covering store→dispatch→delete, callback-throws-then-retries, batching over ExternalIntegrationsDispatchingBatchSize, and concurrent stores. EF-only tests (ExternalIntegrationRequestsDataStoreEFTests, DispatchContextSerializerTests) exercise the new DispatchNow test hook directly rather than polling on timers, plus a serializer round-trip test.

@abparticular
abparticular force-pushed the ef-external-integration branch 4 times, most recently from aa1c321 to af67208 Compare August 20, 2026 05:16
@abparticular
abparticular marked this pull request as ready for review August 24, 2026 02:31
@warwickschroeder

Copy link
Copy Markdown
Contributor

Do these need to be removed?

<!-- External integration event dispatch is not implemented by EF persistence. -->
<Compile Remove="..\ServiceControl.AcceptanceTests\Monitoring\ExternalIntegration\When_a_custom_check_fails.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Monitoring\ExternalIntegration\When_a_custom_check_succeeds.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Monitoring\ExternalIntegration\When_heartbeat_is_restored.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Monitoring\ExternalIntegration\When_heartbeat_loss_is_detected.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_failed_edit_is_resolved_by_retry.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_failed_message_is_archived.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_failed_message_is_resolved_by_retry.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_failed_message_is_resolved_manually.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_failed_message_is_unarchived.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_failed_msg_is_resolved_by_edit.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_group_is_archived.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_message_has_failed_detected.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_a_reedit_solves_a_failed_msg.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\ExternalIntegration\When_encountered_an_error.cs" />
<Compile Remove="..\ServiceControl.AcceptanceTests\Recoverability\When_edited_message_fails_to_process.cs" />

@abparticular
abparticular force-pushed the ef-external-integration branch from acdce9f to fbd11fa Compare August 26, 2026 05:21
@abparticular
abparticular force-pushed the ef-external-integration branch from fbd11fa to 9adc7ec Compare August 27, 2026 03:38
@abparticular
abparticular force-pushed the ef-external-integration branch from 96e336c to 9430603 Compare August 28, 2026 01:55
…hen creating DateTime values as required by PostgreSql
@abparticular
abparticular force-pushed the ef-external-integration branch from 22ae546 to a1257aa Compare August 28, 2026 06:26
@abparticular
abparticular merged commit adbee66 into master Aug 28, 2026
133 of 138 checks passed
@abparticular
abparticular deleted the ef-external-integration branch August 28, 2026 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants