Spike: shrink audit full-text search index

[ramonsmits]

Spike: shrink the audit MessagesView full-text index

Draft/spike — measuring on a real audit database before deciding whether to ship.

Problem

The Query search field of both MessagesViewIndex and MessagesViewIndexWithFullTextSearch indexes every metadata value and every header value. A large share of those are timestamps, durations, sizes, booleans and identifiers. Nobody searches for a timestamp or a ProcessingTime, and because each of those values is unique per message they never share dictionary terms across documents, so they inflate the term dictionary far more than the shared tokens (endpoint names, message types, host names) do.

Changes

1. Exclude temporal, numeric and boolean values from Query — metadata TimeSent, CriticalTime, ProcessingTime, DeliveryTime, ContentLength, BodyUrl, IsSystemMessage, IsRetried, BodyNotStored, OriginatesFromSaga and headers TimeSent, ProcessingStarted, ProcessingEnded, DeliverAt, Timeout.Expire, Retries.Timestamp, ExceptionInfo.TimeOfFailure, TimeOfFailure, NonDurableMessage, TimeToBeReceived. Header values are indexed individually rather than as one joined string; null/empty values dropped; Distinct() removes metadata/header duplicates.
2. Match identifiers exactly instead of via full-text — /messages/search/{keyword} now ORs MessageId = keyword and ConversationId = keyword with the full-text search, and ids (plus their NServiceBus.MessageId/ConversationId/CorrelationId/RelatedTo header copies) are dropped from Query. The OR group is built with the DocumentQuery API because the LINQ provider does not parenthesize Search(..., SearchOptions.Or), which would bind the endpoint/time-range filters to the last OR term only (verified by inspecting the generated RQL).
3. Analyzer referenced by its short name StandardAnalyzer, as the old comment suggested doing whenever the index has to rebuild anyway.
4. New acceptance test When_processed_message_searched_by_conversationid.

Estimated effect

Representative OrderPlaced audit message (21 headers, no body), StandardAnalyzer-style tokenization:

	before	after
tokens / message	83	35
distinct terms / message	42	22
per-message-unique terms	18	0

Real numbers to be measured on an existing audit database (side-by-side trial indexes filtered on ProcessedAt, old vs new Query projection over the same document set).

Behaviour changes / open questions

• Any deployment picking this up rebuilds both MessagesView indexes on startup (unavoidable for any map change).
• CorrelationId and RelatedTo are no longer searchable (no exact-match field). Correlation id usually equals message id. Acceptable?
• The primary instance (ErrorMessagesDataStore) and the EF/SQL persisters still index ids and timestamps as free text — same pattern, follow-up candidate.

Verification

• ServiceControl.Audit.Persistence.Tests.RavenDB 41/41
• ServiceControl.Audit.AcceptanceTests.RavenDB and ServiceControl.Audit.AcceptanceTests search tests 5/5 each