Skip to content

Benchmarks: synthetic v2 harness plus Dapper.AOT variant - #140

Merged
electricessence merged 24 commits into
masterfrom
bench/synthetic-v2
Nov 15, 2026
Merged

electricessence merged 24 commits into
masterfrom
bench/synthetic-v2

Conversation

@electricessence

Copy link
Copy Markdown
Collaborator

What this adds

The benchmark project gets a synthetic, provider-independent harness and a real-provider cross-check, so mapping paths can be compared on equal terms. Every file is under Benchmarks/: no change to Source/, tests, docs or build props.

  • Synthetic provider and catalog (SyntheticProvider.cs and friends): a 20-column typed catalog (widths 1 to 20, rows 1 to 10,000) with a switchable per-row cost, a NextResult cost, an unsealed twin of the reader, and a buffer-boundary refill with consumer pace.
  • Variants measured against each other: the compiled map plan, the shipped Map(Func<IDataRecord,T>) selector, typed-getter ceilings, Select-style two-step projection, expression-tree and emitted-method forms, plain Dapper, and a Dapper.AOT variant (Dapper.AOT 1.1.0, source-generated interceptors, scoped to its own class so the plain Dapper variant stays plain).
  • Metrics: BenchmarkDotNet mean and allocations, plus a reader-open recorder summarised over the measured iterations, with a check script that fails a result whose reader-open time exceeds its mean.
  • Equality checks: --verify proves every variant produces the same rows as the oracle at every width and row count; --verify-rp does the same over SQLite and SQL Server LocalDB; self-tests cover the allocation floor (typed getters at 0 B per row), the open-time recorder, the pace and refill logic.
  • Real-provider benchmarks over SQLite (file database) and SQL Server LocalDB (a dedicated database per run).
  • Console probes (not BenchmarkDotNet): --in6 compares sync reads, ReadAsync and truly async getters, and --s4 reads very large values over a simulated link, one cell per process, with the SqlClient opt-in switches selectable.

Nothing here ships in a package.

Checks

  • Release build of both target frameworks with 0 warnings.
  • --verify, --selftest-alloc, --selftest-s4, --selftest-in6 and the format check pass; the full test gate and a Visual Studio MSBuild of the solution are green.
  • Leak scan clean.

🤖 Generated with Claude Code

electricessence and others added 24 commits November 14, 2026 14:49
…b, V1, V2 to the benchmark project

A deterministic 20-column provider of any width and row count, with an optional cost per row and per NextResult and extra empty result sets,
and a recorder that times the reader from ExecuteReader to its close and the NextResult drain apart. The four variants are checked for equal
final objects by --verify; --selftest-open-time and --selftest-alloc check the timing and that the typed getters allocate nothing. No benchmark is run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… the synthetic suite

Variants over the synthetic provider: V3 (Map then Select, inline), V3s (streamed),
V4a and V4b (reused and pooled intermediate row), V5 (Select after the close), V6a,
V6c and V6d (massaging inside the map delegate, inline, after the close), V7 (Dapper
for the shapes it supports) and V8 (scalars). A consumer-pace parameter busy-waits per
item for the variants that run caller code while the reader is open.

V0 now drains later result sets like the library's default, so a NextResult cost is
charged to every variant alike. --verify runs every variant, and every benchmark method
through the benchmark class's own Setup, against an oracle; --selftest-pace shows
reader-open time growing with pace only for the inline variants. The project targets
net11.0 and net10.0. No benchmark was run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ns that BenchmarkDotNet's Mean uses

The columns were the median of the last 1024 readers, about 6% of the final iteration, so a slow last iteration showed a reader-open time above Mean.
The recorder now sums the readers that close inside each WorkloadActual iteration (the engine's own iteration events, in process, nothing allocated per
reader), and the column takes the mean per operation over the iterations BenchmarkDotNet's Mean is made of: the report's result runs are matched to
their actual iterations, outliers it removed are left out, and a mismatch prints the reason instead of a number. check-readeropen.ps1 fails a
report row whose ReaderOpen exceeds Mean by more than 5%. --selftest-open-time gains checks for the stage filter and the matching. No benchmark is run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…o and the one-off overload timing to the synthetic suite

V9c (Db and Syn), V9d, V9d-M and V9e hand variants over the synthetic provider,
V6b-F, V6b-E and V6b-K (the constructor path), V9a on/off (rented column
buffers, built after the close), V9b (struct rows), V1-fresh (a new RowMap per
operation), a DateTimeOffset scenario outside the width ladder and a small
class timing the one-off Map overloads against a declared map. --verify checks
every new variant against V1, the table or the massaged oracle; the open-time
self-test covers V9a's reader-open window. Nothing is benchmarked here.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…synthetic suite

The synthetic provider can now model a buffer boundary (BufferRows, RefillCostUs): the
sync Read blocks for the refill, ReadAsync returns a truly incomplete task finished from a
timer thread, its continuation queued to the thread pool. ThreadPoolPressure keeps K busy
items on the pool for the duration of a benchmark case. SyntheticBenchmarks takes the three
settings as parameters (ODX_SYN_BUFFER_ROWS, ODX_SYN_REFILL_US, ODX_SYN_PRESSURE).

New check modes: --selftest-refill, --selftest-in6, --calibrate-spin. New console mode
--in6: the three read paths on SqlClient (LocalDB) and on a SQLite file database, with and
without pressure, plus the per-row and per-NextResult calibration. Nothing is run by this
commit; all defaults are off.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…scratch folder

The probe no longer creates its tables in tempdb. It connects to master, creates a database
with a unique name whose data and log files live under the --scratch folder (128 MB data and
64 MB log, 64 MB growth), uses ordinary tables in it, and in a finally sets it to SINGLE_USER,
drops it and checks that both files are gone. If CREATE DATABASE fails it prints a STOP line
and exits non-zero without falling back to tempdb or the default location; the SQLite part
still runs. --scratch is now required; --help and --dry-run show the database and file paths.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nity and priority

--sql-tcp <host,port> with --sql-password-file runs the SQL Server part against a server over
TCP as sa (TrustServerCertificate, default packet size). The password is read from the file at
run time; it and the connection string are never printed, and every printed exception message
is scrubbed. In this mode the dedicated database is created without a FILENAME clause (the
server's default location), dropped in a finally, and the files are counted in sys.master_files
(0 expected). The log shows the server version and net_transport, protocol_type and
net_packet_size of the probe's own connection, and labels the lines SqlServerTcp.
--affinity <hexmask> and --priority set the probe's own process affinity and priority at start;
the processor count printed afterwards (and the auto pressure K) follows the mask. Without
--sql-tcp nothing changes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e, build and verify only)

RealProviderBenchmarks runs V1, V2, V3, V7 (Dapper) and V9c (typed getters on the concrete
reader) over SQL Server LocalDB and a SQLite file database, at width 5 and 20 and 1, 100 and
10,000 rows, with the synthetic catalog's data. The LocalDB tables live in a dedicated database
whose files are in a scratch folder (no tempdb, no fallback); the SQLite file is deleted after the
case. ReaderOpen is taken by the IN6 timing wrapper (V9c: its own stopwatch on the real reader)
and summarized as in the synthetic suite. --verify-rp proves equal final objects per provider
against an oracle built from the seed data, and reports loudly when LocalDB is absent.
All variants are sync: the stage ranks the mapping paths and does not judge sync against async.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ReadAsync wait time

Before S1 the SQL Server part prints the round trip time of SELECT 1 on its own connection
(50 warm-up, 200 timed). New options: --scenarios S1,S2,S3 chooses the SQL Server scenarios,
--ops-s3 N overrides the measured operations of S3 only; the effective values are in the
settings line. The timing wrapper now measures the time spent in incomplete ReadAsync calls
(sum per operation, longest single wait) and each cell line shows wait ms sum X max Y, or n/a
for a path that never calls ReadAsync. A completed read pays only the completed-flag test.
--selftest-in6 checks the wait sum against boundaries x refill on the synthetic provider.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
For every incomplete ReadAsync the timing wrapper notes the row index (the rows already read
in that operation). Each async cell line now ends with wait rows first50 [...] hist10 [...]
over N ops: the 50 smallest distinct indices with repeat counts, and a 10-bucket histogram over
the scenario's rows, pooled over the measured operations. Recording uses preallocated arrays
and allocates nothing; a completed read adds a flag test and a row count. The sync path and
async execute + Read print n/a. --selftest-in6 checks that a refill every 100 rows over 1000
rows gives positions 100 to 900 and the expected histogram, and none without a refill.

--pressure-s3 both|off (default both): with off, S3 runs only its pressure-off cells. The
settings line and the dry-run totals show it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…egate

Adds four synthetic-harness variants that read what the hand-written V2
selector reads, built other ways (no benchmark was run):
- F0: the simple tier, a compiled expression tree with one try/catch around
  the row and a harness-private exception that names the column
- F1: a compiled expression tree over IDataRecord
- F2: the same tree over the sealed concrete reader
- F3: the same logic as a static method emitted into a collectible assembly

--verify checks all four against V1 and the table at every width and rows
1 and 100, plus a planted bad cell that must be reported with the right
column name.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…f|both

The timing wrapper now times, with Stopwatch.GetTimestamp and no allocation, the time inside
every Read and ReadAsync call (before it returns) and inside every column getter. Each cell line
ends with inside ReadAsync ms sum/max, inside Read ms sum/max and getters ms sum (sums are the
median over the measured operations, max the longest single call), so a run can tell whether a
wait sits in ReadAsync, in Read or in the getters. --selftest-in6 checks it on the synthetic
provider: a 500 us refill every 100 rows shows about boundaries x refill inside Read and a small
inside-ReadAsync sum, and a getter-heavy fake shows its getter sum.

--pressure now also takes off or both (default both): off runs every scenario with pressure off
only; auto or an integer still sets K. --pressure-s3 off still applies to S3.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…aReader)

Adds two more synthetic-harness variants next to F0 to F3 (no benchmark
was run):
- F2u: the F2 tree over a copy of the synthetic reader class without
  `sealed`, with a copy of the connection and command that hands it out
- F4: the F1 tree with the reader parameter typed as DbDataReader

--verify checks both against V1 and the table at every width and rows 1
and 100, and that the copied reader class really is not sealed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
P4 reads with ExecuteReaderAsync, ReadAsync and await GetFieldValueAsync<T> for every cell
(IsDBNullAsync only for the columns that can be NULL); P5 is the same with
CommandBehavior.SequentialAccess. They are hand-written in the probe and build the same row
objects as the ODX paths; before a scenario is measured the probe compares the first 5 rows
read by P4 and P5 with the sync path and stops if they differ. The timing wrapper counts
the async getter calls, how many were incomplete, the wait from return to completion (sum,
max) and where they happen (row index: first 50 and a 10-bucket histogram); the cell line shows
them as async getters M incomplete of K calls, wait ms sum X max Y getter wait rows ....
--paths selects the read paths (default the existing three).

The synthetic provider gets SyntheticOptions.BoundaryInGetter (default off): the refill is paid
by the first async column read of the boundary row. --selftest-in6 checks that P4 and P5 read
the same values and that the incomplete getters equal the number of boundaries.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
P6 uses the provider's own connection and command (no ODX, no timing wrapper):
ExecuteReaderAsync, await ReadAsync per row, then sync typed getters, building the same row
objects. The probe loop times it inline with Stopwatch: the time awaiting every ReadAsync
(sum and longest), the time in the getters, the count of ReadAsync calls that were not
complete when checked, and the reader-open time. P6 is selectable with --paths and has the same
equality pre-check and pressure handling as the other paths. --selftest-in6 checks it on the
synthetic provider.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…hod, FastExpressionCompiler tree)

F3r_Sync: F3's emitted static method in a non-collectible (Run) assembly.
F1f_Sync: F1's tree compiled with FastExpressionCompiler.CompileFast (6.0.0),
failing loudly in setup if it cannot compile. Both verified against V1 and
the table; verify also checks the collectibility of the F3 and F3r
assemblies. GlobalSetup prints the tiering environment variables once per case.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds --s4 <scenario>:<path> (one cell per process, one S4RESULT line and one
S4VERIFY line), --s4-setup / --s4-teardown (database and the four tables made
server-side with REPLICATE), --s4-rtt and --selftest-s4 (SQLite, the same
shapes scaled to KB). Scenarios V1, V2, N1, N2; paths R1 to R5 (raw ADO.NET:
Default or SequentialAccess, sync or async reads, R5 with async getters for
the large column) and O1 to O3 (ODX selector, generated mapping async and
sync). Allocation, peak working set, incomplete async reads and getters with
their waits are collected inline.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… --sql-localdb and --s4-scale for smoke tests

REPLICATE of a binary value returns varchar, so the V1 and V2 inserts failed
with an implicit conversion error. The 1 KB random block now goes through
varchar(max) and is converted back with CONVERT; the text tables and the
cell warm-up statement use explicit CONVERT as well. Setup checks rows and
datalength per table against the repeat counts it used.

--s4-scale <divisor> shrinks the values (setup only) and --sql-localdb
<instance> connects with integrated security, so the whole flow can be
smoke-tested against a local instance in seconds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…out=

The timeout is applied to the command of every cell path: the raw paths set
DbCommand.CommandTimeout, the ODX paths call SetTimeout. The warm-up read
uses the same value. The self-test records the commands each path creates
(a pass-through connection) and checks both the timeout and the result line.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… line

With `new`, the first thing the probe does (before any SqlClient type is
touched) is set the AppContext switches UseCompatibilityAsyncBehaviour and
UseCompatibilityProcessSni to false. Every process prints one S4SWITCHES line
with the state read back (unset when not set) and S4RESULT ends with
switches=new|default. The self-test checks the helper: default leaves both
unset, new sets and reads back both, an unknown mode is rejected.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
SyntheticDapperAot.V1 is the same call as SyntheticDapper.V1 (buffered
Query<SynRow>(sql).AsList()), in its own class carrying [DapperAot], so only
that one call site is intercepted and V7 stays on plain Dapper. The csproj
references Dapper.AOT 1.1.0, allows interceptors in the Dapper.AOT namespace
and silences the three advisor codes (DAP219, DAP231, DAP241) that lint the
SQL text of the existing benchmark queries. V7a_V1_Sync is a benchmark next
to V7_V1_Sync; --verify compares V7a with V1 (the table oracle) and with V2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Brings the branch up to date with master before the benchmarks PR; only Benchmarks/ differs from master.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…027)

- CA2020 (In6Probe): the affinity mask goes through new IntPtr(long), which
  throws on overflow in a 32-bit process (the checked behaviour of a mask).
- CA1068 (SyntheticReader.WaitForGetterRefill): the CancellationToken is now
  the last parameter; the two callers are updated, nothing else changes.
- CA1512 (SyntheticTable.Create): ArgumentOutOfRangeException.ThrowIfNegative.
- CA2027 (ChannelStudy): the 30 s Task.Delay gets a cancellation token that is
  cancelled after the measurement, so no timer is left running.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@electricessence
electricessence merged commit adc4076 into master Nov 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant