Skip to content

Commit 21072b6

Browse files
committed
Classify loading and queue operations as writes and reads
Include loading and publishing in Write and every query/receive case in Read. Show RabbitMQ publish and receive throughput from completed full cycles, with explicit cycle labels, and keep one row per database. Verified: 28 WebsiteTests, scoped formatting, static build/governance and browser checks of all 16 databases plus queue values/sorting on desktop/mobile. Full solution restore, build and formatting verification were still running at this checkpoint.
1 parent 73aa8f5 commit 21072b6

5 files changed

Lines changed: 61 additions & 14 deletions

File tree

‎site/AGENTS.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,9 @@
1515
renders a global summary with each database once (`results-summary.mjs`) and
1616
separate tables with each data type's original case columns (`results-view.mjs`).
1717
The summary uses completed operations divided by total measured seconds per
18-
metric and topology; it keeps failures, missing data and case coverage explicit.
18+
metric and topology. Loading/publishing are writes; queries/receiving are reads.
19+
Queue rates retain the full cycle duration and a visible workload label.
20+
The summary keeps failures, missing data and case coverage explicit.
1921
The merged file is published unchanged as `data/results.json`.
2022
`results-controls.mjs` selects the global or type table, sorts fastest first and
2123
shades table edges. Without JavaScript all tables stay visible.

‎site/Features/BenchmarkComparisons/protocol.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,9 @@ The builder validates the merged file with the same contract, renders the benchm
4646

4747
`results-summary.mjs` derives each global rate from the original result files: sum of completed operations divided by sum of measured seconds for the recorded cases in that metric and node count. It does not average rounded rates. Each topology line shows the case count used. Different databases can cover different workloads; the type tables retain the matched scenarios and exact recorded values.
4848

49-
The global metrics map load to Load; update/set/append to Write; point-read/get/pk-select/neighbors/range-read/read to Read; range-query/range-select/traverse/exact-top10/term-query/and-query/aggregate/windows/group-by/filtered-aggregate to Search; delete to Delete; and mixed/cycle to Mixed. A queue cycle is not split into invented independent read or write rates. Index build is a one-time duration and remains in the Search type table.
49+
The global metrics classify operations by what they do. Load remains the bulk-loading column and also contributes to Write, alongside update/set/append and queue publishing. Read includes point-read/get/pk-select/neighbors/range-read/read, all query/search/aggregate cases and queue receiving. The query/search/aggregate cases also contribute to Search. Delete remains its own metric; mixed/cycle contribute to Mixed. A case is counted once within each metric, even when it contributes to multiple different metrics. Index build is a one-time duration and remains in the Search type table.
50+
51+
Each completed queue cycle contains exactly one publish, one receive and one ACK, as defined by `QueueScenario.CycleAsync`. Therefore completed cycles divided by the full measured cycle duration is also the observed publish rate and receive rate in that workload. Write and Read both show this recorded-workload rate with a visible "Full queue cycle" label (or "Includes queue cycle" when combined with other cases). No independent publish/receive duration is inferred and the cycle is not multiplied by two. The same rule applies to RabbitMQ and KeyLoad.
5052

5153
Only recorded jobs contribute to a summary. A recorded job's failed case or job-level error makes that metric read Failed with its error; an absent case within a recorded job reads No data. A metric without a recorded case or a positive measured duration has no number. A dash means the database's declared types have no case for that metric. Missing jobs do not become zero. All original cases and errors remain in the recorded-values tables and the unchanged downloadable results file.
5254

‎site/Features/BenchmarkComparisons/results-summary.mjs‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
import { NODES, TYPES } from './results-contract.mjs';
22

3+
const QUERY_CASES = Object.freeze(['range-query', 'range-select', 'traverse', 'exact-top10',
4+
'term-query', 'and-query', 'aggregate', 'windows', 'group-by', 'filtered-aggregate']);
35
export const SUMMARY_COLUMNS = Object.freeze([
46
{ id: 'load', label: 'Load', kind: 'write', cases: ['load'] },
5-
{ id: 'write', label: 'Write', kind: 'write', cases: ['update', 'set', 'append'] },
6-
{ id: 'read', label: 'Read', kind: 'read', cases: ['point-read', 'get', 'pk-select', 'neighbors', 'range-read', 'read'] },
7-
{ id: 'search', label: 'Search', kind: 'query', cases: ['range-query', 'range-select', 'traverse', 'exact-top10',
8-
'term-query', 'and-query', 'aggregate', 'windows', 'group-by', 'filtered-aggregate'] },
7+
{ id: 'write', label: 'Write', kind: 'write', cases: ['load', 'update', 'set', 'append', 'cycle'] },
8+
{ id: 'read', label: 'Read', kind: 'read', cases: ['point-read', 'get', 'pk-select', 'neighbors', 'range-read', 'read',
9+
...QUERY_CASES, 'cycle'] },
10+
{ id: 'search', label: 'Search', kind: 'query', cases: QUERY_CASES },
911
{ id: 'delete', label: 'Delete', kind: 'write', cases: ['delete'] },
1012
{ id: 'mixed', label: 'Mixed', kind: 'mixed', cases: ['mixed', 'cycle'] },
1113
]);
@@ -30,7 +32,10 @@ function summarize(results, database, column, nodes) {
3032
const operations = sources.reduce((sum, source) => sum + source.entry.operations, 0);
3133
const seconds = sources.reduce((sum, source) => sum + source.entry.seconds, 0);
3234
if (seconds <= 0) return { ...base, status: 'missing', reason: 'no recorded elapsed time' };
33-
return { ...base, status: 'passed', value: operations / seconds, operations, seconds };
35+
const queue = sources.some(source => source.key === 'queue|cycle');
36+
const basis = !queue || !['write', 'read'].includes(column.id) ? null
37+
: sources.length === 1 ? 'Full queue cycle' : 'Includes queue cycle';
38+
return { ...base, status: 'passed', value: operations / seconds, operations, seconds, basis };
3439
}
3540

3641
// Rates describe the recorded workload: completed operations divided by its combined measured duration.

‎site/Features/BenchmarkComparisons/results-view.mjs‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,8 @@ function lineTitle(item, line) {
6464
if (line.status === 'missing') return `${nodesLabel(line.nodes)} · No data: ${line.reason}`;
6565
if (line.status === 'failed') return `${nodesLabel(line.nodes)} · Failed: ${line.error}`;
6666
if (line.summary) return `${nodesLabel(line.nodes)} · ${INTEGER.format(line.operations)} operations / ${FINE.format(line.seconds)} s` +
67-
` · ${line.caseCount} recorded cases · ${INTEGER.format(line.value)} ops/s`;
67+
` · ${line.caseCount} recorded cases · ${INTEGER.format(line.value)} ops/s` +
68+
(line.basis ? ' · One publish and one receive per completed publish/receive/ACK cycle; full cycle duration included' : '');
6869
const { entry } = line;
6970
const batch = entry.batchSize > 1 ? ` · ${INTEGER.format(entry.batchSize)} per call` : '';
7071
const limited = limitedNote(entry);
@@ -95,7 +96,8 @@ function pairLine(model, item, line, one, visible) {
9596
const limited = line.summary ? h('small', { class: 'limited' }, `${line.caseCount} ${line.caseCount === 1 ? 'case' : 'cases'}`)
9697
: line.entry.operations < line.entry.planned
9798
? h('small', { class: 'limited' }, `${compact(line.entry.operations)} of ${compact(line.entry.planned)}`) : null;
98-
return h('span', values, label, h('b', null, item.seconds ? seconds(line.value) : compact(line.value)), note, limited,
99+
const basis = line.basis ? h('small', { class: 'limited' }, line.basis) : null;
100+
return h('span', values, label, h('b', null, item.seconds ? seconds(line.value) : compact(line.value)), note, limited, basis,
99101
h('i', { style: `--size: ${percent(ratio)}` }));
100102
}
101103

@@ -206,7 +208,7 @@ function overview(model, models) {
206208
h('header', { class: 'bench-panel-head' }, h('h3', { id: `bench-${type.id}-title` }, type.label),
207209
h('p', null, model.results.length === 0 ? 'No results in this run.' : subtitle(model.results)),
208210
model.summary ? h('p', { class: 'metric-note' },
209-
'Each database appears once. Rates combine recorded cases: total completed operations divided by total measured seconds. Case counts show coverage; type tabs show the original workloads. The 3-node change is shown only for matching case coverage.') : null),
211+
'Each database appears once. Write includes data loading and publishing; Read includes queries and receiving. Rates use completed operations divided by measured seconds. Queue write/read rates include the full publish/receive/ACK cycle. Case counts show coverage; type tabs retain the original cases.') : null),
210212
h('p', { class: 'table-scroll-hint' }, `Sorted by ${type.cases[0].label}, fastest first (1 node). Select a column to sort. Swipe or scroll horizontally to see all columns.`),
211213
comparison(model), details(model.summary ? models : [model]));
212214
}

‎tests/KeyLoad.WebsiteTests/Cases/SiteResultsSummaryTests.cs‎

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ namespace KeyLoad.WebsiteTests.BenchmarkComparisons;
77
internal sealed class SiteResultsSummaryTests
88
{
99
private const string Write = "write";
10+
private const string Read = "read";
1011
private const string Search = "search";
1112
private const string Mixed = "mixed";
1213

@@ -38,7 +39,7 @@ public async Task WriteSummaryDividesTotalCompletedOperationsByTotalMeasuredSeco
3839
var sources = SiteResultFiles.Artifacts.Select(SiteResultFiles.Read)
3940
.Where(result => (string)result[SiteTokens.Database]! == SiteTokens.KeyLoad && (int)result[SiteTokens.Nodes]! == 1)
4041
.SelectMany(result => result[SiteTokens.Cases]!.AsArray()).Select(item => item!.AsObject())
41-
.Where(item => new[] { "update", "set", "append" }.Contains((string)item[SiteTokens.Id]!)).ToArray();
42+
.Where(item => new[] { "load", "update", "set", "append", "cycle" }.Contains((string)item[SiteTokens.Id]!)).ToArray();
4243
var operations = sources.Sum(item => (double)item[SiteTokens.Operations]!);
4344
var seconds = sources.Sum(item => (double)item[SiteTokens.Seconds]!);
4445
var row = SiteResultsPage.Row(page.Section(), "all", SiteTokens.KeyLoad);
@@ -62,20 +63,55 @@ public async Task SearchSummaryKeepsFailedCasesVisible()
6263
}
6364

6465
[Test]
65-
public async Task QueueSummaryDoesNotInventIndependentReadOrWriteMeasurements()
66+
public async Task QueueWritesAndReadsUseCompletedCyclesWithTheFullMeasuredDuration()
6667
{
6768
var page = await BuildAsync(TestContext.Current!.Execution.CancellationToken);
6869
var row = SiteResultsPage.Row(page.Section(), "all", "RabbitMQ");
6970

70-
foreach (var metric in new[] { "load", "write", "read", "search", "delete" })
71+
foreach (var nodes in new[] { 1, 3 })
7172
{
72-
await Assert.That(row.Cells[metric]).IsEmpty();
73+
var source = SiteResultFiles.Read("benchmark-rabbitmq-queue-n" + nodes.ToString(CultureInfo.InvariantCulture))
74+
[SiteTokens.Cases]![0]!;
75+
var rate = (double)source[SiteTokens.Operations]! / (double)source[SiteTokens.Seconds]!;
76+
foreach (var metric in new[] { Write, Read })
77+
{
78+
var line = row.Cells[metric].Single(line => line.Nodes == nodes);
79+
await Assert.That(line.Status).IsEqualTo(SiteTokens.Passed);
80+
await Assert.That(double.Parse(line.Value, CultureInfo.InvariantCulture)).IsEqualTo(rate);
81+
await Assert.That(line.Text).Contains("Full queue cycle");
82+
}
7383
}
7484

7585
await Assert.That(row.Cells[Mixed].Length).IsEqualTo(2);
7686
await Assert.That(row.Cells[Mixed].All(line => line.Status == SiteTokens.Passed)).IsTrue();
7787
}
7888

89+
[Test]
90+
public async Task EveryDatabaseHasWriteAndReadCasesInTheGlobalComparison()
91+
{
92+
var page = await BuildAsync(TestContext.Current!.Execution.CancellationToken);
93+
94+
foreach (var row in SiteResultsPage.Rows(page.Section()))
95+
{
96+
await Assert.That(row.Cells[Write].Length).IsGreaterThan(0);
97+
await Assert.That(row.Cells[Read].Length).IsGreaterThan(0);
98+
}
99+
}
100+
101+
[Test]
102+
public async Task VectorLoadingIsWritingAndVectorSearchIsReading()
103+
{
104+
var page = await BuildAsync(TestContext.Current!.Execution.CancellationToken);
105+
var row = SiteResultsPage.Row(page.Section(), "all", "Qdrant");
106+
var write = row.Cells[Write].Single(line => line.Nodes == 1);
107+
var read = row.Cells[Read].Single(line => line.Nodes == 1);
108+
109+
await Assert.That(write.Status).IsEqualTo(SiteTokens.Passed);
110+
await Assert.That(read.Status).IsEqualTo(SiteTokens.Passed);
111+
await Assert.That(write.Value).IsEqualTo(row.Cells[SiteTokens.Load].Single(line => line.Nodes == 1).Value);
112+
await Assert.That(read.Value).IsEqualTo(row.Cells[Search].Single(line => line.Nodes == 1).Value);
113+
}
114+
79115
[Test]
80116
public async Task TopologyChangeIsShownOnlyForTheSameCaseCoverage()
81117
{

0 commit comments

Comments
 (0)