Skip to content

Fix .NET 10 support and database cleanup in C# LLM benchmarks#5576

Open
bradleyshep wants to merge 3 commits into
masterfrom
bradley/fix-csharp-llm-bench
Open

Fix .NET 10 support and database cleanup in C# LLM benchmarks#5576
bradleyshep wants to merge 3 commits into
masterfrom
bradley/fix-csharp-llm-bench

Conversation

@bradleyshep

Copy link
Copy Markdown
Contributor

Description of Changes

Updates the LLM benchmark C# project template and generated NuGet configuration to support the repository’s .NET 10 NativeAOT build path.

The repository now selects .NET 10 through global.json, but the LLM benchmark template targeted only net8.0. During C# benchmark runs, spacetime publish therefore selected net10.0 while publishing a net8.0-only project. This caused golden-module preparation to fail with MSBuild error NETSDK1005 before any model benchmarks could run.

The project template now changes from:

<TargetFramework>net8.0</TargetFramework>

to:

<TargetFrameworks>net8.0;net10.0</TargetFrameworks>

After enabling the .NET 10 target, the benchmark exposed a second issue: its generated NuGet.Config did not include the dotnet-experimental feed required by the .NET 10 NativeAOT LLVM packages. Restore consequently failed to find:

  • Microsoft.DotNet.ILCompiler.LLVM
  • runtime.<platform>.Microsoft.DotNet.ILCompiler.LLVM

The generated NuGet configuration now includes the dotnet-experimental feed and package-source mappings for those dependencies.

The first full C# run then exposed a benchmark lifecycle issue. Generated databases remained loaded after scoring, allowing hundreds of C# NativeAOT modules to accumulate on the local server. The server eventually stopped responding, and the remaining publish failures were recorded as zero-score results.

The benchmark harness now:

  • Deletes each generated database immediately after its scoring finishes
  • Keeps golden databases available while scorers compare schemas, reducers, and data
  • Deletes the golden databases after all models and modes finish
  • Reuses the publishing CLI identity when deleting each database
  • Treats cleanup and server availability failures as infrastructure failures
  • Stops remaining work and avoids uploading affected results when the local server fails
  • Reports the local server’s exit status when a benchmark run fails

Together, these changes align the LLM benchmark project with the current C# server templates and prevent generated benchmark databases from accumulating throughout long-running model evaluations.

This is a follow-up to the .NET 10 support introduced in #4915.

API and ABI breaking changes

None.

Expected complexity level and risk

3 — moderate benchmark lifecycle change.

The C# template and NuGet changes are isolated configuration updates. The database cleanup changes affect the benchmark lifecycle for all supported server languages, but do not affect production runtime behavior.

The primary risks are deleting a generated database before its scorers finish or deleting golden databases while they are still needed. The harness keeps each generated database alive through scoring and keeps all goldens alive until every model and mode has completed.

Testing

  • Ran cargo test -p xtask-llm-benchmark — 10 tests passed
  • Ran cargo check -p xtask-llm-benchmark --all-targets
  • Ran cargo clippy -p xtask-llm-benchmark --all-targets -- -D warnings
  • Verified with MSBuild that the template resolves TargetFrameworks to net8.0;net10.0
  • Packed the local C# runtime packages successfully
  • Restored the .NET 10 NativeAOT LLVM dependencies using the generated NuGet configuration
  • Published C# NativeAOT golden modules and produced StdbModule.wasm
  • Verified multiple C# goldens remain published through preparation and are deleted during final cleanup
  • Completed a full C# dry run for all default models in both guidelines and no_context modes
  • Confirmed the dry run completed with 1 successful benchmark run and 0 failures
  • Confirmed the dry run did not upload results
  • Ran git diff --check
  • Confirm the upload-enabled C# benchmark completes successfully

Initial C# validation run: https://github.com/clockworklabs/SpacetimeDB/actions/runs/29789520324

Successful full C# dry run: https://github.com/clockworklabs/SpacetimeDB/actions/runs/29835979746

Upload-enabled C# run: https://github.com/clockworklabs/SpacetimeDB/actions/runs/29848593880

@cloutiertyler

Copy link
Copy Markdown
Contributor

Very low risk, I'm approving this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants