Fix .NET 10 support and database cleanup in C# LLM benchmarks#5576
Open
bradleyshep wants to merge 3 commits into
Open
Fix .NET 10 support and database cleanup in C# LLM benchmarks#5576bradleyshep wants to merge 3 commits into
bradleyshep wants to merge 3 commits into
Conversation
cloutiertyler
approved these changes
Jul 21, 2026
Contributor
|
Very low risk, I'm approving this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
Updates the LLM benchmark C# project template and generated NuGet configuration to support the repository’s .NET 10 NativeAOT build path.
The repository now selects .NET 10 through
global.json, but the LLM benchmark template targeted onlynet8.0. During C# benchmark runs,spacetime publishtherefore selectednet10.0while publishing anet8.0-only project. This caused golden-module preparation to fail with MSBuild errorNETSDK1005before any model benchmarks could run.The project template now changes from:
to:
After enabling the .NET 10 target, the benchmark exposed a second issue: its generated
NuGet.Configdid not include thedotnet-experimentalfeed required by the .NET 10 NativeAOT LLVM packages. Restore consequently failed to find:Microsoft.DotNet.ILCompiler.LLVMruntime.<platform>.Microsoft.DotNet.ILCompiler.LLVMThe generated NuGet configuration now includes the
dotnet-experimentalfeed and package-source mappings for those dependencies.The first full C# run then exposed a benchmark lifecycle issue. Generated databases remained loaded after scoring, allowing hundreds of C# NativeAOT modules to accumulate on the local server. The server eventually stopped responding, and the remaining publish failures were recorded as zero-score results.
The benchmark harness now:
Together, these changes align the LLM benchmark project with the current C# server templates and prevent generated benchmark databases from accumulating throughout long-running model evaluations.
This is a follow-up to the .NET 10 support introduced in #4915.
API and ABI breaking changes
None.
Expected complexity level and risk
3 — moderate benchmark lifecycle change.
The C# template and NuGet changes are isolated configuration updates. The database cleanup changes affect the benchmark lifecycle for all supported server languages, but do not affect production runtime behavior.
The primary risks are deleting a generated database before its scorers finish or deleting golden databases while they are still needed. The harness keeps each generated database alive through scoring and keeps all goldens alive until every model and mode has completed.
Testing
cargo test -p xtask-llm-benchmark— 10 tests passedcargo check -p xtask-llm-benchmark --all-targetscargo clippy -p xtask-llm-benchmark --all-targets -- -D warningsTargetFrameworkstonet8.0;net10.0StdbModule.wasmguidelinesandno_contextmodesgit diff --checkInitial C# validation run: https://github.com/clockworklabs/SpacetimeDB/actions/runs/29789520324
Successful full C# dry run: https://github.com/clockworklabs/SpacetimeDB/actions/runs/29835979746
Upload-enabled C# run: https://github.com/clockworklabs/SpacetimeDB/actions/runs/29848593880