Benchmarks should disable package builds

[zanieb]

We got a report over in astral-sh/uv#17831 that the uv benchmark had significantly regressed so I briefly investigated and as noted in astral-sh/uv#17831 (comment) the issue is that we're resolving to a different version of a package (python-rapidjson) which takes >10s to build (on my machine) and creates a dramatic difference in the benchmark.

I think that --no-build or --only-binary :all: should be used so you're not including package build time in benchmarks.

[zanieb]

Futzing with this in uv, the following change was needed

no-build = true                                                                                                                                                                                                     
no-binary-package = ["petname", "python-u2flib-server"]  

Unfortunately those two packages do not seem to have wheels on Python 3.11 (I didn't futz with trying to figure out which version we are constrained to though)

The Poetry equivalent would be

poetry config installer.only-binary :all:                                                                                                                                                                         
poetry config installer.no-binary petname,python-u2flib-server 

[ipmb]

The requirements file we're using is pretty old. It looks like Sentry is now on Python 3.13 and using uv 🎉

I'd love to update the benchmark to use what's current in their repo now. Some testing locally gets a requirements.txt compatible format from:

curl -fsSL https://raw.githubusercontent.com/getsentry/sentry/refs/heads/06da182ed55ad13f7d91f9a753082ef3e755ffda/pyproject.toml | python -c "import tomllib, sys; [print(r) for r in tomllib.load(sys.stdin.buffer)['project']['dependencies']]"

[dimbleby]

Per #38 (comment) - if tools behave differently then they behave differently, and showing that is kinda the point.

IMO it is more useful to the average user to show default behaviour than it is to fiddle with configuration to avoid slower solutions.

In an ideal world, this benchmark would encourage tool maintainers to improve their default behaviour, rather than encourage them to come here and find improved non-default behaviours.