Some ASV peakmem benchmarks always show the same "improvement"

[Sevans711]

Version

2026.6.0

How did you install UXarray?

Source

What happened?

ASV benchmarking is showing the same improvement (to within ~2%) for these three tests, every single time, regardless of whether the PR should have any effect on them:

Change	Before	After	Ratio	Benchmark (Parameter)
-	578M	391M	0.68	face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
-	708M	390M	0.55	face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
-	498M	384M	0.77	mpas_ocean.Gradient.peakmem_gradient('480km')

Examples of where this occurs:

• Reduce imports inside functions #1603 (comment), for a PR which should not affect memory usage in these tests at all.
• Devalue uxarray to make it less expensive #1588 (comment), for a PR which could maybe have some impact on memory usage?
• Clean up and vectorize the face_areas API #1577 (comment), for a PR which could maybe have some impact on memory usage?
• Fix structured from_xarray n_face mapping (#1410) #1569 (comment), for a PR which probably should not affect these tests at all?

Additionally, there are a few slightly older PRs where the numbers look different by more than a few percent, but still show similar trends as above:

• fix time_face_search benchmark #1546 (comment), for a PR which definitely should not have affected these tests, as it only touched code for unrelated benchmarks.
• Speed up import and load hvplot lazily #1541 (comment), for a PR which probably should not have affected these tests?

What did you expect to happen?

I expected "Benchmarks that have improved" to reflect actual improvements to relevant parts of the code, or at least, to reflect small deviations which occur when running the benchmarking tests multiple times. Instead, these three benchmarks seem to always cost close to the same amount the first time, and "improve" by a similar amount each time, despite the relevant code changes being merged into main.

Pinging @cmdupuis3 for your consideration, when you get the chance!

Can you provide a MCVE to repoduce the bug?

[cmdupuis3]

This thing has been living in my head lol. The only thing I can think of is that maybe there's some kind of file fetching scheme going on, like the first test to touch it gets the file cached somewhere. So a read would run faster, which would lower the peak memory since it doesn't need every chunk at once? Maybe?

[Sevans711]

I agree, it seems like probably something related to caching…

I managed to develop the following understanding (with some help from Claude): numba compilation looks like a likely culprit.

Perhaps what is happening is: numba needs to compile the method the first time it is called, and sometimes recompile with some specialization for input data with different shape or type. In the face_bounds case, this is consistent with largest impact being in the first memory benchmark (hexagon case gets run first), next-largest impact in the next face_bounds memory test (geoflow gets run second), and no noticeable impact for the other two memory tests (scrip and mpas). The method might be cached in a way that it can be re-used the second time the benchmarks are run (for the "After" commit), so that there's no need to recompile.

Sidenote: the memory benchmarks also seem to include the memory involved in import uxarray or something like that? Otherwise, there is no reason for there to be hundreds of MBs of memory being used just to load these tiny example grids.

[cmdupuis3]

Seeing that with Claude also, it seems to be a numba caching issue. The benchmarks are most likely wrong in that light, so I'm having them rewritten to be more reflective of reality.