Repository navigation
bench(cat_ranges): add many-range and large-range microbenchmark scenarios - #1093
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds several new benchmark scenarios to gcsfs/tests/perf/microbenchmarks/cat_ranges/configs.yaml to test performance under various conditions, such as many ranges, mixed range sizes, large ranges, and single large ranges on large files. The feedback suggests adding a batch_sizes configuration to the scenarios with a high number of ranges (up to 5000) to prevent potential connection exhaustion, high memory overhead, or GCS rate limiting.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1093 +/- ##
=======================================
Coverage 90.25% 90.25%
=======================================
Files 16 16
Lines 3458 3458
=======================================
Hits 3121 3121
Misses 337 337 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
zhixiangli
left a comment
zhixiangli
left a comment
There was a problem hiding this comment.
@yuxin00j LGTM!
I would appreciate it if you could add code comments explaining why those tests matter and what real-world scenarios they cover.
The existing scenarios mostly use 200 ranges on 1 GB objects with ranges of at most 16 MB, so the new ones extend each dimension separately: range count (up to 5000), range size (up to 1 GB), and object size (up to 50 GB). These are synthetic coverage cases rather than a specific workload. |
…4MB-1GB) scenarios Adds cat_ranges_many_ranges_sweep, cat_ranges_many_mixed_ranges, cat_ranges_large_ranges_sweep, cat_ranges_single_1gb_range and cat_ranges_huge_file_mixed (50 GB file). Each call stays under ~10 GB.
3fe8890 to
fbb5535
Compare
2 participants
Summary
Adds 11
cat_rangesmicrobenchmark scenarios covering range counts and sizes the current suite does not exercise: 1000/2000/5000 x 1 MB ranges on a 10 GB file, 1000/2000/5000 mixed 64 KB / 1 MB / 4 MB ranges on a 50 GB file, 8 x 64/256/1024 MB ranges, a single 1 GB range, and 32 mixed 1 MB - 1 GB ranges on a 50 GB file.Changes
gcsfs/tests/perf/microbenchmarks/cat_ranges/configs.yaml: newmany_ranges_sweep,many_mixed_ranges,large_ranges_sweep,single_1gb_rangeandhuge_file_mixedscenarios. "Many ranges" and "large ranges" are kept as separate scenarios so a single call stays under roughly 10 GB of returned data.No code changes; benchmark configuration only.