feat(subsystembenchmarks): add per-case rapid_cache_cold and rapid_cache_warm bucket types

[zhixiangli]

Summary

Adds rapid_cache_cold and rapid_cache_warm bucket types to the subsystem benchmark harness so data-loading and checkpointing benchmarks can run against per-case GCS Rapid Cache (Anywhere Cache) instances.

• Per-case Rapid Cache lifecycle (dataloading/rapid_cache.py, dataloading/bucket.py, dataloading/configurator.py, checkpointing/configurator.py):
  • Extends BUCKET_TYPES with rapid_cache_cold (rccold) and rapid_cache_warm (rcwarm). Both create a regional bucket in spec.location and provision an Anywhere Cache in spec.zone via POST b/{bucket}/anywhereCaches (ingestOnWrite=False for rapid_cache_cold, ingestOnWrite=True for rapid_cache_warm), polling GET b/{bucket}/anywhereCaches/{zone} every 10s until state == "RUNNING" (default timeout 3600s).
  • On teardown, _delete calls POST b/{bucket}/anywhereCaches/{zone}/disable, removes all objects under the bucket, and suppresses errors on fs.rmdir(name) because GCS blocks bucket deletion during the 1-hour grace period after disabling an Anywhere Cache.
• Cold single-round timing and Warm multi-pass warmup (dataloading/read_case.py, dataloading/rapid_cache.py, checkpointing/checkpoint_case.py):
  • run_read_case overrides rounds=1 when params.bucket_type == "rapid_cache_cold" so subsequent rounds on the same bucket (which hit the cache after Round 1's admit-on-first-miss) are not averaged into the cold measurement.
  • warm_if_needed (invoked before timed rounds in run_read_case and in run_checkpoint_case when "read" in params.scenario) runs when bucket_type == "rapid_cache_warm" on a gs:// prefix. It reads every object under the prefix in 16 MiB chunks across up to 16 threads for 2 passes (DEFAULT_WARMUP_PASSES = 2) with a 65s settle delay (DEFAULT_WARMUP_SETTLE_SECONDS = 65) after each pass, then invalidates the filesystem cache. This allows asynchronous admit-on-first-miss admission to finish for any shards not admitted during concurrent ingest() writes and separates warmup reads from the timed 60s Cloud Monitoring alignment window.
• Cloud Monitoring amplification enrichment (dataloading/amplification.py):
  • When bucket_type is rapid_cache_cold or rapid_cache_warm and bucket-level network/sent_bytes_count and api/request_count both return None or 0.0 (when reads are served from the cache rather than origin bucket egress), enrich_csv normalizes both to 0.0 so --require-amplification succeeds.
  • Populates gcs_read_bytes, gcs_read_request_count, and gcs_read_amplification_ratio together only when both egress bytes and request count are non-None and ideal > 0.
• CLI, Cloud Build, and schema (run.py, cloudbuild/subsystembenchmarks/*):
  • Requires --zone when --bucket-type is zonal, rapid_cache_cold, or rapid_cache_warm, and adds --rapid-cache-timeout (GCSFS_SUBSYSTEM_RAPID_CACHE_TIMEOUT, default 3600).
  • Updates subsystembenchmarks-cloudbuild.yaml to accept rapid_cache_cold and rapid_cache_warm, pass _RAPID_CACHE_TIMEOUT, increase the job timeout to 18000s (and SSH key TTL to 5h), and disable active Anywhere Caches before cleaning up case buckets in delete-buckets and cleanup-leaked-resources (scoped by regex to <prefix>-(regional|zonal|hns|rapid_cache_cold|rapid_cache_warm)-<8hex>-).
• WebDataset default sweep (dataloading/webdataset/configs.yaml):
  • Parks {axis: "shard_size", file_count: 256, rows_per_file: 196, enabled: false} (the unbuffered 256-shard variant), changing the default enabled WebDataset sweep from 11 to 10 cases.

Tests

• Unit tests for rapid_cache (test_rapid_cache.py), bucket lifecycle (test_bucket.py), run_read_case single-round cold override and warmup ordering (test_read_case.py), run_checkpoint_case warmup hook (test_checkpoint_case.py), amplification enrichment (test_amplification.py), configurator abbreviations (test_configurator.py, checkpointing/tests/test_configs.py, ray_pytorch/tests/test_configs.py), CLI/Cloud Build wiring (test_run_groups.py), and WebDataset default sweep count (webdataset/tests/test_configs.py).

[gemini-code-assist]

Code Review

This pull request introduces support for GCS Rapid Cache (Anywhere Cache) in the subsystem benchmarks by adding two new bucket types: rapid_cache_cold and rapid_cache_warm. It implements the lifecycle management of Anywhere Caches, including creation, polling for RUNNING state, warming for read scenarios, disabling, and cleaning up leaked caches and buckets in Cloud Build. Additionally, configuration schemas, benchmark runners, and tests have been updated to support these new cache types. There are no review comments, so I have no feedback to provide.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.25%. Comparing base (07427a2) to head (e2c5197).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1085      +/-   ##
==========================================
+ Coverage   90.11%   90.25%   +0.14%     
==========================================
  Files          16       16              
  Lines        3408     3458      +50     
==========================================
+ Hits         3071     3121      +50     
  Misses        337      337              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.