Skip to content

fix(perf): use N/A for missing benchmark fields to fix BigQuery schema validation - #1065

Merged
zhixiangli merged 2 commits into
fsspec:mainfrom
Mahalaxmibejugam:fix/benchmark-csv-null-markers-v2
Sep 18, 2026
Merged

zhixiangli merged 2 commits into
fsspec:mainfrom
Mahalaxmibejugam:fix/benchmark-csv-null-markers-v2

Conversation

@Mahalaxmibejugam

Copy link
Copy Markdown
Collaborator

This PR fixes a bug where missing fields in benchmarks (such as batch_size added by the cat_ranges microbenchmarks) were serialized as empty strings instead of "N/A". This caused BigQuery schema validation to fail for INT64 columns. BigQuery expects missing numerical fields to be marked as N/A per the table configuration nullMarker: "N/A".

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates benchmark reporting in both microbenchmarks and subsystem benchmarks to use "N/A" as a default placeholder value instead of empty strings when benchmark metadata, extra info, stats, or percentile values are missing or empty. The feedback suggests simplifying the checks for None and empty strings to the more idiomatic Python expression val not in (None, "") across both modified files to improve readability.

Comment thread gcsfs/tests/perf/microbenchmarks/run.py
Comment thread gcsfs/tests/perf/subsystembenchmarks/_common/report.py Outdated
@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.25%. Comparing base (060d20a) to head (c28672a).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1065   +/-   ##
=======================================
  Coverage   90.25%   90.25%           
=======================================
  Files          16       16           
  Lines        3755     3755           
=======================================
  Hits         3389     3389           
  Misses        366      366           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@zhixiangli

Copy link
Copy Markdown
Collaborator

Does "nullMarkers": ["N/A", ""] help?

@Mahalaxmibejugam

Copy link
Copy Markdown
Collaborator Author

Does "nullMarkers": ["N/A", ""] help?

We could use a "" NullMarker, but if any fields in future contain a legitimate "", they would be parsed as NULL, making it impossible to distinguish between the two. So handling it at python layer might be better? WDYT?

@zhixiangli
zhixiangli merged commit ad3e6f4 into fsspec:main Sep 18, 2026
11 checks passed
@Mahalaxmibejugam
Mahalaxmibejugam deleted the fix/benchmark-csv-null-markers-v2 branch October 5, 2026 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants