Repository navigation
fix(storage): restore emulator credentials - #16687
zhixiangli wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors the storage client's option and credential initialization to properly support gRPC clients that do not use UnifiedCredentialsOption. Specifically, it ensures that DefaultOptions() does not set default credentials, allowing gRPC to configure its own credentials without being overridden by REST defaults. Additionally, it updates signing methods to handle missing unified credentials and adds comprehensive unit tests. The review feedback highlights style guide violations regarding type deduction: 'auto const' should be qualified as a reference ('auto const&') to prevent unnecessary copies of UnifiedCredentialsOption, and 'auto' should be replaced with the explicit 'StatusOrstd::string' type to avoid obscuring the return type.
| auto credentials = WrapRestCredentials(rest_internal::MapCredentials( | ||
| *connection_->options().get<UnifiedCredentialsOption>())); | ||
| // gRPC clients using `GrpcCredentialOption` have no unified credentials. | ||
| auto const unified = connection_->options().get<UnifiedCredentialsOption>(); |
There was a problem hiding this comment.
According to the repository style guide, auto should be explicitly qualified with const and reference (&) to prevent unintended copies of non-fundamental types. Since UnifiedCredentialsOption holds a std::shared_ptr, using auto const& avoids an unnecessary copy and the associated atomic reference-counting overhead.
auto const& unified = connection_->options().get<UnifiedCredentialsOption>();References
- Ensure auto is explicitly qualified with const, reference (&), or pointer () (e.g., auto const&, auto&, auto) to prevent unintended copies or ambiguous mutability. (link)
- When a function returns a reference to a container or a large object (such as std::multimap), use 'auto const&' instead of 'auto' to avoid unnecessary and inefficient copies.
There was a problem hiding this comment.
This line was removed when the PR was rescoped to the REST emulator fix, so URL signing is now unchanged from main. For the record: StorageConnection::options() returns Options by value, so binding auto const& to options().get<...>() would leave a dangling reference (ASan reports a heap-use-after-free); the by-value copy was intentional.
| return SignBlobResponseRaw{credentials.KeyId(), *std::move(signed_blob)}; | ||
| // First try to sign locally. gRPC clients using `GrpcCredentialOption` have | ||
| // no unified credentials, and can only sign using the API. | ||
| auto const unified = connection_->options().get<UnifiedCredentialsOption>(); |
There was a problem hiding this comment.
According to the repository style guide, auto should be explicitly qualified with const and reference (&) to prevent unintended copies of non-fundamental types. Since UnifiedCredentialsOption holds a std::shared_ptr, using auto const& avoids an unnecessary copy and the associated atomic reference-counting overhead.
auto const& unified = connection_->options().get<UnifiedCredentialsOption>();References
- Ensure auto is explicitly qualified with const, reference (&), or pointer () (e.g., auto const&, auto&, auto) to prevent unintended copies or ambiguous mutability. (link)
- When a function returns a reference to a container or a large object (such as std::multimap), use 'auto const&' instead of 'auto' to avoid unnecessary and inefficient copies.
There was a problem hiding this comment.
This line was removed when the PR was rescoped to the REST emulator fix, so URL signing is now unchanged from main. For the record: StorageConnection::options() returns Options by value, so binding auto const& to options().get<...>() would leave a dangling reference (ASan reports a heap-use-after-free); the by-value copy was intentional.
| if (unified) { | ||
| auto credentials = | ||
| WrapRestCredentials(rest_internal::MapCredentials(*unified)); | ||
| auto signed_blob = credentials.SignBlob(signing_account, string_to_sign); |
There was a problem hiding this comment.
According to the repository style guide, auto should not be used when it hides StatusOr<T> return types. Explicitly declaring the type as StatusOr<std::string> improves readability and adheres to the style guide.
StatusOr<std::string> signed_blob = credentials.SignBlob(signing_account, string_to_sign);References
- Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)
There was a problem hiding this comment.
This line was removed when the PR was rescoped to the REST emulator fix, so URL signing is now unchanged from main. For the record: WrapRestCredentials::SignBlob() returns StatusOr<std::vector<std::uint8_t>>, so StatusOr<std::string> would not compile, and this line was existing code that had only moved.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #16687 +/- ##
==========================================
+ Coverage 92.73% 92.75% +0.01%
==========================================
Files 2271 2271
Lines 226963 227072 +109
==========================================
+ Hits 210476 210622 +146
+ Misses 16487 16450 -37 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1025490093
Since v3.0.0, `storage::Client()` ignores `CLOUD_STORAGE_EMULATOR_ENDPOINT` when choosing credentials. `DefaultOptionsWithCredentials()` still built insecure credentials for the emulator, but passed them to a `DefaultOptions()` overload that discarded them, and `DefaultOptions()` then set Application Default Credentials. Off GCE every request retried the metadata server lookup until the retry policy expired. `DefaultOptionsWithCredentials()` now sets `UnifiedCredentialsOption` itself: insecure credentials for the emulator, otherwise Application Default Credentials built with `MakeAuthOptions()`, so token requests keep the client's tracing and logging settings again. `DefaultOptions()` no longer sets credentials, and the unused overload is removed. gRPC clients now call `DefaultOptions()` and keep using Application Default Credentials unless `UnifiedCredentialsOption` is set, as before. The REST emulator endpoint does not change their credentials. Fixes googleapis#16678
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
* The wheel build skips the `*.venv` directories rules_python 2.x creates
for `py_test` targets. Their `python3` symlink does not resolve outside of
runfiles, and the macOS `rsync -L` fails on it.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1025490093
fc10cfc to
3371fd3
Compare
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
* The wheel build skips the `*.venv` directories rules_python 2.x creates
for `py_test` targets. Their `python3` symlink does not resolve outside of
runfiles, and the macOS `rsync -L` fails on it.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1025490093
`DefaultOptionsWithCredentials()` and `DefaultOptionsGrpc()` built the auth options for the default credentials after `DefaultOptions()` added the `CLOUD_STORAGE_ENABLE_TRACING` components. With `CLOUD_STORAGE_ENABLE_TRACING=http` the logs then included the OAuth token requests, including the refresh token. Any value also replaced the `GOOGLE_CLOUD_CPP_ENABLE_TRACING` components that the credentials use, which disabled `auth` logging. Capture the auth options before `DefaultOptions()`, as 2.x did. Logging and tracing options set in code still reach the credentials. In the new tests, spell out the deduced types, and replace test helpers that return a named local: gcov reports their closing brace as never executed. Refs: googleapis#16678
`CLOUD_STORAGE_EMULATOR_ENDPOINT` also makes `storage::Client` use insecure credentials, unless the application sets `UnifiedCredentialsOption`. The variable does not change the gRPC endpoint or credentials of `MakeGrpcClient()` or `AsyncClient`. Refs: googleapis#16678
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
* The wheel build skips the `*.venv` directories rules_python 2.x creates
for `py_test` targets. Their `python3` symlink does not resolve outside of
runfiles, and the macOS `rsync -L` fails on it.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1025490093
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
* The wheel build skips the `*.venv` directories rules_python 2.x creates
for `py_test` targets. Their `python3` symlink does not resolve outside of
runfiles, and the macOS `rsync -L` fails on it.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1025490093
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
* The wheel build skips the `*.venv` directories rules_python 2.x creates
for `py_test` targets. Their `python3` symlink does not resolve outside of
runfiles, and the macOS `rsync -L` fails on it.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1025490093
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
* The wheel build skips the `*.venv` directories rules_python 2.x creates
for `py_test` targets. Their `python3` symlink does not resolve outside of
runfiles, and the macOS `rsync -L` fails on it.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1025490093
| // `CLOUD_STORAGE_ENABLE_TRACING`. Those configure the storage client only: | ||
| // `http` would log the OAuth token requests, and any value would replace the | ||
| // `GOOGLE_CLOUD_CPP_ENABLE_TRACING` components used by the credentials. | ||
| auto auth_options = google::cloud::internal::MakeAuthOptions(opts); |
There was a problem hiding this comment.
Calling MakeAuthOptions(opts) before DefaultOptions() prevents GOOGLE_CLOUD_CPP_OPENTELEMETRY_TRACING from reaching the default credentials. Unlike LoggingComponentsOption, PopulateAuthOptions() does not read GOOGLE_CLOUD_CPP_OPENTELEMETRY_TRACING, so OAuth token requests lose OpenTelemetry spans unless OpenTelemetryTracingOption is set in code.
Consider copying OpenTelemetryTracingOption from opts into auth_options after DefaultOptions() runs in both DefaultOptionsWithCredentials() and DefaultOptionsGrpc(), with a unit test covering that env var.
|
|
||
| /// @test Verify `CLOUD_STORAGE_ENABLE_TRACING` configures the client logging | ||
| /// only. The default credentials use `GOOGLE_CLOUD_CPP_ENABLE_TRACING`. | ||
| TEST(DefaultOptionsGrpc, DefaultCredentialsIgnoreStorageTracingEnvironment) { |
There was a problem hiding this comment.
DefaultCredentialsIgnoreStorageTracingEnvironment passes empty Options{} with GOOGLE_CLOUD_CPP_ENABLE_TRACING=auth, which PopulateAuthOptions() already applies even when MakeGoogleDefaultCredentials() receives empty options. As a result, no test in this file fails if DefaultOptionsGrpc() drops auth_options on line 95.
| // endpoint does not change the gRPC credentials. As in | ||
| // `DefaultOptionsWithCredentials()`, capture the auth options before | ||
| // `DefaultOptions()` adds the `CLOUD_STORAGE_ENABLE_TRACING` components. | ||
| auto auth_options = google::cloud::internal::MakeAuthOptions(options); |
There was a problem hiding this comment.
Per the repository style guide, auto auth_options hides the function return type Options and lacks an explicit qualifier.
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
* The wheel build skips the `*.venv` directories rules_python 2.x creates
for `py_test` targets. Their `python3` symlink does not resolve outside of
runfiles, and the macOS `rsync -L` fails on it.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1025490093
…el 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`,
which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs
several OSS build changes at once:
* The other direct dependencies in `MODULE.bazel` are raised to the versions
riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms,
googletest, pybind11_bazel). The build sets
`--check_direct_dependencies=error`, so they must match.
* CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1
does not load on Bazel 7.
* Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`,
because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has.
Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7,
Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life
together with CentOS 7.
* `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS
too, since the new riegeli removed `PadToBlockBoundary()`.
* macOS CI moves from `macos-14`, which GitHub Actions no longer runs for
this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to
v1.14.3, which fixes GHSA-5xjx-3w67-w9x9.
* The wheel build skips the `*.venv` directories rules_python 2.x creates
for `py_test` targets. Their `python3` symlink does not resolve outside of
runfiles, and the macOS `rsync -L` fails on it.
`ArrayRecordReader` now uses insecure credentials for `gs://` paths when
`CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this
for `storage::Client()`, but its default options replace these credentials with
Application Default Credentials. Outside of GCE those look up the metadata
server, and the read is retried until the retry policy expires. With the new
google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test`
time out. This workaround can be removed once
googleapis/google-cloud-cpp#16678 is fixed (proposed
fix: googleapis/google-cloud-cpp#16687) and the OSS
build uses a google-cloud-cpp release with it.
PiperOrigin-RevId: 1026386371
Summary
Restores insecure credentials for the Cloud Storage REST client when
CLOUD_STORAGE_EMULATOR_ENDPOINT(or the deprecatedCLOUD_STORAGE_TESTBENCH_ENDPOINT) is set, as in 2.x and asgoogle/cloud/storage/doc/environment-variables.doximplies. gRPC and Async client credentials are unchanged.Root Cause
When
Oauth2CredentialsOptionwas removed in v3.0.0,storage::internal::DefaultOptions(std::shared_ptr<oauth2_internal::Credentials> const&, Options)became a pass-through that ignored itscredentialsparameter, andDefaultOptions(Options)setUnifiedCredentialsOptiontoMakeGoogleDefaultCredentials()whenever it was absent. As a result:DefaultOptionsWithCredentials()builtMakeInsecureCredentials()for the emulator, discarded them, and used Application Default Credentials instead. Off GCE, every request retried a failing metadata-server lookup until the retry policy expired (15 minutes by default). With ADC configured, the client sent real OAuth tokens to the plaintext emulator.MakeAuthOptions(opts)passed toMakeGoogleDefaultCredentials()was discarded too, so token requests lost the client's tracing and logging settings.Changes
google/cloud/storage/client.{h,cc}DefaultOptions(Options)no longer sets credentials. The unused two-argument overload is removed.DefaultOptionsWithCredentials(Options)setsUnifiedCredentialsOptionunless the application did:MakeInsecureCredentials()for the emulator, otherwiseMakeGoogleDefaultCredentials(MakeAuthOptions(opts)). As in 2.x, the auth options come from the application's options, captured beforeDefaultOptions()runs.CLOUD_STORAGE_ENABLE_TRACINGtherefore configures only the storage client, and the credentials keep usingGOOGLE_CLOUD_CPP_ENABLE_TRACING.google/cloud/storage/internal/grpc/default_options.ccDefaultOptionsGrpc()calls the credential-neutralDefaultOptions()and uses Application Default Credentials unlessUnifiedCredentialsOptionis set, as before. The REST emulator endpoint therefore does not make gRPC clients insecure, and the gRPC testbench (CLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINT) still selects insecure credentials. The auth options are captured beforeDefaultOptions(), as in the REST client.google/cloud/storage/doc/environment-variables.doxCLOUD_STORAGE_EMULATOR_ENDPOINTmakesstorage::Clientuse insecure credentials unless the application setsUnifiedCredentialsOption, and that it does not change the gRPC endpoint or credentials.client_test.cc,internal/grpc/default_options_test.cc):Clientconstructor,MakeAuthOptionspropagation fromOptions, andCLOUD_STORAGE_ENABLE_TRACINGnot reaching the credentials (which useGOOGLE_CLOUD_CPP_ENABLE_TRACING).CLOUD_STORAGE_ENABLE_TRACINGdoes not reach the credentials.Notes
GrpcCredentialOptioninMakeGrpcClient()andAsyncClient(currently ignored, as onmain) is handled separately in fix(storage): honor GrpcCredentialOption #16694. Changing it also affects URL signing for those clients, so it needs its own change and release note.CLOUD_STORAGE_ENABLE_TRACINGstill configures only the storage client, as onmainand in 2.x. In particular,CLOUD_STORAGE_ENABLE_TRACING=httpdoes not log the OAuth token requests (which contain refresh tokens). UseGOOGLE_CLOUD_CPP_ENABLE_TRACING=authto debug the credentials.Testing
bazel test --test_tag_filters=-integration-test //google/cloud/storage/...: 189/189 pass. A-fno-exceptionsbuild of//:storageand//:storage_grpcsucceeds.Emulator,
storage::Client()with no explicit credentials:mainAuthorizationheaderUNAVAILABLEafter the retry windowReal GCS: REST, gRPC and Async smoke tests and
CreateClientIntegrationTestpass.Real GCS logging: with
CLOUD_STORAGE_ENABLE_TRACING=httpthe logs contain norefresh_token, and withGOOGLE_CLOUD_CPP_ENABLE_TRACING=authplusCLOUD_STORAGE_ENABLE_TRACING=raw-clientthe credential logs are still emitted, both matchingmain.gRPC and signing parity with
mainon real GCS and IAM: data-plane reads, IAMSignBlobsigned URLs (GETreturns 200), and local signing all behave identically.Fixes #16678