Skip to content

fix(storage): honor GrpcCredentialOption - #16694

Open
zhixiangli wants to merge 5 commits into
googleapis:mainfrom
zhixiangli:fix-storage-grpc-credential-option
Open

zhixiangli wants to merge 5 commits into
googleapis:mainfrom
zhixiangli:fix-storage-grpc-credential-option

Conversation

@zhixiangli

@zhixiangli zhixiangli commented Nov 19, 2026 •

Copy link
Copy Markdown

Note: Stacked on top of #16687 (3371fd3586, 34a1381d24, aed8a43aaa). Once #16687 merges into main, only commits 44201459ee and e4ac02f121 remain in this PR's diff.

Summary

Restores GrpcCredentialOption support in storage::MakeGrpcClient() and storage::AsyncClient (regressed in v3.0.0), and handles an unset UnifiedCredentialsOption safely during signed-URL and post-policy generation without silently mixing in ambient Application Default Credentials (ADC).

Root Cause

  1. Shadowed GrpcCredentialOption: Since v3.0.0 (chore: merge prepare-for-v3.0.0 branch into main #15962), DefaultOptionsGrpc() populated UnifiedCredentialsOption unconditionally (first via DefaultOptions(), and after fix(storage): restore emulator credentials #16687 via if (!options.has<UnifiedCredentialsOption>())). Because CreateAuthenticationStrategy() checks UnifiedCredentialsOption before GrpcCredentialOption, explicit grpc::ChannelCredentials passed via GrpcCredentialOption were ignored and replaced by ADC.
  2. Unchecked UnifiedCredentialsOption dereference in signing: Client::SigningEmail() and Client::SignBlobImpl() unconditionally dereferenced *connection_->options().get<UnifiedCredentialsOption>(), which is null when GrpcCredentialOption is used without UnifiedCredentialsOption.

Changes

  • google/cloud/storage/internal/grpc/default_options.cc
    • Skip setting default UnifiedCredentialsOption when options.has<GrpcCredentialOption>() (mirroring google::cloud::internal::PopulateGrpcOptions()). Explicit UnifiedCredentialsOption still wins if both are set, and CLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINT still selects insecure credentials unless GOOGLE_CLOUD_CPP_STORAGE_TESTING_PRESERVE_CREDENTIALS is set.
  • google/cloud/storage/client.cc
    • Guard Client::SigningEmail() and Client::SignBlobImpl() when UnifiedCredentialsOption is unset (holding Options const options = connection_->options(); locally so the const& into options never dangles).
    • Because grpc::ChannelCredentials is opaque (no email or private key), signing on a GrpcCredentialOption client behaves the same way as MakeAccessTokenCredentials():
      • With SigningAccount(...): signs remotely via IAM SignBlob authenticated with the client's GrpcCredentialOption.
      • Without SigningAccount(...): returns StatusCode::kInvalidArgument ("signing account cannot be empty...") without probing or leaking ambient ADC.
  • google/cloud/storage/grpc_plugin.h, google/cloud/storage/signed_url_options.h
    • Document that MakeGrpcClient() accepts GrpcCredentialOption, and that such clients need SigningAccount to create signed URLs or policy documents.
  • Tests (default_options_test.cc, monitoring_project_test.cc, client_sign_url_test.cc, client_sign_policy_document_test.cc):
    • GrpcCredentialOption honored (including with CLOUD_STORAGE_EMULATOR_ENDPOINT set), UnifiedCredentialsOption precedence, and gRPC testbench / PRESERVE_CREDENTIALS interaction.
    • MonitoringProject() with GrpcCredentialOption (with and without ProjectIdOption).
    • CreateV4SignedUrl() and GenerateSignedPostPolicyV4() without UnifiedCredentialsOption (remote SignBlob with SigningAccount, kInvalidArgument without SigningAccount).

Notes

  • In 2.x, a GrpcCredentialOption client that called CreateV4SignedUrl() without SigningAccount happened to read ambient ADC from the legacy Oauth2CredentialsOption slot populated by DefaultOptionsWithCredentials(). That could silently sign URLs with a different principal than the explicit GrpcCredentialOption. Consistent with MakeAccessTokenCredentials() and MonitoringProject(), callers using GrpcCredentialOption must supply storage::SigningAccount when signing URLs or post policies (or configure UnifiedCredentialsOption).

Testing

  • bazel test --test_tag_filters=-integration-test //google/cloud/storage/...: 189/189 pass.
  • AddressSanitizer (-fsanitize=address) on client_sign_url_test, client_sign_policy_document_test, internal_grpc_default_options_test, and internal_grpc_monitoring_project_test: 4/4 pass.
  • -fno-exceptions build of //:storage and //:storage_grpc succeeds.
  • Real GCS and IAM verification:
    • SmokeTest.Rest, SmokeTest.Grpc, SmokeTest.Async, and CreateClientIntegrationTest: 4/4 pass.
    • MakeGrpcClient() and AsyncClient reads with GrpcCredentialOption: valid token succeeds (OK), invalid token is rejected (UNAUTHENTICATED, vs OK on main).
    • Remote signing (CreateV4SignedUrl GET -> HTTP 200; GenerateSignedPostPolicyV4 multipart POST upload -> HTTP 204) succeeds with GrpcCredentialOption(valid token) + SigningAccount, fails with UNAUTHENTICATED on invalid token, and returns INVALID_ARGUMENT without SigningAccount even when GOOGLE_APPLICATION_CREDENTIALS points to an ambient key file on disk.

Refs #16678

Since v3.0.0, `storage::Client()` ignores `CLOUD_STORAGE_EMULATOR_ENDPOINT`
when choosing credentials. `DefaultOptionsWithCredentials()` still built
insecure credentials for the emulator, but passed them to a
`DefaultOptions()` overload that discarded them, and `DefaultOptions()` then
set Application Default Credentials. Off GCE every request retried the
metadata server lookup until the retry policy expired.

`DefaultOptionsWithCredentials()` now sets `UnifiedCredentialsOption` itself:
insecure credentials for the emulator, otherwise Application Default
Credentials built with `MakeAuthOptions()`, so token requests keep the
client's tracing and logging settings again. `DefaultOptions()` no longer sets
credentials, and the unused overload is removed.

gRPC clients now call `DefaultOptions()` and keep using Application Default
Credentials unless `UnifiedCredentialsOption` is set, as before. The REST
emulator endpoint does not change their credentials.

Fixes googleapis#16678
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Nov 19, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors how credentials and options are defaulted in the Google Cloud Storage client, ensuring that gRPC clients configured with GrpcCredentialOption (which lack UnifiedCredentialsOption) are handled correctly and can sign remotely. It also cleans up unused DefaultOptions overloads and adds comprehensive unit tests to verify credential behavior under various emulator and gRPC configurations. The review feedback highlights several instances where auto is used to deduce domain types (such as Options and TestCredentialsVisitor), which violates the repository's style guide on type deduction. Replacing these with explicit types will improve code clarity and consistency.

ScopedEnvironment testbench(
"CLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINT", {});
ScopedEnvironment rest(var, "http://localhost:1234");
auto const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. For consistency with other tests in this file (such as on lines 168 and 182), please use Options const explicitly instead of auto const.

    Options const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());
References
  1. Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)

"CLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINT", "localhost:9999");
ScopedEnvironment preserve(
"GOOGLE_CLOUD_CPP_STORAGE_TESTING_PRESERVE_CREDENTIALS", {});
auto const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. For consistency with other tests in this file, please use Options const explicitly instead of auto const.

  Options const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());
References
  1. Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)

Comment on lines +139 to +142
auto const options = DefaultOptionsGrpc(
Options{}.set<UnifiedCredentialsOption>(MakeAccessTokenCredentials(
"token1", std::chrono::system_clock::now() + std::chrono::hours(1))),
MakeNotOnGcpDetector());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. For consistency with other tests in this file, please use Options const explicitly instead of auto const.

  Options const options = DefaultOptionsGrpc(
      Options{}.set<UnifiedCredentialsOption>(MakeAccessTokenCredentials(
          "token1", std::chrono::system_clock::now() + std::chrono::hours(1))),
      MakeNotOnGcpDetector());
References
  1. Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)

ScopedEnvironment testbench(
"CLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINT", {});
ScopedEnvironment tracing("CLOUD_STORAGE_ENABLE_TRACING", "auth");
auto const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. For consistency with other tests in this file, please use Options const explicitly instead of auto const.

  Options const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());
References
  1. Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)

Comment thread google/cloud/storage/client_test.cc Outdated
TEST_F(ClientTest, EmulatorUsesInsecureCredentials) {
testing_util::ScopedEnvironment endpoint("CLOUD_STORAGE_EMULATOR_ENDPOINT",
"http://localhost:1234");
auto const options = internal::DefaultOptionsWithCredentials(Options{});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. Please use Options const explicitly instead of auto const.

  Options const options = internal::DefaultOptionsWithCredentials(Options{});
References
  1. Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)

Comment thread google/cloud/storage/client_test.cc Outdated
TEST_F(ClientTest, DefaultCredentialsKeepAuthOptions) {
auto const options = internal::DefaultOptionsWithCredentials(
Options{}.set<LoggingComponentsOption>({"auth"}));
auto const visitor = CredentialsVisitorFor(options);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. Please use TestCredentialsVisitor const explicitly instead of auto const.

  TestCredentialsVisitor const visitor = CredentialsVisitorFor(options);
References
  1. Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)

Comment thread google/cloud/storage/client_test.cc Outdated
}

TEST_F(ClientTest, DefaultOptionsHasNoCredentials) {
auto const options = internal::DefaultOptions();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. Please use Options const explicitly instead of auto const.

  Options const options = internal::DefaultOptions();
References
  1. Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)

@zhixiangli
zhixiangli marked this pull request as ready for review November 20, 2026 00:32
@zhixiangli
zhixiangli requested review from a team as code owners November 20, 2026 00:32
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.75%. Comparing base (1ff1740) to head (e4ac02f).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #16694      +/-   ##
==========================================
+ Coverage   92.73%   92.75%   +0.02%     
==========================================
  Files        2271     2271              
  Lines      226963   227181     +218     
==========================================
+ Hits       210476   210732     +256     
+ Misses      16487    16449      -38     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

`DefaultOptionsWithCredentials()` and `DefaultOptionsGrpc()` built the
auth options for the default credentials after `DefaultOptions()` added
the `CLOUD_STORAGE_ENABLE_TRACING` components. With
`CLOUD_STORAGE_ENABLE_TRACING=http` the logs then included the OAuth
token requests, including the refresh token. Any value also replaced the
`GOOGLE_CLOUD_CPP_ENABLE_TRACING` components that the credentials use,
which disabled `auth` logging.

Capture the auth options before `DefaultOptions()`, as 2.x did. Logging
and tracing options set in code still reach the credentials.

In the new tests, spell out the deduced types, and replace test helpers
that return a named local: gcov reports their closing brace as never
executed.

Refs: googleapis#16678
`CLOUD_STORAGE_EMULATOR_ENDPOINT` also makes `storage::Client` use
insecure credentials, unless the application sets
`UnifiedCredentialsOption`. The variable does not change the gRPC
endpoint or credentials of `MakeGrpcClient()` or `AsyncClient`.

Refs: googleapis#16678
Since v3.0.0, DefaultOptionsGrpc() populated UnifiedCredentialsOption unconditionally, shadowing credentials supplied through GrpcCredentialOption in MakeGrpcClient() and AsyncClient.

Skip setting the default UnifiedCredentialsOption when GrpcCredentialOption is present, unless overridden by the gRPC testbench endpoint.

Because gRPC clients using GrpcCredentialOption have no UnifiedCredentialsOption, guard SigningEmail() and SignBlobImpl() against an unset option so signed URLs and post policies sign remotely when SigningAccount is provided and report kInvalidArgument otherwise, matching MakeAccessTokenCredentials().

Refs: googleapis#16678
`MakeGrpcClient()` accepts `GrpcCredentialOption`. These credentials do
not expose a service account, so creating signed URLs or policy
documents with such a client requires `SigningAccount`. Document this on
`MakeGrpcClient()` and on `SigningAccount`.

Refs: googleapis#16678

/// @test Verify `GrpcCredentialOption` is honored and not shadowed by default
/// `UnifiedCredentialsOption`, even when the REST emulator endpoint is set.
TEST(DefaultOptionsGrpc, GrpcCredentialOptionIsHonored) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GrpcCredentialOptionIsHonored tests DefaultOptionsGrpc() in isolation, while grpc_plugin_test.cc always sets UnifiedCredentialsOption.

Consider adding constructor tests for MakeGrpcClient() and AsyncClient with only GrpcCredentialOption set to guard stub and IAM channel creation against unchecked UnifiedCredentialsOption dereferences.

/// @test Verify CreateV4SignedUrl() signs remotely when the connection has no
/// `UnifiedCredentialsOption`, as in a gRPC client using
/// `GrpcCredentialOption`.
TEST_F(CreateSignedUrlTest, V4SignRemoteWithoutUnifiedCredentials) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

V4SignRemoteWithoutUnifiedCredentials and V4SignWithoutUnifiedCredentialsOrSigningAccount cover V4 URL signing, while Client::SignUrlV2() and Client::SignPolicyDocument() remain untested without UnifiedCredentialsOption.

std::shared_ptr<Credentials> const& unified =
options.get<UnifiedCredentialsOption>();
if (!unified) return {};
auto credentials =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: per the repository style guide, auto credentials lacks an explicit const qualifier.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants