Repository navigation
fix(storage): honor GrpcCredentialOption - #16694
zhixiangli wants to merge 5 commits into
Conversation
Since v3.0.0, `storage::Client()` ignores `CLOUD_STORAGE_EMULATOR_ENDPOINT` when choosing credentials. `DefaultOptionsWithCredentials()` still built insecure credentials for the emulator, but passed them to a `DefaultOptions()` overload that discarded them, and `DefaultOptions()` then set Application Default Credentials. Off GCE every request retried the metadata server lookup until the retry policy expired. `DefaultOptionsWithCredentials()` now sets `UnifiedCredentialsOption` itself: insecure credentials for the emulator, otherwise Application Default Credentials built with `MakeAuthOptions()`, so token requests keep the client's tracing and logging settings again. `DefaultOptions()` no longer sets credentials, and the unused overload is removed. gRPC clients now call `DefaultOptions()` and keep using Application Default Credentials unless `UnifiedCredentialsOption` is set, as before. The REST emulator endpoint does not change their credentials. Fixes googleapis#16678
There was a problem hiding this comment.
Code Review
This pull request refactors how credentials and options are defaulted in the Google Cloud Storage client, ensuring that gRPC clients configured with GrpcCredentialOption (which lack UnifiedCredentialsOption) are handled correctly and can sign remotely. It also cleans up unused DefaultOptions overloads and adds comprehensive unit tests to verify credential behavior under various emulator and gRPC configurations. The review feedback highlights several instances where auto is used to deduce domain types (such as Options and TestCredentialsVisitor), which violates the repository's style guide on type deduction. Replacing these with explicit types will improve code clarity and consistency.
| ScopedEnvironment testbench( | ||
| "CLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINT", {}); | ||
| ScopedEnvironment rest(var, "http://localhost:1234"); | ||
| auto const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector()); |
There was a problem hiding this comment.
According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. For consistency with other tests in this file (such as on lines 168 and 182), please use Options const explicitly instead of auto const.
Options const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());References
- Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)
| "CLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINT", "localhost:9999"); | ||
| ScopedEnvironment preserve( | ||
| "GOOGLE_CLOUD_CPP_STORAGE_TESTING_PRESERVE_CREDENTIALS", {}); | ||
| auto const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector()); |
There was a problem hiding this comment.
According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. For consistency with other tests in this file, please use Options const explicitly instead of auto const.
Options const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());References
- Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)
| auto const options = DefaultOptionsGrpc( | ||
| Options{}.set<UnifiedCredentialsOption>(MakeAccessTokenCredentials( | ||
| "token1", std::chrono::system_clock::now() + std::chrono::hours(1))), | ||
| MakeNotOnGcpDetector()); |
There was a problem hiding this comment.
According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. For consistency with other tests in this file, please use Options const explicitly instead of auto const.
Options const options = DefaultOptionsGrpc(
Options{}.set<UnifiedCredentialsOption>(MakeAccessTokenCredentials(
"token1", std::chrono::system_clock::now() + std::chrono::hours(1))),
MakeNotOnGcpDetector());References
- Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)
| ScopedEnvironment testbench( | ||
| "CLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINT", {}); | ||
| ScopedEnvironment tracing("CLOUD_STORAGE_ENABLE_TRACING", "auth"); | ||
| auto const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector()); |
There was a problem hiding this comment.
According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. For consistency with other tests in this file, please use Options const explicitly instead of auto const.
Options const options = DefaultOptionsGrpc(Options{}, MakeNotOnGcpDetector());References
- Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)
| TEST_F(ClientTest, EmulatorUsesInsecureCredentials) { | ||
| testing_util::ScopedEnvironment endpoint("CLOUD_STORAGE_EMULATOR_ENDPOINT", | ||
| "http://localhost:1234"); | ||
| auto const options = internal::DefaultOptionsWithCredentials(Options{}); |
There was a problem hiding this comment.
According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. Please use Options const explicitly instead of auto const.
Options const options = internal::DefaultOptionsWithCredentials(Options{});References
- Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)
| TEST_F(ClientTest, DefaultCredentialsKeepAuthOptions) { | ||
| auto const options = internal::DefaultOptionsWithCredentials( | ||
| Options{}.set<LoggingComponentsOption>({"auth"})); | ||
| auto const visitor = CredentialsVisitorFor(options); |
There was a problem hiding this comment.
According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. Please use TestCredentialsVisitor const explicitly instead of auto const.
TestCredentialsVisitor const visitor = CredentialsVisitorFor(options);References
- Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)
| } | ||
|
|
||
| TEST_F(ClientTest, DefaultOptionsHasNoCredentials) { | ||
| auto const options = internal::DefaultOptions(); |
There was a problem hiding this comment.
According to the repository style guide (Type Deduction section), we should avoid using auto when it hides domain objects or function return types. Please use Options const explicitly instead of auto const.
Options const options = internal::DefaultOptions();References
- Reject Obscured Domain & Return Types: Flag and reject auto when it hides StatusOr, domain objects, protobuf messages/fields, or function return types. (link)
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #16694 +/- ##
==========================================
+ Coverage 92.73% 92.75% +0.02%
==========================================
Files 2271 2271
Lines 226963 227181 +218
==========================================
+ Hits 210476 210732 +256
+ Misses 16487 16449 -38 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`DefaultOptionsWithCredentials()` and `DefaultOptionsGrpc()` built the auth options for the default credentials after `DefaultOptions()` added the `CLOUD_STORAGE_ENABLE_TRACING` components. With `CLOUD_STORAGE_ENABLE_TRACING=http` the logs then included the OAuth token requests, including the refresh token. Any value also replaced the `GOOGLE_CLOUD_CPP_ENABLE_TRACING` components that the credentials use, which disabled `auth` logging. Capture the auth options before `DefaultOptions()`, as 2.x did. Logging and tracing options set in code still reach the credentials. In the new tests, spell out the deduced types, and replace test helpers that return a named local: gcov reports their closing brace as never executed. Refs: googleapis#16678
`CLOUD_STORAGE_EMULATOR_ENDPOINT` also makes `storage::Client` use insecure credentials, unless the application sets `UnifiedCredentialsOption`. The variable does not change the gRPC endpoint or credentials of `MakeGrpcClient()` or `AsyncClient`. Refs: googleapis#16678
Since v3.0.0, DefaultOptionsGrpc() populated UnifiedCredentialsOption unconditionally, shadowing credentials supplied through GrpcCredentialOption in MakeGrpcClient() and AsyncClient. Skip setting the default UnifiedCredentialsOption when GrpcCredentialOption is present, unless overridden by the gRPC testbench endpoint. Because gRPC clients using GrpcCredentialOption have no UnifiedCredentialsOption, guard SigningEmail() and SignBlobImpl() against an unset option so signed URLs and post policies sign remotely when SigningAccount is provided and report kInvalidArgument otherwise, matching MakeAccessTokenCredentials(). Refs: googleapis#16678
`MakeGrpcClient()` accepts `GrpcCredentialOption`. These credentials do not expose a service account, so creating signed URLs or policy documents with such a client requires `SigningAccount`. Document this on `MakeGrpcClient()` and on `SigningAccount`. Refs: googleapis#16678
76eb729 to
e4ac02f
Compare
|
|
||
| /// @test Verify `GrpcCredentialOption` is honored and not shadowed by default | ||
| /// `UnifiedCredentialsOption`, even when the REST emulator endpoint is set. | ||
| TEST(DefaultOptionsGrpc, GrpcCredentialOptionIsHonored) { |
There was a problem hiding this comment.
GrpcCredentialOptionIsHonored tests DefaultOptionsGrpc() in isolation, while grpc_plugin_test.cc always sets UnifiedCredentialsOption.
Consider adding constructor tests for MakeGrpcClient() and AsyncClient with only GrpcCredentialOption set to guard stub and IAM channel creation against unchecked UnifiedCredentialsOption dereferences.
| /// @test Verify CreateV4SignedUrl() signs remotely when the connection has no | ||
| /// `UnifiedCredentialsOption`, as in a gRPC client using | ||
| /// `GrpcCredentialOption`. | ||
| TEST_F(CreateSignedUrlTest, V4SignRemoteWithoutUnifiedCredentials) { |
There was a problem hiding this comment.
V4SignRemoteWithoutUnifiedCredentials and V4SignWithoutUnifiedCredentialsOrSigningAccount cover V4 URL signing, while Client::SignUrlV2() and Client::SignPolicyDocument() remain untested without UnifiedCredentialsOption.
| std::shared_ptr<Credentials> const& unified = | ||
| options.get<UnifiedCredentialsOption>(); | ||
| if (!unified) return {}; | ||
| auto credentials = |
There was a problem hiding this comment.
nit: per the repository style guide, auto credentials lacks an explicit const qualifier.
Summary
Restores
GrpcCredentialOptionsupport instorage::MakeGrpcClient()andstorage::AsyncClient(regressed in v3.0.0), and handles an unsetUnifiedCredentialsOptionsafely during signed-URL and post-policy generation without silently mixing in ambient Application Default Credentials (ADC).Root Cause
GrpcCredentialOption: Since v3.0.0 (chore: merge prepare-for-v3.0.0 branch into main #15962),DefaultOptionsGrpc()populatedUnifiedCredentialsOptionunconditionally (first viaDefaultOptions(), and after fix(storage): restore emulator credentials #16687 viaif (!options.has<UnifiedCredentialsOption>())). BecauseCreateAuthenticationStrategy()checksUnifiedCredentialsOptionbeforeGrpcCredentialOption, explicitgrpc::ChannelCredentialspassed viaGrpcCredentialOptionwere ignored and replaced by ADC.UnifiedCredentialsOptiondereference in signing:Client::SigningEmail()andClient::SignBlobImpl()unconditionally dereferenced*connection_->options().get<UnifiedCredentialsOption>(), which is null whenGrpcCredentialOptionis used withoutUnifiedCredentialsOption.Changes
google/cloud/storage/internal/grpc/default_options.ccUnifiedCredentialsOptionwhenoptions.has<GrpcCredentialOption>()(mirroringgoogle::cloud::internal::PopulateGrpcOptions()). ExplicitUnifiedCredentialsOptionstill wins if both are set, andCLOUD_STORAGE_EXPERIMENTAL_GRPC_TESTBENCH_ENDPOINTstill selects insecure credentials unlessGOOGLE_CLOUD_CPP_STORAGE_TESTING_PRESERVE_CREDENTIALSis set.google/cloud/storage/client.ccClient::SigningEmail()andClient::SignBlobImpl()whenUnifiedCredentialsOptionis unset (holdingOptions const options = connection_->options();locally so theconst&intooptionsnever dangles).grpc::ChannelCredentialsis opaque (no email or private key), signing on aGrpcCredentialOptionclient behaves the same way asMakeAccessTokenCredentials():SigningAccount(...): signs remotely via IAMSignBlobauthenticated with the client'sGrpcCredentialOption.SigningAccount(...): returnsStatusCode::kInvalidArgument("signing account cannot be empty...") without probing or leaking ambient ADC.google/cloud/storage/grpc_plugin.h,google/cloud/storage/signed_url_options.hMakeGrpcClient()acceptsGrpcCredentialOption, and that such clients needSigningAccountto create signed URLs or policy documents.default_options_test.cc,monitoring_project_test.cc,client_sign_url_test.cc,client_sign_policy_document_test.cc):GrpcCredentialOptionhonored (including withCLOUD_STORAGE_EMULATOR_ENDPOINTset),UnifiedCredentialsOptionprecedence, and gRPC testbench /PRESERVE_CREDENTIALSinteraction.MonitoringProject()withGrpcCredentialOption(with and withoutProjectIdOption).CreateV4SignedUrl()andGenerateSignedPostPolicyV4()withoutUnifiedCredentialsOption(remoteSignBlobwithSigningAccount,kInvalidArgumentwithoutSigningAccount).Notes
GrpcCredentialOptionclient that calledCreateV4SignedUrl()withoutSigningAccounthappened to read ambient ADC from the legacyOauth2CredentialsOptionslot populated byDefaultOptionsWithCredentials(). That could silently sign URLs with a different principal than the explicitGrpcCredentialOption. Consistent withMakeAccessTokenCredentials()andMonitoringProject(), callers usingGrpcCredentialOptionmust supplystorage::SigningAccountwhen signing URLs or post policies (or configureUnifiedCredentialsOption).Testing
bazel test --test_tag_filters=-integration-test //google/cloud/storage/...: 189/189 pass.-fsanitize=address) onclient_sign_url_test,client_sign_policy_document_test,internal_grpc_default_options_test, andinternal_grpc_monitoring_project_test: 4/4 pass.-fno-exceptionsbuild of//:storageand//:storage_grpcsucceeds.SmokeTest.Rest,SmokeTest.Grpc,SmokeTest.Async, andCreateClientIntegrationTest: 4/4 pass.MakeGrpcClient()andAsyncClientreads withGrpcCredentialOption: valid token succeeds (OK), invalid token is rejected (UNAUTHENTICATED, vsOKonmain).CreateV4SignedUrlGET -> HTTP 200;GenerateSignedPostPolicyV4multipart POST upload -> HTTP 204) succeeds withGrpcCredentialOption(valid token)+SigningAccount, fails withUNAUTHENTICATEDon invalid token, and returnsINVALID_ARGUMENTwithoutSigningAccounteven whenGOOGLE_APPLICATION_CREDENTIALSpoints to an ambient key file on disk.Refs #16678