Skip to content

Commit e6d1bcf

Browse files
ArrayRecord Teamcopybara-github
authored andcommitted
Move ArrayRecord OSS builds to riegeli 0.0.0-20261117-64d553d and Bazel 8.
The new riegeli release on BCR adds `Reader::NewReader(initial_pos, length_hint)`, which upcoming ArrayRecord changes use for bounded GCS reads. Moving to it needs several OSS build changes at once: * The other direct dependencies in `MODULE.bazel` are raised to the versions riegeli requires (abseil-cpp, protobuf, rules_cc, rules_python, platforms, googletest, pybind11_bazel). The build sets `--check_direct_dependencies=error`, so they must match. * CI and the BCR presubmit move from Bazel 7 to Bazel 8: rules_python 2.4.1 does not load on Bazel 7. * Linux wheels are built on `manylinux_2_28` instead of `manylinux2014`, because Bazel 8 needs a newer libstdc++ than the manylinux2014 image has. Prebuilt Linux wheels now need glibc 2.28 or newer, so CentOS/RHEL 7, Amazon Linux 2, and Ubuntu 18.04 lose them. manylinux2014 is end-of-life together with CentOS 7. * `SequencedChunkWriter` pads with `WritePadding(kImplicitPadding)` in OSS too, since the new riegeli removed `PadToBlockBoundary()`. * macOS CI moves from `macos-14`, which GitHub Actions no longer runs for this repository, to `macos-15`. `pypa/gh-action-pypi-publish` is pinned to v1.14.3, which fixes GHSA-5xjx-3w67-w9x9. `ArrayRecordReader` now uses insecure credentials for `gs://` paths when `CLOUD_STORAGE_EMULATOR_ENDPOINT` is set. google-cloud-cpp 3.x documents this for `storage::Client()`, but its default options replace these credentials with Application Default Credentials. Outside of GCE those look up the metadata server, and the read is retried until the retry policy expires. With the new google-cloud-cpp, this made the GCS emulator tests in `array_record_module_test` time out. This workaround can be removed once googleapis/google-cloud-cpp#16678 is fixed (proposed fix: googleapis/google-cloud-cpp#16687) and the OSS build uses a google-cloud-cpp release with it. PiperOrigin-RevId: 1025490093
1 parent 0ae13f5 commit e6d1bcf

7 files changed

Lines changed: 45 additions & 19 deletions

File tree

‎.bcr/presubmit.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ matrix:
1818
- ubuntu2204
1919
- macos_arm64
2020
bazel:
21-
- 7.x
21+
- 8.x
2222
tasks:
2323
verify_targets:
2424
name: Verify build and test targets

‎.github/workflows/build_and_publish_template.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,10 @@ jobs:
2828
fail-fast: false
2929
matrix:
3030
python-version: ["3.11", "3.12", "3.13", "3.14"]
31-
os: [ubuntu-22.04, ubuntu-22.04-arm, macos-14]
31+
os: [ubuntu-22.04, ubuntu-22.04-arm, macos-15]
3232

3333
env:
34-
USE_BAZEL_VERSION: "7.4.1"
34+
USE_BAZEL_VERSION: "8.8.2"
3535
steps:
3636
- name: Set up Bazel
3737
uses: bazel-contrib/setup-bazel@4fd964a13a440a8aeb0be47350db2fc640f19ca8 # 0.15.0
@@ -54,7 +54,7 @@ jobs:
5454
export PYTHON_VERSION=${{ matrix.python-version }}
5555
export PYTHON_MAJOR_VERSION=$(echo $PYTHON_VERSION | cut -d. -f1)
5656
export PYTHON_MINOR_VERSION=$(echo $PYTHON_VERSION | cut -d. -f2)
57-
export BAZEL_VERSION="7.4.1"
57+
export BAZEL_VERSION="8.8.2"
5858
export OUTPUT_DIR="/tmp/array_record"
5959
export SOURCE_DIR="/tmp/array_record"
6060
. "${SOURCE_DIR}"'/oss/runner_common.sh'
@@ -82,7 +82,7 @@ jobs:
8282
path: dist/
8383
merge-multiple: true
8484
- name: Publish package distributions to PyPI
85-
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
85+
uses: pypa/gh-action-pypi-publish@7245906fb61fa33a13554f9d5fb856fe61758086 # v1.14.3
8686
with:
8787
attestations: false
8888
verbose: true

‎MODULE.bazel‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,19 +20,19 @@ module(
2020
repo_name = "com_google_array_record",
2121
)
2222

23-
bazel_dep(name = "rules_cc", version = "0.2.17")
23+
bazel_dep(name = "rules_cc", version = "0.2.22")
2424
bazel_dep(name = "rules_proto", version = "7.1.0")
25-
bazel_dep(name = "rules_python", version = "1.6.0")
26-
bazel_dep(name = "platforms", version = "1.0.0")
27-
bazel_dep(name = "protobuf", version = "31.1")
25+
bazel_dep(name = "rules_python", version = "2.4.1")
26+
bazel_dep(name = "platforms", version = "1.1.0")
27+
bazel_dep(name = "protobuf", version = "35.1")
2828
bazel_dep(name = "boringssl", version = "0.20260413.0")
2929

30-
bazel_dep(name = "googletest", version = "1.17.0.bcr.2")
31-
bazel_dep(name = "abseil-cpp", version = "20260107.1")
30+
bazel_dep(name = "googletest", version = "1.18.0")
31+
bazel_dep(name = "abseil-cpp", version = "20260817.0")
3232
bazel_dep(name = "abseil-py", version = "2.1.0")
3333
bazel_dep(name = "eigen", version = "3.4.0.bcr.3")
34-
bazel_dep(name = "riegeli", version = "0.0.0-20250717-5b2e77e")
35-
bazel_dep(name = "pybind11_bazel", version = "2.13.6")
34+
bazel_dep(name = "riegeli", version = "0.0.0-20261117-64d553d")
35+
bazel_dep(name = "pybind11_bazel", version = "3.0.0")
3636

3737
SUPPORTED_PYTHON_VERSIONS = [
3838
"3.11",

‎cpp/BUILD‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,11 +43,11 @@ cc_library(
4343
"@riegeli//riegeli/base:initializer",
4444
"@riegeli//riegeli/base:object",
4545
"@riegeli//riegeli/base:status",
46-
"@riegeli//riegeli/base:types",
4746
"@riegeli//riegeli/bytes:writer",
4847
"@riegeli//riegeli/chunk_encoding:chunk",
4948
"@riegeli//riegeli/chunk_encoding:constants",
5049
"@riegeli//riegeli/records:chunk_writer",
50+
"@riegeli//riegeli/records:record_writer",
5151
],
5252
)
5353

‎cpp/sequenced_chunk_writer.cc‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,10 @@ limitations under the License.
2525
#include "absl/strings/str_format.h"
2626
#include "absl/synchronization/mutex.h"
2727
#include "riegeli/base/status.h"
28-
#include "riegeli/base/types.h"
2928
#include "riegeli/chunk_encoding/chunk.h"
3029
#include "riegeli/chunk_encoding/constants.h"
3130
#include "riegeli/records/chunk_writer.h"
31+
#include "riegeli/records/record_writer.h"
3232

3333
namespace array_record {
3434

@@ -118,8 +118,8 @@ void SequencedChunkWriterBase::TrySubmitFirstFutureChunk(
118118
return;
119119
}
120120
if (pad_to_block_boundary_) {
121-
if (!chunk_writer->PadToBlockBoundary())
122-
{
121+
if (!chunk_writer->WritePadding(
122+
riegeli::RecordWriterBase::kImplicitPadding)) {
123123
Fail(riegeli::Annotate(
124124
chunk_writer->status(),
125125
absl::StrFormat("Could not pad boundary for chunk: %d",

‎oss/runner_common.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ build_and_test_array_record() {
2626
else
2727
# Automatically decide which platform to build for by checking on which
2828
# platform this runs.
29-
AUDITWHEEL_PLATFORM='manylinux2014_'"$(uname -m)"
29+
AUDITWHEEL_PLATFORM='manylinux_2_28_'"$(uname -m)"
3030
docker rmi -f array_record:${PYTHON_VERSION}
3131
docker rm -f array_record
3232
DOCKER_BUILDKIT=1 docker build --progress=plain --no-cache \

‎python/array_record_module.cc‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ limitations under the License.
1616
#include <stddef.h>
1717
#include <stdint.h>
1818

19+
#include <cstdlib>
1920
#include <stdexcept>
2021
#include <string>
2122
#include <utility>
@@ -40,6 +41,31 @@ limitations under the License.
4041

4142
namespace py = pybind11;
4243

44+
namespace {
45+
46+
// Returns the GCS client used by `ArrayRecordReader` for `gs://` paths.
47+
//
48+
// When a storage emulator is configured, google-cloud-cpp 3.x still uses
49+
// Application Default Credentials, even though `Client()` documents otherwise.
50+
// Outside of GCE they look up the metadata server and the request is retried
51+
// until the retry policy expires, so the reader appears to hang. Use insecure
52+
// credentials, as emulators expect.
53+
//
54+
// TODO(zhixiangli): Use `google::cloud::storage::Client()` again once
55+
// https://github.com/googleapis/google-cloud-cpp/issues/16678 is fixed in the
56+
// google-cloud-cpp release used by the OSS build.
57+
google::cloud::storage::Client MakeGcsClient() {
58+
google::cloud::Options options;
59+
if (std::getenv("CLOUD_STORAGE_EMULATOR_ENDPOINT") != nullptr ||
60+
std::getenv("CLOUD_STORAGE_TESTBENCH_ENDPOINT") != nullptr) {
61+
options.set<google::cloud::UnifiedCredentialsOption>(
62+
google::cloud::MakeInsecureCredentials());
63+
}
64+
return google::cloud::storage::Client(std::move(options));
65+
}
66+
67+
} // namespace
68+
4369
PYBIND11_MODULE(array_record_module, m) {
4470
using array_record::ArrayRecordReaderBase;
4571
using array_record::ArrayRecordWriterBase;
@@ -104,7 +130,7 @@ PYBIND11_MODULE(array_record_module, m) {
104130
status_or_option->footer_prefetch_bytes();
105131
return new array_record::ArrayRecordReader(
106132
riegeli::Maker<riegeli::GcsReader>(
107-
google::cloud::storage::Client(),
133+
MakeGcsClient(),
108134
riegeli::GcsObject(path), std::move(gcs_reader_options),
109135
footer_prefetch_bytes > 0
110136
? google::cloud::storage::ReadLast(

0 commit comments

Comments
 (0)