Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions .github/scripts/pr-tests.cjs
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
// This controller is loaded only from the trusted workflow commit, never a PR.
const SUITES = new Set(['cloud', 'openwhisk']);
const SUITES = new Set(['cloud', 'openwhisk', 'local']);
const WRITE_PERMISSIONS = new Set(['admin', 'maintain', 'write']);

function parseCommand(body) {
const match = /^\/test (cloud|openwhisk) ([0-9a-f]{40})$/.exec(body.trim());
const match = /^\/test (cloud|openwhisk|local) ([0-9a-f]{40})$/.exec(body.trim());
if (!match) {
throw new Error('Usage: /test <cloud|openwhisk> <full lowercase PR head SHA>');
throw new Error('Usage: /test <cloud|openwhisk|local> <full lowercase PR head SHA>');
}
return { suite: match[1], sha: match[2] };
}
Expand Down Expand Up @@ -87,7 +87,11 @@ async function report({ github, context, env = process.env }) {
if (!SUITES.has(env.TEST_SUITE) || !/^[0-9a-f]{40}$/.test(env.TEST_SHA)) {
throw new Error('Invalid regression status target');
}
const result = env.TEST_SUITE === 'cloud' ? env.CLOUD_RESULT : env.OPENWHISK_RESULT;
const result = {
cloud: env.CLOUD_RESULT,
openwhisk: env.OPENWHISK_RESULT,
local: env.LOCAL_RESULT,
}[env.TEST_SUITE];
const state = result === 'success' ? 'success' : result === 'failure' ? 'failure' : 'error';
await status(github, context, {
sha: env.TEST_SHA, suite: env.TEST_SUITE, state,
Expand Down
14 changes: 12 additions & 2 deletions .github/workflows/pr-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:
suite:
description: Test suite
type: choice
options: [cloud, openwhisk]
options: [cloud, openwhisk, local]
required: true

permissions:
Expand Down Expand Up @@ -93,8 +93,17 @@ jobs:
with:
code_ref: ${{ needs.resolve.outputs.sha }}

local:
needs: resolve
if: needs.resolve.outputs.suite == 'local'
permissions:
contents: read
uses: $/.github/workflows/regression-local.yml
Comment thread
mcopik marked this conversation as resolved.
Resolved
with:
code_ref: ${{ needs.resolve.outputs.sha }}

report:
needs: [resolve, cloud, openwhisk]
needs: [resolve, cloud, openwhisk, local]
if: always() && needs.resolve.result == 'success'
runs-on: ubuntu-latest
permissions:
Expand All @@ -113,6 +122,7 @@ jobs:
TEST_SUITE: ${{ needs.resolve.outputs.suite }}
CLOUD_RESULT: ${{ needs.cloud.result }}
OPENWHISK_RESULT: ${{ needs.openwhisk.result }}
LOCAL_RESULT: ${{ needs.local.result }}
with:
script: |
await require('./.github/scripts/pr-tests.cjs').report({ github, context });
127 changes: 127 additions & 0 deletions .github/workflows/regression-local.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
name: Local Regression

on:
workflow_call:
inputs:
code_ref:
description: Approved commit to test; empty uses the triggering commit
type: string
required: false
default: ""
workflow_dispatch:
push:
branches:
- master
- "feature/**"

jobs:
regression:
strategy:
matrix:
language: [python, nodejs]
architecture: [x64, arm64]
include:
- language: python
version: "3.11"
- language: nodejs
version: "20"
fail-fast: false

name: ${{ matrix.language }} ${{ matrix.version }} ${{ matrix.architecture }}
# Local functions run on the Docker daemon's architecture, so each
# architecture needs a native runner.
runs-on: ${{ matrix.architecture == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-latest' }}
timeout-minutes: 60

env:
RESOURCE_PREFIX: ci
LANGUAGE: ${{ matrix.language }}
LANGUAGE_VERSION: ${{ matrix.version }}
ARCHITECTURE: ${{ matrix.architecture }}

steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.code_ref || github.sha }}
submodules: recursive
persist-credentials: false

- name: Install uv
uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0
with:
enable-cache: ${{ inputs.code_ref == '' }}

- name: Create virtual environment and install SeBS
run: |
uv venv
uv pip install .

- name: Start RustFS and ScyllaDB
run: |
set -o pipefail
uv run sebs storage start all configs/storage.json --output-json storage.json

- name: Create local regression config
run: |
jq \
--arg language "${LANGUAGE}" \
--arg version "${LANGUAGE_VERSION}" \
--arg architecture "${ARCHITECTURE}" \
'
.experiments.deployment = "local"
| .deployment.name = "local"
| .experiments.architecture = $architecture
| .experiments.system_variant = "package"
| .experiments.runtime.language = $language
| .experiments.runtime.version = $version
' configs/example.json > local-regression.json

- name: Run local regression
id: regression
continue-on-error: true
timeout-minutes: 45
run: |
set -o pipefail
uv run sebs benchmark regression test \
--config local-regression.json \
--storage-configuration storage.json \
--deployment local \
--language ${LANGUAGE} \
--language-version ${LANGUAGE_VERSION} \
--architecture ${ARCHITECTURE} \
--selected-architecture \
--resource-prefix ${RESOURCE_PREFIX} \
--filter-output

- name: Collect function container logs
if: steps.regression.outcome == 'failure'
run: |
mkdir -p diagnostics
docker ps -a > diagnostics/containers.txt 2>&1 || true
for container in $(docker ps -a --format '{{.Names}}'); do
docker logs "${container}" > "diagnostics/${container}.log" 2>&1 || true
done

- name: Upload regression artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: local-regression-artifacts-${{ matrix.language }}-${{ matrix.version }}-${{ matrix.architecture }}
path: |
diagnostics/
regression-cache/
storage.json
local-regression.json
regression_*.json
regression_*.log
if-no-files-found: ignore

- name: Fail job when regression failed
if: steps.regression.outcome == 'failure'
run: exit 1

- name: Cleanup
if: always()
run: |
uv run sebs storage stop all storage.json || true
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,16 @@

### Bug Fixes

* `sebs docker build --platform` for another architecture (e.g., x64 images on an ARM machine) failed with Docker's containerd image store, the default since Docker 29: "image ... does not provide the specified platform". Docker's build API runs the legacy builder, so SeBS now builds such images with BuildKit (`docker build --platform`).

* Local functions failed to start when their server needed more than 2.5 seconds, e.g., Node.js containers emulated with QEMU; SeBS now waits up to 30 seconds.

* `sebs docker build` and `sebs docker push` exit with status 1 and list the failed images when an image fails; they always exited with 0.

* Node.js packages built for another architecture (e.g., arm64 functions on an x86 machine) contained native binaries of the build machine: npm selects platform-specific packages, such as sharp's `@img/sharp-linux-arm64`, by `--cpu`, `--os`, and `--libc`, not by `--arch`. The installer now passes all of them. Requires rebuilding the build images.

* Python packages for x64 functions built on ARM machines (e.g., Apple Silicon) contained ARM binaries: multi-platform build images run natively there, so the installer now downloads x86 wheels for x64 targets. Cross-architecture builds accept all manylinux wheels compatible with the build image's glibc instead of only `manylinux_2_17`, which rejected old pinned packages (e.g., Pillow 8.0.0) and new releases (e.g., numpy for Python 3.14). Python dependencies are always installed from wheels.

* Change to input of 120.uploader benchmark to conform with new Wikipedia policies (#308)

* Pull Minio images from the SeBS archive `spcleth/serverless-benchmarks:minio-reproducibility-archive-<release>`, as MinIO no longer publishes images on Docker Hub or quay.io.
Expand All @@ -22,6 +32,8 @@

### Improvements

* Local deployment on ARM machines (Apple Silicon, Graviton): the local build and run images for Python and Node.js are multi-platform (x64 and arm64), and SeBS warns when the configured architecture differs from the host's.

* PAPI hardware counters in local Python images use the distribution's PAPI through a small C binding with overflow sampling (`sebs_papi`). It replaces the pypapi submodule, whose bundled PAPI predates many current CPUs, and is compiled in the run images, which previously shipped pypapi without its native module. Requires the updated local run images.

* Support for multiple variants of the same programming language (#286).
Expand Down
12 changes: 11 additions & 1 deletion configs/systems.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,11 @@
"3.9": "python:3.9-slim",
"3.10": "python:3.10-slim",
"3.11": "python:3.11-slim"
},
"arm64": {
"3.9": "python:3.9-slim",
"3.10": "python:3.10-slim",
"3.11": "python:3.11-slim"
}
},
"images": [
Expand All @@ -48,6 +53,10 @@
"x64": {
"18": "ubuntu:22.04",
"20": "ubuntu:22.04"
},
"arm64": {
"18": "ubuntu:22.04",
"20": "ubuntu:22.04"
}
},
"images": [
Expand All @@ -68,7 +77,8 @@
}
},
"architecture": [
"x64"
"x64",
"arm64"
],
"deployments": [
"package"
Expand Down
4 changes: 2 additions & 2 deletions dockerfiles/nodejs_installer.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ fi
cd /mnt/function

if [ "${TARGET_ARCHITECTURE}" == "arm64" ]; then
npm install --arch=arm64
npm install --arch=arm64 --cpu=arm64 --os=linux --libc=glibc
elif [ "${TARGET_ARCHITECTURE}" = "x64" ]; then
npm install --arch=x64
npm install --arch=x64 --cpu=x64 --os=linux --libc=glibc
else
echo "Unsupported architecture: $TARGET_ARCHITECTURE"
exit 1
Expand Down
30 changes: 28 additions & 2 deletions dockerfiles/python_installer.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,35 @@
set -e
cd /mnt/function

PLATFORM_ARG=""
if [[ "${TARGET_ARCHITECTURE}" == "arm64" ]]; then
PLATFORM_ARG="--platform manylinux_2_17_aarch64 --only-binary=:all:"
TARGET_MACHINE="aarch64"
else
TARGET_MACHINE="x86_64"
fi

# Install only wheels: no package is compiled, so packages never depend on the build machine.
PLATFORM_ARG="--only-binary=:all:"
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
if [[ "$(uname -m)" != "${TARGET_MACHINE}" ]]; then
# Cross-architecture build, e.g., arm64 functions on an x86 machine, or x64 functions
# on Apple Silicon, where the multi-platform build image runs natively.
# Without --platform, pip would install wheels for the build container's architecture.
# pip does not expand --platform to compatible tags, so we list all manylinux tags up to
# the image's glibc: both variants of a build image use the same distribution and glibc.
GLIBC_MINOR=$(python3 -c "import platform; print(platform.libc_ver()[1].split('.')[1])")
LOWEST_MINOR=5
LEGACY_TAGS="manylinux2014 manylinux2010 manylinux1"
if [[ "${TARGET_MACHINE}" == "aarch64" ]]; then
LOWEST_MINOR=17
LEGACY_TAGS="manylinux2014"
fi
for ((minor = GLIBC_MINOR; minor >= LOWEST_MINOR; minor--)); do
PLATFORM_ARG="${PLATFORM_ARG} --platform manylinux_2_${minor}_${TARGET_MACHINE}"
done
for tag in ${LEGACY_TAGS}; do
PLATFORM_ARG="${PLATFORM_ARG} --platform ${tag}_${TARGET_MACHINE}"
done
# Plain Linux wheels, e.g., the PyTorch CPU wheels pinned by 411.image-recognition.
PLATFORM_ARG="${PLATFORM_ARG} --platform linux_${TARGET_MACHINE}"
fi

if [[ "${TARGET_ARCHITECTURE}" == "arm64" ]] && [[ -f "requirements.txt.arm.${PYTHON_VERSION}" ]]; then
Expand Down
11 changes: 7 additions & 4 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,12 @@ SHA, and post one of these commands as a new comment:
```text
/test cloud <full-40-character-lowercase-head-SHA>
/test openwhisk <full-40-character-lowercase-head-SHA>
/test local <full-40-character-lowercase-head-SHA>
```

`cloud` runs the existing AWS/GCP/Azure matrix; `openwhisk` runs the existing
OpenWhisk matrix on kind. Each command must occupy the whole comment. Editing
OpenWhisk matrix on kind; `local` runs the local Docker deployment on x64 and
arm64 runners. Each command must occupy the whole comment. Editing
an existing comment does not launch tests. Bot comments, ordinary issue comments,
and commands from users without repository write access are ignored.

Expand All @@ -52,9 +54,10 @@ targets `master`, and still has the specified head SHA. The dispatched workflow
checks the target again before starting. A later push requires a new review and
command; an earlier success remains attached only to the commit actually tested.

Results appear in the PR's status checks as `regression/cloud` or
`regression/openwhisk`, with a link to the run and its artifacts. For rejected
commands, inspect the **Run PR tests** request run in Actions. The same suites can
Results appear in the PR's status checks as `regression/cloud`,
`regression/openwhisk`, or `regression/local`, with a link to the run and its
artifacts. For rejected commands, inspect the **Run PR tests** request run in
Actions. The same suites can
also be launched using **Actions → Run PR tests → Run workflow**, selecting branch
`master` and supplying the PR number, suite, and reviewed head SHA.

Expand Down
6 changes: 6 additions & 0 deletions docs/platforms.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,18 @@ On Ubuntu-based distributions, this requires installing an OS package and execut
### Multi-platform Docker Images

Build images, which encapsulate package building, are available as both x64 and arm64 for Python and Node.js on AWS Lambda.
The local deployment's build and run images for Python and Node.js are multi-platform too: local functions run on the host, so x86 and ARM machines (e.g., Apple Silicon, Graviton) pull the image of their own architecture.
Set the experiment's `architecture` to the host's (`x64` or `arm64`) for local deployments; SeBS warns about a mismatch, since dependencies are installed for the configured architecture.

To rebuild multi-platform images, an additional flag is needed to enable the internal `docker buildx` command:

```bash
sebs docker build --image-type build --language python --deployment aws --language-version 3.11 --multi-platform
sebs docker build --deployment local --language python --language-version 3.11 --multi-platform
```

A multi-platform build pushes the images for both architectures, and the combined manifest, to the repository. `sebs docker push --multi-platform` skips these images; without the flag, it pushes the locally built single-platform image.

When rebuilding build images (not necessary for regular users, only for developers), make sure that your Docker installation supports multi-platform images, e.g., [you use `containerd` image store](https://docs.docker.com/engine/storage/containerd/) - old Docker installations might not change the storage type after an upgrade to Docker 29.0, where `containerd` is the default.

## Cloud Account Identifiers
Expand Down
Loading
Loading