Add container healthchecks, wait-for-healthy dependencies and automatic restart by MatusBeke · Pull Request #1419 · dataquest-dev/DSpace

MatusBeke · 2026-08-24T11:51:50Z

Why

An incident where nothing crashed but the repository was broken inside. Docker only
knows whether a process is alive — it cannot tell whether Postgres accepts queries,
whether Solr has its cores loaded, or whether Tomcat's connection pool is dead. In all
of those cases the process runs and docker ps stays green.

There were no healthchecks anywhere — not in the compose files, not baked into the
images.

What this adds

Service	Probe	Why this one
`dspacedb`	`pg_isready`	A TCP port check passes during `initdb`; this only passes once the server accepts queries
`dspacesolr`	STATUS of the `search` core	`precreate-core` runs after Solr starts listening, so a port probe goes green while every DSpace query would fail
`dspace`	`/actuator/health/liveness`	See below — the obvious endpoint does not work
`dspace-angular`	`/`	See below — the obvious endpoints do not work either

Plus depends_on: condition: service_healthy throughout, an optional autoheal sidecar,
and OOM handling.

The part that is not obvious

The natural thing to write is curl -f .../actuator/health. It does not work, for two
independent reasons:

It returns HTTP 200 even when the status is DOWN. actuator.cfg maps down to 200
on purpose, because the Angular /health page needs the response body rather than a 503.
So curl -f never fails, no matter how broken the backend is.

The aggregate is DOWN on a clean install. SEOHealthIndicator calls down() when
robots.txt, the sitemap or SSR are missing, and status.order ranks down highest — so
one indicator drags the whole aggregate down. A healthcheck wired to it would restart
DSpace every two minutes over a missing robots.txt.

Hence health groups: liveness (db only) and readiness (db + search/statistics cores),
each mapping DOWN to a real 503 so a plain curl -f is enough, with show-details = never
so nothing leaks and no admin token is needed.

Solr is deliberately not in liveness: restarting Tomcat cannot bring Solr back, so
including it would only produce a restart loop that also kills requests which do not need
Solr at all.

Recovery: two mechanisms, both required

Docker does not restart a container because its healthcheck fails — restart: only
reacts to the main process exiting. So every service now carries both:

restart: unless-stopped — dead process, exited container
autoheal label — container still running while its healthcheck fails

The sidecar is not willfarrell/autoheal, which has no restart budget and will restart
a permanently broken container forever. That happened three times while building this, and
each time the restarts made diagnosis harder because the container kept dying under the
person reading its logs. This one restarts at most AUTOHEAL_MAX_RESTARTS per
AUTOHEAL_WINDOW_SECONDS (3/hour by default), then gives up, logs once, and leaves the
container unhealthy for a human. The budget resets when the container reports healthy.

It lives in a separate overlay because it mounts docker.sock, which grants effective root
on the host — that should be an explicit choice, not something a plain docker compose up
switches on.

OOM

Handled outside the healthcheck. After an OutOfMemoryError the JVM can still answer a
trivial health request while being unable to do real work, so a probe would report healthy
on a dead process. Measured on this image with a worker thread that catches OOM and
continues:

without -XX:+ExitOnOutOfMemoryError:
  WORKER: caught OutOfMemoryError and kept going
  MAIN: still alive (1/3) (2/3) (3/3)
  PROCESS EXITED NORMALLY - nobody learned about the OOM
  EXIT CODE: 0

with -XX:+ExitOnOutOfMemoryError:
  Terminating due to java.lang.OutOfMemoryError: Java heap space
  EXIT CODE: 3

So the JVM is made to die, which restart: unless-stopped then handles. The heap dump goes
to /dspace/log, which becomes a named volume so it survives a recreate instead of being
destroyed exactly when someone needs it.

Also fixed: the stack did not start at all

Unrelated to healthchecks, but it blocks anyone running this compose file.

docker-compose.yml mounts ./dspace/config over the image's config, so the image has to
be built from this tree. Pulling a matching version is not enough — this branch carries
code no published image contains, e.g. dspace/config/ehcache.xml references
org.dspace.external.provider.orcid.xml.CacheLogger, which exists in dspace-api here but
in neither upstream 7.6.5 nor 7.6.8. A pulled image dies with
Error parsing XML configuration at file:/dspace/config/ehcache.xml, and before that with
Could not resolve placeholder 'pubmed.apiKey'.

docker compose up builds a missing image by itself, so a clean machine never sees this.
The trap is a stale image from an earlier pull shadowing the build, which makes the
failure look like a config problem. DSPACE_VER is now pinned to the pom.xml version
instead of the floating dspace-7_x tag, and the file explains this.

How to run it

# 1. base stack - also creates the network the overlays expect to already exist
docker compose -p d7 -f docker-compose.yml up -d

# 2. frontend + autoheal
COMPOSE_PROJECT_NAME=d7 docker compose -p d7 \
  -f docker-compose.yml \
  -f dspace/src/main/docker-compose/docker-compose-angular.yml \
  -f dspace/src/main/docker-compose/docker-compose-autoheal.yml \
  up -d

Two phases because the overlays declare the network as external, so it has to exist
before they are applied.

The autoheal overlay is optional in packaging, not in effect. Leave that -f out and
you still get the healthchecks, the wait-for-healthy ordering and the restart policy — a
dead process is restarted. But a container that keeps running while its healthcheck
fails will not be restarted by anything, and that is the failure mode this PR is mostly
about. It is a separate file because it mounts docker.sock, not because it is a nice-to-have.

Checking state:

docker ps                                          # STATUS column shows (healthy)
docker inspect -f '{{json .State.Health}}' dspace  # .Log keeps the last 5 probe runs
docker logs autoheal                               # restarts and give-up decisions

How it was verified

Full stack from this compose file, image built from the branch. Each service broken a
different way, on purpose, so both recovery mechanisms get exercised:

dspacedb        pg_ctl stop -m immediate   -> container exited, RestartCount 0 -> 1,
                                              healthy again on its own
dspacesolr      search core unloaded       -> failing streak 1..5 -> unhealthy ->
                                              restarted by the sidecar
dspace          chaos flag                 -> unhealthy -> restarted, sidecar logged
                                              "(1/3 in window)" (budget had reset)
dspace-angular  ng serve killed            -> container exited -> restarted

Startup ordering, from the compose output:

Container dspacedb    Waiting -> Healthy
Container dspacesolr  Waiting -> Healthy
Container dspace      Waiting -> Healthy
Container dspace-angular Starting

The restart budget was verified separately, including the give-up path and the reset after
recovery.

Testing hook

The backend probe checks for a flag file first, so the whole chain can be exercised without
taking down a real service:

docker exec dspace touch /tmp/chaos-fail   # report a fault artificially
docker exec dspace rm    /tmp/chaos-fail   # clear it

The flag survives a restart — it lives in the writable layer — so it has to be removed once
the restart has been observed.

Known limits

A restart does not repair an unloaded Solr core: precreate-core only checks whether
the core directory exists, so it logs "Core search already exists" and leaves the core
unregistered. This is exactly the case the restart budget exists for.
liveness covers db; a wedged Tomcat or exhausted thread pool is caught by the probe
timing out, since actuator shares the connector and thread pool with the REST API. Disk
full and misconfiguration are deliberately not covered — a restart cannot fix them.
The sidecar only logs when it gives up. In production that moment should page someone.
readiness is configured but nothing consumes it yet; it becomes useful with a load
balancer or nginx health_check.

🤖 Generated with Claude Code

The aggregated /actuator/health is unusable as a restart signal: it reports DOWN on a clean, fully working install because SEOHealthIndicator calls down() when robots.txt, the sitemap or SSR are missing, and status.order ranks "down" highest. Anything wired to the aggregate would restart forever over a missing robots.txt, which no restart can fix. Health groups let a probe ask a narrower question: liveness = "should this process be restarted?" - db only. Solr is deliberately excluded: restarting Tomcat cannot bring Solr back, it would only loop and take down requests that do not need Solr. readiness = "should the load balancer send traffic here?" - wider, includes the search and statistics cores. Both groups map DOWN to a real 503. The global mapping has to stay 200 because the Angular /health page depends on it, but a group-level mapping means a probe can be a plain `curl -f` with no body parsing. show-details is never, so the groups expose only {"status":...} and need no admin token. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Until now the stack had no Docker healthchecks at all - not in the compose files, not baked into the images. Docker therefore only knew whether a process was alive, which is why a repository could serve nothing while `docker ps` stayed green. Healthchecks: dspacedb pg_isready. Unlike a TCP port check it only succeeds once the server accepts queries; the port listens during initdb already. dspacesolr asks for the "search" core, not the port. The entrypoint creates cores via precreate-core AFTER Solr starts listening, so a port probe goes green while DSpace would fail every query. A missing core returns {"status":{"search":{}}} with no "name". dspace the liveness group (see previous commit), with --max-time 5. depends_on now uses condition: service_healthy for both the database and Solr, which makes the TCP poll in the entrypoint redundant - and it was the weaker check anyway, so it is removed. OOM is handled outside the healthcheck on purpose. After an OutOfMemoryError the JVM can still answer a trivial health request while being unable to do real work, so a probe would report healthy on a dead process. Measured on this image with a worker thread that catches OOM and continues: without the flag: process survives, exit code 0, nobody learns about the OOM with the flag: "Terminating due to java.lang.OutOfMemoryError", exit code 3 So ExitOnOutOfMemoryError turns an invisible internal failure into a container exit, which restart: unless-stopped then handles. HeapDumpOnOutOfMemoryError keeps the evidence, and /dspace/log becomes a named volume so the dump survives a recreate rather than dying with the writable layer. The healthcheck also carries a chaos hook (/tmp/chaos-fail) so the whole unhealthy -> restart chain can be exercised without taking down a real service. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Docker does not restart a container just because its healthcheck says unhealthy - the restart policy only reacts to the main process exiting. So a container can sit unhealthy indefinitely while `docker ps` shows it running. The optional autoheal overlay closes that gap for containers labelled autoheal=true. It is deliberately NOT willfarrell/autoheal. That image has no restart budget: if a container can never become healthy it restarts it forever. That happened three times while building this - once on the backend (a config/version mismatch meant Spring never booted) and twice on the frontend (`ng serve` needs around 15 minutes to build and the start_period was too short, so the build was killed mid-flight). In every case the restarts made the problem harder to diagnose, because the container kept dying under the person reading its logs. This sidecar restarts at most AUTOHEAL_MAX_RESTARTS times per AUTOHEAL_WINDOW_SECONDS, then gives up, logs once, and leaves the container unhealthy for a human. The budget resets when the container reports healthy. It stays in a separate overlay rather than the base compose because it mounts docker.sock, which grants effective root on the host - that should be an explicit choice, not something a plain `docker compose up` turns on. The frontend probe targets / and not: /app/health - a pure proxy to the backend actuator (server.ts healthCheck), so it reports the backend's health, not the UI's. It would mark the UI unhealthy whenever the backend is down, and restarting the UI cannot fix the backend. /robots.txt - that express route only exists in the SSR server, i.e. the dist image. This overlay uses the development image, where `ng serve` returns 404 for it. Measured: / -> 200 in 35ms, /robots.txt -> 404. start_period is 900s because `ng serve` compiles bundles at container start and does not answer the first request for many minutes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…f pulling Two problems surfaced the first time this compose file was run end to end. 1. The stack did not start at all, and it had nothing to do with healthchecks. docker-compose.yml mounts ./dspace/config over the config inside the image, so the image has to be built from this tree. It is not enough to pull a matching version: this branch carries code no published image contains - for instance dspace/config/ehcache.xml references org.dspace.external.provider.orcid.xml.CacheLogger, which is in dspace-api here but in neither upstream 7.6.5 nor 7.6.8. A pulled image dies with "Error parsing XML configuration at file:/dspace/config/ehcache.xml", and before that with "Could not resolve placeholder 'pubmed.apiKey'". `docker compose up` builds a missing image on its own, so a clean machine never sees this. The trap is a stale image from an earlier pull silently shadowing the build, which makes the failure look like a config problem. DSPACE_VER is now pinned to the pom.xml version instead of the floating dspace-7_x tag so a newer upstream image cannot take its place, and the file says so in a comment. 2. Only the backend and the frontend could actually recover. dspacedb and dspacesolr had neither a restart policy nor the autoheal label, so "the database goes down and comes back on its own" simply did not happen - it stayed down. Every service now carries both, because they cover different failures: the restart policy handles a dead process and an exited container, the sidecar handles a container that still runs while its healthcheck fails. Verified by breaking each service in a different way: dspacedb pg_ctl stop -m immediate -> container exited, RestartCount 0 -> 1, back to healthy on its own dspacesolr search core unloaded -> failing streak 1..5 -> unhealthy -> restarted by the sidecar dspace chaos flag -> unhealthy -> restarted dspace-angular ng serve killed -> container exited -> restarted One correction to an earlier claim in this file: a restart does NOT repair an unloaded Solr core. precreate-core only checks whether the core directory exists, so it logs "Core search already exists" and leaves the core unregistered. The comment now says that, and it is precisely the case the sidecar's restart budget exists for - it gives up and asks for a human instead of looping. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot

Pull request overview

Adds container-level healthchecks and restart behavior to the local Docker Compose stack so that “running” containers are also functionally healthy (Postgres accepting queries, Solr core registered, backend liveness responding), with optional automated recovery for unhealthy-but-running containers.

Changes:

Added healthchecks + depends_on: condition: service_healthy ordering for dspacedb, dspacesolr, dspace, and dspace-angular, plus restart: unless-stopped.
Introduced Actuator health groups (liveness / readiness) with 503 mappings for probe-friendly behavior without leaking details.
Added an optional “autoheal with restart budget” sidecar overlay and persisted /dspace/log via a named volume for OOM heap dumps.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
`docker-compose.yml`	Adds healthchecks, restart policies, startup ordering, OOM JVM exit/heapdump handling, and a persistent log volume.
`dspace/config/modules/actuator.cfg`	Defines `liveness`/`readiness` health groups with strict HTTP mappings suitable for container probes.
`dspace/src/main/docker-compose/docker-compose-angular.yml`	Adds restart policy, waits for backend to be healthy, and introduces a frontend probe appropriate for the dev image.
`dspace/src/main/docker-compose/docker-compose-autoheal.yml`	New optional sidecar that restarts unhealthy containers with a per-container restart budget.

Suppressed comments (1)

docker-compose.yml:252

This Solr healthcheck uses curl | grep inside the dspacesolr container, but the repository’s dspace-solr/Dockerfile is based on solr:${SOLR_VERSION}-slim and does not install curl/grep. If the base image doesn’t include these tools, Solr will never become healthy, which will block dspace startup via depends_on: condition: service_healthy and may trigger autoheal restarts. Please ensure the Solr image explicitly includes the required tooling or implement the probe using a built-in Solr/Java mechanism that’s guaranteed to be present.

      test: ["CMD-SHELL", "curl -fsS 'http://localhost:8983/solr/admin/cores?action=STATUS&core=search' | grep -q '\"name\":\"search\"'"]

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Review feedback: both healthchecks shell out to curl, but neither Dockerfile installs it. Both base images do ship curl today, so this was not a live failure: solr:8.11-slim -> /usr/bin/curl, /usr/bin/grep tomcat:9-jdk (built image) -> /usr/bin/curl The objection is still right, because the probes relied on that by accident and the failure mode would be misleading rather than obvious: the container would never report healthy, depends_on: condition: service_healthy would stop dspace-angular from starting at all, and the autoheal sidecar would keep restarting a container that is fine. Nothing in the output would point at a missing binary. So the dependency is now explicit in both images, with a comment saying what breaks without it. Fixing the Solr image also required fixing its build. `docker compose build dspacesolr` could not work at all, independently of this branch: failed to compute cache key: "/scripts/log4j2.solr.xml": not found The compose build context was ./dspace/src/main/docker/dspace-solr/, a directory holding nothing but the Dockerfile, while the Dockerfile copies scripts/log4j2.solr.xml from the repository root. CI never noticed because .github/workflows/docker.yml builds the same Dockerfile with the repo root as context - compose and CI were building the image two different ways and only CI's way worked. Compose now matches CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

dspace/src/main/docker-compose/docker-compose-autoheal.yml:39

The usage example omits setting COMPOSE_PROJECT_NAME, but this file uses ${COMPOSE_PROJECT_NAME}_dspacenet for the external network name. If COMPOSE_PROJECT_NAME isn’t exported, Compose will substitute an empty string and try to join _dspacenet (likely failing in a confusing way). Update the example to export COMPOSE_PROJECT_NAME (or otherwise document it as required).

This issue also appears on line 44 of the same file.

# Usage:
#   docker compose -p d7 -f docker-compose.yml \
#     -f dspace/src/main/docker-compose/docker-compose-autoheal.yml up -d

dspace/src/main/docker-compose/docker-compose-autoheal.yml:45

networks.default.name uses ${COMPOSE_PROJECT_NAME}_dspacenet. When COMPOSE_PROJECT_NAME is unset, this silently becomes _dspacenet, which makes startup/debugging unnecessarily confusing. Consider making the variable required so Compose errors early with a clear message.

    name: ${COMPOSE_PROJECT_NAME}_dspacenet
    external: true

Follow-up to review feedback that `grep` is as implicit a dependency as `curl` was. It is not, strictly - in Debian the two differ: grep: Essential=yes (apt refuses to remove it; it is part of the base system) curl: Essential=no (genuinely optional, which is why it is now installed explicitly) But the objection points at something better than installing another package: the probe does not need grep at all. Asking the core's own ping handler is enough, because an unregistered core is not a route: curl -fsS -o /dev/null http://localhost:8983/solr/search/admin/ping registered core -> HTTP 200, exit 0 unloaded core -> HTTP 404, exit 22 Verified by unloading and recreating the core against the running container. This is a better probe than the CoreAdmin STATUS + grep it replaces: - depends on curl alone, no pipe and no second binary - does not match on Solr's JSON, which a Solr upgrade could reformat - ping executes a real query against the core, so it checks the core answers rather than merely appearing in a list Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

Dockerfile.test:58

Minor typo in comment: "containger" -> "container".

# Copy the /dspace directory from 'ant_build' containger to /dspace in this container

dspace/src/main/docker/dspace-solr/Dockerfile:44

The comment above the curl install no longer matches the actual Solr healthcheck: docker-compose.yml now probes the search core via /solr/search/admin/ping, not the CoreAdmin API. This makes the Dockerfile documentation misleading for future maintenance/debugging.

# curl: used by the container healthcheck in docker-compose.yml, which asks the CoreAdmin
# API whether the "search" core is registered. The solr:*-slim base ships curl today, but
# the probe must not silently depend on that: without it Solr would never report healthy,
# service_healthy would block the backend, and autoheal would restart a healthy container.

MatusBeke and others added 4 commits August 24, 2026 13:18

MatusBeke requested a lite review from Copilot August 24, 2026 11:52

Copilot started reviewing on behalf of MatusBeke August 24, 2026 11:52 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Comment thread docker-compose.yml

MatusBeke requested review from milanmajchrak and vidiecan and a lite review from Copilot August 24, 2026 12:16

github-actions Bot added the REVIEW-in-progress label Aug 24, 2026

Copilot started reviewing on behalf of MatusBeke August 24, 2026 12:17 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Comment thread dspace/src/main/docker/dspace-solr/Dockerfile

MatusBeke requested a lite review from Copilot August 24, 2026 12:30

Copilot started reviewing on behalf of MatusBeke August 24, 2026 12:31 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add container healthchecks, wait-for-healthy dependencies and automatic restart - #1419

Add container healthchecks, wait-for-healthy dependencies and automatic restart#1419
MatusBeke wants to merge 6 commits into
dtq-devfrom
feat/container-healthchecks

MatusBeke commented Aug 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

MatusBeke commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What this adds

The part that is not obvious

Recovery: two mechanisms, both required

OOM

Also fixed: the stack did not start at all

How to run it

How it was verified

Testing hook

Known limits

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MatusBeke commented Aug 24, 2026 •

edited

Loading