Extract Traefik certificates with traefik-certs-dumper

[AndrewSav]

With more than one certificate resolver in acme.json, the mailserver extracts no certificate at all and silently serves the snake-oil one instead. That is #62, open since January 2024, and it is not specific to the Traefik version in that report — the run below is a file written by Traefik 3.6:

[INFO] Search for SSL certificates generated by Traefik
[INFO] acme.json found with Traefik v2 format, dumping into pem files
[ERROR] The certificate for mail.domain.tld or the private key was not found !

# /var/mail/ssl/acme_dump.log
Could not find private key from <stdin>
40C732A8407F0000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:
../crypto/encode_decode/decoder_lib.c:104:No supported data to decode.

Fixes #62

• What is the current behavior (you can also link to an open issue here) ?

  dumpcerts.traefik.v2.sh reads the ACME account key with jq -e -r '.[].Account.PrivateKey', which returns one key per certificate resolver. Both keys are then piped into openssl rsa at once, openssl refuses the input, and set -o errexit aborts the script — before the loop that extracts the certificates has run. One resolver works; two produce nothing.

• What is the new behavior (if this is a feature change) ?

  The acme.json written by Traefik 2 and 3 is extracted with traefik-certs-dumper, which handles any number of resolvers. Everything downstream is unchanged: same file names, same wildcard handling, same reload path.

The key it dies on is never used

The account key is written to private/letsencrypt.key and nothing in this repository reads it — not certs_helper.sh, not Postfix, not Dovecot. The certificates the mailserver actually needs sit in .[].Certificates[] and are extracted after the line that fails.

Why a tool rather than a patch

dumpcerts.traefik.v2.sh is an upstream file (Brian Harrington's dumpcerts.sh), so patching it here would fork it. traefik-certs-dumper is in the base image since debian-mail-overlay:1.1.0 (so in mailserver:1.2.0), and is refreshed to its latest release every time the overlay image is rebuilt.

It is a drop-in: its default layout is the one the script produced, certs/<domain>.crt and private/<domain>.key, so certs_helper.sh keeps working unchanged around it.

acme.json	dumpcerts.traefik.v2.sh	traefik-certs-dumper
one resolver	certs/<domain>.crt, private/<domain>.key	identical bytes
two resolvers	nothing, exit 1	every certificate
same domain under two resolvers	both certificates concatenated into one file, both keys into another	one certificate
wildcard entry	certs/*.domain.tld.crt	same
resolver with "Certificates": null	–	skipped

The script is deleted rather than left unused.

The two older formats keep their scripts

traefik-certs-dumper cannot read the Traefik 1.5/1.6 acme.json (.PrivateKey with .DomainsCertificate.Certs[]): given one, it writes an empty dump and exits 0, which would turn a broken extraction into a silent fallback to the self-signed certificate. So the detection chain in certs_helper.sh is untouched and only its third branch changes.

For the Traefik 1.7 format (.Account.PrivateKey) the tool's --version v1 output is identical to dumpcerts.acme.v2.sh except for the PEM header of the unused account key, so that branch could move too — deliberately not done here, as it is not part of the bug.

--clean=false, and a temporary directory that was never emptied

The destination is /tmp/ssl, which already contains normalized/, the directory the final certificates are assembled in; the tool's default --clean would delete it. Cleaning the directory is certs_helper.sh's own job, except that it never did it:

rm -rf "$CERT_TEMP_PATH/*"   # quoted, so it only ever removed a file named *
rm -rf "$LIVE_CERT_PATH/*"

Both globs are now outside the quotes, so stale files from an earlier run are really removed before a new dump.

Say "cannot read" rather than "unknown format"

An unreadable acme.json was reported as an unknown format, and that message was only ever appended to the dump log, never printed — so docker logs mailserver showed three raw jq: error: ... Permission denied lines and nothing else. This is what the reporter of #62 spent a year chasing, on a host with SELinux enforcing. It now says what is wrong:

[INFO] Search for SSL certificates generated by Traefik
[ERROR] /etc/letsencrypt/acme/acme.json is not readable, check its ownership and permissions
[ERROR] The certificate for mail.domain.tld or the private key was not found !

An unreadable acme folder was worse: [ -f "$ACME_FILE" ] fails, so the mailserver reported "Traefik SSL certificates not used" and generated a self-signed certificate as if no volume had been mounted at all. A mounted but unreadable folder is now called out before the fallback:

[ERROR] /etc/letsencrypt/acme is not readable, check its ownership and permissions
[INFO] Traefik SSL certificates not used

Neither message can misfire when the mount is absent: /etc/letsencrypt does not exist in the image.

Two more lines in the same block were wrong rather than unclear: the failure hint pointed at /mnt/docker/traefik/acme/dump.log, which nothing has ever written — the log is acme_dump.log in the mail volume — and the advice mentioned a Traefik 1 "frontend rule".

The format everyone uses had no test coverage

test/share/traefik/acme.v1 is the Traefik 1.5/1.6 format and acme.v2 is the Traefik 1.7 one. The format every current user has — what Traefik 2 and 3 write — was not covered by any suite, which is why this bug survived two years of green CI.

traefik_v3 covers it, with a fixture generated by Traefik 3.6 against a local Pebble ACME server, holding two certificate resolvers (mail.domain.tld and spam.domain.tld) and so reproducing #62 directly. One of its assertions is that the fixture still has more than one resolver, so the regression cannot be quietly removed by a later edit of the fixture.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Test (adding missing tests or correcting existing ones)

Status

• Ready

Todo List

• Implementation
• Tests
• Documentation — the user-facing Traefik 3 documentation is Document Traefik 3 configuration #91; nothing here changes documented behaviour

How has this been tested ?

Suite by suite, not as root, after git submodule update --init --recursive and make build:

• traefik_v3 — 9 tests, new
• traefik_acmev1 — 11 tests
• traefik_acmev2 — 12 tests
• ecdsa — 4 tests
• default — 128 tests

0 failures. reverse, ldap, ldap2 and sieve were not run locally: they take the same self-signed certificate path as default, and CI runs them.

The new suite fails on the current image, which is what makes it a regression test rather than a description of the fix. Same fixture, NAME=mailserver2/mailserver:1.2.0:

	1.2.0	this branch
acme.json exist	ok	ok
acme.json has more than one certificate resolver	ok	ok
one write to acme.json causes exactly one reload	not ok	ok
the dump log was removed	not ok	ok
all certificates were generated	not ok	ok
check private key	not ok	ok
private key matches the certificate	not ok	ok
the certificate from acme.json is the one served	not ok	ok
/var/log/mail.err does not exist	not ok	ok

Beyond the suite, certs_helper.sh update_certs was run against files written by Traefik 3.6 in three shapes, checking each time that /ssl/privkey.pem matches /ssl/cert.pem:

acme.json	before	after
one resolver	certificate extracted	unchanged
two resolvers	[ERROR] The certificate for mail.domain.tld ... was not found !	certificate extracted
two resolvers, one holding a wildcard	same error	[INFO] Let's encrypt wildcard certificate found, certificate extracted

The fixture is reproducible without a public domain or a Let's Encrypt account: run ghcr.io/letsencrypt/pebble with httpPort 80 and tlsPort 443, point two certificatesResolvers (one tlsChallenge, one httpChallenge) at https://pebble:14000/dir, give the Traefik container --network-alias mail.domain.tld --network-alias spam.domain.tld so Pebble resolves the challenge through Docker's DNS, and let Traefik trust Pebble with LEGO_CA_CERTIFICATES=/pebble.minica.pem.