[bugfix] Speak decoded UTF-8 resource names on the wire by joewiz · Pull Request #54 · eXist-db/existdb-openapi

joewiz · 2026-06-10T01:43:48Z

[This PR was co-authored with Claude Code. -Joe]

Update This PR predated eXist-db/exist#6463 - where the more fundamental fixes for resource names in eXist-db (and open questions that need to be resolved) are described. This PR is still valuable in the interim, ensuring existdb-openapi and clients that talk to it consume and supply resource names correctly.

Summary

The /api/db endpoints spoke eXist's stored, percent-encoded form on the wire. This PR makes the API speak decoded UTF-8 in both directions: a client sends and receives café-ünïcode.xml, never caf%C3%A9-%C3%BCn%C3%AFcode.xml.

eXide and TEI Publisher already work this way (decode for display, encode before storage). This brings existdb-openapi in line, so a name from a listing can be echoed straight back as an operation path and always resolve to the same resource.

What changed

modules/db.xqm — two boundary helpers, applied uniformly across every handler:

db:to-stored — an incoming wire path → the stored form, applied once at the top of each handler before any doc()/collection()/xmldb:*-available/xmldb:* call. It uses fn:iri-to-uri.
db:to-display — a stored path/name → the decoded wire form, applied to every name and path leaving the API.

Why `fn:iri-to-uri` and not `xmldb:encode`

iri-to-uri is the inbound encoder because it reproduces exactly what eXist's storage layer writes. Verified against a live instance:

name	`xmldb:store` stores	`fn:iri-to-uri`	`xmldb:encode`
`café.xml`	`caf%C3%A9.xml`	`caf%C3%A9.xml` ✅	`caf%C3%A9.xml`
`quote'name.xml`	`quote'name.xml`	`quote'name.xml` ✅	`quote%27name.xml` ❌
`a&b.xml`	`a&b.xml`	`a&b.xml` ✅	`a%26b.xml` ❌
`a b.xml`	throws `FORG0001`	`a%20b.xml` ✅	`a%20b.xml`

iri-to-uri percent-encodes spaces and non-ASCII but leaves sub-delims (' & + @) and existing %XX untouched. So it (a) matches the stored form xmldb:store produces — including the literal-sub-delim names xmldb:store leaves un-encoded, which xmldb:encode would instead store as %27/%26 and then fail to resolve; (b) additionally encodes the space that xmldb:store rejects outright; and (c) is idempotent on already-encoded input, so older clients still sending caf%C3%A9.xml keep working with no double-encoding.

On the decode side, db:to-display protects a literal + (as %2B) before calling xmldb:decode-uri. xmldb:decode-uri otherwise form-decodes + to a space (the x-www-form-urlencoded convention; eXist-db/exist#1824), but a + in a stored name is always a literal + (spaces are stored as %20), so without this protection naïve+test.xml would read back as naïve test.xml. This mirrors URIUtils.decodeForURI (the core fix in eXist-db/exist#6451) at the API layer, so it is correct independent of the core build and forward-compatible once #6451 lands.

URLs in responses (e.g. runPath) stay encoded, since they are clickable links, not operation keys.

Test plan

New Cypress coverage (db.cy.js) — store / read / list / remove a café déjà.xml, an o'brien.xml, and a naïve+test.xml by their decoded names, asserting listing names are never percent-encoded. (The existing db tests use ASCII names and don't exercise this boundary.)
Real-client end-to-end (existdb-oxygen-plugin, against a beta3 instance carrying this fix): display, read, create, rename, and move/copy all round-trip on space, non-ASCII, CJK, Cyrillic, &, parentheses, and apostrophe — with zero plugin-side encode/decode logic.
On-disk verification (bypassing the API): every stored name is the exact iri-to-uri form of the user-typed name, all resolve, and the full decoded-path round-trip returns content (including Cyrillic).

Known boundary (out of scope, documented)

Literal % can't be disambiguated from a percent-escape in Phase 1 (needs a bijective encoding) — iri-to-uri leaves a literal % untouched. This is the one remaining edge of the broader resource-naming contract work; this PR is the existdb-openapi piece of it.

(The literal-+ case is handled here directly; see the decode note above.)

The /api/db endpoints spoke eXist's stored, percent-encoded form on the wire: listings returned "caf%C3%A9.xml", so the existdb-oxygen-plugin and other clients displayed encoded names and could not tell which form was canonical. eXide and TEI Publisher already decode for display and encode before storage; this brings existdb-openapi in line. Every handler now encodes the incoming wire path once (db:to-stored) before any doc()/collection()/xmldb:* call, and decodes every name and path leaving the API (db:to-display). fn:iri-to-uri is used inbound (not xmldb:encode) because it matches eXist's own storage escaping: it percent-encodes spaces and non-ASCII but leaves sub-delims (' & + @) and existing %XX untouched. Verified against a live instance: xmldb:store("café.xml") -> caf%C3%A9.xml and store("quote'name.xml") -> quote'name.xml (literal), while xmldb:store throws outright on a raw space. iri-to-uri reproduces store's output where store succeeds, additionally encodes the space store rejects, and is idempotent on already-encoded paths -- so older encoded-path clients keep working. xmldb:encode would full RFC-3986 encode the sub-delims and fail to resolve names xmldb:store left literal. Proven end-to-end with "späce & quøte'd.xml" (space + sub-delim + non-ASCII): store -> list shows decoded -> get by decoded path returns content -> delete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The existing db.cy.js tests use ASCII names only, so they pass but never exercise the encode-on-input / decode-on-output boundary in db.xqm. Add an awkward-names block: store, read, list, and remove resources named "café déjà.xml" (non-ASCII + space) and "o'brien.xml" (sub-delim apostrophe, which xmldb:store leaves literal) — all addressed by their DECODED name. Assert the read echoes the decoded path, content is intact, and the listing shows decoded names (no %XX), confirming the round trip. Verified on a live instance (existdb-openapi on an ft:fields bed): both names store and round-trip with decoded path/content and decoded listing names. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

db:to-display decoded names with xmldb:decode-uri, which form-decodes "+" to a space (the x-www-form-urlencoded convention; eXist-db/exist#1824). But a "+" in a stored name is always a literal "+" -- spaces are stored as %20 -- and db:to-stored (fn:iri-to-uri) leaves "+" untouched on the encode side, so a name like "naïve+test.xml" stored correctly but read back as "naïve test.xml". Protect a literal "+" as %2B before xmldb:decode-uri so it decodes back to "+", restoring symmetry with the encode side. This mirrors what URIUtils.decodeForURI (the core fix in eXist-db/exist#6451) does, applied at the API layer so it is correct independent of the core build, and forward-compatible once #6451 lands. Spaces (%20) are unaffected. Verified end-to-end against a live instance: "naïve+test.xml" stores as na%C3%AFve+test.xml on disk, lists and reads back as naïve+test.xml. Adds the "+" case to the Cypress awkward-name coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

line-o

As I understand the current situation we are working around a limitation or issue in xmldb-module.

Would it be feasible to tackle it there? Since we are in the process of releasing a major version of exist-db this might be a good opportunity.

joewiz · 2026-06-15T19:15:14Z

[This response was co-authored with Claude Code. -Joe]

You're exactly right, the root fix belongs in core, not in each API that wraps it. This PR is a boundary workaround until the core is fixed: db:to-stored / db:to-display translate between the wire's decoded UTF-8 and eXist's percent-encoded storage at the edge of every handler, purely because doc() / collection() / xmldb:* don't agree on a decoded form today.

The path to that root fix is laid out in eXist-db/exist#6463 — "Define eXist's resource-naming contract: name vs URI (5 decisions)". The issue lays out five questions needing maintainer agreement (with recommended answers) that would unblock the implementation across REST / WebDAV / XML-RPC / existdb-openapi and core's own xmldb:store vs doc(). Some of it has already started landing piecemeal. This PR's decode side mirrors URIUtils.decodeForURI from eXist-db/exist#6451.

So this PR is an interim boundary fix, ensuring that existdb-openapi speaks correct decoded UTF-8, until eXist-db/exist#6463's contract is decided and implemented core-side. Once core's doc() / xmldb:* speak the decided form, existdb-openapi's db-core simply inherits it and the db:to-stored / db:to-display helpers will be removed. I'll track that retirement so the workaround doesn't outlive its purpose. The Cypress round-trip tests (café-ünïcode.xml, quote'name.xml) stay either way, as regression cover for the wire contract.

Would value your take on eXist-db/exist#6463's five decisions — that's where the durable fix gets settled.

Split modules/db.xqm into a roaster-independent core (db-core.xqm) and a thin roaster wrapper, so the same db-resource CRUD + naming-correctness implementation can be shared in-process by other apps (e.g. eXide) without an HTTP hop or re-auth. db-core.xqm holds all the logic: list / get-resource / store / create-collection / remove-resource / remove-collection / move / copy / properties / set-permissions / sync / modules. Functions take a wire path plus an options map and return plain maps; failures are signalled as typed errors in the http://exist-db.org/api/db-core/error namespace (bad-request / not-found / forbidden / conflict / server-error) rather than HTTP responses. db:to-stored / db:to-display (the iri-to-uri encode/decode boundary, incl. the literal-"+" guard) now live here as db-core:to-stored / db-core:to-display — the single home for naming correctness. db.xqm is now purely HTTP framing: each handler unpacks the roaster $request, calls db-core, and maps a typed error to a status via a small error-response helper. Store's 201-vs-200 (new vs existing) is conveyed by a `created` flag the wrapper strips before responding; create-collection keeps its 201. No api.json or route changes — behavior-preserving. Verified against a live eXist: all 46 db Cypress tests pass, including the awkward-name (café / o'brien / naïve+test) encode/decode coverage from eXist-db#54, and the 400/403/404/409 error mappings match the pre-refactor responses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

duncdrum

as the core issue doesn't see much traction, I m in favour of merging this.

Split modules/db.xqm into a roaster-independent core (db-core.xqm) and a thin roaster wrapper, so the same db-resource CRUD + naming-correctness implementation can be shared in-process by other apps (e.g. eXide) without an HTTP hop or re-auth. db-core.xqm holds all the logic: list / get-resource / store / create-collection / remove-resource / remove-collection / move / copy / properties / set-permissions / sync / modules. Functions take a wire path plus an options map and return plain maps; failures are signalled as typed errors in the http://exist-db.org/api/db-core/error namespace (bad-request / not-found / forbidden / conflict / server-error) rather than HTTP responses. db:to-stored / db:to-display (the iri-to-uri encode/decode boundary, incl. the literal-"+" guard) now live here as db-core:to-stored / db-core:to-display — the single home for naming correctness. db.xqm is now purely HTTP framing: each handler unpacks the roaster $request, calls db-core, and maps a typed error to a status via a small error-response helper. Store's 201-vs-200 (new vs existing) is conveyed by a `created` flag the wrapper strips before responding; create-collection keeps its 201. No api.json or route changes — behavior-preserving. Verified against a live eXist: all 46 db Cypress tests pass, including the awkward-name (café / o'brien / naïve+test) encode/decode coverage from eXist-db#54, and the 400/403/404/409 error mappings match the pre-refactor responses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ource Retire the JSON envelope and the path-in-URL endpoints; all resource content now flows through the query-param /api/db/resource, identical in shape to its db siblings. - GET /api/db/resource?path= → raw content: binary streamed as-is; XML/text serialized from the node tree (eXist has no raw byte form for XML). Content-Type is the stored mime. download=true → Content-Disposition: attachment. - PUT /api/db/resource?path= → raw request body (binary-safe); mime from Content-Type or inferred from the name; returns { path } (201/200) so the caller can reconcile name normalization. - Serialization params: full W3C vocabulary + eXist extensions via the output: namespace (eXist-db/exist#6447 — expand-xincludes, highlight-matches, add-exist-id, process-xsl-pi, jsonp, insert-final-newline). Unsupported params on a pre-#6447 eXist return a clean 400. - Removed: JSON envelope, meta=full + X-Resource-* headers, runPath (derivable client-side), the /api/resource/{path} and /api/db/resource/{path} endpoints, and dbc:get-run-path. - Metadata stays at GET /api/db/properties. Supersedes eXist-db#38, eXist-db#56; folds + extends eXist-db#48. Depends on eXist-db#54, eXist-db#55. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Split modules/db.xqm into a roaster-independent core (db-core.xqm) and a thin roaster wrapper, so the same db-resource CRUD + naming-correctness implementation can be shared in-process by other apps (e.g. eXide) without an HTTP hop or re-auth. db-core.xqm holds all the logic: list / get-resource / store / create-collection / remove-resource / remove-collection / move / copy / properties / set-permissions / sync / modules. Functions take a wire path plus an options map and return plain maps; failures are signalled as typed errors in the http://exist-db.org/api/db-core/error namespace (bad-request / not-found / forbidden / conflict / server-error) rather than HTTP responses. db:to-stored / db:to-display (the iri-to-uri encode/decode boundary, incl. the literal-"+" guard) now live here as db-core:to-stored / db-core:to-display — the single home for naming correctness. db.xqm is now purely HTTP framing: each handler unpacks the roaster $request, calls db-core, and maps a typed error to a status via a small error-response helper. Store's 201-vs-200 (new vs existing) is conveyed by a `created` flag the wrapper strips before responding; create-collection keeps its 201. No api.json or route changes — behavior-preserving. Verified against a live eXist: all 46 db Cypress tests pass, including the awkward-name (café / o'brien / naïve+test) encode/decode coverage from #54, and the 400/403/404/409 error mappings match the pre-refactor responses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ource Retire the JSON envelope and the path-in-URL endpoints; all resource content now flows through the query-param /api/db/resource, identical in shape to its db siblings. - GET /api/db/resource?path= → raw content: binary streamed as-is; XML/text serialized from the node tree (eXist has no raw byte form for XML). Content-Type is the stored mime. download=true → Content-Disposition: attachment. - PUT /api/db/resource?path= → raw request body (binary-safe); mime from Content-Type or inferred from the name; returns { path } (201/200) so the caller can reconcile name normalization. - Serialization params: full W3C vocabulary + eXist extensions via the output: namespace (eXist-db/exist#6447 — expand-xincludes, highlight-matches, add-exist-id, process-xsl-pi, jsonp, insert-final-newline). Unsupported params on a pre-#6447 eXist return a clean 400. - Removed: JSON envelope, meta=full + X-Resource-* headers, runPath (derivable client-side), the /api/resource/{path} and /api/db/resource/{path} endpoints, and dbc:get-run-path. - Metadata stays at GET /api/db/properties. Supersedes eXist-db#38, eXist-db#56; folds + extends eXist-db#48. Depends on eXist-db#54, eXist-db#55. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

joewiz and others added 3 commits June 9, 2026 20:22

line-o requested review from a team and line-o June 15, 2026 09:28

line-o reviewed Jun 15, 2026

View reviewed changes

line-o requested a review from a team June 15, 2026 09:33

duncdrum approved these changes Jun 18, 2026

View reviewed changes

duncdrum merged commit 52df653 into eXist-db:develop Jun 18, 2026
1 check passed

joewiz deleted the fix/resource-name-encoding branch July 13, 2026 17:16

joewiz mentioned this pull request Jul 14, 2026

Adopt Unicode-native resource and collection names: remove percent-encoding from the storage layer (7.0) eXist-db/exist#6575

Open

Navigation Menu

Uh oh!

[bugfix] Speak decoded UTF-8 resource names on the wire - #54

[bugfix] Speak decoded UTF-8 resource names on the wire#54
duncdrum merged 3 commits into
eXist-db:developfrom
joewiz:fix/resource-name-encoding

joewiz commented Jun 10, 2026 •

edited

Loading

Uh oh!

line-o left a comment

Uh oh!

joewiz commented Jun 15, 2026 •

edited

Loading

Uh oh!

duncdrum left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

joewiz commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Why fn:iri-to-uri and not xmldb:encode

Test plan

Known boundary (out of scope, documented)

Uh oh!

line-o left a comment

Choose a reason for hiding this comment

Uh oh!

joewiz commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

duncdrum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joewiz commented Jun 10, 2026 •

edited

Loading

Why `fn:iri-to-uri` and not `xmldb:encode`

joewiz commented Jun 15, 2026 •

edited

Loading