feat(db): path-in-URL streaming endpoint /api/db/resource/{path} (closes #35)

[joewiz]

[This PR was co-authored with Claude Code. -Joe]

Closes #35.

Summary

Adds a path-in-URL sibling endpoint at /api/db/resource/{path} that streams via eXist's REST servlet, complementing the existing JSON-envelope endpoint:

URL shape	Use for	Behavior
PUT /api/db/resource (path in JSON body)	Text + metadata in a single roundtrip	JSON envelope; Roaster handler; body buffered (tens-of-MB ceiling)
PUT /api/db/resource/{path} (path in URL)	Binary or any large upload	Raw bytes, mime from Content-Type header — streams via /exist/rest

Same URL prefix (/exist/apps/existdb-openapi/api/db/resource…), no web.xml changes, no second URL space. controller.xq routes the path-in-URL variant to /exist/rest/{path} via <forward url="/rest{$path}" absolute="yes"/>. The absolute="yes" flag escapes the controller context so the URL resolves to the servlet root rather than /db/apps/existdb-openapi/rest/….

Why this shape

The original PR design (base64 inside the JSON envelope) was rethought after surveying peer systems:

System	Wire format for binary	Same endpoint as text?
eXist WebDAV (Jackrabbit)	Raw bytes over HTTP, streamed	Yes
eXist XML-RPC	base64 inside XML-RPC envelope (wire-format quirk)	No (separate methods)
eXide (/api/storage)	Raw bytes via fetch(url, {body: <File>})	Yes
xst CLI	XML-RPC (base64 in envelope)	n/a
atom-editor-support	Raw bytes via request:get-data()	Yes
fusion-studio-api	Raw bytes via RESTXQ %rest:PUT("{$body}")	Yes

Five of six peers use raw bytes over HTTP for binary; nobody uses base64-in-JSON. The path-in-URL forward matches the eXide / fusion-studio-api / WebDAV idiom (path-in-URL, raw bytes, mime from Content-Type) while keeping the JSON envelope available at the bare URL for text-with-metadata.

A second concern that tipped this design: Roaster's request-body pipeline always materializes the body in JVM memory (see roaster:content/body.xqm's body:parse — every Content-Type branch ends up reading the full body into XQuery values before the handler runs), so the JSON-envelope endpoint inherits a memory ceiling that grows with file size. A path-in-URL forward to /exist/rest sidesteps Roaster entirely and streams through eXist's RESTServer.java, where the body becomes an InputStream that broker.storeDocument(InputSource) consumes directly — no JVM buffer.

controller.xq dispatch

else if (matches($exist:path, "^/+api/db/resource/.+")) then
    let $db-path := replace($exist:path, "^/+api/db/resource", "")
    return
        <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
            <forward url="/rest{$db-path}" absolute="yes">
                <set-header name="Access-Control-Allow-Origin" value="*"/>
                <set-header name="Access-Control-Allow-Methods" value="GET, PUT, DELETE, HEAD, OPTIONS"/>
                <set-header name="Access-Control-Allow-Headers" value="Content-Type, Authorization"/>
            </forward>
        </dispatch>

The discovery that made this work: eXist URL-rewriting docs explicitly note that absolute="yes" resolves the url against the servlet context root rather than the controller context. Without it, /rest/... resolves to /db/apps/existdb-openapi/rest/... and the forward silently misfires (returns 201 but the request never reaches EXistServlet — an interesting URLRewrite framework quirk I hit while developing this).

End-to-end verification

Manual curl with a 68-byte transparent PNG:

$ curl -u admin: -X PUT -H "Content-Type: image/png" --data-binary @pixel.png \
    http://localhost:8080/exist/apps/existdb-openapi/api/db/resource/db/pixel.png
HTTP/1.1 201

$ curl -u admin: http://localhost:8080/exist/apps/existdb-openapi/api/db/properties?path=/db/pixel.png
{"mime-type": "image/png", "size": 68, ...}

$ curl -u admin: http://localhost:8080/exist/apps/existdb-openapi/api/db/resource/db/pixel.png -o back.png
$ diff pixel.png back.png      # ✓ byte-identical
$ xxd -l 8 back.png             # 89 50 4e 47 0d 0a 1a 0a  ← canonical PNG signature

DELETE round-trips too:

$ curl -u admin: -X DELETE http://localhost:8080/exist/apps/existdb-openapi/api/db/resource/db/pixel.png
HTTP/1.1 200
$ curl -u admin: http://localhost:8080/exist/apps/existdb-openapi/api/db/properties?path=/db/pixel.png
HTTP/1.1 404

Cypress

5 new tests in db.cy.js for the path-in-URL endpoint covering PUT / GET / DELETE routing with text content. The controller.xq forward logic is body-type-agnostic; text exercises the same routing path as binary. (cy.request can't cleanly send raw binary bytes — Cypress's Buffer ↔ JSON serialization mangles them — so the binary round-trip is verified via the curl recipe above. Considered using cy.task to invoke node-side HTTP for the binary case but the marginal CI value didn't justify the plugin overhead.) Full cypress suite: 110/110 passing.

OpenAPI spec

/api/db/resource/{path} documented as a sibling endpoint with PUT/GET/DELETE. The request/response bodies are described as application/octet-stream (type: string, format: binary) per OpenAPI conventions. The bare /api/db/resource endpoint's description gained a note pointing at the path-in-URL variant for binary uploads.

Out of scope / known limits

• Roaster's request-body buffering ceiling still applies to the JSON-envelope endpoint. Clients hitting that ceiling for large text uploads should also use the path-in-URL variant (with Content-Type: text/plain or appropriate).
• The JSON envelope GET on a binary resource is still lossy — util:binary-to-string interprets bytes as UTF-8 and mangles anything non-UTF-8. Recommendation in the OpenAPI description: use the path-in-URL variant for binary GET.
• Multipart upload is not added; clients that want to upload many resources at once should make multiple requests.

Dependency on #34

This PR is branched off #34 (fix/store-resource-mime-auto-detect). The two PRs share modules/db.xqm but the path-in-URL endpoint is implemented entirely in controller.xq (no db.xqm changes specific to this PR beyond the comment update). Merge #34 first → this rebases cleanly.

Considered and rejected: wrapping REST's response in a JSON envelope via <view>

A natural follow-up question: could a <forward> + <view> pipeline normalize REST's minimal response (empty body, just an HTTP status) into the same {stored, runPath} / {removed} / {error} JSON envelope the bare /api/db/resource handler returns? The eXist URL-rewriting framework supports exactly this kind of two-stage chain (the documentation app and eXide both use it for HTML templating).

Tried it on a side branch. The pattern composes cleanly when the first forward targets a DB-backed XQuery — the framework already knows how to dispatch through to a view handler in that case. But our first forward targets a servlet (EXistServlet for streaming) which lives outside the URL-rewriting tree, and the view's subsequent <forward url="modules/wrap-rest-response.xq"/> fails to resolve cleanly:

• absolute path ({$exist:controller}/modules/wrap-rest-response.xq) → 500 with NPE in XQueryServlet.process line 358 (Paths.get(path) with null path — neither xquery.source nor xquery.url attribute is set, so XQueryServlet falls through to the filesystem-path branch)
• relative path (modules/wrap-rest-response.xq) → 400 with empty body; the framework resolves relative paths against the current request path, which after the rest forward is /exist/rest/db/..., so the lookup never finds the XQuery

There's likely a way to make it work (manually setting xquery.source or xquery.url attributes on the view forward, or routing through a different servlet mapping) but it would be reverse-engineering framework internals to glue together what isn't designed to chain. The cost outweighs the benefit, given that REST's HTTP status codes already convey what clients need to know and the peer survey shows no precedent for envelope-wrapping binary responses (none of WebDAV, XML-RPC, eXide, xst, atom-editor-support, or fusion-studio-api do this).

If we ever decide the response envelope matters enough to justify the work, the cleaner path would be a thin Java handler that takes raw bytes via servlet InputStream and emits JSON directly — but that requires a web.xml change, which would prevent existdb-openapi from being a self-contained XAR install (the same reason we rejected the "bypass roaster with a Java servlet" option earlier in the binary-transport discussion).

[line-o]

I would rather like to see us come up with a way to serve binaries from roaster efficiently.

If this is done within the controller I would want to avoid to have it within the api path.

[joewiz]

Agreed that roaster would be the best option. An analysis of options is underway.

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Agreed on both points — the Roaster-native path is the right north star, and parking a /rest-forward under /api/* does muddy the contract. Let me lay out what I'm seeing in Roaster as the actual blockers, in case it's useful for shaping the work.

The two ends are blocked by different things

Upload side, roaster/content/body.xqm:29-73. Every branch of body:parse() ends in request:get-data() (or request:get-uploaded-file-data() for multipart) — all of them materialize the full body into a JVM value (xs:base64Binary, parsed JSON map, XML tree) before the handler runs at router.xql:288. So even a handler that wanted to stream to xmldb:store-as-binary-resource() can't — by the time it runs, the body is already on the heap. The XQuery request:* API simply doesn't expose an InputStream handle. Nothing in Roaster can fix that on its own.

Download side, router.xql:454-477. router:write-response() returns $response?($router:RESPONSE_BODY) as the final expression, which eXist then runs through the XQuery serializer based on output:method. For binary that means the body must be a complete value already in memory. eXist does have response:stream-binary(), but that still takes a fully-materialized xs:base64Binary — there's no zero-copy "stream from stored BinaryDocument to HTTP response" path callable from XQuery today.

What I think Roaster would need

Three layers, roughly:

1. Opt-out from body:parse(). An OpenAPI extension like x-roaster-raw-body: true on the operation tells Roaster to skip parsing and hand the handler a stream handle in $request?body-raw instead. This is the easy part — Roaster-side change is ~15 lines. But it's only useful once initial release #2 exists.

2. An eXist-level streaming primitive. Either request:get-input-stream() returning a usable handle (for uploads) or response:stream-binary-resource($path, $media-type) that opens the BinaryDocument and copies bytes to the response OutputStream without an intermediate XQuery value (for downloads). These belong in exist-core, not Roaster — Roaster just exposes them through the route contract.

3. A "dispatch to servlet" passthrough. The deepest version: an OpenAPI extension like x-roaster-passthrough: { servlet: "exist", path: "/rest{path}" } that lets Roaster honor the spec for routing/validation/docs but hand the actual byte handling off to EXistServlet without ever crossing the XQuery boundary. This is basically what RESTServer.java already does well — Roaster would just route to it. It's also what the controller workaround in this PR is achieving externally; lifting it into Roaster proper would make it a first-class spec-driven feature instead of a per-app side door.

(1) gives the API surface, (2) gives it teeth, (3) is the "do what /exist/rest does without a controller" version. They can land independently — (1) alone is harmless, (2) alone is useful even for non-Roaster apps, (3) is the eventual destination.

On /api/* specifically

Hear you on the path concern. If we keep #38 as a controller workaround in the interim, I'd move the path-in-URL variant out of /api/* — something like /files/{path} under the app root — so /api stays Roaster's. Want me to revise the PR along those lines, or hold it pending the Roaster work above?

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Surveyed the five neighboring systems for prior art on this — the convergence is striking and points at why a Roaster-side fix alone won't fully close the gap.

Two patterns exist, nothing in between

Pattern A — Java-servlet-direct, fully streaming, never crosses XQuery:

• eXist REST (RESTServer.java) — PUT is request.getInputStream() → CachingFilterInputStream → broker.storeDocument(InputSource). Binary GET is broker.readBinaryResource(BinaryDocument, response.getOutputStream()) — direct byte copy from internal store to the response. No XQuery layer involved either direction.
• eXist WebDAV — pure Java (Milton on eXist 6, Jackrabbit on eXist 7). Streaming by definition; never touches XQuery.
• BaseX REST — docs explicitly call out application/octet-stream PUT as curl -T file … with raw bytes; servlet-direct in the same fashion.

Pattern B — XQuery-value binding, materialized to xs:base64Binary or equivalent:

• eXist RESTXQ (HttpServletRequestAdapter.java) — body bound via %rest:POST("{$body}"). Backed by CachingFilterInputStreamCache, so large bodies disk-spill rather than blowing JVM heap, but the XQuery handler still sees an xs:base64Binary value (just disk-backed instead of heap-backed).
• BaseX RESTXQ — %rest:POST("{$body}") with $body as xs:base64Binary. Docs: unrecognized content types "default to xs:base64Binary." Multipart parts "are represented as a sequence, and each part is converted to an XQuery item." Full materialization.
• eXist XML-RPC — base64 inside the XML-RPC envelope. Worst case: whole body in heap plus base64 inflation tax.
• Roaster — same as Pattern B; one rung worse than eXist RESTXQ because Roaster's body:parse() reaches request:get-data() for every content type, without the caching layer.

Nothing in any of these surveyed systems offers a spec-driven binding (annotation-based or OpenAPI-based) that also exposes a raw InputStream to the handler. The choice everywhere has been: servlet-direct = streaming + no spec, or spec-driven = value materialization. The two haven't been bridged yet, in any of them.

BaseX Request module specifically

No streaming hook there either — full list of body-adjacent functions is request:parameter*, request:header*, request:cookie*, request:attribute*, request:method, plus URI/connection accessors. No request:get-input-stream or equivalent. Same gap as eXist's request:*.

What that means for the Roaster work

A genuine free improvement Roaster can grab today: swap body:parse()'s request:get-data() for the CachingFilterInputStream pattern eXist RESTXQ uses. Same $request?body contract for handlers, but large bodies disk-spill instead of OOM-ing. Roaster-only change, no exist-core dependency.

The bigger win — handler sees a real InputStream and can pipe directly to broker.storeDocument() without an intermediate value — needs the eXist-side request:get-input-stream() primitive I mentioned in the previous reply, and then a thin Roaster opt-out (x-roaster-raw-body: true) to surface it. That would put Roaster ahead of both RESTXQ implementations on this axis, since neither offers the InputStream escape hatch.

If both of those land, the x-roaster-passthrough idea (route directly to EXistServlet, skip XQuery entirely) becomes less urgent — though still potentially worth having for cases where you want to keep an OpenAPI-documented surface but defer 100% of byte handling to existing servlet code.

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Closing as superseded by #59, which consolidates all resource-content transport onto a single binary-safe /api/db/resource and retires the path-in-URL /api/db/resource/{path} shape introduced here — the outcome of the design review captured in #59. Planned merge order: #54 → #55 → #59.