diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml index 5d06dc51..79dacb3e 100644 --- a/.github/workflows/main.yaml +++ b/.github/workflows/main.yaml @@ -70,6 +70,7 @@ jobs: pytest tests/test_manifests.py --tests=./specifications/json-ld-framing/tests --loader=${{ matrix.loader }} pytest tests/test_manifests.py --tests=./specifications/normalization/tests --loader=${{ matrix.loader }} pytest tests/test_manifests.py --tests=./specifications/rdf-canon/tests --loader=${{ matrix.loader }} + pytest tests/test_manifests.py --tests=./specifications/yaml-ld/tests --loader=${{ matrix.loader }} pytest --ignore ./tests/test_manifests.py env: LOADER: ${{ matrix.loader }} @@ -106,4 +107,4 @@ jobs: uses: MishaKav/pytest-coverage-comment@ae0e8a539a3f310aefb3bfb6a2209778a21fa42b with: pytest-coverage-path: ./pytest-coverage.txt - junitxml-path: ./pytest.xml \ No newline at end of file + junitxml-path: ./pytest.xml diff --git a/.gitignore b/.gitignore index 820ee873..1bed7b5b 100644 --- a/.gitignore +++ b/.gitignore @@ -19,3 +19,9 @@ tests/data/test_caching.json # Local version file for pyenv .python-version + +# Local lock file for uv +uv.lock + +# Codex CLI +.codex diff --git a/.gitmodules b/.gitmodules index 87e7ce95..824bc216 100644 --- a/.gitmodules +++ b/.gitmodules @@ -10,3 +10,6 @@ [submodule "specifications/rdf-canon"] path = specifications/rdf-canon url = https://github.com/w3c/rdf-canon.git +[submodule "specifications/yaml-ld"] + path = specifications/yaml-ld + url = https://github.com/w3c/yaml-ld.git diff --git a/AGENTS.md b/AGENTS.md index fd178e3a..07063a07 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,7 +29,7 @@ Read [CONTRIBUTING.md](CONTRIBUTING.md) for code style, linting (e.g. `make lint ### Documentation validation - **No in-repo Playwright.** Do not add `@playwright/test`, `playwright.config.js`, or e2e test dependencies. Live browser checks use **only** the Playwright MCP server (`user-playwright`). -- After doc changes, run `make docs-build` (strict). For interactive checks, run `make docs-serve` and validate with Playwright MCP: `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_wait_for`. Prefer `browser_run_code_unsafe` with `page.screenshot({ animations: 'disabled', timeout: 60000 })` over `browser_take_screenshot` (font load timeouts). +- After doc changes, run `make docs-build` (strict). Before launching a preview server, first use Playwright MCP to check the configured local docs URL (normally `http://127.0.0.1:8000/pyld/`) and reuse it when it is responsive; run `make docs-serve` only when no responsive preview server is available. Validate interactive checks with Playwright MCP: `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_wait_for`. Prefer `browser_run_code_unsafe` with `page.screenshot({ animations: 'disabled', timeout: 60000 })` over `browser_take_screenshot` (font load timeouts). ## Committing diff --git a/README.md b/README.md index fca45e7f..805f2f59 100644 --- a/README.md +++ b/README.md @@ -388,13 +388,14 @@ git submodule update #### Cloning manually You can also avoid using git submodules by manually cloning the `json-ld-api`, -`json-ld-framing`, and `normalization` repositories hosted on GitHub using the -following commands: +`json-ld-framing`, `normalization`, and `yaml-ld` repositories hosted on GitHub +using the following commands: ```bash git clone https://github.com/w3c/json-ld-api ./specifications/json-ld-api git clone https://github.com/w3c/json-ld-framing ./specifications/json-ld-framing git clone https://github.com/json-ld/normalization ./specifications/normalization +git clone https://github.com/w3c/yaml-ld ./specifications/yaml-ld ``` Note that you can clone these repositories into any location you wish; however, diff --git a/docs/project/decisions/choose-where-to-host-yaml-ld-support/implement-yaml-ld-in-pyld.md b/docs/project/decisions/choose-where-to-host-yaml-ld-support/implement-yaml-ld-in-pyld.md new file mode 100644 index 00000000..6e3b56ad --- /dev/null +++ b/docs/project/decisions/choose-where-to-host-yaml-ld-support/implement-yaml-ld-in-pyld.md @@ -0,0 +1,112 @@ +# :material-package-variant: Implement YAML-LD in PyLD + +!!! info "Conditional roadmap" + + This is the implementation universe in which [:fontawesome-brands-github: `digitalbazaar/pyld`](https://github.com/digitalbazaar/pyld) owns YAML-LD support. It applies only if the parent decision selects this alternative. + +## :material-target: Delivered contract + +`PyLD[yaml-ld]` adds YAML-LD document loading to the existing PyLD API; `PyLD[cli,yaml-ld]` adds that capability to the CLI. The implementation supports the YAML-LD JSON profile only: YAML 1.2 Core-schema values that produce JSON values. It does not implement the YAML-LD extended profile: `processingMode='yaml-ld-extended'`, or a profile token exactly `http://www.w3.org/ns/json-ld#extended`, fails with `jsonld.LoadDocumentError`, code `profile-error`. Ignore unknown profile tokens; do not reject an arbitrary `profile` parameter. YAML tags outside the Core schema are discarded when constructing the JSON representation, as required by the JSON profile. A missing `ruamel.yaml` import fails only when a YAML document is actually parsed, with an actionable `yaml-ld` extra message. + +Raw YAML `str` and `bytes` are not new public inputs to `expand`, `compact`, `flatten`, `frame`, `to_rdf`, or `normalize`: a string remains a document URL and a mapping/list remains an already-parsed JSON-compatible document. YAML enters those APIs through a document URL and its document loader. `from_rdf` remains raw RDF input only. + +Do not add `pyld.yaml_ld.expand()` or sibling transformation functions. A +`pyld.yaml_ld` namespace may expose parsing helpers such as `loads()` and +`loads_all()`, while `pyld.jsonld.*` remains the sole transformation API. + +## :material-source-branch: Implementation baseline + +Build this work on [:fontawesome-brands-github: `306-complete-pyld-cli`](https://github.com/digitalbazaar/pyld/tree/306-complete-pyld-cli), which adds [`lib/pyld/cli/input.py`](https://github.com/digitalbazaar/pyld/blob/306-complete-pyld-cli/lib/pyld/cli/input.py) and the command test contract. Either rebase after that branch merges, or create this as a stacked PR with `306-complete-pyld-cli` as its base. Do not reimplement the CLI in this change. + +## :material-clipboard-check-outline: Architectural decisions + +### One media parser at the PyLD loading boundary + +Add `lib/pyld/documentloader/media.py`, with a single idempotent entry point used by [`jsonld.load_document`](https://github.com/digitalbazaar/pyld/blob/master/lib/pyld/jsonld.py): `parse_remote_document(remote_doc, options, profile) -> RemoteDocument`. + +1. Normalize `contentType` by lowercasing the media type and removing parameters for dispatch, while retaining the parameter map to inspect `profile`. Recognize `application/ld+yaml`, `application/yaml`, `application/x-yaml`, and `*+yaml`; recognize JSON as `application/ld+json`, `application/json`, and `*+json`; retain the existing HTML types. +2. If `remote_doc['document']` is a mapping or list, return it unchanged. This makes the boundary idempotent and keeps custom loaders which already return parsed JSON compatible. +3. Built-in loaders always supply raw bytes. A custom loader may supply text, which the boundary first encodes as UTF-8 and then decodes using the same optional-BOM path; an unencodable string or malformed byte sequence is `invalid-encoding`. Any other value is a `loading document failed` error. Dispatch YAML and JSON to the media parser; dispatch HTML to the PyLD HTML folding algorithm below. The parser replaces `remote_doc['document']` with only ordinary Python `dict`, `list`, `str`, `int`, `float`, `bool`, or `None` values. +4. Call this helper exactly once, immediately after `options['documentLoader'](url, options)` in `load_document`, before the null-document check and before processing API code observes the result. It owns parsing, HTML traversal, base handling, profile selection, and fragment selection. Built-in and custom loader output therefore follows the same path. + +Change [`FileDocumentLoader`](https://github.com/digitalbazaar/pyld/blob/master/lib/pyld/documentloader/file.py), [`RequestsDocumentLoader`](https://github.com/digitalbazaar/pyld/blob/master/lib/pyld/documentloader/requests.py), and [`AioHttpDocumentLoader`](https://github.com/digitalbazaar/pyld/blob/master/lib/pyld/documentloader/aiohttp.py) to return raw bytes in `document`, plus `contentType`, `contextUrl`, and final `documentUrl`. In particular, replace Requests’ `response.json()` and aiohttp’s `response.json(content_type=None)` with raw-body reads. `SqliteCacheRequestsDocumentLoader` continues to inherit the Requests behavior. Do not change the documented `RemoteDocument` fields or require third-party loaders to change: their parsed mapping/list output is the idempotent case. + +### YAML parsing and errors + +Use `ruamel.yaml >=0.19`, configured for YAML 1.2 and the Core schema. Decode once, safely compose/load every document in a stream, and recursively convert it to JSON values: keys must be strings; aliases are copied; undefined aliases and cyclic aliases fail; `.inf` and `.nan` are rejected because they are not JSON numbers. A top-level scalar fails and an empty stream fails. With `extractAllScripts=False`, a YAML stream returns its first document; with `extractAllScripts=True`, it returns an array of all stream documents, including an array of one document. This rule applies equally to direct YAML and YAML script bodies. + +Production errors use the current YAML-LD vocabulary: decode errors are `invalid-encoding`; non-string mapping keys are `mapping-key-error`; and the extended profile is `profile-error`. Syntax errors, scalar roots, empty streams, undefined/cyclic aliases, non-JSON floats, and unsupported document shapes are `loading document failed`. JSON is parsed only by the JSON dispatch branch and YAML only by YAML dispatch: do not describe or implement a “valid JSON but invalid YAML” fallback. + +### HTTP, files, links, and negotiation + +Extend `CONTENT_TYPES` with `.yamlld` → `application/ld+yaml` and `.yaml` → `application/yaml`. The exact default `Accept` header, unless the caller supplied `options['headers']`, is: + +```text +application/ld+yaml, application/ld+json;q=0.9, application/yaml;q=0.8, application/x-yaml;q=0.8, application/json;q=0.7, text/html;q=0.5, application/xhtml+xml;q=0.5 +``` + +This intentional YAML-first ordering implements the YAML-LD alternative’s preference; test the literal value in both HTTP loaders. When `requestProfile` is set, prepend the RFC-quoted `application/ld+json;profile="", `, preserving the remaining order. + +Normalize a response `Content-Type` before link decisions. Continue to reject multiple JSON-LD context links. A `rel=alternate` is followed only if its normalized `type` is a supported YAML or JSON media type and the response is neither a supported YAML nor JSON media type; resolve it against the original request URL and recurse through the same loader with the same options. A context link is retained only when the response is neither `application/ld+json` nor `application/ld+yaml`; do not synthesize a context link from a YAML document. Preserve final URLs, redirect behavior, secure-mode enforcement, supplied headers, cache keys, and cache hit/miss behavior. + +### HTML folding, owned by PyLD + +`parse_remote_document` calls a refactored `load_html` that takes the raw HTML text, `documentUrl`, `profile`, and `options`; no transport loader parses HTML or script bodies. + +1. Parse HTML and resolve the first `` against `options['base']` or `documentUrl`; store the resolved base in `options['base']` and replace the remote document URL as current `load_document` does. +2. If `documentUrl` has a fragment, select exactly the script whose `id` equals that fragment. It must have a supported JSON or YAML media type; otherwise raise `loading document failed`. Ignore `extractAllScripts` in this case and return only that script’s parsed document(s). +3. Otherwise select `