Repository navigation
ZEP 12: explicit array bounds (draft) - #72
rhoadesScholar wants to merge 6 commits into
Conversation
Adds `bounds` as an alternative to `shape` and an optional `origin` on the regular chunk grid, so an array can grow before its first element with a metadata write. Companion spec text goes to zarr-specs.
Backward Compatibility: a reader that predates the ZEP refuses `bounds` on two grounds (unknown member, missing `shape`); name neuroglancer as the reader that ignores unknown members; argue the 3.2 minor bump from the stability policy's own example; add writer guidance (explicit opt-in to the first prepend, keep `shape` where possible, `origin` is permanent). Alternatives: state normanrz's single-field form as he proposed it, with his arguments, and say the working group has not chosen. `origin`: MUST NOT be present without `bounds`, readers MUST refuse `shape` + `origin`; give it its own justification (jbms's two uses) and say it is severable; drop the zero `origin` from the example. Also: sharding and signed-index sentences, benchmark caption limited to what was measured (NVMe, zarr-python 3.4.1; 256 KiB chunks), one read site per prototype, zarrs name collision, d-v-b in the core-preference list, citations for zarr-indexing and OME-NGFF, v2 encoding in the references, duplicated paragraphs cut.
zarr-specs#400 carries the spec text; zarr-extensions#79 stays listed under Discussion as the thread the proposal came from.
Sharding: an inner chunk outside the bounds is marked empty in the shard index, never absent, and the index covers every inner chunk; "array shape" in the sharding codec is read as the bounds. "Implementation" for the refusal rule, matching the core. Discussion URL in angle brackets as the other ZEPs do. Two Alternatives bullets shortened to pointers at the arguments made earlier.
| - an optional `origin` in the `regular` chunk grid configuration, and negative chunk grid indices; | ||
| - how the `default` and `v2` chunk key encodings write a negative index. | ||
|
|
||
| Out of scope: the rectilinear chunk grid, which is a registered extension and would gain the |
There was a problem hiding this comment.
Something to consider is whether an origin could be applied as a chunk grid that layers over other another chunk grid. Similar in spirit to zarr-developers/zarr-extensions#28
"chunk_grid": {
"name": "offset",
"configuration": {
"offset": [100, 100, 100],
"base_chunk_grid": {
"name": "regular",
"configuration": {"chunk_shape": [10, 10, 10]}
}
}
}That is a lot of metadata, but then adjusting the origin of a grid is generalised. The rectilinear or any other chunk grid for that matter wouldn't even need special metadata or code to support negative chunks if the offset was used appropriately alongside array bounds.
I can think of more chunk grids that don't fit under a regular / rectangular mode, so I think it is good to think about generalising herel. E.g. I've been thinking of something like this zarrs/zarrs#453
There was a problem hiding this comment.
I'd be fine with that. It takes origin out of the core change entirely, so this ZEP would be just bounds and negative grid indices, and the wrapper goes through the registry on its own clock, rectilinear included (this draft punts on it). The price is one more level of nesting on every array that uses it, which the two-field version from #79 avoids. Added it under Alternatives for now. If you and @jbms both prefer the wrapper, I'll cut origin from here and open the grid in zarr-extensions.
|
|
||
| An array metadata document MUST contain exactly one of `shape` and `bounds`. | ||
|
|
||
| `bounds` is an array with one entry per dimension. Each entry is a two-element array of integers |
There was a problem hiding this comment.
I would love to hear thoughts on whether bounds should be an object instead: "bounds": { "inclusive_min": [...], "exclusive_max": [...]}
There was a problem hiding this comment.
I like it better too: nobody can misread which end is open, and it's what tensorstore already writes for an index domain. Switched in this PR and in #400; pairs are under Alternatives now.
Taken from review on zeps#72: the member names state the half-open convention and match tensorstore's index-domain JSON. States the constraints the pair form carried implicitly (equal length, min <= max, the zero-dimensional form), moves per-dimension pairs to Alternatives and adds the wrapper-grid alternative raised in the same review.
Draft ZEP 12: explicit array bounds.
Adds
boundsto the array metadata as an alternative toshape(a half-open coordinate range per dimension) and an optionaloriginto the regular chunk grid, so that an array can grow before its first element by editingzarr.json. Existing chunks keep their keys and existing elements keep their coordinates. Arrays written withshapeare unchanged; readers that do not implement the ZEP refuse arrays that use it, sinceshapeis absent.Follows zarr-developers/zarr-extensions#79, where @jbms and @normanrz preferred a core change to the chunk grid route prototyped in zarr-developers/zarr-extensions#81. The thread's open question,
boundsin place ofshapeversus a single field besideshape, is laid out under Alternatives; the draft takes theboundsform because it is the one every reader checked refuses today, and it flips if the working group prefers the other.Spec text: zarr-developers/zarr-specs#400. Prototypes, currently on the chunk grid form: zarr-developers/zarr-python#4516, google/tensorstore#296, google/neuroglancer#1133.
The number 12 is the next unassigned (6, 7, 8 and 10 are in open PRs); renumber as needed.
Rendered: https://github.com/rhoadesScholar/zeps/blob/zep-0012-array-bounds/draft/ZEP0012.md
What changes for a user
Growing a 3-D uint8 array by two chunk rows in front of axis 0 today means a resize and a move of every chunk; with this ZEP it is one metadata write. Measured on a local filesystem store with 64³ chunks (details in the ZEP):