feat: v1 semantic convention#717
Draft
FBumann wants to merge 95 commits into
Draft
Conversation
The design goals and transitioning goals for linopy's v1 arithmetic convention, under arithmetics-design/goals.md. The convention itself and the bug catalogue (meta issue #714) follow separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Placeholder for the v1 convention document, to be written. Goals are in arithmetics-design/goals.md; the bug catalogue is the meta issue #714. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
FBumann
force-pushed
the
feat/arithmetic-convention
branch
from
May 21, 2026 12:48
40af9d6 to
1e336a4
Compare
Flesh out convention.md from the placeholder into the full spec — thirteen numbered sections in three groups: absence (§1–§7), coordinate alignment (§8–§11), and constraints and reductions (§12–§13). Covers the strict exact-match alignment model and the propagate-don't-fill NaN/absence convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The convention governs coordinate alignment, absence/NaN handling, constraints, and reductions — not just arithmetic operators — so retitle convention.md and goals.md to "The v1 convention". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce linopy.options["semantics"] — legacy (default) or v1 — with LinopySemanticsWarning, a FutureWarning shown to users by default and exported at top level. Add the autouse `semantics` conftest fixture that runs every test under both conventions, plus legacy/v1 markers to pin a test to one. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 22, 2026
Merged
`_align_constant` branches on `options["semantics"]`: v1 uses exact alignment via `xr.align(join="exact")`; legacy keeps the size-aware positional/left-join behaviour and emits `LinopySemanticsWarning` when v1 would diverge. `_add_constant`/`_apply_constant_op` raise on a NaN in a user-supplied constant under v1, warn under legacy. `Variable.__mul__(DataArray)` now routes through `to_linexpr() * other` so the LinearExpression checks fire; the scalar fast-path is preserved (a NaN scalar diverts to the expression path so v1 raises). Marks the bug-class test groups `TestCoordinateAlignment` (#708/#586/ #550), `TestConstraintCoordinateAlignment`, `TestNaNMasking`, `test_auto_mask_constraint_model`, and four piecewise NaN-padding tests as `@pytest.mark.legacy` — they assert the very behaviour v1 forbids. v1 coverage of those bug classes accretes via later slices. `test/test_legacy_violations.py` (new) adds 22 paired tests covering §5/§8/§9 plus the PyPSA #1683 `0*inf=NaN` case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`merge` now pre-validates that all operands agree on the labels of every shared *user* dimension before concatenating. Helper dims (`_term`, `_factor`) and the concat dim itself are excluded — those legitimately vary between operands. v1 raises on mismatch; legacy keeps current size-based override/outer behaviour and emits `LinopySemanticsWarning` when v1 would diverge. The check uses a new `_merge_shared_user_coords_differ` helper. The existing override/outer decision is unchanged for the actual `xr.concat` call — the new check only gates whether legacy/v1 accept the merge, never how the concat itself runs. Adds 8 paired tests for var+var, var-var, expr+expr, broadcast guard, and warning emission on the merge path. Reclassifies as `@pytest.mark.legacy`: `test_non_aligned_variables` (deliberately disjoint coords), `test_linear_expression_sum` / `test_linear_expression_sum_with_const` (assert `v.loc[:9]+v.loc[10:]` merges), `TestJoinParameter` cases that build `a*b` from mismatched- coord vars, and two SOS2 reformulation tests. File-level legacy mark on `test_piecewise_constraints.py` + `test_piecewise_feasibility.py` until `linopy/piecewise.py` itself is made v1-aware (tracked as Slice P). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Variable.to_linexpr() now produces a LinearExpression whose absent slots (labels == -1) carry NaN coeffs and NaN const under v1, so downstream arithmetic has something to propagate. The expression constant operators (_add_constant, _apply_constant_op) no longer fillna(0) self.const / self.coeffs under v1 — NaN flows through. `merge` sums const along _term with skipna=False under v1, so a slot that's absent in any operand stays absent in the result. Legacy paths keep the silent-fill behaviour verbatim. LinearExpression.isnull() now returns `const.isnull()` under v1: a slot is absent iff its const is NaN. ``vars == -1`` is a dead-term signal (the slot can still be a present constant after fillna), not a slot-level absence marker. Legacy keeps the historical ``(vars == -1).all() & const.isnull()`` formula for byte-for-byte compatibility. Variable.fillna(numeric) now returns a LinearExpression (a constant isn't a variable). Variable.fillna(Variable) stays Variable, as before. Adds 11 tests for §6 propagation (mul/add/sub/div preserve absence, absent-vs-zero distinguishable, present + absent propagates) and §7 resolution (fillna numeric on expr / Variable, present-zero revival). Reclassifies test_masked_variable_model as @pytest.mark.legacy — its assertion "x bound to 10 at masked-y slots" only holds because legacy collapses absent y to 0. The v1 way is x + y.fillna(0) >= 10; a counterpart test in test_legacy_violations.py pins this. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The convention spec names ``reindex`` and ``reindex_like`` among the absence-creating mechanisms (alongside ``mask=``, ``.where()``, ``.shift()``, and ``.unstack()``), but master only had them on ``LinearExpression``. Add them on ``Variable``, with the sentinel fill values (``labels=-1``, ``lower=upper=NaN``) so new positions slot cleanly into §6 propagation. The methods work the same way under both semantics — under legacy the sentinels exist but downstream arithmetic still collapses them back to 0 (the #712 bug), so the user-visible effect of reindex-as- absence only really lands under v1. Adds 5 tests: extend with absent, subset drops, reindex_like with another Variable, and the §4 + §6 hand-off (a reindex-introduced absent flows through ``* 3`` and is visible via ``isnull()``). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Slice C propagated NaN const cleanly but left the storage half-absent after a merge: `(1*x) + xs` at the absent slot kept the `1*x` term's valid coefficient and label even though `const` was NaN there. The §1/§2 promise "absence is one concept, whatever the dtype" only holds if `const.isnull()` at a slot ⇒ every term at that slot has `coeffs = NaN`, `vars = -1`. Add `_absorb_absence(ds)` and call it at the end of `merge` under v1. The constant-operand paths (`_add_constant`, `_apply_constant_op`) don't need explicit absorption — their NaN-propagation naturally preserves the invariant when the input is already v1-compliant (NaN * anything = NaN; dead terms stay dead). Only `merge` opens the gap by concatenating one operand's live term with another operand's absent slot along `_term`. `convention.md` §2 now states the invariant explicitly and introduces the *dead term* terminology, so `fillna(value)` reviving a slot while leaving the sentinel term in place reads as a feature, not a glitch. Adds `test_outer_fillna_then_add_collapses_to_just_added` pinning `(x + y.shift()).fillna(0) + x` — at the previously-absent slot the result has exactly one live term (`1·x[0]`) with `const = 0`, algebraically equal to `x[0]`. At present slots all three terms stay live (`2·x[i] + y[i-1]`), so fillna placement is load-bearing — moving it inside (`x + y.shift().fillna(0) + x`) would double-count `x` at the absent slot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`.add/.sub/.mul/.div/.le/.ge/.eq` already accepted a `join=` argument; this slice's job is just §12's RHS handling under v1. `to_constraint` branches on `options["semantics"]`. Under v1 it skips the legacy `reindex_like(self.const, fill_value=NaN)` step that silently padded a subset RHS, so a coord mismatch with the LHS now flows through `self.sub(rhs)` and gets caught by §8's exact alignment. A NaN in a user-supplied constant RHS raises at construction (§5) — including the PyPSA #1683 case of `min_pu * nominal_fix` with `p_nom=inf` and `p_min_pu=0`. An absent slot in the LHS (propagated from §6) still produces a NaN RHS at that row; downstream auto-mask drops the constraint there, which is exactly §12's "absent slot yields no row." Legacy keeps the old auto-mask path verbatim and adds a `LinopySemanticsWarning` whenever a NaN RHS is observed, so users get the rollout signal without behaviour change. Adds 11 paired tests: TestNamedMethodJoin (inner/outer/left across .add/.mul/.le, plus a "bare op still raises" guard) and TestConstraintRHS (subset RHS raises, NaN RHS raises, PyPSA #1683 on the constraint side, §6→§12 hand-off where the absent LHS slot yields NaN RHS, plus the paired legacy auto-mask documentation and warning-emission tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Coming next, in order:
Then a final pass on docs (user-facing migration / rollout — deferred so far). |
Three internal patterns were violating §8 / §11:
1. ``_add_incremental`` in ``linopy/piecewise.py`` builds
``delta_hi <= delta_lo`` from two ``.isel(piece_dim=slice)`` slices
of the same variable. ``drop=True`` is a no-op for slice indexers
so ``piece_dim`` stays on both with *different* labels (first n-1
vs last n-1 of piece_index) — v1 §8 rejects. Relabel the high
slice onto the low slice's labels so the comparison aligns by
label (the explicit-positional path of §10). Same fix for
``binary_hi <= delta_lo``.
2. ``_incremental_weighted`` computes ``bp0 = bp.isel({dim: 0})``
without ``drop=True``, leaving the breakpoint dim as a scalar
coord on the resulting expression. When that expression appears
as the RHS of ``links.eq_expr == ...`` it conflicts with the LHS,
which has no such coord — §11 aux-coord conflict. Add ``drop=True``.
3. ``reformulate_sos2`` builds its first/last constraints from
scalar isels at different positions on ``sos_dim`` (``x``/``M`` at
``n-1`` paired with ``z`` at ``n-2``, etc.). All without
``drop=True``, so the scalar ``sos_dim`` coord differs across
operands — §11 aux-coord conflict. Add ``drop=True`` to all three
sites.
Removes the module-level ``pytestmark = pytest.mark.legacy`` from
``test_piecewise_constraints.py`` and ``test_piecewise_feasibility.py``
and the method-level marks from the two SOS2 multidim tests. Suite is
+598 tests under v1 vs Slice E (legacy → v1 broadened coverage),
0 failures under either semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
§13 falls out of xarray's ``skipna=True`` default; no code changes needed. Adds 4 tests so future drift is caught: sum over a dim, sum without a dim, sum of all-absent (the zero expression), and groupby.sum across heterogeneously-present groups. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `_conflicting_aux_coord(datasets)` and wires it into both `merge` and `_align_constant`. When two operands carry an aux coord of the same name with disagreeing values, v1 raises with a pointer to the explicit resolutions (``.drop_vars(...)`` or ``.assign_coords(...)``). xarray silently drops the conflict — the #295 bug — and legacy keeps that behaviour but now emits a `LinopySemanticsWarning`. The helper guards against string-dtype coord values (no `equal_nan=True` there) so the multiindex case keeps working. `_merge_shared_user_coords_differ` refactored to compare bare ``d.indexes[k]`` instead of ``d.coords[k]``: aux coords no longer leak into the §8 check, so §11 owns aux-coord conflicts cleanly and §8 owns dim-coord mismatches with a separate message. Convention §11 expanded from one paragraph: aux coords are validated and propagated but never computed with — they describe the data, they don't enter the math. Goal #4 in `goals.md` picks this up: user-attached auxiliary coordinates are the user's, linopy never silently rewrites them. `test_linear_expression.py::test_merge` adds ``drop=True`` to its ``.sel`` setup — the test was leaving a leftover scalar coord that v1 now correctly catches as a §11 conflict; the fix preserves the test's intent of exercising merge with differing term counts. Conflict-raising tests (TestAuxCoordConflict) cover expr+const, var+var, scalar-isel-without-drop, the ``drop=True`` escape hatch, plus the paired legacy left-wins documentation and warning-emission tests. Propagation guarantees land in a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Regression coverage on the half of §11 that wasn't tested before: non-conflicting aux coords carry through every binary operator and into constraints. xarray already preserves them; the tests guard against future drift (e.g. a reduction or helper accidentally dropping a non-dim coord). TestAuxCoordPropagation covers ``3*v``, ``v+5`` (single-operand, fast paths), ``v+v`` with matching aux (the merge path), ``v<=10`` (the constraint path), ``x*a`` / ``x+a`` / ``x/a`` / ``x<=a`` where only the constant DataArray carries the coord (the ``_align_constant`` path), and the var+var case where only one side has the coord. Together: every operator times every "one side / both sides" arrangement, since only conflicts on both sides raise. Runs under both semantics — the legacy behaviour matches the v1 behaviour for the non-conflict cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… solve Fills the convention-coverage gaps surfaced by review of the branch: - §1/§2 dead-term storage invariant: pin that after a merge with an absent slot, coeffs=NaN AND vars=-1, not just const=NaN. The existing propagation tests read through isnull() which only checks const, so a regression in _absorb_absence would have passed them. Multi-operand variant catches binary-only-absorption regressions. - §12 equality: mirror the existing <=/>= TestConstraintRHS coverage for ==. Subset RHS raises, NaN RHS raises, absence in LHS drops the row. - §11 extra operators: add mul-constant and == constraint cases to the existing TestAuxCoordConflict. The class already covered +-constant and var+var; these extend coverage to the other call-site shapes. - §13 scope note: mean/resample/coarsen aren't yet on LinearExpression (tracked in #703); the spec text is the rule those will follow when implemented. Docstring note in TestReductionsSkipAbsent makes this explicit so the gap doesn't read as missing coverage. - End-to-end v1 solve: test_masked_variable_model_v1_drops_constraint pins the v1 outcome at the solver layer — con0 masked at absent slots (solver-independent) and x bound to 0 where the constraint still binds. _v1_fillna_binds confirms the §7 escape hatch recovers the legacy outcome. Catches the regression where v1 silently produces wrong solutions instead of raising. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls the seven v1-specific helpers and the user-NaN message out of ``expressions.py`` and into a dedicated ``linopy/semantics.py`` module — a single home for "what v1 means" that imports cleanly from ``config`` and ``constants`` only. Adds a tiny ``is_v1()`` predicate so the 16 scattered ``options["semantics"] == V1_SEMANTICS`` checks collapse to a one-line call. Helpers (renamed to drop the leading underscore now that they're a real module API): ``check_user_nan_scalar``, ``check_user_nan_array``, ``dim_coords_differ`` (was ``_shared_coords_differ`` — clearer name, matches ``merge_shared_user_coords_differ``), ``merge_shared_user_coords_differ``, ``conflicting_aux_coord``, ``absorb_absence``, plus ``is_v1``. No behaviour change — same checks, same warnings, same raises. The diff is mechanical: imports flipped, two local ``is_v1 = options[...]`` bindings replaced by the imported predicate, one missed ``_USER_NAN_MESSAGE`` reference in ``to_constraint`` routed through ``check_user_nan_array`` for consistency. ``expressions.py`` shrinks by ~105 lines. Future v1-only API surface (e.g. exposing ``is_v1()`` as ``linopy.is_v1()`` for downstream code) and the eventual legacy removal at 1.0 both reduce to deletions of ``semantics.py`` and its import sites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three test clusters in ``test_legacy_violations.py`` had near-identical
``test_add_X``, ``test_mul_X``, ``test_div_X`` triples that varied only
by which binary operator they exercised. Collapse each into a single
``@pytest.mark.parametrize("op", ...)`` test:
- TestExactAlignmentConstant: same-size-different-labels and
subset-constant raises, parameterized over add/sub/mul/div.
- TestUserNaNRaises: NaN-DataArray raises over add/sub/mul/div, NaN
scalar over add/sub/mul (div scalar shares the same ``_apply_constant_op``
code path as mul, but ``x / nan`` trips ``__div__``'s unary-negate
TypeError before our check fires; the dispatch needs a separate
fix that's not worth pulling into this refactor).
- TestAbsencePropagation: ``shifted OP scalar`` preserves absence,
parameterized over add/sub/mul/div. Adds a per-op present-slot
value check so the parameterization broadens rather than narrows
the assertion.
Adds a module-level ``_OPS`` dict mapping name → ``operator``
callable so the parameter is the readable name (``"add"``,
``"div"``) while the test still calls the actual operator.
Cuts ~50 lines off ``test_legacy_violations.py`` and makes adding a
new operator a one-line change. Test IDs become e.g.
``test_same_size_different_labels_raises[v1-add]`` — slightly less
self-describing than the explicit-method names but cheap to read.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both methods had v1 and legacy logic interleaved via a ``fillna0`` closure that was identity under v1 and ``da.fillna(0)`` under legacy. Pull them apart into: - ``_add_constant`` / ``_apply_constant_op`` — two-line dispatchers. - ``*_v1`` — v1's implementation, reads as a single coherent story. - ``*_legacy`` — legacy's implementation, ``# LEGACY: remove at 1.0`` marker on each. At 1.0 the removal is mechanical: delete the ``_legacy`` methods and inline the ``_v1`` body into the dispatcher (or rename it back to the public name). Future readers don't have to mentally subtract the legacy branches to understand what v1 does. Add ``LEGACY: remove at 1.0`` marker comments at the other mixed sites in ``expressions.py`` so ``grep`` finds every place that needs touching: ``_align_constant``'s size-aware default fallback, ``to_constraint``'s auto-mask fallthrough, ``LinearExpression.isnull``'s historical AND, and the two warn-on-divergence sites in ``merge``. New ``arithmetics-design/legacy-removal.md`` is the master checklist for the 1.0 cut: every file, function, test, doc edit, and the safe order to do them in. The intent is that the eventual legacy removal takes an afternoon, not a week of grep-archaeology. No behaviour change — same checks, same warns, same raises. Suite is 7282 passed, 0 failures under both semantics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two distinct CI failures both rooted in the v1 harness commit: 1. **Test collection crash on every linopy/*.py module.** ``test/conftest.py`` imported ``linopy.config`` at module top, which loaded linopy from site-packages before pytest's ``--doctest-modules`` collection walked the source tree. The resulting __file__ mismatch broke all 22 module collections. ``pyproject.toml`` already documents this exact failure mode in the ``filterwarnings`` block. Fix: keep the constant *values* (``"legacy"`` / ``"v1"``) inline in conftest as ``_LEGACY_SEMANTICS`` etc. so the parametrize decorator doesn't force an import, and defer the ``LinopySemanticsWarning`` / ``options`` import into the fixture body. The original import comment in pyproject is now mirrored at the top of conftest. 2. **mypy: 72 "no-untyped-def" errors in test_legacy_violations.py.** The new tests were missing parameter type annotations on the fixture-injected params (``x``, ``xs``, ``op``, ``unsilenced``, ``subset``, ``A``, ``da_aux_B``, ...). ``disallow_untyped_defs`` is set globally, so test files need them too. Filled in the types (``Variable``, ``str``, ``None``, ``xr.DataArray``, ``pd.Index``), added an ``isinstance(result, LinearExpression)`` narrowing in ``test_variable_fillna_zero_revives_slot_as_present_zero`` so mypy can pick the right branch of ``fillna``'s return union. Local: 7282 passed, 0 failures under both semantics; ``mypy .`` Success. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three v1 raises were under-informative — naming the rule violated but not the operand, dim, or values involved. Make each message carry the information the helper already has: - **§5 user-NaN**: the old message conflated the two intents the user might have had — *data error* (fix with ``.fillna(value)``) vs *intended absence* (mark on the variable with ``mask=`` / ``.where`` / ``.reindex`` / ``.shift``). The new message separates them and points each to its own remedy. - **§8 merge mismatch**: rename ``merge_shared_user_coords_differ`` (bool) to ``merge_shared_user_coord_mismatch`` (tuple ``(dim, left, right) | None``). Raise text now includes the offending dim name and both sides' labels (truncated), plus the full set of resolution paths from §10: ``.sel`` / ``.reindex`` / ``.assign_coords`` / ``linopy.align`` / ``join=`` on ``.add`` / ``.sub`` / ``.mul`` / ``.div`` / ``.le`` / ``.ge`` / ``.eq``. - **§11 aux-coord conflict**: ``conflicting_aux_coord`` returns ``(name, left_vals, right_vals) | None``. Raise text includes the coord name, both value snippets, and all three resolution paths (``.drop_vars`` / ``.assign_coords`` / ``isel(drop=True)`` — ``.assign_coords`` was previously omitted). The text is now centralized in ``semantics.py`` so the two raise sites in ``expressions.py`` (``_align_constant`` and ``merge``) share one voice instead of paraphrasing each other. New ``TestErrorMessageContent`` pins the rich content in three tests — that the §5 message names both intents, that the §8 message names the dim and both label lists, and that the §11 message names the coord, both value lists, and lists all three §11 fixes (the ``.assign_coords`` omission would have slipped through ``match= "Auxiliary coordinate"`` substrings). Section references (``§5``, ``§8``, ``§11``) deliberately omitted from user-visible text — spec jargon, not a navigation aid for downstream callers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the small-but-real holes in the §1–§13 coverage map. New tests
mostly, plus one code fix that the test surfaced.
§4 — absence creation
- test_where_creates_absence: §4 names ``.where(cond)`` but only
``mask=`` / ``.reindex`` were tested.
- test_unstack_creates_absence_at_missing_combinations: the
non-rectangular MultiIndex case (``stack`` preserves, ``unstack``
fills) is the asymmetry that earns its own test. Hit a real bug
on the way — ``Variable.unstack`` was producing float NaN in the
integer ``labels`` field instead of the ``FILL_VALUE`` sentinel
(-1), violating §2. Fixed by passing ``fill_value=_fill_value``
to the underlying ``Dataset.unstack`` (same pattern as ``shift``).
Audited the rest of the varwrap calls — only ``shift`` and
``unstack`` introduce new positions; the others either preserve
shape (``assign_*``, ``rename``, ``swap_dims``, ``set_index``,
``roll``, ``stack``), select existing positions (``sel`` /
``isel`` / ``drop_*``), or broadcast existing data without fill
(``broadcast_like``, ``expand_dims``).
- test_data_preserving_methods_do_not_create_absence: parameterized
over ``.roll`` / ``.sel`` / ``.isel``, regression-guards §4's
explicit contrast against the creators.
§10 — named-method join= argument
- test_add_join_override_aligns_positionally: positional-mode is the
surprising one in the join= set; pin it explicitly.
- test_reindex_like_resolves_mismatch_before_bare_op and
test_assign_coords_resolves_mismatch_before_bare_op: §10 names
these as the canonical user fixes; pin that the post-fix bare
operator actually accepts the once-mismatched operand.
§11 — auxiliary-coordinate conflicts
- test_assign_coords_resolves_conflict: §11 lists three escape
hatches; only ``.drop_vars`` / ``isel(drop=True)`` were tested.
- test_multi_operand_merge_aux_conflict_raises: the merge-path
check inspects all operands; a 3-way ``v + w + u`` with the
third disagreeing exercises that.
§12 — constraints follow the same rules
- Parameterize the existing subset / NaN / absence-propagation
tests in ``TestConstraintRHS`` over the three signs (``le`` /
``ge`` / ``eq``) via a new module-level ``_SIGNS`` dispatch.
Folds the previous ``<=`` and ``==`` duplicates together and
fills in ``>=`` for each rule (which was the explicit gap).
The PyPSA #1683 test stays separate — it's tied to ``>=`` by
the real-world case it documents.
Suite: 7303 passed, 515 skipped, 0 failures under both semantics.
``mypy .`` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v1): don't warn on exactly-aligned @/dot under legacy (#849) The legacy alignment path in LinearExpression._align_constant warned whenever the constant operand's size dict differed from the expression's. But a size difference can come purely from disjoint dims (e.g. `x @ C`, where C carries its own contracted-out dim) — that is ordinary broadcasting, not a shared-dim misalignment. v1 accepts it, so the legacy LinopySemanticsWarning was a false positive on already-aligned operands. Warn only when first_mismatched_dim reports a real shared-dim disagreement, mirroring the sizes-equal branch, and collapse the duplicated warn/branch into one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: simplify legacy _align_constant branch (single return) Name the aligned result and return once instead of repeating the (self.const, ..., False) triple, and move the "positional when sizes match" comment onto the branch it describes. Behaviour unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…entions (#847, #848) (#851) * fix(v1): make Variable.fillna(scalar) resolve absence under both conventions (#847, #848) Variable.fillna(<scalar>) is the documented resolution for absent slots (from shift/where/reindex/mask), but under legacy it misbehaved two ways: - #847: it warned. fillna internally calls to_linexpr(), whose legacy path emits the masked-variable LinopySemanticsWarning — even though fillna is itself the resolution that warning points to. So the documented fix couldn't be written warning-free on both conventions. - #848: it silently dropped the fill value. Legacy to_linexpr marks absent const as 0 (not NaN), so the subsequent LinearExpression.fillna had nothing to fill; fillna(5) left 0 at absent slots while v1 put 5. The v1 path already behaved correctly, so this is a pure legacy workaround (marked LEGACY: remove at 1.0): keep the clean one-liner under v1, and under legacy place the value at the -1 labels directly and skip the absence warning. Result: `var.shift(1).fillna(v)` is now a single form, identical under both conventions (same vars/const; only the immaterial phantom coeff differs), so downstream migrating to v1 no longer has to version-gate the expression. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(v1): note legacy no-op of LinearExpression.fillna on absent slots Document why var.to_linexpr().fillna(v) is a no-op under legacy (absence is already materialised as const=0, so there is no NaN to fill) and point at Variable.fillna as the cross-convention resolution. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Polish for the #717 "polish the tests" checklist, scoped to the v1/legacy groupby-MultiIndex divergence tests in test_linear_expression.py: - Replace the manual try/finally toggle of options["semantics"] in test_group_multiindex_reset_index_matches_v1 with the `with options as o` context manager, so semantics state is restored even if the assertion fails (it was the only test bypassing the conftest marker mechanism). - Swap internal `.data.indexes` / `.data.coords` reaches for the public `.indexes` / `.coords` accessors, matching the already-public sibling tests in the same class. No behaviour change; 584 passed, ruff clean. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FBumann
added a commit
that referenced
this pull request
Jul 20, 2026
…sages Follow-up polish for the #717 "polish the tests" checklist: - Move the four masked-addend §6 tests (absence propagation of a fully masked term in a sum) out of test_linear_expression.py into TestAbsencePropagation in test_legacy_violations.py, where the rest of the §6 coverage lives. They now share one `ab_all_masked` fixture instead of rebuilding an inline two-variable model four times. - Add `match="only supported for"` to the four bare `pytest.raises(ValueError)` guards on the `use_fallback=True` DataFrame-grouper path, so they assert *why* the fallback raises rather than accepting any ValueError. The fifth guard (issue #351) keeps its precise `(KeyError, IndexError)` — its message is an incidental numpy index error, brittle to pin. No behaviour change; test_legacy_violations.py 294 passed, test_linear_expression.py green, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sages (#853) Follow-up polish for the #717 "polish the tests" checklist: - Move the four masked-addend §6 tests (absence propagation of a fully masked term in a sum) out of test_linear_expression.py into TestAbsencePropagation in test_legacy_violations.py, where the rest of the §6 coverage lives. They now share one `ab_all_masked` fixture instead of rebuilding an inline two-variable model four times. - Add `match="only supported for"` to the four bare `pytest.raises(ValueError)` guards on the `use_fallback=True` DataFrame-grouper path, so they assert *why* the fallback raises rather than accepting any ValueError. The fifth guard (issue #351) keeps its precise `(KeyError, IndexError)` — its message is an incidental numpy index error, brittle to pin. No behaviour change; test_legacy_violations.py 294 passed, test_linear_expression.py green, ruff clean. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FBumann
added a commit
that referenced
this pull request
Jul 20, 2026
…rop dead code Three follow-ups from the #717 review: - §5 user-NaN check on the scalar fast path used `isinstance(other, float)`, which misses `np.float32`/`np.float16` NaN scalars (they don't subclass Python `float`). Such a scalar was silently added/multiplied into the expression instead of raising (v1) / warning (legacy). Add `semantics.is_nan_scalar` (float | np.floating) and use it at all four scalar sites. - `conform_merge_dims` called `Index.get_indexer` on a shared dim with non-unique labels, raising an opaque `InvalidIndexError` — a regression vs. master, which aligned duplicate labels positionally. Guard on `idx.is_unique` so a non-unique differing index is reported as a §8 mismatch: legacy aligns positionally + warns, v1 raises the canonical "Coordinate mismatch" ValueError. - Remove the unused `LEGACY_SEMANTICS_MESSAGE` constant and the stale `EvolvingAPIWarning` filter cells in the piecewise notebooks (obsolete under `semantics="v1"`; the warning was renamed to LinopySemanticsWarning). Adds regression tests for the NaN-scalar dtypes and the duplicate-label merge on both conventions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rop dead code (#854) Three follow-ups from the #717 review: - §5 user-NaN check on the scalar fast path used `isinstance(other, float)`, which misses `np.float32`/`np.float16` NaN scalars (they don't subclass Python `float`). Such a scalar was silently added/multiplied into the expression instead of raising (v1) / warning (legacy). Add `semantics.is_nan_scalar` (float | np.floating) and use it at all four scalar sites. - `conform_merge_dims` called `Index.get_indexer` on a shared dim with non-unique labels, raising an opaque `InvalidIndexError` — a regression vs. master, which aligned duplicate labels positionally. Guard on `idx.is_unique` so a non-unique differing index is reported as a §8 mismatch: legacy aligns positionally + warns, v1 raises the canonical "Coordinate mismatch" ValueError. - Remove the unused `LEGACY_SEMANTICS_MESSAGE` constant and the stale `EvolvingAPIWarning` filter cells in the piecewise notebooks (obsolete under `semantics="v1"`; the warning was renamed to LinopySemanticsWarning). Adds regression tests for the NaN-scalar dtypes and the duplicate-label merge on both conventions. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…to MI row (#855) The migration guide's fix table covered the core absence + alignment + aux-coord + MultiIndex-dim + groupby-result cases, but three issue-backed v1 changes were under-covered. Add rows and mirror two into the release-notes v1 summary: - Unlabeled operand (numpy / list / polars) pairs to dims by size; ambiguous or no-size-match raises (#736) — the object-scope premise. - A reordered/mismatched groupby grouper raises rather than reindexing positionally (#827). - MultiIndex-level projection: v1 rejects the MultiIndex *dimension* at construction, so the per-level-input case is folded into the existing MultiIndex row — decompose with reset_index, then project by the level aux coord. (The convention.md §11 example still shows the legacy-shaped `.get_level_values` snippet, which cannot run under v1 — flagged for the spec pass.) Release notes already carry the level-projection deprecation separately, so only #736 and the grouper point are mirrored there. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raduation (#856) The §11 example used `expr.indexes["snapshot"].get_level_values("period")`, which cannot run under v1: v1 rejects a first-class `pd.MultiIndex` dimension at construction, so `snapshot` is a flat dim with `period`/`timestep` as aux coords and `.indexes["snapshot"]` is a flat index, not a MultiIndex. Replace it with the working aux-coord mapping (verified on the branch) and add a one-line note on the flat representation. Also record in open-items.md that convention.md graduates into the rendered Sphinx docs at the 1.0 reframe (add myst-parser, toctree, internal cross-refs); until then it stays a design-record linked by URL — the rules are permanent but the transitional framing would only be re-done, so rendering now is wasted. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 20, 2026
…e) (#858) Positional indexing leaves a scalar coordinate behind: `x.isel(time=0)` drops `time` as a dimension but keeps it as a scalar coord (the first label). So a cyclic/boundary constraint like `x.isel(time=0) == x.isel(time=-1)` — extremely common in storage/energy models — hits §11's aux-coord conflict under v1 (`time` = first vs last) and raises, where legacy silently dropped it. The aux-coord row already covered this in principle, but listed only `.drop_vars` / `.assign_coords` as fixes and didn't name the scalar-leftover cause. Broaden the migration-guide row and add the natural fix — drop the coord at the indexing site with `.isel(..., drop=True)` / `.sel(..., drop=True)` — and add the same note to convention.md §11. Verified on the branch: `x.isel(time=0) == x.isel(time=-1)` raises "Auxiliary coordinate 'time' has conflicting values ... left=0, right=3" under v1; `drop=True` on both sides builds cleanly. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(v1): polish nan-scalar + duplicate-label merge tests (dedup, trim comments) * test(v1): polish the #717 convention test suite Applies the maintainer's polish guidelines across the v1/legacy semantics tests (test_legacy_violations.py, test_convention.py): - Assert FULL error/warning message text (hardcoded literals) instead of partial `match="..."` substrings, so every message change surfaces as a test diff. Covers all v1 ValueError raises and legacy LinopySemanticsWarning sites across the 17 convention classes. - DRY: dedupe repeated operand/coord setup into fixtures; merge near-identical tests via @pytest.mark.parametrize (per-operator NaN/mismatch cases, quadratic build paths, join modes, object-scope operand kinds, ...). - Trim essay-length docstrings/inline comments to one crisp line where the rationale isn't load-bearing. Coverage is preserved or expanded (500 vs 480 collected invocations); no tests removed. Suite green under both semantics; ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@FabianHofmann I fixed all issues you found. We need to decide where to place the documentation of the convention. |
This was referenced Jul 21, 2026
Collaborator
you are a hero. this sounds good, let's do it. I added two tracker prs
I will gather feedback from those here. atm it looks like there is not big issues |
FBumann
added a commit
to flixOpt/flixopt
that referenced
this pull request
Jul 21, 2026
linopy's upcoming "v1" arithmetic convention (PyPSA/linopy#717) replaces legacy's silent coordinate realignment and NaN-fill with strict, explicit rules that raise on ambiguity. flixopt's constraint construction relied on the legacy behaviour in three places; this makes every site explicit so the model builds and solves identically under both `linopy.options["semantics"] = "legacy"` and `= "v1"`. Categories fixed: - Coordinate mismatch on a shared dim (adjacent-step constraints). Sites that combine `x.isel(dim=slice(1, None))` with `x.isel(dim=slice(None, -1))` carried different labels on the shared dim; legacy aligned by position, v1 raises. New `modeling._lead(var, dim)` relabels the leading slice onto the trailing slice's labels (linopy's recommended `.assign_coords` fix), which is identical under both semantics. Applied to the storage energy balance, on/off duration forward/backward/lb, state- and continuous-transition bounds, level tracking, and linked-period sizing. - Conflicting scalar auxiliary coordinate. Endpoint constraints like `x.isel(time=0) == x.isel(time=-1)` leave a scalar `time`/`scenario`/ `cluster_boundary` coord whose values disagree; legacy drops it silently, v1 raises. Added `drop=True` at each scalar-index site (cyclic/initial storage & status constraints, scenario-equality, intercluster linking, previous-state selection). - Deprecated `Constraint.lhs` setter. Replaced `con.lhs +=/-=` with `con.update(lhs=...)` (linopy 0.8 deprecation) at the transmission-loss, bus-balance, and share-allocation sites. The white-box `assert_conequal` tests in tests/deprecated reconstruct these constraints; updated their expected expressions to the same explicit form. Verified: full suite (tests/, excl. slow tutorials) is identical under both semantics — 1615 passed, same 4 pre-existing clustering failures (solution degeneracy in cyclic-cluster SOC level, unrelated to the semantics flag). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
|
@FBumann quick update, making good progress on the pypsa compatibility side. will continue tomorrow, but if it continues like that, I would say we should be able to merge this by the end of the week |
Collaborator
Author
|
I also updated flixOpt here for reference including a reference of transferring patterns |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The strict v1 semantic convention for linopy — predictable coordinate alignment and NaN handling.
This is the master PR for the new semantic convention in linopy. It starts with our
Design & transitioning goals, which is carried out in ourNew Semantics spec. Both files are tracked in this branch. WHat you read is the current state.Both might change until this PR is merged
The concrete rollout checklist — the three stages (opt-in → default → 1.0) and their per-stage status — is tracked in
arithmetics-design/open-items.md.Scope
The convention ships behind
linopy.options["semantics"]—v1opt-in,legacythe default. This PR carries the design, spec, tests and implementation; documentation notebooks follow separately.Testing
All tests in linopy will be executed for both semantics.
Differing behaviour will be tested using
pytest.markers.This will increase ci time temporarily until v1 is released.
Defered to a follow up PR
Docs: Documentation, Migration guide, Release notes
Note
Issue links below added by AI. These are the milestone issues the v1
convention fixes; they auto-close when this PR merges. #703 (extend xarray
method coverage) stays open — it is a deferred follow-up (§13), not fixed here.
Closes #714
Closes #713
Closes #712
Closes #711
Closes #708
Closes #670
Closes #586
Closes #570
Closes #295
Closes #257
Closes #736
Closes #256