Skip to content

fix: return upsert identity keys and correlate on the full key set so bulk upserts don't drop records - #812

Merged
zachdaniel merged 2 commits into
ash-project:mainfrom
sephianl:fix/bulk-upsert-drops-records-upstream
Aug 11, 2026
Merged

fix: return upsert identity keys and correlate on the full key set so bulk upserts don't drop records#812
zachdaniel merged 2 commits into
ash-project:mainfrom
sephianl:fix/bulk-upsert-drops-records-upstream

Conversation

@DGollings

Copy link
Copy Markdown
Contributor

Problem

Since #786, bulk upserts can silently drop records from the returned result while still
writing them to the database — so after_action hooks and notifiers never run for the
dropped records. This surfaced on the 2.10.0 → 2.11.0 upgrade (same regression as #808/#809).

bulk_create/3 correlates each returned row back to its changeset by the upsert identity's
keys. Two things break that correlation:

  1. The identity keys aren't selected. When the action's select excludes them
    (e.g. upsert_fields [] with select: [], as Ash.Reactor's bulk_create step passes),
    the returned rows don't contain the identity columns, so nothing correlates and every
    record is dropped. This is the fix from @marcnnn's fix: return upsert identity keys so bulk upserts don't drop records #808, with the regression tests from test: cover bulk upsert dropping records when select excludes identity keys #809.

  2. Unset (nil) identity keys are missing on the changeset side. For a multi-field identity
    that leaves a key unset — a nullable field left nil, common with nils_distinct?: false
    natural keys such as [barcode, timestamp, order_id, route_id] — the changeset key is built
    with Map.take(changeset.attributes, keys), which drops the unset key. The changeset's
    correlation key then has fewer fields than the returned row's key (whose Map.take(row, keys)
    includes the key as nil), so it still never matches even after (1).

Fix

  • Add the identity's attribute keys to the upsert RETURNING list so returned rows always
    carry them. Ash.Actions.Helpers.select/2 masks the extra fields back out of the records,
    so this doesn't widen what callers observe. Identity keys that aren't attributes (e.g. an
    identity over a calculation) have no column and are left out.
  • Build the per-changeset correlation key over the full identity key set, defaulting unset
    keys to nil, so both sides have the same shape.

Tests

The fix in (1) and the first two tests are originally by @marcnnn (#808, #809); this PR carries
them and adds the multi-field / nil-identity correlation fix and test on top.

Note for reviewers

On PostgreSQL 17 (MERGE path) the pre-existing test returns a skipped upsert whose identity contains nil (added by #794) fails with a unique-constraint error — that's a separate
nil-identity issue in the return_skipped_upsert? path, unrelated to and not addressed by
this PR.

Disclosure

Drafted with AI assistance, but every change was reviewed and validated: the failure is
reproduced by a failing test first, the fix is verified against those tests, and the whole
change was exercised end-to-end against a real 8k-test downstream suite on PostgreSQL 17.


  • Bug fixes include regression tests

Bulk upserts correlate each returned row back to its changeset by the upsert
identity keys. Since ash-project#786 those keys are no longer read back when the actions
select does not include them (e.g. `upsert_fields []` with `select: []`, as
Ash.Reactors bulk_create step passes), so nothing correlates and every record
is silently dropped from the result while still being written to the database.

Add the identity attribute keys to the upsert RETURNING list so correlation
always succeeds; `Ash.Actions.Helpers.select/2` masks the extra fields back out.

Fix and regression tests originally by @marcnnn (ash-project#808, ash-project#809).
Returning the identity keys (previous commit) is not enough for multi-field
identities that leave a key unset: `Map.take(changeset.attributes, keys)` drops
the unset key, so the changeset correlation key has fewer fields than the
returned rows key and still never matches - the record is silently dropped.

Build the changeset correlation key over every identity key, defaulting unset
ones to nil, so both sides have the same shape. Covers natural-key identities
like [barcode, timestamp, order_id, route_id] with `nils_distinct?: false` where
trailing keys are commonly nil.
@zachdaniel
zachdaniel merged commit 9ecd4fd into ash-project:main Aug 11, 2026
116 of 126 checks passed
@zachdaniel

Copy link
Copy Markdown
Contributor

🚀 Thank you for your contribution! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants