Flaky ScyllaDB storage tests: Org_Domain_Operations and User_SCIM_Fields fail on read-after-write #813

Activity

1. lakhansamani

   added 2 commits that reference this issue on Oct 15, 2026

   docs(storage): correct the LWT timestamp-domain rationale

   ...

   a0f627d

   fix(storage): stop mixing LWT and plain writes on cassandra partitions (

   ...

   ff07f45
2. lakhansamani commented on Oct 15, 2026

   

   lakhansamani

   on Oct 15, 2026

   Last edited by lakhansamani

   ContributorAuthor

   More actions

   Root cause found for one of the two failures, measured

   Org_Domain_Operations — fixed by #818

   AddOrgDomain is an INSERT … IF NOT EXISTS (a lightweight transaction, deliberately — the normalized domain is the partition key and first-writer-wins is the atomicity guarantee). DeleteOrgDomain was a plain DELETE.

   An LWT takes its mutation timestamp from the Paxos ballot, i.e. the server clock. A plain DELETE writes a row tombstone carrying gocql's client-side timestamp (DefaultTimestamp defaults to true). A tombstone only shadows a row whose timestamp it exceeds — so when the server clock runs ahead of the client, the delete is discarded with no error and the row survives. That is precisely the reported symptom: the deleted domain was still found, and the list returned 2 instead of 1.

   Measured, 50 LWT-insert-then-delete pairs per cell, both engines in Docker with the engine clock ~5-6ms ahead of the host:

   Engine	plain DELETE	DELETE … IF EXISTS
   scylladb/scylla 3.0.8 (what make test-scylladb runs)	46/50 silently dropped	0/50
   cassandra:4.1 (4.1.12)	0/50	0/50

   92% on the engine the suite actually runs against. The intermittency in CI comes from the clock lead being ~0.6ms — when the gap between the two statements happens to exceed it, the delete lands.

   Apache Cassandra escapes it only because its ballot timestamp is truncated to whole milliseconds, which floors it behind the later client timestamp. The two clock domains are confirmed on both engines — on Cassandra by precision (50/50 of its LWT writetimes land on an exact millisecond against 0/50 for plain writes), on Scylla by direction (45/50 land ahead of a host clock reading taken after Exec() returned, by up to 1393µs, which a client-stamped timestamp cannot do). A margin, not a guarantee.

   The same shape existed on the user table — AddUser's IF NOT EXISTS against a plain DeleteUser. #818 removes that guard (it was dead: it only constrained a freshly generated UUID partition key, and its applied flag was never read) and adds IF EXISTS to the two org_domain deletes.

   User_SCIM_Fields — still open, and not this

   IsActive=false failing to persist is a cell update, not a row delete. A plain UPDATE after an LWT insert measured 0/50 lost on both engines, in the identical loop shape that drops 46/50 of deletes — so whatever breaks User_SCIM_Fields, it is not reproducible as this. Worth saying plainly that I also cannot explain why updates escape the race when deletes don't; under the timestamp model above they should lose too. Keeping this issue open for that failure alone.

   Also ruled out for it, for whoever picks it up:

   • fix(storage): make cassandra index readiness probe actually wait #810's index-readiness fix. The failing queries bypass secondary indexes entirely — ListOrgDomainsByOrg uses ALLOW FILTERING on the base table and GetOrgDomainByDomain is WHERE id = ?.
   • The omitempty write-drop class. IsActive carries no omitempty tag, so the field is always serialized.

   Unexplored leads: shared state between subtests, a host clock adjustment mid-run, and behaviour that only appears under full-suite load rather than on the isolated path.

   Related, and worth its own issue

   Session_Token_Operations failed on a separate run against main at provider_test.go:1073 — "a repeat delete must not claim the row". That is the mirror shape: AddSessionToken is a plain INSERT (session_token.go:25) while the delete is an LWT (:68), and the LWT's applied flag is the exactly-once arbiter behind refresh-token rotation. #818 leaves that table on a documented allow-list rather than guessing, because converging it means lifting the insert to an LWT and adding a Paxos round-trip to every login — a performance decision, not a cleanup.
3. lakhansamani

   added a commit that references this issue on Oct 15, 2026

   test(storage): stop the user suite depending on an almost-empty table

   ...

   18d44f7
4. lakhansamani commented on Oct 15, 2026

   

   lakhansamani

   on Oct 15, 2026

   Last edited by lakhansamani

   ContributorAuthor

   More actions

   Both failures accounted for. Two causes, neither of them flakiness in the usual sense.

   Org_Domain_Operations — a real storage bug, fixed in #818

   AddOrgDomain's lightweight transaction was stamped from the Paxos ballot (server clock) while DeleteOrgDomain was a plain row delete carrying gocql's client timestamp. A tombstone only shadows a row whose timestamp it exceeds, so with the server clock ahead the delete was discarded with no error and the domain stayed claimed. Details and measurements in #818.

   Re-measured today on a fresh container with a ~4ms engine clock lead: 48/50 plain deletes silently dropped, 0/50 with the fix. Forcing the skew makes it 50/50. Note the rate tracks the container's instantaneous clock offset, which is why it looked intermittent — the same probe on a different container start showed 0/50.

   User_SCIM_Fields — I could not reproduce it, and found why the suite looks flaky anyway

   0 failures in 100 targeted runs, before and after #818. I'm not able to call this one diagnosed.

   What I did find is that the suite fails hard on repeated local runs, for reasons that have nothing to do with storage. Running the scylladb storage suite in a loop gave 15 consecutive full-suite failures — on main, and equally on pre-#818:

   --- FAIL: .../User_Operations
       created user should appear in ListUsers
   

   Two self-inflicted causes, fixed in #824:

   1. ListUsers first-page assumption. User_Operations asserted the just-created user appears in ListUsers(Limit: 10, Offset: 0) unfiltered. That only holds while the table is nearly empty. At 463 accumulated rows it fails every time — and the failure reads exactly like a write that never landed, which is how it would get filed as a flaky storage test. Now filtered by the user's own email.
   2. Cascading pollution. That failure aborts before the subtest's cleanup, leaving a fixed +1234567891 phone literal in the table. Every later run then failed earlier with user with given phone number already exists — one failure mutating into a different, misleading one. Both phone numbers are unique per run now.

   Invisible in CI, which gets a fresh container per run. Deterministic locally.

   Same container, keyspace holding 463+ users throughout:

   	before	after
   full storage suite × 15	15/15 failed	0/15 failed
   User_SCIM_Fields / Org_Domain_Operations / Session_Token_Operations × 100	0 failed	0 failed

   Two more things came out of this

   #824 — the cassandradb provider had never been tested. It was missing from the storage suite's backend list, so the provider that serves both cassandradb and scylladb was only ever exercised through Scylla. Pointing it at cassandra:4.1 failed on the first run: three queries use CQL LIKE, which Apache Cassandra accepts only against a SASI index. GetWebhookByEventName errored out on every Apache Cassandra deployment, so webhook dispatch by event name never fired there. Same for two session-token delete methods. DeleteAllSessionTokensByUserID was also matching unanchored, so a delete for 42 took out tokens for 142 — live on ScyllaDB too. make test-cassandradb added.

   Also in #824 — the session_tokens / oauth_states tables had the mirror of the org_domain bug: plain insert (client clock) against DELETE … IF EXISTS (ballot). With the client clock ahead, the tombstone sits below the insert, the row survives, and Paxos still reports applied=true — so a single-use refresh token can be claimed twice. Measured 20/20 with the client clock 10ms ahead, 0/20 after. That is the Session_Token_Operations failure at provider_test.go:1073 ("a repeat delete must not claim the row") reported on main in this thread.

   Proposed disposition

   Keep this open until #824 merges, then close it. User_SCIM_Fields stays unreproduced — if it resurfaces after those land, it deserves a fresh issue rather than being tracked under a title that now describes something understood.
5. lakhansamani

   added 3 commits that reference this issue on Oct 22, 2026

   test(storage): cover the \z half of the mongo anchor

   ...

   6be312c

   docs(changelog): record #824 and retract the 818 margin claim

   ...

   d7563b9

   fix(storage): make the cassandradb provider correct on apache cassand…

   ...

   6e9a96c