Skip to content

ENG-653: Flash-native payment splits for incoming payments (API, phase 1) - #526

Merged
islandbitcoin merged 13 commits into
mainfrom
jabariennis/eng-653-payment-splits
Oct 12, 2026
Merged

islandbitcoin merged 13 commits into
mainfrom
jabariennis/eng-653-payment-splits

Conversation

@islandbitcoin

@islandbitcoin islandbitcoin commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem

ENG-653: a merchant or user wants an incoming payment split automatically, for example 10% to a partner wallet. Flash had no rules for this. Splits must go through the normal send path so ENG-573 limits, KYC level, ENG-530 idempotency, notifications and history all apply.

The two input plans assumed Fygaro top-ups, referral payouts and split payouts never reach the invoice webhook. They do. intraledgerPaymentSendWalletId mints the recipient invoice with Ibex.addInvoice, and addInvoice always binds onReceive.invoice. Without a marker, every system credit would trigger splits, and a split payout would trigger the next split in a loop.

Change

Everything sits behind paymentSplits.enabled, which defaults to false.

  • Config: paymentSplits { enabled, maxAttempts (5), minShareCents (1), loopIntervalMs (60s), executorTimeBudgetMs (45s), reconcileIntervalMs (15 min), cronIntervalMs (24h) }.
  • Marker (payment-critical code): Flash-to-Flash payments never trigger splits (decision 1). send-intraledger writes an InternalTransferInvoice row (origin: "intraledger") for the recipient invoice's payment hash between addInvoice and payInvoice. The cash-wallet cutover does the same with origin: "cutover". If the write fails, the send returns InternalTransferMarkerError and nothing is paid. That error is not cached, so the same idempotency key can retry. The hash is decoded from the bolt11 being paid, with IBEX's invoice.hash as fallback. The marker has a 30-day TTL. Bolt11 pays (lnInvoicePaymentSend, lnNoAmountUsdInvoicePaymentSend) mark the invoice hash (origin: "ln-send") before paying, fail-closed, because the payee may be a Flash wallet; an undecodable bolt11 is paid unmarked since every IBEX-minted invoice decodes. lnurlPaymentSend cannot know the hash before IBEX pays, so it marks the hash IBEX returns (origin: "lnurl-send"), and the executor re-checks the marker before every payout (skipped_internal).
  • Data: new PaymentSplitRule, PaymentSplitExecution (unique (triggerKey, walletId, destinationWalletId) index, one payout per receive per target, so a webhook replay after the rules were re-saved (new rule ids, same destinations) is still a duplicate) and InternalTransferInvoice collections. Their repos return typed RepositoryErrors, and a duplicate insert is a typed no-op.
  • Domain: validateSplitSet accepts an empty set (clears all splits) and otherwise checks that the sum is in (0, 100], percents are integers from 1 to 100, there are at most 10 rules, no wallet splits to itself, destinations are unique, and every destination uses the source wallet's currency. Source wallets must be USD or USDT. computeShare floors in minor units (USD cents, USDT micros) and then to whole cents. A share below minShareCents is recorded as skipped_min.
  • Webhook: a recordPaymentSplits middleware runs on the invoice and lnurl routes only. It records rows and never sends. On any error it logs and moves on, so the response is still 200; the split reconcile (below) records what it missed. Receives with a marker record nothing, and replays hit the unique (triggerKey, walletId, destinationWalletId) index and do nothing, including a replay after a rule re-save. The webhook passes the receive's receivedAt, so a rule never fires on a receive that settled before the rule was created. A re-save keeps createdAt for every destination that survives it, so it never hides an earlier unrecorded receive from the reconcile.
  • Split loop (startPaymentSplitsLoop, runs inside the long-running ibex-webhook workload every loopIntervalMs). The galoy cron is a daily k8s Job (chart 0 2 * * *), too slow to pay or retry splits, so no chart change is needed or relied on:
    • Reconcile, once per reconcileIntervalMs across replicas (Redis NX lease, fail-open): reads IBEX type-1 receives on wallets with rules, paging newest-settled first until a page passes the lookback. The lookback is derived from the cadence (cadence plus a margin: 25 min here) and uses the same payment-hash key as the webhook. Type 9, ETH USDT crypto receives, is ignored per decision 2.
    • Executor: claims rows with an atomic pending → sending update until none are due or executorTimeBudgetMs is spent (no fixed batch cap), then pays through intraledgerPaymentSendWalletIdForUsdWallet with authorizeSend (kind intraledger) and idempotencyKey: split:<executionId>. Results map as follows:
      • Success becomes sent and pushes the owner.
      • Pending becomes ibex_pending and is never re-paid.
      • Limit, KYC, insufficient-balance and inactive-account errors become failed on the first attempt and push the owner.
      • Rate-limit, guard-unavailable, marker and IBEX 429 errors, and a lock that was never acquired (ResourceAttemptsLockServiceError before the send guard passed), retry after 1, 2, 4 and 8 minutes, then become failed at the 5-attempt budget.
      • Any other lock error, or any lock error after the send guard passed (redlock can report one after IBEX paid, when the routine throws or the release fails), becomes unknown with an ops alert and is never resent.
    • Stale sweep: moves a row stuck in sending for more than 10 minutes to unknown and sends an ops alert. It never resends.
  • Cron: keeps only a backstop reconcile for the loop being down, with a lookback of cronIntervalMs plus a margin (26h).
  • GraphQL:
    • walletSplitsSet and walletSplitDelete are registered under atWalletLevel, so the wallet-id middleware checks ownership. The app layer re-checks ownership too.
    • Wallet.splits is on UsdWallet and UsdtWallet, visible only to the wallet's owner. BTCWallet always returns []. It is not added to PublicWallet.
    • API-key scopes are set: both mutations need write:wallet, and the nested splits field needs read:wallet. The SDL is regenerated.
  • Notifications: notification.paymentSplit.sent and notification.paymentSplit.failed, in en and es.

How it was tested

  • yarn tsc-check-noimplicitany, yarn tsc-check, yarn eslint-check, yarn build, yarn check:sdl, yarn check-yaml, yarn madge-check and typos --config typos.toml all pass.
  • Full unit suite (the command make unit-in-ci runs): 290 suites, 3673 passed, 3 skipped.
  • New unit specs:
    • domain validation and share math
    • set and delete splits: username vs walletId resolution, same-currency wallet pick, ownership re-check, lock held
    • record: replay dedupe, marked hash, flag off
    • executor: success, pending, terminal failures, backoff, stable idempotency key, concurrent claim
    • sweep and reconcile
    • webhook routes
    • mutation resolvers and the Wallet.splits resolver
    • send-intraledger and cutover marker ordering and fail-closed behavior
    • a guard test that lists every Ibex.addInvoice call site
  • api-key-scope-enforcement.spec.ts now treats splits as a sensitive field.
  • New integration spec test/flash/integration/payment-splits: unique (triggerKey, walletId, destinationWalletId) index (a rule-rewrite replay is a no-op, another destination is not; index shape asserted), rule createdAt carried across a re-save and a reconcile that records a receive settled before the re-save, one winner among 5 concurrent claimers, marker TTL index, rule replace semantics. Locally I ran it against a throwaway mongo:6.0.5 container with a minimal mongoose-only setup. CI runs it under the normal integration harness.

Follow-ups

  • Trigger on ETH USDT (type 9) deposits that do not match a BridgeDeposits tx hash (decision 2).
  • Write an ops runbook for resolving unknown and ibex_pending rows.
  • Mobile UI.
  • The marker is a denylist. The call-site test fails when a new addInvoice caller appears. CashoutManager is classified as a Flash-owned recipient, but nothing stops someone from adding split rules on a Flash-owned wallet.
  • Split sends use the owner's ENG-573 attempt budget and daily limit.
  • The executor uses a 5-attempt budget, so the 16-minute step is only reached if maxAttempts is raised to 6.
  • cronIntervalMs mirrors the chart's cron schedule; if the chart schedule ever becomes less frequent than daily, raise it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6

bobodread876 and others added 8 commits October 11, 2026 22:57
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
Every IBEX addInvoice is bound to the on-receive webhook, so system credits
(Fygaro top-ups, referral payouts) and split payouts would otherwise trigger
the recipient's payment splits. The marker write fails closed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
@linear

linear Bot commented Oct 12, 2026

Copy link
Copy Markdown

ENG-653

bobodread876 and others added 5 commits October 11, 2026 23:14
…o-Flash, empty set

- Cadence: run reconcile, executor and stale sweep every minute inside the
  ibex-webhook workload (startPaymentSplitsLoop, Redis lease for reconcile).
  The daily galoy cron keeps only a backstop reconcile. The reconcile lookback
  is derived from the caller's cadence (cadence plus margin: 25 min for the
  loop, 26 h for the cron) and pages through IBEX until it passes the lookback.
  The executor drains due rows within executorTimeBudgetMs instead of a fixed
  batch, so the 1/2/4/8 minute backoff is real.
- Lock errors: retry only ResourceAttemptsLockServiceError raised before the
  send guard passed. Any other lock error, or one after the guard passed (the
  payment may have gone out), marks the row unknown and alerts ops.
- Flash-to-Flash (decision 1): bolt11 pays mark the hash (origin ln-send)
  before paying, fail-closed; LNURL pays mark the IBEX-returned hash after
  paying, and the executor re-checks the marker before every payout
  (skipped_internal). A call-site test classifies every IBEX pay call.
- walletSplitsSet with an empty list clears all splits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
…s, drain loop on SIGTERM

- payMarkedBolt11 marks the bolt11, pays, and deletes its own ln-send marker
  when IBEX rejects the payment outright (insufficient balance, corroborated
  FAILED, other 4xx). Pending, unreadable, 5xx, timeouts and already-paid
  keep the marker. A rejected Flash attempt no longer turns off the payee's
  splits for a later external payment of the same invoice.
- New pending split executions become claimable only after
  paymentSplits.claimGraceMs (default 2 min), so a post-payment LNURL
  internal marker lands before the executor's pay-time re-check.
- startPaymentSplitsLoop returns stop(): no new tick, the executor claims no
  new row, and the in-flight send finishes. The ibex-webhook server awaits it
  on SIGTERM/SIGINT (bounded at 25s) and re-raises the signal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
The InternalTransferInvoice marker was unique on paymentHash alone. When two
Flash wallets paid the same bolt11 concurrently, the second insert was a
duplicate and owned no row; if the first payment was then rejected by IBEX,
its withdrawal deleted the only marker and the second (successful)
Flash-to-Flash payment could trigger the payee's splits (decision 1).

The unique index is now (paymentHash, senderWalletId, origin). Each payer owns
its own row and deleteInternalTransferInvoice still removes only the caller's
row; isInternalTransferInvoice stays a hash-only lookup, so the hash is
internal while any row exists. No reader decides policy from a single row's
sender or origin.

The collection is new in this PR and unreleased; graphql-main-server boots
with syncIndexes(), which drops the old paymentHash_1 index wherever the
branch may already have run, so no migration is needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
…BEX errors, wall-clock stamps

Review fixes for ENG-653:

- Execution dedupe is now unique on (triggerKey, walletId,
  destinationWalletId) instead of (triggerKey, ruleId). walletSplitsSet
  rewrites every rule with a fresh id, so a webhook replay after the
  merchant re-saved a split paid the same partner twice. The webhook also
  passes the receive time (settledAt/createdAt) as receivedAt, so the
  pre-rule guard covers it too.
- After the send guard passed, an IBEX error that is not a definitive
  rejection (5xx, timeout, transport error) marks the row unknown and
  alerts ops instead of failing it and telling the owner it failed.
- Claims and outcomes are stamped with wall-clock time, and each loop
  phase reads the clock when it starts, so a slow reconcile cannot let
  another replica's stale sweep take a row still being paid. A lost
  outcome (finishClaimed matched nothing) is logged at error level.
- CompletedInvoice is a definitive rejection: the caller's ln-send marker
  is withdrawn, so a Flash user cannot suppress a merchant's splits by
  trying to pay an already-paid bolt11.
- Transient repository errors before the send retry within the attempt
  budget; only a definite not-found fails at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
…cross re-saves

- Thread a beforePay hook through the intraledger send. Only an IBEX error
  after payInvoice was reached is ambiguous (`unknown` + ops alert). An
  addInvoice failure or missing bolt11 moved nothing and is retried within
  maxAttempts (a definitive 4xx still fails).
- replaceForWallet carries createdAt forward for destinations that survive
  the replace, so a re-save no longer drops a settled-but-unrecorded receive
  at reconcile time.
- Integration spec: dedupe is on (triggerKey, walletId, destinationWalletId);
  a rule-rewrite replay is a no-op, another destination is not; index shape
  asserted; createdAt carry-forward and the reconcile-after-re-save case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
@islandbitcoin
islandbitcoin merged commit aeef6b7 into main Oct 12, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants