Repository navigation
ENG-653: Flash-native payment splits for incoming payments (API, phase 1) - #526
Merged
Merged
Conversation
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
…ores Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
Every IBEX addInvoice is bound to the on-receive webhook, so system credits (Fygaro top-ups, referral payouts) and split payouts would otherwise trigger the recipient's payment splits. The marker write fails closed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
…ives Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6
…o-Flash, empty set - Cadence: run reconcile, executor and stale sweep every minute inside the ibex-webhook workload (startPaymentSplitsLoop, Redis lease for reconcile). The daily galoy cron keeps only a backstop reconcile. The reconcile lookback is derived from the caller's cadence (cadence plus margin: 25 min for the loop, 26 h for the cron) and pages through IBEX until it passes the lookback. The executor drains due rows within executorTimeBudgetMs instead of a fixed batch, so the 1/2/4/8 minute backoff is real. - Lock errors: retry only ResourceAttemptsLockServiceError raised before the send guard passed. Any other lock error, or one after the guard passed (the payment may have gone out), marks the row unknown and alerts ops. - Flash-to-Flash (decision 1): bolt11 pays mark the hash (origin ln-send) before paying, fail-closed; LNURL pays mark the IBEX-returned hash after paying, and the executor re-checks the marker before every payout (skipped_internal). A call-site test classifies every IBEX pay call. - walletSplitsSet with an empty list clears all splits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
…s, drain loop on SIGTERM - payMarkedBolt11 marks the bolt11, pays, and deletes its own ln-send marker when IBEX rejects the payment outright (insufficient balance, corroborated FAILED, other 4xx). Pending, unreadable, 5xx, timeouts and already-paid keep the marker. A rejected Flash attempt no longer turns off the payee's splits for a later external payment of the same invoice. - New pending split executions become claimable only after paymentSplits.claimGraceMs (default 2 min), so a post-payment LNURL internal marker lands before the executor's pay-time re-check. - startPaymentSplitsLoop returns stop(): no new tick, the executor claims no new row, and the in-flight send finishes. The ibex-webhook server awaits it on SIGTERM/SIGINT (bounded at 25s) and re-raises the signal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
The InternalTransferInvoice marker was unique on paymentHash alone. When two Flash wallets paid the same bolt11 concurrently, the second insert was a duplicate and owned no row; if the first payment was then rejected by IBEX, its withdrawal deleted the only marker and the second (successful) Flash-to-Flash payment could trigger the payee's splits (decision 1). The unique index is now (paymentHash, senderWalletId, origin). Each payer owns its own row and deleteInternalTransferInvoice still removes only the caller's row; isInternalTransferInvoice stays a hash-only lookup, so the hash is internal while any row exists. No reader decides policy from a single row's sender or origin. The collection is new in this PR and unreleased; graphql-main-server boots with syncIndexes(), which drops the old paymentHash_1 index wherever the branch may already have run, so no migration is needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
…BEX errors, wall-clock stamps Review fixes for ENG-653: - Execution dedupe is now unique on (triggerKey, walletId, destinationWalletId) instead of (triggerKey, ruleId). walletSplitsSet rewrites every rule with a fresh id, so a webhook replay after the merchant re-saved a split paid the same partner twice. The webhook also passes the receive time (settledAt/createdAt) as receivedAt, so the pre-rule guard covers it too. - After the send guard passed, an IBEX error that is not a definitive rejection (5xx, timeout, transport error) marks the row unknown and alerts ops instead of failing it and telling the owner it failed. - Claims and outcomes are stamped with wall-clock time, and each loop phase reads the clock when it starts, so a slow reconcile cannot let another replica's stale sweep take a row still being paid. A lost outcome (finishClaimed matched nothing) is logged at error level. - CompletedInvoice is a definitive rejection: the caller's ln-send marker is withdrawn, so a Flash user cannot suppress a merchant's splits by trying to pay an already-paid bolt11. - Transient repository errors before the send retry within the attempt budget; only a definite not-found fails at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
…cross re-saves - Thread a beforePay hook through the intraledger send. Only an IBEX error after payInvoice was reached is ambiguous (`unknown` + ops alert). An addInvoice failure or missing bolt11 moved nothing and is retried within maxAttempts (a definitive 4xx still fails). - replaceForWallet carries createdAt forward for destinations that survive the replace, so a re-save no longer drops a settled-but-unrecorded receive at reconcile time. - Integration spec: dedupe is on (triggerKey, walletId, destinationWalletId); a rule-rewrite replay is a no-op, another destination is not; index shape asserted; createdAt carry-forward and the reconcile-after-re-save case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4SaLJF5SWq6sjBEaWJFow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ENG-653: a merchant or user wants an incoming payment split automatically, for example 10% to a partner wallet. Flash had no rules for this. Splits must go through the normal send path so ENG-573 limits, KYC level, ENG-530 idempotency, notifications and history all apply.
The two input plans assumed Fygaro top-ups, referral payouts and split payouts never reach the
invoicewebhook. They do.intraledgerPaymentSendWalletIdmints the recipient invoice withIbex.addInvoice, andaddInvoicealways bindsonReceive.invoice. Without a marker, every system credit would trigger splits, and a split payout would trigger the next split in a loop.Change
Everything sits behind
paymentSplits.enabled, which defaults to false.paymentSplits { enabled, maxAttempts (5), minShareCents (1), loopIntervalMs (60s), executorTimeBudgetMs (45s), reconcileIntervalMs (15 min), cronIntervalMs (24h) }.send-intraledgerwrites anInternalTransferInvoicerow (origin: "intraledger") for the recipient invoice's payment hash betweenaddInvoiceandpayInvoice. The cash-wallet cutover does the same withorigin: "cutover". If the write fails, the send returnsInternalTransferMarkerErrorand nothing is paid. That error is not cached, so the same idempotency key can retry. The hash is decoded from the bolt11 being paid, with IBEX'sinvoice.hashas fallback. The marker has a 30-day TTL. Bolt11 pays (lnInvoicePaymentSend,lnNoAmountUsdInvoicePaymentSend) mark the invoice hash (origin: "ln-send") before paying, fail-closed, because the payee may be a Flash wallet; an undecodable bolt11 is paid unmarked since every IBEX-minted invoice decodes.lnurlPaymentSendcannot know the hash before IBEX pays, so it marks the hash IBEX returns (origin: "lnurl-send"), and the executor re-checks the marker before every payout (skipped_internal).PaymentSplitRule,PaymentSplitExecution(unique(triggerKey, walletId, destinationWalletId)index, one payout per receive per target, so a webhook replay after the rules were re-saved (new rule ids, same destinations) is still a duplicate) andInternalTransferInvoicecollections. Their repos return typedRepositoryErrors, and a duplicate insert is a typed no-op.validateSplitSetaccepts an empty set (clears all splits) and otherwise checks that the sum is in (0, 100], percents are integers from 1 to 100, there are at most 10 rules, no wallet splits to itself, destinations are unique, and every destination uses the source wallet's currency. Source wallets must be USD or USDT.computeSharefloors in minor units (USD cents, USDT micros) and then to whole cents. A share belowminShareCentsis recorded asskipped_min.recordPaymentSplitsmiddleware runs on theinvoiceandlnurlroutes only. It records rows and never sends. On any error it logs and moves on, so the response is still 200; the split reconcile (below) records what it missed. Receives with a marker record nothing, and replays hit the unique(triggerKey, walletId, destinationWalletId)index and do nothing, including a replay after a rule re-save. The webhook passes the receive'sreceivedAt, so a rule never fires on a receive that settled before the rule was created. A re-save keepscreatedAtfor every destination that survives it, so it never hides an earlier unrecorded receive from the reconcile.startPaymentSplitsLoop, runs inside the long-running ibex-webhook workload everyloopIntervalMs). The galoy cron is a daily k8s Job (chart0 2 * * *), too slow to pay or retry splits, so no chart change is needed or relied on:reconcileIntervalMsacross replicas (Redis NX lease, fail-open): reads IBEX type-1 receives on wallets with rules, paging newest-settled first until a page passes the lookback. The lookback is derived from the cadence (cadence plus a margin: 25 min here) and uses the same payment-hash key as the webhook. Type 9, ETH USDT crypto receives, is ignored per decision 2.pending → sendingupdate until none are due orexecutorTimeBudgetMsis spent (no fixed batch cap), then pays throughintraledgerPaymentSendWalletIdForUsdWalletwithauthorizeSend(kindintraledger) andidempotencyKey: split:<executionId>. Results map as follows:sentand pushes the owner.ibex_pendingand is never re-paid.failedon the first attempt and push the owner.ResourceAttemptsLockServiceErrorbefore the send guard passed), retry after 1, 2, 4 and 8 minutes, then becomefailedat the 5-attempt budget.unknownwith an ops alert and is never resent.sendingfor more than 10 minutes tounknownand sends an ops alert. It never resends.cronIntervalMsplus a margin (26h).walletSplitsSetandwalletSplitDeleteare registered underatWalletLevel, so the wallet-id middleware checks ownership. The app layer re-checks ownership too.Wallet.splitsis onUsdWalletandUsdtWallet, visible only to the wallet's owner.BTCWalletalways returns[]. It is not added toPublicWallet.write:wallet, and the nestedsplitsfield needsread:wallet. The SDL is regenerated.notification.paymentSplit.sentandnotification.paymentSplit.failed, in en and es.How it was tested
yarn tsc-check-noimplicitany,yarn tsc-check,yarn eslint-check,yarn build,yarn check:sdl,yarn check-yaml,yarn madge-checkandtypos --config typos.tomlall pass.make unit-in-ciruns): 290 suites, 3673 passed, 3 skipped.Wallet.splitsresolverIbex.addInvoicecall siteapi-key-scope-enforcement.spec.tsnow treatssplitsas a sensitive field.test/flash/integration/payment-splits: unique(triggerKey, walletId, destinationWalletId)index (a rule-rewrite replay is a no-op, another destination is not; index shape asserted), rulecreatedAtcarried across a re-save and a reconcile that records a receive settled before the re-save, one winner among 5 concurrent claimers, marker TTL index, rule replace semantics. Locally I ran it against a throwawaymongo:6.0.5container with a minimal mongoose-only setup. CI runs it under the normal integration harness.Follow-ups
BridgeDepositstx hash (decision 2).unknownandibex_pendingrows.addInvoicecaller appears.CashoutManageris classified as a Flash-owned recipient, but nothing stops someone from adding split rules on a Flash-owned wallet.maxAttemptsis raised to 6.cronIntervalMsmirrors the chart's cron schedule; if the chart schedule ever becomes less frequent than daily, raise it.🤖 Generated with Claude Code
https://claude.ai/code/session_01WxkKbrg2Xohf3mQj7BLrs6