feat(fygaro): float runway alerts + stranded-credit retry sweep

[islandbitcoin]

Problem

On 2026-10-03 the bankowner treasury (the Fygaro auto-credit funding source) ran dry. A $70 card top-up was captured, the credit failed with insufficient-treasury-float, and the customer was charged with nothing delivered until ops hand-credited $66.02 and clicked Mark Completed.

Three gaps made that worse than it needed to be:

1. The warning was ignorable. The float monitor fired a warning every day for seven weeks (balance under $2000 since the day after the last refill). It carried no runway figure, never escalated, and shared nothing with the per-event CRITICAL that only fires after a customer has been charged.
2. The cadence was not ours. The check ran from the cron Job, whose schedule the chart owns. In prod that was daily at 02:00.
3. A failed credit never retried. payment.ts deliberately leaves the row Fiat Received with no failure_reason to invite a provider retry, but Fygaro does not redeliver a 200. Every float failure was a manual credit + a manual status stamp.

What changed

Float monitor (float-monitor.ts)

• Computes runway = balance / trailing-7-day net credited (ERPNext Completed rows, final_amount). Burn read only when balance < 2x floor.
• Two tiers: warning below floorUsd; critical below criticalFloorUsd (default $500 = the auto-credit single-payment limit) or under criticalRunwayDays (default 3). Critical has its own dedup key and its own Redis marker so an open warning window cannot swallow it.
• Alert detail/context carry balance, floor, daily burn, runway, and the fund URL (ERPNext System Accounts page).
• ERPNext is an input to runway only. A failed or throwing burn read still fires the floor alert with runway "unknown".
• Returns the reading so the loop can decide whether to sweep.

Stranded-credit sweep (stranded-credit-sweep.ts, new)

• Candidates: ERPNext rows provider=Fygaro, Topup, Fiat Received, USD, attributed, no failure_reason, not email-attributed, within retry.lookbackDays, oldest first, capped at retry.maxPerSweep. Rows refused by a gate carry a failure_reason and are never touched.
• Processed markers are read first, before any gate or coverage check: the ERPNext row (ops may have hand-completed it) and a durable Redis "credited" marker (credited-marker.ts, written the moment a send succeeds, 30-day floor TTL decoupled from lookbackDays). A marker-credited row that is still not Completed pages critical ("promote by hand") and is never re-sent.
• The FULL webhook gate is re-run (evaluateCreditGate over live settings, account level, trailing-24h gross excluding this row, operator fee-discount whitelist) — including the daily cap. Why: payment.ts answers 500 with no stamp for the transient settings-unavailable / history-unavailable refusals, so a candidate may never have passed the cap at all; crediting it blind could put a customer over a compliance cap with no refusal row and no alert. Outcomes:
  • transient reasons (settings-unavailable, history-unavailable): no stamp, retried next tick;
  • deterministic refusals (daily-limit-exceeded, over auto-credit limit, under minimum, non-USD, non-positive net, level with no cap): failure_reason stamped exactly as the webhook stamps it, warning alert per payment, row left for manual (and dropped from the daily-allowance sum). A failed stamp pages critical.
• Send is the same creditFygaroTopup (idempotent on fygaro:<transactionId>) with the gate's own fee figure.
• Coverage: availableUsd is balance minus the critical floor (the loop passes the reserve-adjusted figure). A row whose net exceeds what is left is reported uncovered (warning, per payment, "available above reserve") and skipped so a big row never blocks a small one. A float failure from the credit itself stops the sweep and pages fygaro:float-exhausted.
• Aging out is never silent. Candidates are windowed on last_seen_at, which a failed or skipped retry never touches, so a row that keeps failing drops out of the list on day lookback+1. Every sweep also counts the same-shape rows older than the window (countAgedOutUncreditedFygaroTopups) and pages critical once per dedup window with the count and oldest request_id ("credit by hand").
• On success: Redis marker written before the ERPNext promotion; row promoted with the full fee split (no more "Pending" fees on hand-completed rows); checkout intent stamped credited when the payload carries one; ops-feed success tagged retry: stranded-credit-sweep; customer pushed once under the same fygaro-credit-push: claim the webhook uses. A failed promotion after a successful send pages critical and is not retried by re-sending.

Treasury loop (treasury-loop.ts, new; started from webhook-server/index.ts)

• Float check + sweep every fygaro.float.checkIntervalMs (default 15 min) inside the long-running fygaro-webhook pod. Redis NX lease per tick so one replica runs. First run 30s after boot. Sweep only when balance >= critical floor, and the sweep is handed balance − criticalFloorUsd: the floor is a reserve for live card traffic, so a backlog can never drain the float through it.
• The cron Job keeps calling the float check too; both paths dedup through the same Redis markers.

Config (schema.ts, schema.types.d.ts, base-config.yaml) — all defaulted, no override required to deploy:

knob	default
fygaro.float.floorUsd	2000 (unchanged)
fygaro.float.criticalFloorUsd	500
fygaro.float.criticalRunwayDays	3
fygaro.float.checkIntervalMs	900000
fygaro.float.fundUrl	https://erp.flashapp.me/app/system-accounts
fygaro.retry.enabled	true
fygaro.retry.lookbackDays	7 (must stay below the 30-day credited-marker TTL; see schema comment)
fygaro.retry.maxPerSweep	20

ERPNext readers (ErpNext.ts): sumFygaroCompletedNetCentsSince, listUncreditedFygaroTopups, countAgedOutUncreditedFygaroTopups. Reads only, existing fields only.

Deploy notes

• No frappe-flash-admin change required. Both readers use fields already on Bridge Transfer Request.
• No chart change required for the defaults. Set fygaro.float.fundUrl in the TEST overlay if the alert should point at the TEST ERP.
• The fygaro-webhook workload needs Redis reachable (it already does, for checkout intents). Lease and markers are fail-open: a Redis blip means duplicate alerts, never silence.
• critical severity routes wherever alertBridge sends criticals today. Confirm that path pages a human; the whole point is that warning did not. New criticals from this PR: float critical, float exhausted during retry, credited-but-not-promoted, refusal-not-stamped, aged-out stranded rows.
• Do not raise fygaro.retry.lookbackDays to 30 or above without first raising MIN_CREDITED_MARKER_TTL_DAYS in credited-marker.ts.
• First deploy will sweep any Fiat Received / no-reason / attributed Fygaro rows from the last 7 days. As of 2026-10-03 there were none outstanding (verified against prod ERP), so expect a no-op.

Test plan

• yarn tsc-check clean, eslint clean on touched files, prettier --check (3.6.2) clean, typos clean.
• Full unit suite: 277 suites / 3487 passed (3 skipped).
• New/extended specs: float-monitor.spec.ts (30: runway math, tiering, separate critical marker, ERP fail-open on error and on throw, defaults), stranded-credit-sweep.spec.ts (gates incl. ERPNext kill-switch, processed markers first, full gate re-run incl. daily cap with transient-vs-deterministic outcomes, fee math + whitelist, reserve coverage ($600 float / $500 floor / $280 row → uncovered), float decrement, stop-on-float-exhausted, per-payment failure alert, aged-out paging incl. zero-candidate and error paths, never throws), credited-marker.spec.ts (new: 30-day TTL floor independent of lookback, fail-open write, unknown-on-read-error), treasury-loop.spec.ts (tick gating on critical floor, balance − floor handed to the sweep, boot delay + interval, lease TTL, lease held skip, Redis fail-open, clamp, survives throw), ErpNext.spec.ts (+20 for the three readers), BridgeTransferRequestWriter.spec.ts (+6), webhook-server/index.spec.ts (+1 boot wiring).

Follow-ups (not in this PR)

• Gate fygaroCheckoutCreate on float coverage so a card is never charged for a credit that cannot be delivered. Inert until mobile calls that mutation (ENG-443/444).
• Auto-rebalance bankowner from funder/dealer via the existing transfer_between_system_wallets.
• ENG-545 reconciliation remains the backstop for payments that never reach the webhook at all.

🤖 Generated with Claude Code