draft: Better payment value distribution with rebalancing

[carlaKC]

We know that the way that we generate traffic isn't great - see #301.

The former attempt at fixing this was abandoned because it caused simulations to un-balance, which was problematic for the long simulations we were running. A solution here is to allow an "arbitrary auto rebalance" in the network so that payments can keep flowing.

This is useful for the channel jamming case, because we just care about there being some activity that is reproducible across networks.

TBD:

• Check the math for payment amounts
• We need to be able to exclude some channels from auto-rebalance

[carlaKC]

Went through this with Clara and it seems to output far more reasonable values: - sample output from test below:

Setup — two nodes, 1 BTC deployed capacity each, default E = 3,800,000 msat:

payment limit L: 50,000,000,000 msat, E: 3,800,000 msat, L/E: 13158
mu: 14.8039, sigma: 0.8326          ← sigma at MAX_SIGMA, as expected at this L/E

First 1000 amounts (opening rows; full grid in the run above):

    0 │  4,000,314   8,158,938   2,254,367   3,994,875   1,754,324   1,234,714   1,166,366   5,763,817
    8 │ 15,717,203   1,477,272   2,758,127   6,413,662  14,589,930   3,612,185   4,330,396   1,346,217
   16 │  3,415,592   1,741,610   2,633,663   1,377,395   1,958,138   5,389,653     656,615     225,431
   24 │  1,553,823   6,790,277   4,355,790   3,854,209   3,953,198   5,629,915     657,087     760,481

Summary:

                                     msat     sample      theory
 ---------------------------------------------------------------
 mean                            3,625,855     0.954E      1.000E
 median                          2,733,290     0.719E      0.707E
 std dev                         3,481,424     0.916E      1.000E

 mean − 1 sd                       144,431     0.038E      0.000E
 mean + 1 sd                     7,107,280     1.870E      2.000E

 16th pct (median ÷ e^σ)         1,169,808     0.308E      0.308E
 84th pct (median × e^σ)         5,755,897     1.515E      1.626E

 min                               114,362     0.030E
 max                            52,013,022    13.688E
 ---------------------------------------------------------------
 CV: 0.960 (sample) vs 1.000 (theory)
 payments above the limit: 0/1000

[only1dreamgene]

Skimmed through this, mostly looking at the payment-amount distribution logic and the new rebalancer. The log-normal / sigma-cap approach for payment_amount looks solid and the math checks out (including the sigma=0 degenerate case). Left two inline comments on things worth a look before this comes out of draft.

[only1dreamgene]

This holds channels_lock for the entire scan of scids (lines 1239-1253), not just per-channel. propagate_payment needs the same channels mutex to add/remove HTLCs per hop, so every in-flight payment forward will stall until the whole rebalance scan finishes. With a large network and a short interval_secs, this could show up as periodic simulation-wide latency spikes that scale with network size. Might be worth releasing/reacquiring the lock per-channel (or per-batch) instead of holding it across the full loop.

[only1dreamgene]

This changed from an exact equality check (payments_list.lock().unwrap().as_ref() == expected_payment_list) to a .take(expected_payment_list.len()) prefix comparison. That means the test no longer catches cases where the simulation emits fewer payments than expected, or extra/reordered payments beyond the first N — it'll pass as long as the prefix matches. Given this PR is changing the payment-amount and fee-limit logic that feeds this exact test, it'd be good to double check this loosening was intentional and not just masking a timing flake.

[carlaKC]

Likewise here if you'd be interested in picking this up @only1dreamgene it's certainly up for grabs!

@elnosh maybe you'd be interested in grabbing this one if you're running sims concerned with routability? I reckon it'll give us better payment amounts to think about.