[afnan-mojumder]

Closes #466

Summary

While Pingora allows configuring an upper bound on retries globally via conf.max_retries, proxy operators currently face three major limitations when implementing retry and backoff strategies:

1. Inaccessible Retry Count: In HttpProxy::process_request, the retries counter was maintained only as a local stack variable. Neither Session nor user filters (upstream_peer, fail_to_connect, error_while_proxy, logging, etc.) could inspect which attempt was currently in progress. As noted in Make MAX_RETRIES configurable #466 and community discussions, users had to manually invent and increment custom counter fields across filters, which breaks whenever an error occurs outside fail_to_connect.
2. Static Retry Limits: max_retries could not be configured per-request or per-route based on path, headers, or client identity.
3. Synchronous Sleep Anti-Pattern: Because fail_to_connect is synchronous, users wanting a delay before retrying resorted to std::thread::sleep, which blocks Tokio worker threads and harms proxy concurrency. In examples/backoff_retry.rs, sleep was temporarily hacked into upstream_peer.

This pull request introduces clean, non-breaking abstractions to resolve all three issues.

---

Changes

1. Attempt & Retry Tracking on Session:

   • Added Session::tries(&self) -> usize: Returns 0 before upstream attempts begin, 1 on the initial attempt, and increments on each retry.
   • Added Session::retry_attempts(&self) -> usize: Returns 0 on the initial attempt, 1 on the first retry, etc. (self.tries.saturating_sub(1)).
   • This allows any filter or peer selector to query the current attempt directly from session.

2. Per-Request max_retries Hook:

   • Added ProxyHttp::max_retries(&self, session: &Session<DS>, ctx: &Self::CTX) -> Option<usize>:
     • Returns None by default to preserve existing behavior (falling back to conf.max_retries).
     • Returning Some(n) allows setting per-request limits (e.g. Some(1) to disallow retries on specific routes, or higher limits for cold-starting backends).

3. Asynchronous retry_backoff Hook:

   • Added ProxyHttp::retry_backoff(&self, session: &mut Session<DS>, ctx: &mut Self::CTX, error: &Error) -> bool:
     • Called asynchronously before initiating the next upstream attempt when an error is deemed retryable.
     • Allows performing non-blocking async delay (e.g. tokio::time::sleep).
     • Allows dynamic retry cancellation (e.g. when an elapsed time budget / max duration is exceeded) by returning false.
     • Returns true by default (no-op).

4. Updated Example & Documentation:

   • Updated pingora-proxy/examples/backoff_retry.rs to show the idiomatic use of retry_backoff, max_retries, and session.retry_attempts().

5. Unit Tests:

   • Added comprehensive tests under mod retry_configuration_and_backoff:
     • session_tries_and_retry_attempts_initial
     • max_retries_override_and_retry_backoff_called
     • max_retries_override_allows_single_attempt
     • retry_backoff_can_abort_retry_early
     • max_retries_falls_back_to_server_conf

---

Verification

• cargo check -p pingora-proxy: PASS
• cargo clippy --all-targets -p pingora-proxy -- -D warnings: PASS (0 warnings)
• cargo fmt --all -- --check: PASS
• cargo test -p pingora-proxy --lib: PASS (all 227 unit tests pass)
• cargo build --example backoff_retry: PASS