Skip to content

feat(transport): Enter probing mode upon connection failure - #260

Open
j-g00da wants to merge 1 commit into
mainfrom
j-g00da/worker-probing-mode
Open

j-g00da wants to merge 1 commit into
mainfrom
j-g00da/worker-probing-mode

Conversation

@j-g00da

@j-g00da j-g00da commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Introduces a probing mode to Worker.

If all transport channels failed, we enter "probing" mode, which defers everything immediately until a retry succeeds.

This way we prevent large delays on group chats when some relay is down. (Transport handler won't have to wait whole 9 minutes until timeout.)

Related: #252

Comment thread src/transport/worker.rs
Introduces a probing mode to Worker.

If all transport channels failed, we enter "probing" mode,
which defers everything immediately until a retry succeeds.

This way we prevent large delays on group chats when some relay is down.
(Transport handler won't have to wait whole 9 minutes until timeout.)

Related: #252
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
@j-g00da
j-g00da force-pushed the j-g00da/worker-probing-mode branch from eb20831 to 7d7c767 Compare October 23, 2026 10:50
Comment thread src/transport/worker.rs
return rx;
}
// timeout here is so that the task can exit, even if no message arrive
if let Ok(Some(m)) = timeout(Duration::from_secs(5), rx.recv()).await {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to read from deferrer_rx and rx at the same time, whatever comes first, you probably want tokio::select! instead of arbitrary timeouts.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But deferrer_rx is a oneshot receiver, so it would be moved in the first iteration of the loop. Only thing I can think of is changing it to mpsc, but perhaps i'm missing something?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then maybe replace oneshot channel with cancellation token? This is how core cancels IMAP/SMTP loops.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL, thanks

Comment thread src/transport/worker.rs
&https_client,
dns_resolver.clone(),
destination.clone(),
&message.envelope,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the same message tried in a loop until it is delivered? It does not look good to try unreachable destination every minute if no messages are sent to it. It could be that the message itself is bad, e.g. too large, and this is why we get timeouts, but we may be able to deliver smaller messages still. And the messages sent to the worker are immediately discarded at the same time.

I think it's better to try the queued messages once. If sending fails, we don't exit probing mode, but wait for the next message to be queued. If nothing is queued, then worker should not send anything and quit eventually.

Also WORKER_KEEPALIVE_DURATION is now on the outer receiver, so if you only send a single message to a destination that resolves to an IP which does not respond, you get a worker that never quits and tries to connect forever every minute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants