Skip to content

[SYSTEMDS-3960] Fix federated worker readiness probe on delayed connections - #2587

Open
gaturchenko wants to merge 1 commit into
apache:mainfrom
gaturchenko:worker-coord-network-delay
Open

[SYSTEMDS-3960] Fix federated worker readiness probe on delayed connections#2587
gaturchenko wants to merge 1 commit into
apache:mainfrom
gaturchenko:worker-coord-network-delay

Conversation

@gaturchenko

Copy link
Copy Markdown
Contributor

With any delay on the worker's interface, currently every federated test fails before it starts, while the worker is listening the whole time. FederatedWorkerUtils.tryConnect allocated 25ms per connect attempt, but a TCP handshake needs two traversals of the link which are ~100ms under the delay above. Every attempt expired with a SocketTimeoutException, which is an IOException and therefore indistinguishable from a closed port, so the probe never succeeded.

The following changes were implemented:

  • Per-attempt connect budget increased from 25ms to 2s. A closed port is refused by the kernel immediately, so it only applies once a handshake is in progress. Additionally, 2s covers one lost SYN, retransmitted after ~1s
  • Each attempt is capped by the time left until the overall deadline (attemptTimeout), so a slow connect cannot exceed it. The returned value of 0 implies the attempt was never made
  • The waitForWorkers now rechecks the deadline per port, which can now cost up to 2s each
  • Six new tests are added verifying the change

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant