Repository navigation
bus: log a redis queue lock that fails instead of dropping it silently - #156
Open
cosmin-staicu wants to merge 1 commit into
Open
cosmin-staicu wants to merge 1 commit into
cosmin-staicu wants to merge 1 commit into
Conversation
A queue subscription decides which receiver handles a message with a SET NX lock on the payload hash. When that command failed, read() skipped the message exactly as it does when another receiver won the lock, with nothing logged. A queue RPC whose only server hit an error there was lost without a trace, and the client saw "no response from servers". Log the error before skipping the message, unless the subscription is closing. Losing the lock to another receiver is the normal outcome and stays silent. The message is still not retried: go-redis already retries connection errors, and after an ambiguous reply this receiver cannot tell whether it holds the lock. The tests install a recording logger once for the package, since the psrpc logger is a global that running bus goroutines read, and use a go-redis hook to fail SET on demand: a failed lock is logged once and the next message is delivered, and a lock lost to a second receiver logs nothing. Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A queue subscription picks the receiver for a message with a SET NX lock on the payload hash. When that command fails,
read()skips the message exactly as it does when another receiver won the lock, and logs nothing. A queue RPC whose only server hit an error there is lost without a trace, and the client sees "no response from servers".The error is now logged before the message is skipped, unless the subscription is closing. Losing the lock to another receiver is the normal case and stays silent. The message is still not retried: go-redis already retries connection errors, and after an ambiguous reply this receiver can't tell whether it holds the lock.
The psrpc logger is a global that running bus goroutines read, so the tests install a recording logger once in
TestMain. A go-redis hook fails SET on demand. The tests check that a failed lock is logged once and the next message is still delivered, and that a lock lost to a second receiver logs nothing.