drpc: classify connection-teardown errors at the source#76
Open
shubhamdhama wants to merge 2 commits into
Open
drpc: classify connection-teardown errors at the source#76shubhamdhama wants to merge 2 commits into
shubhamdhama wants to merge 2 commits into
Conversation
suj-krishnan
reviewed
Jul 2, 2026
| // do not recognize. | ||
| func isTransportError(err error) bool { | ||
| var opErr *net.OpError | ||
| return errors.As(err, &opErr) || errors.Is(err, net.ErrClosed) |
There was a problem hiding this comment.
I'm wondering if this should be a catch-all ("gRPC maps every transport failure"), since the net package has other kinds of errors as well. I haven't checked the gRPC code though.
var netErr net.Error
if errors.As(err, &netErr) {
return true
}
Author
There was a problem hiding this comment.
I've changed the whole way of detecting the connection failure. Right at the source we will appropriately classify the failure.
5d6ae53 to
b633dc9
Compare
drpc.ToRPCErr maps drpc's connection-teardown errors (ConnectionError,
ClosedError) to codes.Unavailable so callers can uniformly detect a dead
connection. The read path wraps teardown errors accordingly
(drpcwire.Reader.read, drpcmanager.Manager.terminate), but the mux write
path did not: when MuxWriter's flush to the wire failed it stored and
propagated the raw *net.OpError, which fell through ToRPCErr to
codes.Unknown.
With stream multiplexing a single long-lived connection is reused, so the
client writes before the read side observes the close and the write hits
EPIPE ("broken pipe") first. Connection-liveness checks keyed on
Unavailable (and the retry and circuit-breaker logic built on them) then
fail to recognize the write-side teardown.
Wrap the failed write in drpc.ConnectionError at its source in
MuxWriter.run, symmetric with the read path, so ToRPCErr's existing
ConnectionError branch maps it to codes.Unavailable. The underlying error
is preserved in the chain. This keeps the classification with the code
that operated the transport rather than re-deriving it from net error
types at the gRPC boundary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
b633dc9 to
46e2e3d
Compare
The pattern here is simple: classify errors where they happen, and only
translate them at the edge.
drpc.ToRPCErr is just a translator from drpc errors to gRPC status codes.
It only recognizes errors that were already tagged with a class it knows
about: ConnectionError and ClosedError map to Unavailable (matched even
through wrapping), the context and EOF sentinels map to their own codes
(matched by identity), and everything else becomes Unknown. The boundary
has no way to recover intent that was never attached to the error, so any
subsystem that can tear a connection down has to classify its own error at
the source. The edge only translates.
The manager left two teardown causes unclassified, so they reached the
boundary as Unknown and defeated liveness checks like
grpcutil.IsClosedConnection. We saw this as a TestDrain flake under DRPC
stream multiplexing: the server kept reporting "not yet refusing RPC"
because the teardown error carried Unknown instead of Unavailable. The two
causes were a deliberate Manager.Close(), which showed up as "manager
closed: Close called", and an abrupt read-side EOF.
The fix classifies each one by its cause, never with a catch-all:
- Close() means the connection is gone, so it terminates with a
ClosedError (Unavailable) and keeps the "Close called" cause underneath.
- A read EOF means the peer hung up. For a client the connection is gone
(ClosedError, Unavailable); for a server a client hang-up is really a
canceled RPC (context.Canceled), which is how gRPC behaves too.
- Every other reader error is only wrapped in managerClosed, which is
transparent to ToRPCErr. An error the reader already classified as a
ConnectionError still maps to Unavailable, while a protocol or internal
fault correctly stays Unknown instead of being mistaken for a retryable
connection loss.
With this, terminate() no longer derives anything on its own. It just fans
the single, already-classified cause out to everyone: the writer, the
in-flight streams, and any stream created after teardown. Because the cause
now lives in the term signal already classified, NewServerStream, which
returns term.Err() directly, is fixed for free. It used to leak the raw
cause.
One caveat worth calling out: never bury a sentinel like io.EOF or
context.Canceled inside an opaque wrapper such as managerClosed. ToRPCErr
matches sentinels by identity, so it would miss them through the wrapper.
Connection classes are matched through wrapping, so wrapping those is fine.
This also removes isConnectionReset. The reader already classifies every
non-EOF I/O error, including connection resets, as ConnectionError, which
ToRPCErr maps to Unavailable, so the reset-sniffing was dead code.
The pattern is now documented where the next person will run into it:
ToRPCErr (the translator and the source-classification contract), the
error-class definitions in drpc.go (what the edge understands, and liveness
vs. fault), and readError in the manager.
This is the read and lifecycle half of the fix. The write half landed in
"drpcwire: classify write failures as ConnectionError", where MuxWriter now
tags a failed write as ConnectionError at the source, mirroring the reader.
Together, both I/O directions and the lifecycle path classify at the source,
so ToRPCErr's Unknown fallback is left for genuine faults.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
da26d7f to
b00dae2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The idea across this PR is simple: classify an error where it happens, and
only translate it at the edge.
drpc.ToRPCErris the translator from drpc errors to gRPC status codes, forcallers that work in terms of codes (for example
grpcutil.IsClosedConnection). It only recognizes errors that were alreadytagged with a class it knows about:
ConnectionErrorandClosedErrormap tocodes.Unavailable(matched even through wrapping), the context and EOFsentinels map to their own codes (matched by identity), and everything else
becomes
codes.Unknown. The boundary has no way to recover intent that wasnever attached to the error, so anything that can tear a connection down has to
classify its own error at the source. The edge only translates.
A few teardown paths did not do this and reached the boundary as
codes.Unknown, which defeated the connection-liveness checks. We saw it as aTestDrainflake under DRPC stream multiplexing: the server kept reporting"not yet refusing RPC" because the teardown error carried
Unknowninstead ofUnavailable. With multiplexing a single long-lived connection is reused, soboth the write side and the lifecycle path get exercised in practice.
This PR fixes both I/O directions and the lifecycle path, so that
Unknownisleft for genuine faults.
drpcwire: classify write failures as ConnectionErrorWhen
MuxWriter's flush to the wire failed, it stored and passed along the raw*net.OpError(EPIPE), which fell throughToRPCErrtocodes.Unknown. Nowthe failed write is wrapped in
drpc.ConnectionErrorat its source inMuxWriter.run, mirroring the read path, so the existingConnectionErrorbranch maps it to
codes.Unavailable. The underlying error stays in the chain.drpcmanager: classify teardown errors at the source, by causeManager.Close(), which showed up as "manager closed: Close called", and anabrupt read-side EOF were both unclassified. They are now classified by cause,
never with a catch-all. A
Close()means the connection is gone, so it becomesa
ClosedError. A read EOF means the peer hung up, which is aClosedErrorfora client and a
context.Canceledfor a server, matching how gRPC treats aclient hang-up. Every other reader error is left transparent, so a real
protocol or internal fault still surfaces as
Unknownrather than beingmistaken for a retryable connection loss.
terminate()now just fans the onealready-classified cause out to everyone, which also fixes a raw-error leak in
NewServerStream. This commit drops the now-deadisConnectionReset.The pattern is documented where the next person will run into it:
ToRPCErr(the translator and the source-classification rule), the error-class
definitions in
drpc.go, andreadErrorin the manager.