pick_first (legacy): channel permanently stuck in READY with a stale picker after resolver refresh burst during transient network failure; wait-for-ready RPCs black-hole until deadline

[JamesXNelson]

grpc-java-pickfirst-wedge.tar.gz

What version of gRPC-Java are you using?

1.76.2 (grpc-netty, netty 4.1.136.Final)

What is your environment?

Linux, Java 17 (also observed under Java 11 in production).
Load balancing policy: pick_first (single-address target).
Active LB implementation: legacy io.grpc.internal.PickFirstLoadBalancer (its
RequestConnectionPicker appears in the channel logs; the new
PickFirstLeafLoadBalancer was not enabled — see "Additional data point" below).

What did you expect to see?

After a transient network failure heals, a pick_first channel either recovers or
transitions out of READY so that the LB policy or the application (via
notifyWhenStateChanged) can react.

What did you see instead?

The channel is left permanently reporting READY while every RPC — including
withWaitForReady() RPCs — buffers forever and fails only by its own deadline:

DEADLINE_EXCEEDED: ... Name resolution delay 0 ... [buffered_nanos=..., waiting_for_connection]

No channel state transition ever fires again, so nothing can observe that the channel
is broken. ManagedChannel.enterIdle() reliably recovers it.

Mechanism, from FINEST ChannelLogger logs (attached):

1. The transport to the current address dies (TLS handshakes reset — in production,
   a rolling container upgrade left the port published while the backend reset
   handshakes for ~10 seconds).
2. gRPC's reconnect logic calls NameResolver.refresh() repeatedly. The resolver
   answers asynchronously with a mix of onError and onResult deliveries (it is
   backed by a discovery service that sits behind the same failing network, so
   deliveries arrive out of order relative to subchannel state changes).
3. Each onResult causes PickFirstLoadBalancer to create a replacement subchannel
   and schedule delayed shutdown (~5s) of the previous one.
4. Within the delayed-shutdown window, the old subchannel completes its connection
   against the recovering backend and reaches READY; the LB publishes a READY picker
   referencing it.
5. The delayed shutdown then fires (SHUTDOWN with UNAVAILABLE(Subchannel shutdown invoked)), terminating the transport — but the published READY picker is never
   replaced. The channel stays READY forever with a picker whose subchannel has no
   transport.

Steps to reproduce the bug

Self-contained repro attached: one Java file (BurstRepro.java, only grpc-java +
netty dependencies, in-process backend server), a small Python TCP proxy that models
the network failure (reset mode: accept + immediately close, emulating TLS resets),
and a driver script. Typically reproduces within a few iterations.

Per iteration: verify a healthy RPC through the proxy, switch the proxy to reset
for a few seconds (driving reconnect/backoff and resolver refreshes), heal the proxy,
then probe with a withWaitForReady() RPC with a deadline.

Two wedge variants observed:

• logs/run2-READY-signature/ — channel READY, wait-for-ready probe black-holes to
  DEADLINE_EXCEEDED; enterIdle() immediately recovers it. This matches our
  production incident signature.
• logs/run1-connecting-wedge/ — channel wedged in CONNECTING with no reconnect
  attempt scheduled.

Additional data point: the new PickFirstLeafLoadBalancer does not wedge

Running the identical repro with GRPC_EXPERIMENTAL_ENABLE_NEW_PICK_FIRST=true
(confirmed active: PickFirstLeafLoadBalancer in the logs), the permanent black-hole
does not reproduce in 40 iterations. The same post-burst sequence still leaves the
channel claiming READY while the picked transport is dead, but the RPC fails fast
with UNAVAILABLE and gRPC immediately schedules a resolver refresh, after which the
channel recovers (logs/run3-newpf-no-wedge/).

So the catastrophic silent wedge appears specific to the legacy
PickFirstLoadBalancer. Since the new PF also has a known stuck-in-IDLE issue
(#12395) with a draft fix that appears stalled (#12403), may we suggest revisiting
that PR — fixing #12395 and graduating the new PF would retire both wedge variants,
whereas today users are stuck choosing between a legacy PF with this READY wedge and
an experimental PF with a known IDLE wedge.

Related issues

• PickFirstLeafLoadBalancer may get stuck in Idle on backend address change #12395 — PickFirstLeafLoadBalancer stuck in IDLE on backend address change (Java
  analog of pickfirstleaf: Avoid getting stuck in IDLE on connection breakage grpc-go#8615). Same family: LB internal state diverging from the
  published picker when a resolver update races a subchannel state change. Differs
  in that it wedges in IDLE and affects the new leaf PF; this report is the legacy
  PF wedging in READY, which is strictly worse (wait-for-ready RPCs black-hole and
  no observable state ever changes).
• PickFirstLeafLoadBalancer does not emit TRANSIENT_FAILURE states #11082 — PF not emitting TRANSIENT_FAILURE (missed state propagation).
• Client requests fail with NullPointerException in PickFirstLeafLoadBalancer after temporary DNS resolution failures #11227 — NPE from inconsistent PF internal state after resolution errors (fixed in
  1.66); shows the resolver-error path mutating LB state non-atomically.

Production impact

Our etcd-backed auth-service channel black-holed all RPCs for 60+ seconds (until
process restart) during a rolling upgrade. The server saw zero incoming RPCs during
the window while the client channel claimed READY throughout. We now run an
application-level watchdog that calls enterIdle() when RPCs time out while the
channel claims READY.

Attachment

grpc-java-pickfirst-wedge.tar.gz — repro sources, driver, and FINEST channel logs
for all three runs described above.