Google Cloud engineers resolved a three-layer networking mystery that caused intermittent Kubernetes API server timeouts across GKE clusters in europe-west. The root cause combined packet reordering in Private Service Connect, Envoy's half-close keeping sockets open, and a Linux kernel bug in COS that prevented FIN processing — requiring a kernel patch to fully fix.
Three bugs converged to block API access
The outage began in January 2026 when pods across multiple GKE clusters in europe-west started failing to reach the Kubernetes API server with TCP connection timeouts. Initial suspects — custom operators, External Secrets Operator, security agents, conntrack exhaustion, and control plane load — were all ruled out after weeks of investigation. The issue affected DaemonSets, Deployments, system pods, and external clients alike, pointing to a fundamental problem in the network path between everything and the API server.
Packet captures taken simultaneously at the KCP VM's NIC and inside the kube-apiserver container revealed that SYN packets arrived at the control plane but received an out-of-place ACK instead of the expected SYN-ACK. The ACK carried sequence numbers from a previous connection that had used the same source port minutes earlier. Wireshark flagged these as "TCP Port numbers reused," indicating the KCP VM had never fully released the old connection state.
A spike in the Private Service Connect metric "Proxy protocol drop: invalid ACK packet" correlated with a USPS version rollout in February 2026. The new USPS code was dropping challenge ACKs — a legitimate RFC 5961 mechanism where a server sends an ACK with the old sequence number when a new SYN arrives on a recently-closed 5-tuple. The client should respond with RST to clear state, but the dropped challenge ACK left the new connection hanging until timeout.
Why connections stuck in FIN_WAIT_2 for minutes
The deeper mystery was why the KCP VM held connections in FIN_WAIT_2 for over two minutes — long enough for Private Service Connect's NAT layer to reuse the source port. Analysis of simultaneous captures at the host NIC versus the PIGv virtual NIC showed packets arriving in order at the host but reordered at the PIGv. Private Service Connect's miss-thread processing sends new flows through a slower path that adds proxy protocol headers, causing FIN packets to arrive after subsequent ACKs.
Envoy, the proxy in front of kube-apiserver on the KCP VM, performed a half-close by calling shutdown(SHUT_WR). This kept the socket open for reading while closing the write side — valid TCP behavior. However, the Linux kernel's tcp_fin_timeout (60 seconds on COS) only applies to orphaned sockets with no process holding them. Because Envoy retained the socket, the timeout never started, leaving the connection in FIN_WAIT_2 indefinitely.
The final piece was a Linux kernel bug in version 6.1 (used by Container-Optimized OS on KCP VMs). When the reordered client FIN arrived, the kernel's tcp_ack() function returned zero for an old-but-acceptable ACK per RFC 5961. The FIN processing path checked this return value, saw zero, and silently discarded the FIN without calling tcp_fin(). The connection never transitioned to TIME_WAIT. The fix (commit 795a7dfbc3d9) landed in kernel 6.9.7 and was backported to 6.6.y and 6.1.y LTS branches.
Resolution required fixes at three layers
The immediate mitigation in February 2026 rolled back the USPS change that dropped challenge ACKs, restoring connectivity for affected clusters by early March. This was a band-aid that restored the behavior the system had accidentally depended on, but left the underlying FIN_WAIT_2 persistence unaddressed. A proper correctness fix replaced the USPS logic to handle challenge ACKs properly rather than dropping them, with a PacketDrill test added to prevent regression.
The root cause fix required backporting the upstream Linux kernel patch to COS images used by GKE control plane VMs. The patch, authored by Eric Dumazet, ensures tcp_ack() properly handles old-but-acceptable ACKs during connection close so that received FIN packets are processed in FIN_WAIT_2 state. This fix ships in GKE 1.33.5 and later, resolving the kernel-level bug for all affected COS versions.
The case demonstrated how three independently benign behaviors — packet reordering from PSC's NAT architecture, Envoy's correct half-close implementation, and a kernel edge case in RFC 5961 handling — combined to create a systemic failure. Simultaneous multi-point packet captures, correlation with infrastructure metrics, and lab reproduction were essential to untangling the interaction across the Private Service Connect fabric, the KCP VM userspace, and the OS kernel.
What to do next
GKE clusters running versions 1.33.5+ include the kernel backport that resolves the FIN_WAIT_2 stall. Teams on older versions should plan upgrades to COS images with the 6.1.y or 6.6.y LTS backports. The USPS correctness fix is deployed globally. For similar mysteries, capture packets at multiple hops simultaneously — host NIC, virtual NIC, and container — before the trail goes cold.
Source: When the Kubernetes API went silent: A three-layer networking mystery (GCP).
