GKE Dataplane V2 solves the network identity bottleneck that stalls AI agent sandbox creation at scale. By moving unique per-pod metadata from labels to annotations and applying Cilium label exclusion filters, clusters avoid the 65,536 identity limit and achieve sub-second provisioning for 300 sandboxes per second, keeping GPUs busy instead of waiting on network setup.
Move high-cardinality metadata from labels to annotations
GKE Dataplane V2 calculates Cilium network identities exclusively from pod labels, not annotations. When agent orchestrators assign unique identifiers like agent-id, timestamp, or claim-uid to every sandbox pod as labels, each distinct label combination forces a new identity allocation, serializing pod creation while eBPF maps update. The fix is architectural: keep only labels that participate in service selectors or core resource grouping, and migrate everything else — sandbox names, request IDs, timestamps — into the pod's annotations field where the dataplane ignores them for identity purposes.
This separation preserves operational visibility because annotations remain queryable via kubectl and monitoring tools, but they no longer trigger identity churn. Teams running reinforcement learning workloads with thousands of pod transitions per minute see immediate relief: identical agent types now share a single Cilium identity regardless of how many unique metadata values accompany each pod. The result is linear scaling of sandbox density without linear growth in network identity consumption.
Implementation starts with auditing current pod templates. Identify every label key that varies per pod instance rather than per agent type. For each, add the same key-value pair to the pod's annotations section and remove it from labels. Validate that service selectors, pod disruption budgets, and cluster autoscaler grouping still function using only the remaining stable labels. This change requires no GKE control plane upgrade and takes effect on the next pod rollout.
Configure Cilium label exclusion filters as a safety net
Even after migrating known high-cardinality keys to annotations, developer frameworks or upstream charts may introduce new dynamic labels unexpectedly. GKE Dataplane V2 supports Cilium label exclusion filters via the cilium-config-emergency-override ConfigMap in the kube-system namespace. Adding rules such as !agents.x-k8s.io/sandbox-name-hash or !timestamp instructs the dataplane to ignore these specific label keys during security identity calculation, preventing them from generating new identities even if they appear in pod specs.
The filter syntax follows Cilium's identity-relevant label specification: prefix a label key with an exclamation mark to exclude it. Multiple exclusion rules can coexist in the same ConfigMap. This approach is particularly valuable in multi-tenant clusters where different teams deploy agent workloads with varying labeling conventions. A platform team can maintain a centralized exclusion list that protects the entire cluster's identity budget without requiring every workload owner to audit their pod templates.
To apply filters, edit the ConfigMap with kubectl patch or kubectl edit, adding the excluded label keys under the appropriate Cilium configuration key. Changes propagate to the dataplane within seconds without pod restarts. Monitor the Cilium identity count via cilium identity list or the GKE networking dashboard to confirm that identity growth stabilizes while new sandboxes continue launching. This defense-in-depth layer catches labeling regressions before they exhaust the 65,536 identity ceiling.
Validate sub-second provisioning and plan for Agent Substrate
After applying label migration and exclusion filters, measure sandbox startup latency end-to-end: from pod admission to network policy enforcement readiness. GKE Dataplane V2 benchmarks show 300 sandboxes per second at sub-second latency on a single cluster, with Mistral AI reporting spikes over 30,000 concurrent sandboxes. Use the GKE networking dashboard or custom metrics on pod scheduled-to-ready duration to verify your cluster achieves comparable throughput. If latency spikes persist, check for remaining high-cardinality labels slipping through or Cilium agent resource saturation on nodes.
For workloads demanding even higher density — hundreds of thousands of agents with suspend/resume cycles — Agent Substrate on GKE builds on these Dataplane V2 optimizations while bypassing Kubernetes scheduling limits entirely. In validation runs with 200,000 agents, Agent Substrate sustained 500 activations per second at under 500 ms suspend and resume latency. It integrates the same label and annotation best practices natively, so clusters already tuned for Dataplane V2 can adopt Agent Substrate incrementally without re-architecting their network identity strategy.
Both Agent Sandbox and Agent Substrate are deployable on existing GKE clusters today. Start by enabling GKE Dataplane V2 if not already active, then apply the label-to-annotation migration and exclusion filters documented here. For net-new agent platforms, adopt Agent Substrate from day one to capture its scheduling and state-rehydration advantages. The combined approach keeps expensive GPU and TPU capacity executing model code rather than idling during network identity negotiation.
What to do next
Audit your pod labels this week: move per-instance metadata to annotations, add Cilium exclusion filters for any remaining dynamic keys, and benchmark sandbox startup. Then evaluate Agent Substrate if your roadmap includes suspend/resume workflows or >50K concurrent agents.
FAQ
Which label keys should I move to annotations versus keep as labels?
Keep only labels used in service selectors, pod disruption budget selectors, or cluster autoscaler grouping. Move everything else — sandbox names, request IDs, timestamps, agent-instance UUIDs — to annotations.
Do Cilium label exclusion filters require pod restarts to take effect?
No. Updates to the cilium-config-emergency-override ConfigMap propagate to the dataplane within seconds. Existing pods retain their current identities; new pods immediately respect the updated filter rules.
Source: How GKE networking scales to support high densities of concurrent AI agents (GCP).
