To serve Gemma 3 with high availability, deploy two GKE clusters in different regions, each with TPU v6e node pools enabled for managed DRANET, mount model weights via Cloud Storage FUSE, and route traffic through a multi-cluster Inference Gateway that fails over automatically when a region becomes unavailable.
Provision Networking and Storage Foundations
Create a custom VPC with subnets in the zones where you will reserve TPU resources. Set up firewall rules to allow health check traffic and inter-node communication. Reserve static internal IP addresses in each region for the GKE Inference Gateway endpoints. These IPs must be reserved before gateway creation to ensure stable addressing across failover events.
Provision a Cloud Storage bucket and enable the Cloud Storage FUSE CSI driver. Create a dedicated IAM service account with storage.objectViewer permissions, bind it to a Kubernetes Workload Identity, and configure pods to mount the bucket at /mnt/models. This allows TPU workers to load Gemma 3 weights directly from Cloud Storage without copying to local disks.
Verify that your project has sufficient TPU v6e quota in both target regions. Use the ct6e-standard-4t machine type, which provides four TPU v6e chips per VM. Request quota for at least two VMs per region to support a 2x2 TPU slice per cluster. Without confirmed quota, TPU node pool creation will fail silently during autoscaling.
Deploy GKE Clusters with TPU and DRANET
Create two GKE clusters, one in each region (e.g., us-east5 and europe-west4). During cluster creation, enable the Gateway API flag (--gateway-api=standard) and install the Cloud Storage FUSE CSI driver (--addons GcsFuseCsiDriver). These components are required for Inference Gateway integration and model access.
Add dedicated node pools for TPU workloads using the ct6e-standard-4t template. Label these node pools with cloud.google.com/gke-networking-dra-driver=true and set --accelerator-network-profile=auto to activate managed DRANET. This enables low-latency, dedicated accelerator interconnects between TPUs within the slice while isolating them from general pod traffic.
Register both clusters to a GKE Fleet. Enable Multi-Cluster Service Discovery and Multi-Cluster Ingress within the fleet configuration. Designate one region as the control plane hub to propagate Gateway configurations and health policies consistently across regions.
Serve Model and Configure Failover Routing
Deploy a temporary Job to download the Gemma 3 (gemma-3-27b-it) weights into the Cloud Storage bucket. Then run your inference server (e.g., vLLM) as a Deployment on TPU node pools. Use node selectors to target the TPU nodes, request exactly four TPUs per pod, and mount the netdev ResourceClaim to access DRANET-enabled networking. Mount the model storage at /mnt/models so the server can load weights on startup.
Install the Inference Gateway CRDs and deploy an AutoscalingMetric named tpu-cache that exports vLLM's KV cache usage percentage. Group deployments from both regions into an InferencePool via Helm. Deploy a cross-region Gateway using the gke-l7-cross-regional-internal-managed-mc class, with regional IP references and an HTTPRoute pointing to the pool.
Apply a HealthCheckPolicy and GCPBackendPolicy to the pool, configuring backend timeout to 100 seconds and balancingMode to CUSTOM_METRICS. Reference the tpu-cache metric with a maxUtilizationPercent of 60. Finally, create an InferenceObjective that routes traffic to the pool with the lowest KV cache usage, enabling automatic failover when one region's TPUs become saturated or unavailable.
What to do next
Validate the setup by sending inference requests and then draining one region’s TPU node pool. Observe how the Inference Gateway shifts traffic to the healthy region within seconds, maintaining zero dropped requests. For hands-on practice, follow the linked codelab which provides Terraform and Helm configurations to reproduce this architecture end-to-end.
FAQ
Do I need to manually sync model weights between regions?
No. The model is stored once in Cloud Storage and mounted via FUSE in both regions, ensuring all pods read from the same authoritative source without replication overhead.
Can I use GPUs instead of TPUs with this setup?
Yes. Managed DRANET supports both GPUs and TPUs; simply change the node pool accelerator type and ensure the deviceClassName in the ResourceClaim matches your accelerator’s network profile.
Source: Experimenting with TPUs, GKE Managed DRANET, and Multi-cluster Inference Gateway (GCP).
