When your federated identity provider goes down, you lose access to Amazon EKS clusters unless you have break-glass credentials ready. This guide shows how to provision a secure, MFA-enforced cross-account IAM role using infrastructure-as-code, validate it works, and follow a recovery procedure to restore access during an outage—without creating a circular dependency.
Design the break-glass IAM role with MFA enforcement
Create a dedicated IAM role in a separate AWS account from your EKS cluster to avoid shared failure points. Attach the AmazonEKSClusterPolicy and AmazonEKS_CNI_Policy managed policies to grant necessary cluster access. Configure the role’s trust policy to allow sts:AssumeRole only when MultiFactorAuthPresent is true in the request context, ensuring MFA is required at assume time. Use an external ID for added security against confused deputy problems.
Limit the role’s session duration to a maximum of 1 hour to reduce exposure if credentials are compromised. Tag the role with break-glass:purpose=emergency-access and break-glass:account-id=<cluster-account> for auditability. Avoid attaching AdministratorAccess; instead, scope permissions to eks:DescribeCluster, eks:AccessKubernetesApi, and related actions needed for recovery tasks like updating ConfigMaps or checking node status.
Store the role ARN externally—such as in a password manager or offline document—so it remains accessible even if your AWS console or identity tools are unreachable. Do not embed the role ARN in automation scripts that depend on the same identity provider you’re trying to bypass.
Provision and validate with infrastructure-as-code
Use AWS CloudFormation or Terraform to define the IAM role, trust policy, and permissions boundary in version-controlled templates. Include a condition in the trust policy that checks for MultiFactorAuthPresent using the StringEquals operator. Example: "Condition": {"BoolIfExists": {"aws:MultiFactorAuthPresent": "true"}}. Deploy the template to a dedicated security or tools account, not the workload account.
After deployment, validate the role requires MFA by attempting to assume it without a token—this should fail with AccessDenied. Then assume it using the AWS CLI or STS with a valid MFA token from a hardware or virtual MFA device. Confirm you receive temporary credentials and can call eks:DescribeCluster on the target cluster. Log the assumed role session for audit trails.
Test the end-to-end flow by simulating an IdP outage: temporarily disable your federated identity provider’s connectivity (e.g., via security group or network ACL), then use the break-glass role to access the cluster. Verify you can run kubectl commands to view nodes or apply a recovery ConfigMap. Document the exact CLI commands and expected outputs in your runbook.
Execute recovery and rotate credentials post-incident
During an actual federated identity outage, retrieve the break-glass role ARN and MFA device from your secure store. Assume the role using aws sts assume-role with --serial-number and --token-code parameters. Export the temporary credentials and configure kubectl to use them via AWS_IAM_PROFILE or environment variables. Perform only the minimal actions needed to restore identity access, such as fixing IAM roles for service accounts or correcting OIDC provider configurations.
After access is restored, immediately revoke the break-glass session by not reusing the assumed credentials and consider rotating the MFA seed if compromise is suspected. Review CloudTrail logs for the assumed role session to validate activity scope and duration. Do not leave the break-glass role assumed longer than necessary for the recovery task.
Conduct a post-incident review to update the break-glass procedure: verify the role still works, check that MFA enforcement is intact, and ensure the external ID and trust policy haven’t drifted. Rotate any long-term secrets used to store the role ARN and schedule quarterly drills to validate the entire break-glass process remains functional.
What to do next
Store your break-glass role ARN and MFA details offline, test access quarterly, and limit its use to identity restoration only. Treat break-glass credentials like a fire alarm: hope you never need them, but verify they work when it matters. Update your runbook with the exact commands and validation steps from this guide.
FAQ
Can I use the break-glass role for routine cluster administration?
No. The break-glass role should only be used when federated identity is unavailable and you need to restore access. Using it for routine tasks defeats its purpose and increases risk.
What if I lose access to my MFA device during an outage?
Register a secondary MFA device (e.g., virtual and hardware) for the break-glass role in advance and store recovery codes securely. Never rely on a single MFA factor for emergency access.
Source: Break-glass access for Amazon EKS when federated identity fails (AWS).