CircleCI Secrets Breach (Jan 2023): Rotate Everything

Retrospective: this article looks back at events from January 2023, written in 2026 with the benefit of hindsight.

On January 4, 2023, CircleCI, a widely used continuous integration and delivery platform, told customers to rotate all secrets stored in its platform immediately — environment variables, contexts, API tokens and SSH keys.

What happened

CircleCI later explained that malware on an engineer's laptop had stolen a valid, SSO-backed session cookie. The malware wasn't detected by the company's antivirus. With the session, the attacker impersonated the engineer — bypassing two-factor authentication, which had already been completed — and accessed production systems. The attacker exfiltrated customer data, including environment variables and keys. Although that data was encrypted at rest, the attacker also extracted encryption keys from a running process, potentially allowing decryption.

Why it mattered

CI/CD platforms hold secrets for deploying to production: cloud credentials, API keys, signing keys, database passwords. A breach at a CI/CD provider becomes a breach risk for every customer's production environment.

The required response — rotating every secret — was enormous. Many organizations discovered they didn't know where all their CI/CD secrets were used.

Lessons in hindsight

• Session theft bypasses MFA. Device-bound sessions, short session lifetimes for privileged access and EDR on engineer devices matter.
• Eliminate stored secrets in CI/CD using OIDC federation to cloud providers.
• Inventory secrets so rotation is possible.
• Practice rotation before you need to do it under pressure.
• Monitor use of rotated credentials — failed attempts with old keys indicate theft.

In hindsight

Organizations that had moved to OIDC federation for AWS and Azure deployments had far less to rotate after CircleCI. The incident became a strong argument for secretless pipelines.