Rotate Northflank Secrets Without Breaking Production
Matthew Leo · Published November 6, 2026 · Guides
Northflank is telling customers to assume that all environment variables stored on its platform were compromised. That makes this a secret-rotation incident, not a routine password-change exercise.
The order of operations matters. Revoking everything at once can create a second outage; changing values only inside Northflank can leave the original credentials valid at their issuers. The goal is to contain abuse while moving production through known-good credentials in a controlled sequence.
What is confirmed
Northflank’s incident status page advises customers to rotate workload variables, secret groups and global secrets. It has also told customers to revoke credentials at the issuing provider, generate replacements, update Northflank and review usage and billing for abuse.
On October 31, the company said audit logs had been enabled for all customers and added rotation support for database add-ons. Some customers reported unauthorized use of artificial-intelligence service keys. BleepingComputer independently reported those complaints and noted that the entry vector and number of affected customers were still unknown.
First, establish an incident boundary
Create an incident record with an owner, a time-stamped action log and a list of affected projects. Freeze non-essential deployments so routine changes do not obscure evidence or overwrite useful logs.
Inventory every secret that could have been present in Northflank, including:
- cloud-provider access keys and service-account credentials;
- database users, connection strings and certificates;
- source-control, container-registry and deployment tokens;
- payment, email, messaging and analytics API keys;
- signing keys, webhook secrets and session secrets;
- credentials copied into build arguments, job definitions or global secret groups.
Do not limit the inventory to variables currently attached to a running service. Include old environments, preview deployments, jobs and credentials removed during the suspected exposure window.
Rotate in dependency order
1. Contain high-impact credentials
Start with credentials that can create more credentials, change infrastructure or access source code. Cloud administrator keys, identity-provider tokens, repository tokens and secret-manager credentials usually outrank a read-only analytics key.
Where the issuer allows two active credentials, create a replacement before revoking the old one. Apply the new credential to Northflank, deploy, verify and then revoke the old value. If a credential is already being abused, revoke it immediately and accept the controlled outage.
2. Rotate data-store access
Create a new database user or credential with the minimum required permissions. Update the application and background workers, confirm successful connections, then disable the old user. Check replicas, scheduled jobs and reporting services before deletion.
3. Replace third-party API keys
Revoke keys at the provider, not just in Northflank. Review provider-side usage, billing, IP history and model or endpoint access before generating the replacement. Set spend caps and alerts where supported.
4. Rotate trust and signing material carefully
Webhook secrets, cookie-signing keys and token-signing keys can invalidate active sessions or break integrations. Use a documented overlap period only when the protocol supports multiple valid keys and there is no evidence the old key is actively abused. Otherwise force reauthentication and communicate the effect.
Deploy, verify and revoke
For each credential, record four states: replacement issued, workload updated, production verified and old credential revoked. A secret is not rotated when only the configuration value has changed.
Verification should cover health checks, background jobs, queues, scheduled tasks and outbound integrations. Watch authentication failures and error rates during the transition, then confirm at the issuer that the old credential can no longer be used.
Hunt for abuse after rotation
Review Northflank’s newly available audit logs alongside logs from cloud, source-control, database and API providers. Look for unfamiliar locations, new tokens, changed permissions, unusual exports and spend spikes. Preserve relevant logs outside the affected platform.
Mapletechie’s earlier analysis of the Cursor and Claude Code extension incident explains the same important distinction: removing a compromised component does not invalidate credentials it may already have exposed.
Canadian privacy obligations may apply
If investigation shows that personal information under a Canadian organization’s control was exposed, the response expands beyond credential rotation. The Office of the Privacy Commissioner says organizations subject to PIPEDA must keep a record of every breach of security safeguards and assess whether it creates a real risk of significant harm. Qualifying breaches must be reported and affected individuals notified.
Use the OPC’s breach-reporting guidance and obtain legal advice for the organization’s jurisdiction and facts. An exposed secret is not automatically a reportable privacy breach, but it can be the access path to one.
What to improve after containment
- Keep production secrets in a managed vault and inject short-lived credentials where the platform permits.
- Separate development, staging and production identities.
- Give every service its own least-privilege credential.
- Set provider-side spending and usage alerts.
- Practise secret rotation before an incident so teams know the dependency order.
The Canadian Centre for Cyber Security also recommends reviewing third-party integrations, revoking unused tokens and monitoring unusual API activity. Those controls reduce both the number of secrets exposed in the next provider incident and the time needed to contain them.
Tags: cybersecurity, cloud computing, software supply chain, incident response