- A single inventory of machine and agent credentials, each with an owner and an expiry
- Long-lived cloud keys removed from CI in favour of short-lived federated credentials
- AI agents given their own identities with narrow, short-lived permissions
- Revoking all machine credentials became a rehearsed action rather than an improvised one
- Evidence for identity and access controls ready for the regulator and auditors
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A South African platform offered investment accounts and payments to retail customers and small businesses. Staff identity was in good shape: single sign-on, multi-factor authentication and quarterly access reviews. Software identity was not. Pipelines deployed with long-lived cloud access keys. Services read database passwords from configuration files. Partner API keys were shared between several services, and nobody was sure which. Over the past year, teams had added AI agents: a support assistant that looked up account status, and internal agents that triaged alerts and drafted reconciliation reports, each with its own set of keys.
As a financial institution in scope of Joint Standard 2 of 2024, issued by the FSCA and the Prudential Authority and effective from 1 June 2025, the company had to maintain cybersecurity and cyber resilience controls, including identity and access management34. A security compromise involving personal information would also trigger notification duties under POPIA. The company's access controls were written around people; its regulator would not see the distinction as an excuse.
Why now. In late October, Chainalysis disclosed an intrusion by an actor using "offensive agentic techniques"2. On 6 November it said the attack had exposed password hashes and live tokens for all users, and it took key products offline1. The lesson for the board was simple: automated attackers find and use every credential they can reach, quickly. Industry research suggested the company was not unusual: a 2026 identity survey reported that only a small minority of organisations automate the lifecycle of machine identities5.
What had to be true
- Known: every machine and agent credential in one inventory.
- Owned: each with a named team and a purpose.
- Short-lived: static keys replaced by credentials that expire in minutes or hours wherever possible.
- Narrow: each identity allowed only what its job needs.
- Revocable: all machine credentials can be rotated or revoked quickly, in a tested way.
- Visible: every action attributable to a specific identity, and unusual use detected.
Architecture
Pipelines stopped storing cloud keys and instead exchanged a signed identity token from the CI system for short-lived cloud credentials. Services on Kubernetes used their service account identities to obtain credentials from the cloud and the secrets manager, which issued database users that expired after a few hours. Kubernetes itself follows the same principle for image pulls, where credential provider plugins fetch registry credentials on demand instead of storing them6. Partner API keys that could not be made short-lived were split so each service had its own, stored only in the secrets manager and rotated on a schedule.

The data flow, end to end
- ProveA pipeline, service or agent presents its workload or agent identity
- CheckThe issuer checks policy: which identity, which target, which action
- IssueA credential scoped to the job is issued with a short expiry
- ActThe caller uses it; agents acting for a customer carry that delegation
- RecordEvery action is logged against the identity, and the user where delegated
- ExpireThe credential lapses; nothing long-lived is left behind to steal
The hard part: agents, delegation and revoking without an outage
Agents acting for customers. The support assistant needed to look up a specific customer's account, but must never be able to look up any account. The token broker issued it a token only after the customer had authenticated in the conversation, scoped to that customer, read-only and expiring with the session. Internal agents had their own identities and could only call tools on an approved list. Every action was logged against both the agent and, where relevant, the person it acted for. See agentic infrastructure.
Revoking everything, safely. The scenario the board cared about was "assume every machine credential is compromised". The first tabletop exercise showed that revoking everything at once would take the platform down, because some services could not obtain new credentials without a restart and a few partner keys had no rotation process at all. Fixing that became its own workstream: services that refresh credentials automatically, a documented rotation for each partner key, and a staged revoke-all runbook rehearsed in a test environment.
If you cannot revoke a credential without breaking production, you do not control it. Make rotation routine before you need it in an emergency.
How it ran
- Inventory and owners
- Policy and token broker
- Alerts and drills
- CI federation
- Workload identity
- Dynamic credentials
- Agent tool lists
- Delegation flows
- Credential refresh in services
- Map to Joint Standard
- Breach playbooks
- Evidence for the regulator
- Scan code and config
- List cloud and API keys
- Assign owners
- CI federation
- Dynamic database users
- Split shared keys
- Agent identities
- Token broker
- Delegation and logging
- Rotation drills
- Revoke-all runbook
- Detection tuning
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Inventory before tooling | Buy a non-human identity product first | A tool cannot assign owners or decide scopes |
| Federated, short-lived CI credentials | Rotate static keys more often | Removes the key rather than shrinking its life |
| One identity per agent | Agents share a service account | Actions attributable and revocable per agent |
| Customer-scoped tokens for the support agent | Broad read access with prompt rules | Limits enforced in code, not in instructions |
| Staged revoke-all runbook | Revoke everything at once | Contains an incident without causing an outage |
| Partner keys split per service | One shared key per partner | A leak exposes one service, not all |
What I'd do again, and differently
Again: start with the inventory and owners. It found credentials nobody remembered creating, and it made every later decision concrete. Differently: run the revoke-all tabletop in the first month; it exposed the services that could not refresh credentials, which shaped the whole plan. Related: LLMjacking and stolen AI keys, securing MCP servers, the 8 November tech news and where I work.
If your services and AI agents hold credentials you cannot list, own or revoke, see my fractional CTO work, or analyse your challenge and bring the result to a call.
- 1Chainalysis Breach Exposes All Users' Credentials, Forces Shutdown of Key ServicesThe Hacker News · Nov 2026
- 2Chainalysis discloses data breach from "offensive agentic techniques"BleepingComputer · Oct 2026
- 3
- 4
- 5Identity security planning for 2026 is shifting under pressureHelp Net Security · 7 Jan 2026
- 6Kubernetes v1.26: GA Support for Kubelet Credential ProvidersKubernetes blog · 22 Dec 2022