The first AI hire is easy. By the fifth, someone usually asks where they should sit. One option is a central AI team that builds everything; it becomes a bottleneck and a queue. The other scatters AI engineers across product teams; each one reinvents model access, evaluation and guardrails, and nobody owns the shared risks. Neither works for long.
A useful vocabulary
Team Topologies, the framework from Matthew Skelton and Manuel Pais, describes four team types1:
- Stream-aligned teams deliver value along a flow of work, usually a product or customer journey.
- Platform teams provide internal services that reduce the load on others.
- Enabling teams help other teams build new capabilities, for a limited time, then move on.
- Complicated-subsystem teams own parts that need deep specialist knowledge.
And three ways for teams to interact: collaboration, working closely towards a shared goal, mainly during discovery; X-as-a-Service, consuming what another team provides with minimal coordination; and facilitating, one team helping another learn12. Interactions that fit none of these are often a sign that team boundaries are drawn badly2.
How I map AI work onto it
| AI work | Team type | Why |
|---|---|---|
| AI features in a product | Stream-aligned | The team that owns the journey owns the AI in it, including its quality |
| Model gateway, evaluation tooling, observability, GPU capacity | Platform | Shared once, consumed as a service by every product team |
| Teaching teams to build, evaluate and secure AI features | Enabling | Time-boxed help that leaves capability behind |
| Custom models, fine-tuning, a governed ontology | Complicated-subsystem | Genuinely specialist work that product teams shouldn't carry |
The question isn't "central or embedded". It's which AI work is product work, which is platform work, and which is temporary teaching.
Interaction modes over time
The modes should change as capability matures. Early on, an AI platform or enabling team collaborates closely with one or two product teams to discover what's needed. Once patterns stabilise, the platform offers them as a service: an approved gateway, evaluation pipelines, templates. Enabling help moves on to the next team. If a product team still needs constant hand-holding a year later, the platform's interfaces are probably the problem, which is exactly what the platform engineering maturity model helps diagnose.
Conway's law applies to AI too
Melvin Conway observed in 1968 that organisations design systems that mirror their own communication structures3. In AI, the effect is easy to spot: a central AI team produces a central AI service that every product must queue for; fully scattered teams produce five slightly different gateways and no shared evaluation. Design the team structure you want the architecture to reflect.
What I put in place first
- Name owners for AI features in each product team, including their evaluation and incident handling.
- Create a small AI platform capability for the gateway, evaluation and observability, run as a product with internal users; see LLM model routing and AI cost allocation for what typically lives there.
- Time-box enabling work, with explicit exit criteria for each team supported.
- Declare interaction modes between teams, so expectations are clear and revisited.
- Record the design as an architecture decision record and revisit it each half year.
How I can help
As a fractional CTO, I help companies organise teams and architecture together, so AI work scales without bottlenecks or duplication. Or analyse your challenge and bring the result to a call.
- 1Key conceptsTeam Topologies
- 2Team TopologiesOpen edX wiki
- 3How do committees invent?Melvin E. Conway · 1968
- 4Platform Engineering Maturity ModelCNCF TAG App Delivery