Separate who decides what

The cleanest organisations separate concerns: a platform team owns the gateway and the approved routes, and application teams choose from those routes without negotiating contracts themselves. Kubernetes networking made the same separation explicit, with infrastructure providers, cluster operators and application developers each owning their own resources⁵.

Buy models through the route your compliance needs, not the one that is easiest to sign. A gateway lets you hold several routes without your applications noticing.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A South Korean recruitment platform served employers and job seekers. It used machine learning to match candidates to roles and, more recently, a language model to summarise applications and draft screening notes for recruiters. Employers liked the speed. Nobody had written down which steps used AI, how they were tested, or what a candidate could do if they disagreed.

Why now. South Korea's AI Basic Act took effect on 22 January 2026¹². Commentary describes core transparency duties for generative and high-impact AI, with high-impact uses, including areas such as hiring, bringing duties around risk management and human oversight²³. Observers note the Act tries to balance innovation with social risk, and that its details are being filled in through implementing rules⁴. With enforcement maturing and enterprise customers asking for assurances, the platform chose to build for the high-impact obligations now.

What had to be true

• Every AI use known and classified, with hiring uses treated as high-impact.
• Candidates informed when AI is involved, with a route to human review.
• A person decides: AI can rank and summarise, but no rejection without a recruiter's decision.
• Models tested before release, including for differences across groups of candidates.
• Evidence on demand for any application: inputs, outputs, model version and decision.

Architecture

The data flow, end to end

The hard part: evidence without slowing recruiters down

Oversight fails when it becomes paperwork. The platform made evidence a by-product of normal work: every AI call was traced with OpenTelemetry's GenAI conventions, capturing the input, the output, the tools used and the model, so any application could be reconstructed later⁵. The reviewer workbench put the AI summary next to the original CV with citations, so checking it took seconds rather than minutes.

Personal data in traces was minimised and access-controlled; summaries were stored with the application, not in a separate analytics copy. Evaluation followed the approach in evaluation sets that predict production, with checks for differences across candidate groups added to the release gate. See also LLM observability with OpenTelemetry.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: build the inventory and tracing first; everything else depended on them. Differently: design the candidate-facing review process earlier. Explaining an AI-assisted outcome in plain language was harder than producing the technical evidence. For broader governance context, see an AI governance programme across jurisdictions and ISO 42001 vs NIST AI RMF.

If you deploy AI in hiring, lending or other high-impact decisions in Asia-Pacific or elsewhere, my AI architecture work and regional pages explain how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A UK managed service provider looked after IT for several hundred organisations: small businesses, schools, professional services firms and a few public-sector suppliers. Its technicians held administrator access to customer networks, cloud tenants and line-of-business systems. Over the years, access had accumulated in shared accounts, remote management tools and scripts with stored credentials.

Why now. The Cyber Security and Resilience Bill would bring medium and large managed service providers within the scope of the NIS regulations, because of the privileged access they hold to customer systems¹². It also proposes duties to report significant incidents and to identify and notify customers likely to be affected³. In October 2026 the Bill was still before Parliament⁴, but the direction was clear, and customers were already asking about it in renewals.

What had to be true

• No standing administrator access to customer environments.
• Every privileged session attributable to a named technician, approved and recorded.
• Incidents detected and escalated fast, with a process that can support early notification and a fuller report.
• Affected customers identifiable within minutes, not days of spreadsheet work.
• Automation under the same rules as people, including AI-assisted tools.

Architecture

The data flow, end to end

The hard part: one front door for hundreds of environments

Access was spread across per-customer accounts in dozens of tools, each with its own login and its own way of revoking access. The design moved to a single identity provider for all technician and automation access, with customers' systems trusting it for scoped, time-limited sessions. The same principle is spreading in AI tooling: the Model Context Protocol's Enterprise-Managed Authorization lets an organisation provision access to many servers through its identity provider, scoped to users' groups and roles, instead of authenticating to each one separately⁶.

Automation got no exemption. Scripts and AI-assisted tools used scoped service identities through the same broker, with no stored administrator passwords. More on that in securing MCP servers and sandboxing AI agents.

How it ran

Controls were mapped to the NCSC Cyber Assessment Framework, which organises security into outcomes for managing risk, protecting against attack, detecting events and minimising impact⁵. That gave customers and, later, a regulator a recognised way to read the programme.

Key decisions and trade-offs

What I'd do again, and differently

Again: start with the privileged access inventory, because it surfaced most of the risk. Differently: run the first notification drill earlier. It showed that knowing which customers were affected was harder than detecting the incident. Related reading: defender-first AI and patch cycles and an AI copilot for the security operations centre.

If you run or rely on managed services in the UK, my work with UK companies explains how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

Let the scheduler enforce your priorities

On Kubernetes, the scheduler decides where each workload runs, and pod priority and preemption decide what happens when resources run out: higher-priority pods are scheduled first, and lower-priority ones can be evicted to make room⁴⁵. Giving customer-facing inference a higher priority class than batch jobs means a capacity squeeze lands on work that can wait.

In a shortage, something will wait. Decide in advance what it is, and encode that decision in the platform rather than in a meeting during the incident.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A manufacturer of connected heat pumps and ventilation units sold across the European Union. Its devices streamed temperatures, energy use, fault codes and runtime data to the cloud, where service teams used it for diagnostics and the product team used it to improve designs. Homeowners, installers and energy advisers saw only a slice of it in an app, and there was no way for a customer to take their data to someone else.

Why now. The EU Data Act gives users rights over the data their connected products and related services generate, including access and sharing with third parties they choose¹². Its access-by-design obligation applies to connected products placed on the market after 12 September 2026, and commentators stress that it is an engineering obligation that has to be built into the product³. Separately, the Act's cloud switching rules prohibit switching charges from 12 January 2027³. For a product line launching new models, both dates landed in the middle of the roadmap.

What had to be true

• Users can get their data easily, in a usable format, without raising a support ticket.
• Users can share it with an installer, energy adviser or app of their choice, and revoke that.
• Personal data handled under the GDPR, because household energy patterns can identify people⁴.
• Trade secrets protected through agreed measures, not used as a blanket excuse.
• No cloud lock-in in the platform itself, in line with the switching rules.

Architecture

The data flow, end to end

The hard part: ingestion that scales with the fleet

Access by design starts at ingestion: if data is dropped, summarised too early or locked into a proprietary pipeline, users cannot get it later. The team standardised device telemetry on OTLP and placed a tier of OpenTelemetry Collectors in gateway mode behind a load balancer in each region, so data keeps flowing as the fleet grows and individual collectors can be replaced without loss⁵.

The second hard part was the line between raw data and know-how. Raw readings went into the user-accessible store; the predictive maintenance models and efficiency algorithms built from them stayed separate, with sharing partners bound by terms to protect anything sensitive. More on related patterns in OpenTelemetry Collector deployment patterns and data contracts.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: treat the catalogue as a product, because it made the data useful to customers and partners, not just compliant. Differently: settle the trade secret approach with legal before building the stores, since it shaped the data model more than any technical choice. Related reading: lakehouse catalogue strategy and the EU AI Act roadmap.

If you build connected products for the European market, my work with European companies explains how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A general insurer in Australia had moved quickly with AI. Claims intake used a hosted model to read documents and photos, the contact centre used it to draft responses, and the fraud team used it to summarise case files. Almost all of it ran through one external model provider, wired directly into each application.

CPS 230, APRA's prudential standard on operational risk management, has applied to banks, insurers and superannuation entities since 1 July 2025, with existing service provider contracts brought in at renewal or by 1 July 2026¹. It expects entities to identify critical operations, set tolerance levels for disruption, and manage material service providers². The board asked a fair question: if the model provider has an outage, runs out of capacity or changes terms, what happens to claims?

Why now. Capacity is not a theoretical risk. Reporting on Microsoft's latest results notes that demand for its cloud still exceeds supply⁴, and AI capacity is the tightest part of that market. See the tech news update for 29 October.

What had to be true

• AI dependencies visible in the same register as other material service providers.
• Tolerance levels for each AI-assisted critical operation: how long, how degraded, before customers are harmed.
• A tested alternative for every critical use, not a slide.
• Privacy obligations met on every path, including fallbacks.
• No rebuild of the applications: changes behind a common interface.

Architecture

The data flow, end to end

The hard part: deciding what to drop when capacity is short

Not every AI use is equally important, and pretending otherwise means everything fails together. The team tagged each request with the criticality of the operation it served, so the gateway could protect claims intake and shed or delay lower-priority work. The same idea appears in the Kubernetes Gateway API inference extension, where scheduling considers whether a request is critical and whether there is capacity before choosing a model server, and can drop non-critical requests when there is not⁵.

Every fallback path also had to meet privacy obligations. The OAIC's guidance on commercially available AI products expects due diligence on each product and ongoing review, not a one-off check³, so the second provider and the private model went through the same assessment as the primary. Routing patterns are covered in LLM model routing and Kubernetes inference gateways.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: put a gateway in front of every model call before adding a second provider; it made everything else cheap. Differently: build the evaluation set for each critical operation first. The slowest part of adding a provider was proving its answers were good enough, not wiring it in. See evaluation sets that predict production and AI incident postmortems.

If you run AI in a regulated Australian business, my work with Australian companies explains how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

Assume exploitation, and watch behaviour

Patching will sometimes be late. The Kubernetes community has long argued for treating every microservice as vulnerable and monitoring how it behaves, because an exploited service makes irregular requests, returns irregular responses and takes irregular time to respond⁶. Those signals do not depend on knowing the vulnerability in advance.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A travel marketplace serving Singapore and its neighbours sold hotels, flights and activities through a website and app. Its traffic already included price scrapers and comparison bots, handled with blocking and rate limits. Its booking flow assumed a person clicking through screens.

Why now. In late October 2026, Meta made its personal AI agent, Muse, available in Singapore, with partners including Shopee, Grab, Singapore Airlines, Agoda and Trip.com¹². Coverage describes the agent building a basket and waiting for the person's approval before ordering, and asking for approval before sensitive steps such as purchases². For a travel business, that means some customers will arrive through an assistant rather than the app, and the marketplace either offers agents a good way in or watches them book elsewhere. See the tech news update for 29 October.

What had to be true

• Agents read the truth: inventory, total prices, fees and cancellation rules available as structured data, not scraped from pages.
• No booking without a person's approval, and no duplicate bookings from retries.
• Agent traffic isolated, so a busy agent platform cannot slow down human customers.
• Clear data protection basis for using personal data in recommendations under Singapore's PDPA.
• One interface for many agents, not a custom integration per assistant.

Architecture

The data flow, end to end

The agent-facing tools followed the Model Context Protocol so different assistants could use the same interface⁵. Security for those tools followed the practices in securing MCP servers, and the idempotent booking design reused patterns from idempotent payment pipelines.

The hard part: serving agents without hurting people

Agent platforms can generate bursts of searches as they compare options for thousands of users at once. The answer was to route agent traffic to its own backends with its own limits and capacity, so a spike from one assistant never slowed the app. The Kubernetes Gateway API made this declarative: routes can match on headers, such as an authenticated agent identifier, and split or direct traffic to different services⁴.

On personal data, the PDPC's advisory guidelines explain how consent and notification obligations, and their exceptions, apply when personal data is used in AI recommendation systems³. Recommendations to agents used only the data needed for the request, and anything personalised was tied to a recorded basis. More on the rules in fractional CTO in Singapore.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: make prices and policies machine-readable first; it improved the human site too. Differently: involve finance and customer service earlier. Approval, refund and dispute flows for agent bookings raised more questions than the APIs did. Related reading: structured data for AI search and MCP registries for enterprises.

If your customers are starting to arrive through AI assistants, my AI architecture work and work in Singapore and Southeast Asia explain how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

An update to this morning's brief, covering stories published on 28 October that it did not include.

Read on asksumanta.com for diagrams, references and related reading.

How to decide whether to switch

1. BaselineMeasure current quality and cost per completed task, not per token
2. ReplayRun your evaluation set and recorded tasks against the new model
3. Compare by stepFind which step types match quality and which degrade
4. RouteMove only the matching steps, behind a feature flag
5. WatchTrack cost, latency and task success in production telemetry
6. DecideKeep, widen or roll back based on your numbers, not the launch post

The measurement layer is where most teams fall short. OpenTelemetry's GenAI conventions record each model call with the requested model, token usage split into input and output, and latency, so you can compare models side by side and estimate cost per request⁵.

Price per token is a vendor's number. Cost per completed task is yours. Only the second one should drive a model switch.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A digital lender in India served customers through a mobile app, with credit decisions supported by bureaus, co-lending partners, verification vendors, a collections agency and marketing platforms. Consent was a set of checkboxes at sign-up. Personal data flowed to partners through file drops and APIs built over years by different teams, and nobody could answer a simple question for a single customer: who holds this person's data, for which purpose, and what happens if they withdraw?

Why now. The Digital Personal Data Protection Rules were notified on 13 November 2025, bringing the DPDP Act, 2023 into force in stages over 18 months¹². Registration of consent managers applies after the first year, which falls in November 2026, and most core obligations for data fiduciaries apply from around May 2027³. For a lender whose business runs on personal and financial data, that is not much time to rebuild how consent works.

What had to be true

• Consent per purpose, not one blanket checkbox, with the notice version recorded.
• Withdrawal as easy as giving consent, and actually honoured by every system and partner.
• Retention that deletes: data removed when its purpose ends, with evidence.
• Breach detection and reporting fast enough to meet notification duties.
• Room for consent managers, so customers can manage consent through a registered intermediary.

Architecture

The data flow, end to end

The hard part: proving a withdrawal reached everyone

Recording consent is easy. Proving that a withdrawal stopped processing everywhere is not, because data has already been copied to analytics, partners and vendors. The design treated each withdrawal as an event on the stream, with every consumer and partner integration required to acknowledge it. Tracing producers and consumers with OpenTelemetry links a published event to each receiving service in one trace⁴, which turns propagation into something you can query and show to an auditor.

Partners without event integrations received withdrawals through an authenticated API with mandatory acknowledgement and a retry queue. Unacknowledged withdrawals raised alerts to the partner owner, not just to engineering. Related reading: change data capture and event streams and data contracts.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: start with the data and partner inventory, and make withdrawal the first feature built. Differently: bring partners in earlier. The longest delays came from contract changes and vendors without any way to receive a withdrawal, not from the engineering. For broader context on rules across markets, see where I work and how another data protection law shapes AI architecture.

If you are preparing an Indian platform for the DPDP Rules, my work with Indian companies explains how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A specialty insurer in the United States had grown its digital estate over fifteen years. The agent portal and quoting tools ran on PHP, alongside dozens of marketing and campaign sites built by agencies, a document-generation service, and vendor appliances that embedded PHP without anyone realising. Some runtimes were current; others were several releases out of support.

Why now. On 26 October 2026, CISA added two PHP remote code execution flaws, known as PHP2Shell, to its Known Exploited Vulnerabilities catalog and gave US federal agencies days to remediate¹³. Researchers reported successful exploitation in the wild². For a regulated insurer, the first question from the board was simple: are we exposed? The honest answer was "we don't know yet", and that was the real problem to solve.

What had to be true

• Exposure closed within days: every affected runtime patched to a fixed release or isolated.
• One trustworthy inventory of PHP runtimes, versions and owners, refreshed automatically.
• No outage for agents during quoting hours.
• Visibility before change: know what each legacy app actually does in production before touching it.
• A durable fix, so the next advisory is routine work, not a crisis.

Architecture

The data flow, end to end

The patched releases (8.5.12, 8.4.27, 8.3.36 and 8.2.35) were available from 21 October². Anything on PHP 8.1 or older could not simply be patched, because those branches no longer receive security fixes⁴; CISA's advice for such systems was to take them offline until they could be upgraded¹.

The hard part: changing systems nobody fully understood

The quoting tool had no tests and few people who knew it well. Before upgrading or migrating anything, the team added request tracing at the web server. The OpenTelemetry module for Apache HTTP Server records each incoming request as a trace, with spans for the modules involved, so you can see what the server is doing without changing application code⁵.

A few weeks of traces showed which routes agents actually used, which were dead, and which called downstream systems. That turned an unknown monolith into a ranked list of routes to move.

With that map, quoting functions moved behind a routing facade one route at a time, the strangler fig approach⁶: new services took over a route, traffic shifted gradually, and the old code path was removed once unused. More on the observability side in OpenTelemetry Collector deployment patterns.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: inventory first, then golden images, then tracing before touching code. Differently: put patch SLAs and the exceptions register in place before an emergency, not after. The scramble was mostly the cost of not knowing what was running. For the incident-review side, see running useful postmortems, and for the week's context, tech news for 29 October.

If your estate has systems nobody can fully map, my work with US companies and technical due diligence pages explain how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

A sandbox architecture for agents

Agent

Planner and modelno secrets in contextTask budgettime, steps, spend

Policy

Tool gatewayallowlist per taskHuman approvalfor consequential actions

Execution

Ephemeral sandboxfresh per run, no ambient credentialsShort-lived tokensscoped, issued per task

Network

Default-deny egressnamed destinations via proxyNo route to productionseparate accounts and networks

Assurance

Audit logevery tool call and resultKill switchstop a run in seconds

On Kubernetes, the network layer is straightforward to express: a NetworkPolicy that selects agent pods and allows egress only to an approved proxy and DNS, so every other destination is blocked by default⁷. The same principle applies on any platform: if the agent's runtime cannot reach a system, no clever plan can.

Read on asksumanta.com for diagrams, references and related reading.

A short, sourced round-up of the technology news that matters to engineering and product leaders, with my view on each. This first edition covers 26 to 28 October 2026.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A financial services group ran a security operations centre that took alerts from endpoint protection, identity providers, cloud platforms, email security and network tools. Each tool had its own data format and console. Analysts spent much of their time gathering context: which user, which host, what else happened nearby, has this been seen before. Experienced analysts carried that knowledge in their heads; newer ones escalated more than they needed to.

Leadership wanted to apply AI. The security team's concerns were specific: an assistant that invents findings, one that attackers can manipulate through the very logs it reads, and anything that acts on production systems without a person deciding.

What had to be true

• Every statement cited to the events, alerts or runbooks it came from.
• Read-only by default: the copilot investigates; it does not change systems.
• Human approval for every response action, with the action and its reason logged.
• Consistent data across sources, so the same question gives the same kind of answer.
• Aligned with the incident response process the organisation already followed and was audited against.

Architecture

The data flow, end to end

Normalisation made the rest possible. OCSF is an open, vendor-agnostic schema for security events, with a framework of data types, an attribute dictionary and a taxonomy², and it moved under Linux Foundation governance in 2024³. MITRE ATT&CK supplied a shared vocabulary for adversary behaviour, so summaries described detections in terms analysts already used⁴.

The hard part: an assistant that reads attacker-controlled text

Most AI copilots read trusted documents. A security copilot reads email subjects, file names, command lines and web requests, many written by attackers. Prompt injection, listed in the OWASP Top 10 for LLM applications⁵, is not a theoretical risk here. The design assumed it would be attempted:

• All event content treated as untrusted data, clearly separated from instructions in every prompt.
• Tools read-only by default, with narrow scopes; no tool able to change a system directly.
• Action proposals limited to steps from approved runbooks, and every proposal approved by an analyst.
• Generated queries shown to the analyst, so the reasoning can be checked rather than trusted.
• An evaluation set of past incidents, including deliberately manipulated log content, run before each change.

The general controls are covered in prompt injection defence for agents and evaluating AI agents.

How it ran

Key decisions and trade-offs

Incident records and handoffs were aligned with NIST SP 800-61 Revision 3, which organises incident response recommendations around the Cybersecurity Framework 2.0 functions¹, so the copilot fitted the process auditors already expected. Lessons from incidents fed the approach in AI incident postmortems.

What I'd do again, and differently

Again: normalise data and build the evaluation set from real incidents before writing prompts. Differently: start runbook clean-up earlier. Many proposals were only as good as the runbooks behind them, and some runbooks were out of date.

If you're considering AI in security operations, my AI architecture work covers how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A UAE utility handled a large volume of customer questions about bills, connections, moving home and service interruptions. Customers wrote in Modern Standard Arabic, Gulf Arabic, English, and frequently a mix of Arabic and English in the same message, sometimes in Latin script. The knowledge base was inconsistent: some policies existed only in English, some only in Arabic, and translations drifted out of date.

An earlier chatbot pilot had worked well in English and poorly in Arabic. Leadership wanted an assistant that treated Arabic speakers as first-class users, answered from current policy and never exposed customer data outside the country.

What had to be true

• Answer in the customer's language, whichever language the source document was written in.
• Cite the policy behind every answer, so agents and customers can check it.
• Handle real Arabic: dialect, spelling variation, diacritics and code-switching.
• Measure quality per language, not as one blended score that hides Arabic failures.
• Keep personal data in the country and out of prompts where it isn't needed.

Architecture

The data flow, end to end

Retrieval-augmented generation grounds answers in retrieved documents rather than the model's memory, which is what makes citations possible³. Multilingual embedding models make the cross-language part work: models such as multilingual E5 were trained on large collections of multilingual text pairs so that equivalent meaning in different languages lands close together². Embedding choice is covered in choosing embedding models and hybrid search in hybrid search for RAG.

The hard part: Arabic that real customers write

Benchmarks use clean Modern Standard Arabic. Customers don't. The work that made the difference was unglamorous:

• Normalising letter variants and removing diacritics for keyword search, while keeping the original text for display.
• Adding a keyword index alongside vector search, because product names, area names and account terms are often written inconsistently.
• Building the evaluation set from real anonymised questions in Gulf Arabic, Modern Standard Arabic, English and mixed messages, scored separately.
• Choosing the answering model on that evaluation set. Arabic-centric models such as Jais were developed precisely because Arabic is underserved, with training that combined a large Arabic corpus with English data¹; the right choice is whichever performs best on your own questions.

The general approach to evaluation is in evaluation sets that predict production.

How it ran

Key decisions and trade-offs

The federal Personal Data Protection Law governs personal data processing in the UAE⁴. Commentary notes it does not apply in the DIFC and ADGM free zones or to health data covered by separate law, and that transfers abroad are generally limited to jurisdictions with adequate protection⁵. Redaction and in-country hosting kept the design simple under those rules; see UAE PDPL and AI architecture.

What I'd do again, and differently

Again: build the bilingual evaluation set first and launch to agents before customers. Differently: involve content owners earlier. Many "Arabic quality" problems were really missing or outdated Arabic source content, not model weaknesses.

If you serve customers in Arabic and English, my AI architecture work and the fractional CTO in the UAE page describe how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

Most real knowledge graph questions are not just "is this true?" but "according to whom, since when, and how sure are we?". A supplier relationship came from a contract system; a diagnosis came from a coded record; a link between two companies was extracted by a language model with some confidence. In classic RDF, attaching that context to a single fact meant reification patterns that were verbose, inconsistent between teams and awkward to query.

:acme :supplies :globex {| :source :contractSystem ;
                           :since "2024-03-01"^^xsd:date ;
                           :confidence 0.92 |} .

Read on asksumanta.com for diagrams, references and related reading.

The situation

A fintech serving customers in the UAE offered budgeting and cash-flow insights built on account data. Getting that data had meant a patchwork: screen scraping for some banks, bilateral integrations with others, and manual statement uploads as a fallback. Each connection broke differently, security reviews were difficult to pass, and the approach had no future as regulation matured.

The Central Bank of the UAE's Open Finance Regulation changed the picture by establishing a regulated framework for open finance services¹. The question for leadership was how to move onto it cleanly, while continuing to ship features.

What had to be true

• One integration path through the regulated framework, replacing scraping and bespoke connectors.
• Consent as a product feature: customers see what they shared, with whom, for how long, and can revoke it.
• Financial-grade security that a bank's or regulator's reviewer would recognise.
• Data minimisation: store only what each feature needs, for as long as it needs it.
• Clear regulatory boundaries, including where free zone permissions do and do not apply.

The framework is described as having three parts: a trust framework, an API hub and common infrastructure services². Commentary highlights that the API hub is operated by a Central Bank subsidiary, Nebras, as a single gateway between providers and licensed institutions, that participation is mandatory for licensees in respect of in-scope products, and that DFSA or FSRA permissions in the financial free zones do not by themselves grant access to the onshore hub³.

Architecture

The data flow, end to end

The hard part: security that reviewers recognise

Financial APIs attract determined attackers, and generic OAuth setups leave room for token theft and replay. The OpenID Foundation's FAPI 2.0 security profile defines a hardened profile of OAuth 2.0 and OpenID Connect for exactly these APIs; its security profile and attacker model were approved as final specifications, and conformance tests and certification are available⁴⁵. Building to a recognised profile meant security discussions with partners started from a shared baseline rather than a custom design.

• Sender-constrained tokens and strong client authentication, so a stolen token alone is not enough.
• Keys held in a key management service or hardware security module, never in application configuration.
• Conformance testing in CI against the profile, run before every release of the connector.
• Product services given no direct access to tokens or the hub; they consume events and minimised data.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: treat consent as the centre of the data model and build to a recognised security profile from the start. Differently: settle the licensing and regulatory perimeter before detailed design, especially the onshore and free zone question. It shaped which institutions and products were reachable more than any technical choice. Related reading: UAE PDPL and AI architecture and idempotent payment pipelines.

If you're building fintech products in the UAE, the fractional CTO in the UAE page explains how I work, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The pattern is familiar in most engineering organisations by now. Engineers use AI assistants every day. Pull requests are larger and arrive faster. Reviewers are stretched, and occasionally someone merges code that nobody on the team can fully explain. There is usually no policy, just a vague sense that "we should be careful".

A Kubernetes maintainer described the core problem well: AI has made generating code fast, but there has been very little improvement in maintaining codebases¹. That asymmetry is exactly what a policy needs to address.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A company had grown partly through acquisitions, and each acquired team had brought its own cloud and its own way of using Terraform. The result was infrastructure across AWS, Azure and GCP, described by hundreds of folders. Modules had been copied rather than shared, so fixes landed in one place and not the others. Some state files described entire environments, so a careless plan could touch hundreds of resources. Database passwords and keys sat in state, readable by anyone with access to the bucket.

Engineers had learned to avoid infrastructure changes. That was the real cost: features waited on a handful of people brave enough to run apply.

What had to be true

• Small, composable modules per cloud, versioned and shared, rather than copies.
• Small blast radius: no single plan able to damage unrelated systems.
• No plaintext secrets in state, and narrow access to state that remains.
• Every change through a pipeline, with plan review and policy checks; no laptop applies to production.
• Teams self-serve common infrastructure through a paved road, with the platform team owning the road, not every change.

Architecture

The data flow, end to end

The hard part: resisting the universal module

The first instinct was a set of cloud-agnostic modules: one "database" module that worked on all three clouds. It would have hidden the features that made each managed service worth using and created a large abstraction nobody could debug. HashiCorp's guidance favours composition instead: a root module assembling small building blocks and passing dependencies between them as inputs and outputs¹.

• Building blocks were provider-specific, thin and opinionated about security defaults.
• Shared interfaces covered what really is common: naming, tagging, ownership labels and output shapes.
• Templates composed blocks into patterns teams actually use, such as "service with database and queue".
• Each block had examples and tests, and was released with semantic versions.

Secrets were the other urgent fix. Terraform 1.10 introduced ephemeral values that are not persisted in plan or state, and 1.11 added write-only arguments on managed resources, so secrets can flow from a secrets manager into resources without landing in state²³. Every secret that had ever been in state was rotated as part of the move. The checklist is in keeping secrets out of Terraform state.

How it ran

Key decisions and trade-offs

The tooling question came up early. OpenTofu was forked from Terraform under the Linux Foundation after HashiCorp's 2023 licence change⁴. Rather than let that decision block the refactor, modules were kept compatible with both while legal and procurement reached a view; the trade-offs are in OpenTofu or Terraform in 2026.

What I'd do again, and differently

Again: put pipelines, policy and state boundaries in place before migrating anything, and rotate every secret that touched state. Differently: publish the module interface conventions (names, tags and outputs) in the first week. Teams started writing new stacks immediately, and every one written before the conventions existed needed rework. More on structure in Terraform module design for multi-cloud.

If your infrastructure code has become something people avoid touching, my fractional CTO work covers estates like this, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

An AI software company sold to enterprises in Europe, the United States and the Gulf. Its products used large language models for summarisation, classification and agent-style automation. Every serious deal now came with a security questionnaire, an AI questionnaire and often a data protection review. The questions overlapped heavily but never matched exactly, so the CTO and a few senior engineers answered them by hand, deal after deal.

Meanwhile the rules kept moving. The EU AI Act's obligations were phasing in, then partly delayed; US buyers asked about SOC 2 and the NIST AI framework; Gulf buyers asked about data residency and local regulation. The company needed one answer, not three.

What had to be true

• One inventory of every AI system and feature, with owner, purpose, models, data and risk rating.
• One control set, mapped to each framework, rather than a programme per regulation.
• Evidence by design: logs, evaluations and approvals produced by the delivery pipeline, not gathered by hand.
• Proportionate: governance effort matched to risk, so low-risk features ship without ceremony.
• Owned by engineering and product, with legal and security as partners rather than gatekeepers.

Architecture

The data flow, end to end

The frameworks fit together better than their language suggests. The NIST AI Risk Management Framework organises AI risk work into govern, map, measure and manage functions⁴. ISO/IEC 42001 specifies a certifiable AI management system³. SOC 2 reports on controls against the trust services criteria for service organisations⁵. One control can satisfy all three when it is written once and mapped carefully, as discussed in ISO 42001 vs the NIST AI RMF and SOC 2 for AI startups.

The hard part: a moving regulatory timeline

The EU AI Act applies different obligations at different times¹. AI literacy duties already applied⁶, transparency obligations under Article 50 applied from 2 August 2026, and, as reported after the Digital Omnibus, stand-alone high-risk obligations moved to December 2027 with Annex I products following in August 2028². Building for a single deadline would have meant rework every time it moved.

• Obligations were attached to inventory entries by role and use case, not to a calendar, so a date change updated a field, not a plan.
• Controls that are valuable regardless of deadlines, such as inventory, evaluation, logging and human oversight, were built first.
• High-risk-specific controls were designed early but switched on per product as classification and timelines firmed up.
• Each EU AI Act question was tracked as provider or deployer, because the obligations differ.

The background is in the Digital Omnibus roadmap, when fine-tuning makes you a provider and, for Gulf buyers, the UAE's Federal Authority for AI and Data.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: start with the inventory and let engineering pipelines produce the evidence. Differently: involve sales engineering from the start. They see every questionnaire, and their list of repeated questions is the fastest way to prioritise which controls and evidence to build first.

If AI questionnaires are slowing your deals, my fractional CTO work covers governance programmes like this, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

Every service that calls another service has to prove who it is. Inside Kubernetes, the default proof has been a service account token, a signed JWT that the kubelet mounts into the Pod and keeps fresh². It works well, which is why it underpins workload identity federation with the major cloud providers¹. It also has one structural weakness, and Kubernetes 1.37 shipped a GA foundation for addressing it.

How Pod Certificates work

A Pod requests certificates through projected volume sources in its spec¹³. When the Pod is scheduled, the kubelet generates a private key, creates a PodCertificateRequest addressed to a named signer, and waits for the signer controller to issue a certificate chain. The kubelet writes the key and certificate into the container, alongside trust anchors collected from matching Cluster Trust Bundles, and refreshes everything before it expires¹.

Two details matter in practice. Rotation is automatic and frequent: signers eventually shipped in core are expected to issue certificates with a maximum lifetime of 24 hours, and other signers are capped at 91 days, so applications must reload credentials when files change. And security checks sit in the API server, where node restriction stops a compromised node requesting certificates for Pods not scheduled to it¹.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A biotech with several programmes in clinical development worked, like most of its peers, through vendors: one for electronic data capture, others for central labs, patient-reported outcomes, wearable devices and imaging. Each study's data team built its own transfers, mappings and checks. Submission datasets were produced by statistical programmers working from extracts, and the reasoning behind many transformations existed only in their code and memory.

As the pipeline grew, leadership saw the risk: every new study cost the same effort as the last, and an inspection question about where a value came from could take days to answer.

What had to be true

• Traceability from every submission value back to its source record and every transformation in between.
• Reuse across studies: shared ingestion, mapping patterns and checks, not bespoke pipelines per protocol.
• Quality visible early, to data managers during the study, not at database lock.
• Audit trails and access control by design, suitable for computerised system validation.
• Vendor independence: changing an EDC or lab vendor should not mean rebuilding the platform.

The regulatory direction supported this. ICH finalised the revised Good Clinical Practice guideline, E6(R3), in January 2025, and commentary highlights its risk-proportionate approach and a new data governance section that acknowledges digital trial data handling¹. In the EU, the principles and Annex 1 applied from 23 July 2025².

Architecture

The data flow, end to end

Submission requirements anchored the design. The FDA lists the study data standards it supports and requires, including SDTM versions, with dates from which each version is supported and then required³. Building SDTM mappings as versioned, reviewable assets meant a standards update became a controlled change rather than a scramble.

The hard part: one mapping layer for many protocols

Every protocol is different, and the temptation is to treat every study as a special case. The platform separated what genuinely varies by study, such as visit schedules, endpoints and specific forms, from what doesn't, such as demographics, adverse events, concomitant medications and lab structures.

• A shared library of mappings and derivations for common domains, with study-level extensions rather than copies.
• Controlled terminology managed centrally, so the same concept is coded the same way across studies.
• Mapping specifications stored with the code, reviewed like code and versioned with each data cut.
• Blinded and unblinded data separated by access control and environment, not convention.

How it ran

Key decisions and trade-offs

For systems that create or modify records required by FDA regulations, 21 CFR Part 11 sets expectations for electronic records, including secure, computer-generated, time-stamped audit trails⁴. Designing those in from the first phase was far cheaper than retrofitting them.

What I'd do again, and differently

Again: one study end to end before building the shared library, and quality at the table from the first week. Differently: negotiate transfer agreements with vendors as part of platform work, not study start-up. Consistent vendor formats are what make shared mappings possible. Related approaches are in data contracts and FHIR to OMOP architecture.

If your clinical data is growing faster than your ability to trace it, the healthcare data architecture page describes how I work, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A pharmaceutical company's drug safety team handled case processing well; their safety database and regulatory reporting were mature. Signal management was not. Potential signals arrived from periodic report reviews, literature, partners and occasional regulator queries, and were tracked in a spreadsheet. Evaluating one meant a custom data request, an analyst working for weeks, and conclusions summarised in slides that did not record which data or method they came from.

The head of safety wanted three things: consistent screening, faster evaluation and a record any inspector could follow from first observation to final decision.

What had to be true

• Every signal has one record, from source to decision, with who did what and why.
• Screening is consistent: the same methods on the same schedule, not ad hoc queries.
• Detection is not confused with evidence. Report counts start investigations; they don't conclude them.
• Evaluation is fast and reproducible, using standardised real-world data and versioned study code.
• Validated where it matters, with the computerised system controls a regulated safety function expects.

The EU's good pharmacovigilance practice guidance on signal management frames the lifecycle the platform needed to support, and EMA presentations on it emphasise that marketing authorisation holders should keep an audit trail of their signal management activities¹.

Architecture

The data flow, end to end

The hard part: not over-reading spontaneous reports

Spontaneous reports are the backbone of signal detection and also easy to misuse. The FDA states plainly that a report does not mean the product caused the event, that report counts cannot establish how often an event occurs, and that reports can be duplicated, incomplete or stimulated by publicity². The platform's design reflected that:

• Disproportionality scores were labelled as screening statistics, never as risk estimates, everywhere they appeared.
• Every flag required case-level review by a safety physician before becoming a validated signal.
• Evaluation used designed studies on population data, not more report counting.

For evaluation, self-controlled designs suited many acute safety questions because they compare time periods within the same people, removing confounding by stable characteristics³. Negative control outcomes and empirical calibration showed whether residual bias could explain a result⁴. The designs are compared in real-world evidence study designs.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: fix the record-keeping first, then add analytics. Differently: agree data access with real-world data partners at the start. Contracts and governance for external data take longer than building the screening pipeline, and evaluations wait on them. The data side is described in federated health analytics and OMOP data quality checks.

If your safety, evidence or medical teams need a platform like this, the healthcare data architecture page describes how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

Most early MCP servers were built like chat connections: a client connected, shook hands, held a session open and the server pushed requests back when it needed something. That worked on a laptop. It worked less well behind a load balancer, in serverless runtimes, or anywhere a connection could drop. The 2026-07-28 specification addresses that head on¹.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A software company had moved quickly on AI. Support had an answer assistant, sales had an email drafter, the core product had a summariser and a natural-language search box. Each team had picked prompts, vendors and settings independently. When a model provider released a new version, nobody knew which features would get better and which would quietly break. Quality problems surfaced as customer tickets. Finance saw one large model invoice and could not split it by feature.

Leadership did not want to slow the teams down. They wanted the same confidence in AI releases that they already had in ordinary code releases.

What had to be true

• Every AI feature has an evaluation set that reflects real use, owned by the team that ships it.
• Changes ship through gates: a prompt, model or retrieval change runs the evals before it reaches customers.
• Every model and tool call is traceable, with latency, tokens and errors, using an open standard rather than a vendor format.
• Sensitive content is not logged by default.
• Cost and quality are visible per feature, so product managers can make trade-offs.

Architecture

The data flow, end to end

Tracing followed the OpenTelemetry semantic conventions for generative AI, which standardise how model calls are recorded, including the model, input and output token counts and, when explicitly enabled, prompt and completion content¹². Standard metrics for token usage and operation duration made cost and latency comparable across features and vendors³. The detail is in LLM observability with OpenTelemetry.

The hard part: evaluation sets that predict production

Building the pipeline was straightforward. Building evaluation sets that actually predicted how features behaved for customers was not. Generic benchmarks said little about whether the support assistant would answer a specific customer's question correctly.

• Cases drawn from real, anonymised usage, grouped by intent, with a slice of known hard and sensitive cases.
• Deterministic checks wherever possible: format, required fields, citations present, forbidden content absent.
• Rubric-based model graders only where judgement was needed, calibrated against human labels. Research on LLM-as-a-judge found strong models can match human preference agreement, but also documented position and verbosity biases that need controlling⁴.
• Regular refresh from production misses, so the set evolves with the product.

More on this in evaluation sets that predict production and, for agentic features, evaluating AI agents.

How it ran

Key decisions and trade-offs

The programme was mapped to the NIST Generative AI Profile, which gave the risk team a recognised structure for issues such as confabulation, information security and data privacy⁵.

What I'd do again, and differently

Again: start with tracing and cost per feature, because visibility creates demand for evaluation. Differently: write a model upgrade playbook in the first phase. Provider version changes arrive on their schedule, not yours, and a rehearsed path from new model to gated rollout takes the drama out of them.

If your AI features are multiplying faster than your confidence in them, my AI architecture work covers evaluation and observability platforms, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

A familiar pattern in health data projects: the team gets access to a new OMOP database, and within a week someone is training a risk model. A month later the model behaves strangely, and the cause turns out to be something basic. Half the patients have under a year of observation, a whole drug class is missing from one source, or the "new users" are mostly people transferring in from another system.

Characterisation is how you find those things first. The Book of OHDSI describes four methods, each answering a different question¹.

Incidence, defined precisely

Incidence measures new occurrences of an outcome in a population during a time at risk. The book defines two metrics¹:

• Incidence proportion: people in the cohort with a new outcome during the time at risk, divided by people in the cohort with time at risk.
• Incidence rate: people with a new outcome divided by the total person-time at risk, where each person's time stops when the outcome occurs.

Most disagreements about "the rate" of something trace back to different choices of cohort, time at risk or outcome definition. Writing those down explicitly is what makes incidence comparable across sites and studies.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A fast-growing technology company ran most of its products on a handful of shared Kubernetes clusters. Teams shipped quickly, which was the point. But cloud spend had become one of the largest lines in the budget, and the monthly review was a guessing game: the bill showed compute, storage and network by account, while the organisation thought in teams and products. Requests and limits had been copied between services for years. Some workloads ran around the clock for jobs that happened once a day.

A previous clean-up project had cut costs for a quarter, then spend crept back. The leadership question was simple: how do we make this stay fixed?

What had to be true

• Every workload has an owner: a team and a product, enforced, not hoped for.
• Cost is visible where decisions are made: per team, per product, in a format finance accepts.
• Efficiency is the default: new services start right-sized and autoscaled without teams becoming Kubernetes experts.
• Shrinking is cheap and safe, so teams actually do it.
• No slowdown in delivery; controls live in the paved road, not in tickets.

Architecture

The data flow, end to end

OpenCost provides workload-level cost allocation for Kubernetes, attributing cluster costs to namespaces, labels and workloads¹. Joining those allocations with billing data in the FinOps Foundation's FOCUS format gave finance and engineering one view of the same numbers²; the reasoning is in FOCUS and AI spend.

The hard part: making shrinking safe

Visibility alone rarely changes behaviour. Teams knew some services were over-provisioned; they left them because changing requests meant restarts, and restarts meant risk. Two Kubernetes changes removed most of that friction:

• In-place Pod resize, stable since Kubernetes 1.35, lets CPU and memory change on running Pods, and the Vertical Pod Autoscaler's in-place mode uses it³. Recommendations could finally be applied continuously, starting with CPU on stateless services. Details in in-place Pod resize.
• Scale to zero with the HorizontalPodAutoscaler, available from Kubernetes 1.37 for workloads driven by suitable object or external metrics such as queue length⁴. Workers that waited idle most of the day stopped costing anything while idle. See Kubernetes FinOps and scale to zero.

Namespace quotas then capped the total each team could request, which turned capacity into a visible, negotiable budget rather than an unlimited pool⁵.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: start with ownership and allocation, and put every improvement into the templates so it spreads automatically. Differently: agree unit cost metrics with product leaders earlier. Cost per team tells you who spends; cost per customer action tells you whether the spend is worth it.

If cloud costs are outgrowing the business, my fractional CTO work covers platform and FinOps programmes like this, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

An organisation had embraced AI assistants faster than it had governed them. Developers connected IDE assistants to MCP servers for the code host, the ticketing system and the data warehouse. Some servers were built internally in an afternoon; others were installed from public repositories. Many used personal access tokens with wide permissions, stored in local configuration files. Nobody could say which servers were in use, who had approved them, or what they had done.

Security's first instinct was to block MCP outright. Leadership's instinct was the opposite: agents were becoming central to engineering and operations, and a ban would simply push usage further out of sight.

What had to be true

• One place to find approved servers, discoverable by the clients people already use.
• No long-lived shared tokens. Access flows through the identity provider, per user, with the narrowest scopes that work.
• Every tool call is attributable to a person, an agent, a server and a version.
• Writes and destructive actions need explicit, visible consent.
• Publishing a new internal server is easy if it follows the standard; exceptions are rare and visible.

Architecture

The design followed the ecosystem's own shape: the official MCP Registry is intended as an upstream that organisations build private subregistries on, adding internal servers that should never appear publicly³. The approach is described in more depth in running an enterprise MCP registry.

The data flow, end to end

Step five matters most. The MCP security guidance names token passthrough, where a server forwards a token it received from the client to a downstream API, as an anti-pattern that breaks OAuth security boundaries, and the authorisation specification forbids it¹². Every server on the platform validates that tokens were issued for it, and calls downstream systems with its own, separately scoped credentials.

The hard part: scopes people would actually accept

The technical controls were well understood. The difficult part was scope design. The MCP guidance warns that over-broad scopes granted up front expand the blast radius of a stolen token, make revocation disruptive and create consent dialogs that users abandon; it recommends a progressive, least-privilege model starting with a minimal read-only set¹. In practice that meant:

• A small, named set of scopes per server, starting with read-only discovery.
• Write and admin scopes requested only when a user first invokes a tool that needs them.
• Central control through the identity provider using the authorisation extensions, so administrators can grant and revoke access in one place⁴.
• Metrics on which scopes are requested and used, so creep is visible.

How it ran

Key decisions and trade-offs

The prompt injection and excessive agency risks in the OWASP Top 10 for LLM applications shaped the server template, with tool descriptions treated as untrusted and write tools gated by consent⁵. The detailed controls are in securing MCP servers, prompt injection defence for agents and MCP extensions strategy.

What I'd do again, and differently

Again: start with an inventory and make the approved path faster than the unapproved one before enforcing anything. Differently: publish the server template and its CI checks in the first phase. Teams wanted to build compliant servers early, and every week without a template produced another one-off that later needed migrating.

If agents are spreading faster than your controls, my AI architecture work describes how I approach agent platforms, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

Ask a platform team how their services configure OpenTelemetry and the honest answer is usually "it depends on the service". One team sets a dozen OTEL_ variables in a Helm chart, another wires exporters in code, a third copied a sampler setting from a blog post two years ago. The result is telemetry that differs in quality across services for no good reason, and changes that require touching every repository.

OpenTelemetry's declarative configuration is designed to fix that, and the core of it is now stable.

file_format: "1.0"
resource:
  attributes:
    - name: service.name
      value: checkout
tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_http:
            endpoint: ${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:-http://collector:4318/v1/traces}

Read on asksumanta.com for diagrams, references and related reading.

The situation

A multi-line insurer in the United States handled first notice of loss through a mix of call centre notes, web forms, emails with attachments and broker submissions. Triage was manual: a team read each new claim, decided its complexity, picked a queue and flagged anything suspicious. At peaks, the backlog meant simple claims waited behind complex ones, and the quality of triage depended on who happened to read the file.

Leadership wanted AI in the loop. The claims and compliance teams wanted something else first: confidence that no model would quietly decide a claim, and evidence they could show a regulator about how AI was governed.

What had to be true

• Adjusters decide. AI summarises, extracts and suggests; coverage, payment and denial decisions stay with licensed staff.
• Every flag has a reason a person can read and challenge.
• Testing for error and unfair outcomes before release and on a schedule, by line of business.
• One inventory of AI systems with owners, purpose, data and controls.
• No change to the claims system of record; AI writes suggestions alongside it, not into it.

These map closely to the NAIC model bulletin, which asks insurers to maintain a written programme for the responsible use of AI systems, including governance, risk management controls and testing for errors, bias and unfair discrimination¹. A growing number of states have adopted it, so building to it was the safest baseline across jurisdictions²³.

Architecture

The data flow, end to end

Extraction used structured outputs validated against a schema, so a malformed response fails fast rather than reaching an adjuster, the pattern described in structured outputs for reliable LLM integration. Fraud signals drew on relationships between parties, addresses, repairers and prior claims, using the graph approach in entity resolution for knowledge graphs.

The hard part: fairness and explainability, not extraction

Getting a model to read a claim form well is now the easy part. The difficult questions were about outcomes. Does a fraud signal fire more often for claims from certain postcodes for reasons unrelated to fraud? Does a severity hint systematically under-prioritise one line of business? The bulletin expects insurers to test for exactly these risks¹, and the NIST AI Risk Management Framework gave a practical structure for mapping, measuring and managing them⁴.

• An evaluation set of historical claims with known outcomes, refreshed regularly, scored for extraction accuracy and routing agreement.
• Outcome comparisons across segments the compliance team defined, reviewed before each release.
• Fraud signals limited to features the special investigations team could explain, with every flag showing its contributing reasons.
• Override rates tracked by signal; a signal adjusters routinely override gets investigated, not ignored.

How it ran

Key decisions and trade-offs

For insurers with European operations, the EU AI Act adds another layer: its Annex III lists AI used for risk assessment and pricing of natural persons in life and health insurance as high-risk⁵. Claims triage sits next to that boundary, so classification should be confirmed with counsel; the governance described here is a strong starting point either way. See ISO 42001 vs the NIST AI RMF and Colorado's AI law reset for the wider landscape.

What I'd do again, and differently

Again: write the AI programme and the evaluation set before writing the first prompt, and run in shadow mode. Differently: involve the special investigations team in designing fraud signals from the first week. Their sense of what makes a signal actionable shaped the design more than any model choice.

If you're bringing AI into claims, underwriting or service, my AI architecture work covers how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A seed-stage B2B software company had found traction. The founders were strong builders and the small engineering team shipped quickly, but the things that matter at the next stage were falling behind. Architecture choices were made in chat and forgotten, the release process depended on one person, there was no shared view of reliability, and enterprise prospects were sending security questionnaires nobody had time to answer well. A Series A process was on the horizon, and technical diligence would follow.

The company did not yet need, or want to pay for, a full-time CTO. It needed senior technical leadership for part of each week, with a clear plan for handing over.

What had to be true

• A roadmap the team believed, connecting product goals to platform work and technical debt.
• Decisions written down, so they can be reviewed, challenged and learned by new hires.
• Delivery and reliability measured, as signals for improvement rather than targets to game.
• Security that stands up to scrutiny, with controls and evidence rather than promises.
• A team that outlasts the engagement: senior hires and habits that keep working after hand-off.

Architecture

Here the architecture was as much the operating model as the system. The technical work rested on a light set of leadership practices, each producing an artefact the team and investors could see.

Architecture decision records followed the lightweight format Michael Nygard described: context, decision and consequences, kept with the code¹. How I use them in a fractional role is in architecture decision records for fractional CTOs.

The data flow, end to end

The cadence mattered more than any single document. Each cycle followed the same loop:

Delivery was tracked with DORA's software delivery metrics, which cover throughput and instability. DORA itself warns against turning them into blanket targets, so they were used per service and only to find bottlenecks². See DORA metrics for AI-assisted engineering. Reliability used a small number of SLOs on the journeys customers cared about, with an agreed policy for what happens when an error budget is spent³.

The hard part: adding structure without slowing the team

A fast seed team is right to be suspicious of process. Every practice had to earn its place by saving more time than it cost. That meant:

• ADRs only for decisions that are expensive to reverse, written in under an hour, not for every library choice.
• Automating the evidence: delivery metrics from the pipeline, access reviews from the identity provider, change history from pull requests.
• Sequencing security work against real demand. The SOC 2 trust services criteria (security, availability, processing integrity, confidentiality and privacy) gave the structure⁴, with security controls first because every enterprise questionnaire asked about them. More in SOC 2 for AI startups.
• Hiring senior engineers who could own areas, not just add capacity, so leadership could be handed over rather than borrowed.

How the engagement ran

Key decisions and trade-offs

What I'd do again, and differently

Again: write decisions down from the first week and keep the cadence predictable, even when the week is busy. Differently: start the search for the permanent engineering leader earlier. Good senior hires take longer than any technical workstream, and the hand-off is only as strong as the person receiving it.

If your team is at a similar stage, the fractional CTO page describes how I work, when you need one covers the signals, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A private equity investor was preparing to invest in a software company whose growth story rested on AI: models that automated part of its customers' work and data assets that, the deck said, competitors could not replicate. The commercial and financial diligence was well under way. What the deal team lacked was an independent view of whether the technology, the data rights and the team matched the story, and what it would cost to fix anything that did not.

The timeline was set by the deal, so the review had to be structured, evidence-led and focused on findings that could change price, terms or the post-close plan.

What had to be true

• Evidence over assertion: every finding tied to something inspected, not only something said in an interview.
• Proprietary versus borrowed made explicit: which capabilities the company built, and which depend on third-party models or APIs.
• Data rights traced from source to model to product, including customer data used for training.
• Costs that scale: inference and cloud spend modelled against the growth plan.
• A report the deal team can use: risk-rated, with recommended responses, readable by non-engineers.

Architecture

The review was organised into workstreams, each with its own evidence requests and interview plan. AI governance was assessed against the functions of the NIST AI Risk Management Framework (govern, map, measure, manage)¹, with ISO/IEC 42001 used as the reference for what a management system for AI looks like². The comparison is in ISO 42001 vs NIST AI RMF.

The data flow, end to end

Secure development practices were reviewed against the NIST Secure Software Development Framework, which groups practices into preparing the organisation, protecting the software, producing well-secured software and responding to vulnerabilities³. The full checklist I use is in the AI startup technical due diligence checklist, and supply chain questions in AI model supply chain security.

The hard part: proving data rights, not just data

Showing that a company has data is easy. Showing that it may use that data for the purpose the plan depends on is not. The provenance workstream asked, for each dataset behind a model:

• Where it came from, under which contract or licence, and whether that permits training and commercial use.
• Whether customer data was used to train shared models, and whether customer contracts allow it.
• Whether the dataset is documented in a form like a datasheet, covering motivation, composition and collection⁴.
• Whether models carry documentation of intended use and evaluation, in the spirit of model cards⁵.

Regulation was mapped to where the company sells. For customers in the European Union, the review considered how the EU AI Act might classify the product and which provider or deployer obligations could apply⁶.

How the engagement ran

Key decisions and trade-offs

What I'd do again, and differently

Again: insist on repository and billing access early, and separate verified findings from representations in the report. Differently: start the data rights tracing on the first day, in parallel with legal counsel. It is the slowest workstream and the one most likely to change the deal.

If you're planning an investment, the technical due diligence page describes how I run these reviews, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A telecom operator ran customer-facing digital services (self-care apps, online ordering, payments and activation) on top of a mix of modern microservices and long-lived back-office systems. Monitoring had grown team by team. Each group had its own agents, dashboards and alert rules, and the same incident lit up several tools at once without any of them saying whether customers were affected.

The pattern on bad days was familiar: complaints arrived through support channels first, several teams investigated in parallel from their own dashboards, and the root cause sat at a boundary nobody's tool covered.

What had to be true

• One vendor-neutral telemetry pipeline, so instrumentation is done once and backends can change.
• Traces across boundaries, from the app request through the API layer to the back-office call.
• Objectives tied to customer journeys, agreed with the people who own those journeys.
• Alerts that mean action: page when users are being hurt, ticket when a budget is slowly eroding, and nothing else.
• Cost under control, with sampling and cardinality managed centrally rather than service by service.

Architecture

The core was an OpenTelemetry Collector gateway: services and node-level agents send OTLP to a central endpoint per cluster or region, where processing, sampling and export happen once¹. Deployment patterns are compared in OpenTelemetry Collector deployment patterns.

For older services that could not be changed quickly, OpenTelemetry eBPF Instrumentation (OBI) captured spans and rate, error and duration metrics for HTTP and gRPC traffic without code changes². It cannot see business context, so revenue-critical services also received SDK instrumentation for custom attributes. More in OpenTelemetry eBPF instrumentation.

The data flow, end to end

Prometheus held the SLI time series and evaluated recording and alerting rules, which suits numeric operational data well, though not anything that needs exact per-request accounting³.

The hard part: deciding what to promise

The pipeline was engineering. The objectives were negotiation. Each journey (sign in, view bill, pay, activate) needed an indicator measured close to the user, a target the business could defend, and an owner who accepted the consequences of the error budget⁵. In practice that meant:

• Choosing indicators at the edge of the journey, not on internal components that customers never see.
• Starting with targets the service already met most of the time, then tightening, rather than declaring aspirational numbers nobody could hold.
• Writing a short error budget policy per journey that said what happens when the budget runs low.
• Retiring old component alerts deliberately, with each team agreeing what replaced them.

Alerting followed the multiwindow, multi-burn-rate approach in the Google SRE Workbook, which pairs a long window for significance with a short window for fast reset⁴. The practical version for smaller teams is in SLOs and error budgets for small teams.

How the engagement ran

Key decisions and trade-offs

What I'd do again, and differently

Again: trace one journey end to end before instrumenting everything, and make product owners co-own the targets. Differently: set attribute naming and cardinality rules before the first service onboards. Cleaning up inconsistent labels later cost more than agreeing them early.

If you're planning something similar, the fractional CTO page describes how I lead platform and reliability work, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A property operator onboarded tenants and paid suppliers through a core property management system. Everything that fed it arrived as documents: signed leases, identity documents, proof of address, supplier invoices and service contracts. Each was opened, read and keyed by hand. Onboarding waited on the queue, keying errors surfaced later as billing disputes, and the team spent its time transcribing rather than checking.

Earlier attempts with template-based OCR had struggled because the documents varied too much: different lease templates, scanned and photographed IDs, and invoices in every layout a supplier could invent.

What had to be true

• Every value validated before it reaches the core system, by rules the business can read.
• The model never writes: it proposes a record, and deterministic code decides what is posted.
• Uncertain documents go to people, with the reason and the source region visible.
• Personal data handled deliberately: limited access, defined retention and no identity documents in logs or prompts beyond what extraction needs.
• Traceability from any posted value back to the page it came from.

Architecture

Each document type had a versioned JSON Schema describing its fields, types, formats and required properties¹. The model was asked to return output conforming to that schema, and the same schema validated the response, so a malformed answer failed loudly instead of slipping through. More on the pattern in structured outputs for reliable LLM integration.

The data flow, end to end

The audit and staging tables in Postgres carried their own check, not-null and foreign key constraints, so invalid states were rejected by the database as well as by the application³. Retries were safe because posting was idempotent, the same discipline described in idempotent payment and order pipelines.

The hard part: confidence you can act on

Extraction quality was rarely the blocker. Deciding when to trust it was. A model's own statement of certainty is not a reliable signal, so confidence was built from evidence instead:

• Arithmetic and consistency checks: invoice lines must sum to totals, lease dates must be in order, rent schedules must match the term.
• Reference checks: the unit, tenant or supplier must exist and match what the document claims.
• Field-level thresholds set per document type from a labelled sample, with stricter rules on fields that move money or identify people.
• Reason codes on every exception, so the team could see which rule or field caused most reviews and fix the cause.

Documents are also untrusted input. Text inside an uploaded file can carry instructions aimed at the model, a form of indirect prompt injection². The defence was architectural: the extraction model had no tools and no credentials, its output was confined to the schema, and posting was done by code that ignored anything outside defined fields. See prompt injection defence.

How the engagement ran

Ongoing monitoring of exception rates, corrections and drift by document type followed the measure-and-manage approach of the NIST AI Risk Management Framework⁴.

Key decisions and trade-offs

What I'd do again, and differently

Again: keep the model away from credentials and let validation, not the model, decide what is posted. Differently: build the labelled sample before the pipeline. Every threshold discussion went faster once there was a shared set of real documents to measure against.

If you're planning something similar, the AI architect page describes how I run these engagements, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A media organisation held decades of video, audio and photography. The material was valuable for new programmes, licensing and research, but most of it was hard to find. Cataloguing conventions had changed several times, many items carried only a title and a broadcast date, and the same subject was described in a dozen ways. Search worked if you already knew the exact words a cataloguer had used years earlier, and otherwise depended on a few long-serving people who remembered where things were.

The organisation wanted to use AI to describe the backlog. The concern, rightly, was that a model left to itself would produce confident, inconsistent tags at a scale nobody could review.

What had to be true

• One controlled vocabulary, so a subject, place or person is described the same way across the archive.
• Machine suggestions, human authority: models propose, people decide where confidence is low or the subject is sensitive.
• Provenance on every field, so anyone can see whether a label came from a cataloguer, a model or a reviewer.
• Search by meaning and by exact term, for editors who know what they want and for those who are exploring.
• A taxonomy the archive team owns, not one hidden inside prompts or code.

Architecture

The vocabulary was modelled as a SKOS concept scheme, which gives each concept a preferred label, alternative labels and broader or narrower relationships¹. Descriptive fields such as creator, date, subject and rights were aligned with DCMI Metadata Terms so that records stayed portable between systems².

The data flow, end to end

For semantic search, transcripts and summaries were embedded with a sentence-embedding model, which produces vectors that can be compared with cosine similarity³. Nearest-neighbour lookup used an HNSW index, a graph-based approximate search method that scales well to large collections⁴. Model choice and storage trade-offs are covered in choosing embedding models and pgvector or a vector database.

The hard part: a vocabulary people agree on

Running models over the archive was the easy part. Agreeing what the labels should be took far longer. The approach that worked:

• Seed the concept scheme from the terms cataloguers already used most, then merge duplicates as alternative labels rather than deleting them, so old records still match.
• Use SKOS mapping properties to link local concepts to external vocabularies where they existed, instead of copying those vocabularies wholesale¹.
• Never let the pipeline create a concept. A model suggestion with no match becomes a proposal for the taxonomy owner, with example items attached.
• Treat people and sensitive topics as always-reviewed fields, whatever the model's confidence.

The taxonomy design choices are discussed further in SKOS vs OWL for enterprise taxonomies.

How the engagement ran

Thresholds were set against a hand-labelled evaluation set drawn from the pilot collection, not against the model's own sense of certainty. The method is in LLM evaluation sets that predict production.

Key decisions and trade-offs

What I'd do again, and differently

Again: put the vocabulary at the centre and make provenance a field, not a log file. Differently: involve the cataloguers in designing the review screen from the first week. The speed and quality of review decided the throughput of the whole system, far more than the model did.

If you're planning something similar, the ontology and knowledge graph page describes how I run these engagements, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

An education platform served structured courses to school-age students, aligned to a defined syllabus. Students were already using general-purpose chatbots for homework help, and teachers could see the result: correct answers with methods the class had never been taught, explanations pitched at university level, and no sign the student had learned anything. The platform wanted its own tutor, built around how its courses actually teach, and one that teachers could see into and trust rather than work around.

What had to be true

• Grounded in the syllabus: the same methods, notation and vocabulary as the lessons.
• Teaches, not tells: hints and questions first, with solutions only where the teaching design allows.
• Safe for the age range, on what students type and on what the tutor says.
• Useful to teachers, as insight into misconceptions rather than transcripts to read.
• Affordable at classroom scale, when a whole year group logs in at once.

UNESCO's guidance on generative AI in education calls for data privacy protections, an age limit for independent use and validation of tools for pedagogical and ethical suitability¹. Those became design inputs, not an afterthought: the tutor was designed for supervised use within the platform, with schools in control of access.

Architecture

Grounding followed the retrieval-augmented generation pattern, combining a model with retrieval from a specific corpus², here the platform's own lessons and worked examples filtered by the student's course and unit. See the RAG to production checklist.

The data flow, end to end

The hard part: hinting without leaking the answer

Getting a model to be helpful is easy. Getting it to be helpful in the right way, and to resist a determined student asking for "just the answer" five different ways, is the real work.

• The tutoring policy was explicit and versioned: which exercise types allow worked solutions, how many hints before a fuller explanation, when to ask the student to show their step.
• Teachers authored a misconception library per objective, so hints targeted the likely error rather than repeating the method.
• An output check compared tutor responses against the exercise's answer key to catch leaked final answers.
• Attempts to override the tutor's instructions were treated as expected behaviour, not edge cases; prompt injection is the first entry in the OWASP Top 10 for LLM applications⁴. See prompt injection defence.

How the engagement ran

Key decisions and trade-offs

Tutoring prompts share a long, stable prefix of instructions, policy and lesson context. Prompt caching reuses a processed prompt prefix across requests, so only the changing tail is processed fresh, and any change to the prefix is a cache miss³. Structuring prompts so the stable parts come first made caching effective. Details in prompt caching for cost and latency.

What I'd do again, and differently

Again: write the tutoring policy with teachers before writing prompts, and let teachers grade the test set. Differently: build the teacher dashboard earlier. Seeing where whole classes got stuck changed the content roadmap as much as the tutor.

If you're building AI for learners, the AI architect page describes how I run these engagements, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A legal team handled a steady flow of commercial agreements: supplier contracts, NDAs, services agreements, licences. Each one was checked against an internal playbook that set out preferred positions, acceptable fallbacks and red lines for clauses such as liability caps, indemnities, termination, governing law and data protection. The work was skilled but repetitive, and it bunched up at quarter end when everything was urgent.

The team had tried generic AI chat tools. The answers sounded plausible, cited nothing, and didn't know the playbook. Nobody trusted them, rightly.

What had to be true

• The lawyer decides. The system prepares and proposes; it never sends a redline on its own.
• Every suggestion is cited to the exact contract text and the playbook position it relies on.
• Consistent structure, so outputs can be validated and compared, not just read.
• Measured accuracy per clause type, on the team's own contracts, before anyone relies on it.
• Confidentiality: contract text handled under the firm's existing data controls.

Contract review is a well-studied task: the CUAD dataset, built with legal experts, frames it as finding the clauses a reviewer needs to see across dozens of categories¹. That framing, extraction first and judgement second, shaped the design.

Architecture

Extraction used structured outputs, where the model's response is constrained to a supplied JSON Schema². Each clause came back with its type, section reference, verbatim quote and confidence. A deterministic check then confirmed that every quote actually appeared in the source document. See structured outputs for reliable LLM integration.

The data flow, end to end

The hard part: long context versus retrieval

A single contract usually fits within a modern model's context window, so the temptation is to put everything in: the contract, the whole playbook, and a pile of precedents. Research on long contexts shows models use information at the start and end of the input more reliably than information in the middle³, which is a poor property when the clause that matters is on page fourteen.

The design split the jobs:

• The contract is read whole for extraction, because cross-references and definitions matter.
• The playbook is retrieved per clause, in the spirit of retrieval-augmented generation⁴, so each comparison sees only the relevant positions.
• Comparison runs clause by clause, with small, focused prompts that are easier to evaluate.

The trade-offs are covered in long context vs RAG.

Contracts from counterparties are also untrusted input. Prompt injection tops the OWASP Top 10 for LLM applications⁵, so contract text was always treated as data, the model had no tools that could act on it, and outputs were constrained by schema. See prompt injection defence.

How the engagement ran

Key decisions and trade-offs

Building evaluation sets that hold up in production is covered in evaluation sets that predict production.

What I'd do again, and differently

Again: turn the playbook into data before writing a prompt, and treat lawyer decisions as the evaluation signal. Differently: involve lawyers in schema design from the start. The clause taxonomy is a legal decision, and the earlier they own it, the fewer rework cycles follow.

If you're considering AI for review work like this, the AI architect page describes how I run these engagements, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A manufacturer with several plants had good systems and poor connections. Quality inspection results lived in a central quality system. Production parameters lived in each plant's execution and historian stack, configured differently at every site. Supplier certificates and incoming inspection data arrived as spreadsheets and portal exports. When a customer complaint came in, tracing it back to a material lot, a line, a shift and a process setting took days of manual joining, and by then more product had shipped.

Previous attempts at a central warehouse had stalled. Every plant change broke a pipeline, and every report defined "first-pass yield" slightly differently.

What had to be true

• Joinable at the right grain: batch, lot, line and time, across quality, production and supplier data.
• Plants accountable for their feeds, with a clear owner and a written promise for each.
• Open storage, so different teams and engines could use the same tables.
• One definition per metric, shared by dashboards, analysts and any future AI use.
• No rip and replace of plant systems.

ISA-95 framed the boundaries. It addresses the interface between enterprise business systems and manufacturing operations and control¹, which is exactly where the lakehouse sat: reading from plant-level systems, serving enterprise-level analysis, owning neither.

Architecture

Apache Iceberg tracks table state through metadata files replaced by atomic swaps, with snapshots and schema evolution defined in the specification². That gave reproducible reads for investigations and safe schema changes when a plant added a parameter. See Apache Iceberg v3 for the lakehouse and lakehouse catalog strategy.

The data flow, end to end

Capture patterns are covered in change data capture and event streams.

The hard part: getting plants to own a promise

The technology was well understood. The hard part was organisational: plants had never been asked to promise anything about their data, and a central team had always absorbed the breakage. Contracts changed that, but only once they were written in terms plants recognised.

We used the Open Data Contract Standard, a YAML-based, platform-neutral format for agreements between data producers and consumers³; version 3 organises a contract into sections that include schema, data quality, service levels and team⁴. In practice:

• Contracts started small: schema, owner, freshness and a handful of rules the plant itself agreed were reasonable.
• A failed check went to the plant's named owner, not to the central team's backlog.
• Breaking changes required a new contract version and a notice period, agreed in a short change process.
• Plant-specific codes stayed in the feed; mapping to shared reference data was a central, versioned step.

More in the data contracts playbook.

How the engagement ran

Key decisions and trade-offs

Why metrics belong in a semantic layer, especially if AI will query them later, is covered in text-to-SQL and the semantic layer.

What I'd do again, and differently

Again: start by tracing one real complaint end to end, and make plants owners rather than suppliers of data. Differently: agree metric definitions with quality leads before building dashboards, not alongside them. Every disagreement found late cost a rebuild.

If you're facing something similar, the fractional CTO page describes how I run these engagements, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

An energy operator ran a large estate of rotating equipment, vessels, pipework and instrumentation. When something failed, the reliability team asked the obvious questions: has this happened before, on this unit or on similar ones, what did the sensors show beforehand, and what did we change last time? Answering meant searching the maintenance system, exporting historian trends, finding the right revision of a drawing and reading free-text work order notes. Each investigation rebuilt the same picture from scratch.

The data existed. The connections between it did not.

What had to be true

• One identity per asset, reconciled across the register, the maintenance system, tag lists and documents.
• A shared vocabulary for equipment classes, failure modes and maintenance actions, so "similar assets" and "same failure" mean something consistent.
• Systems of record untouched: link to them, don't replace them.
• Answers with evidence, because engineers will not act on an unexplained answer.
• Owned by the business: reliability engineers, not just the data team, maintain the model.

Architecture

The ontology borrowed its structure from ISO 14224, which organises reliability and maintenance data into equipment, failure and maintenance categories with a defined equipment taxonomy¹. Sensor concepts followed the W3C and OGC Semantic Sensor Network ontology, with SOSA as its lightweight core², rather than inventing our own model of observations.

Curated tables sat in Apache Iceberg, whose snapshot-based metadata gives versioned, reproducible inputs to every graph build⁵. See lakehouse catalog strategy and ontology vs knowledge graph vs semantic layer.

The data flow, end to end

SHACL let us express the rules ("every pump has exactly one parent system", "every work order references a resolved asset") as shapes that are validated on every build³. More in SHACL validation for knowledge graphs.

The hard part: deciding that two things are the same asset

Entity resolution took more effort than the graph itself. The same pump appeared under a tag number in the register, a slightly different tag in the maintenance system, a description in free-text notes and a drawing number in the document store. Some tags had been reused after equipment was replaced.

• Deterministic rules first (exact tag matches within a site), probabilistic matching second.
• Every match carries a confidence and a method, so low-confidence links are visible, not hidden.
• A review queue where reliability engineers confirm or reject uncertain matches; their decisions train the rules.
• Equipment replacement modelled explicitly, so history follows the functional location and the physical unit separately.

The patterns are described in entity resolution for knowledge graphs.

How the engagement ran

Key decisions and trade-offs

The assistant combined graph traversal with document retrieval. GraphRAG research shows graph-based indexes help with broad, corpus-level questions that plain vector retrieval handles poorly⁴; here the graph already existed, so retrieval followed its edges. See when GraphRAG beats vector RAG.

What I'd do again, and differently

Again: write competency questions with engineers before modelling anything, and make every link explainable. Differently: put the review queue in front of engineers sooner. Their corrections improved resolution more than any tuning we did.

If you're connecting operational data like this, the ontology and knowledge graph page describes how I run these engagements, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A UAE government entity had the usual pattern: enthusiasm for generative AI in every department, a handful of disconnected pilots, and a security team quietly worried about staff pasting documents into public chatbots. Policy said sensitive data could not leave national infrastructure. Nobody wanted to say no to AI; they wanted a safe way to say yes.

The national direction supports that stance. The UAE Charter for the Development and Use of Artificial Intelligence puts privacy, data security and compliance with local law alongside innovation as guiding principles².

What had to be true

• In-country by design: models, prompts, retrieved documents, logs and evaluation data all hosted on national infrastructure.
• One front door, so identity, redaction and audit are enforced once rather than reimplemented by each team.
• Known applicable rules, mapped to controls before the first use case went live.
• Measured quality: no use case in production without an evaluation set and an owner.
• Operable by the internal team after hand-off.

The rules question needed care. The federal Personal Data Protection Law sets aside several categories from its scope, including government data and government entities¹. That does not mean anything goes: it means the entity's own data classification, emirate-level policy and sector rules carry the weight, with PDPL principles a sensible baseline. We confirmed the applicable regime with legal counsel first. More context in UAE PDPL and AI architecture and the federal authority for AI and data.

Architecture

Serving used vLLM, whose PagedAttention design manages the KV cache in pages to reduce memory waste and raise throughput on shared GPUs³. That matters when a fixed, in-country GPU budget has to serve many departments. See LLM inference serving with vLLM and where to run models in the UAE.

The data flow, end to end

The hard part: one platform, many trust levels

The engineering was not the hard part. The hard part was that a single shared service had to respect different data classifications, department boundaries and risk appetites. The OWASP Top 10 for LLM applications lists prompt injection and sensitive information disclosure among the leading risks⁴, and on a shared platform both translate into one question: can a user see something through the model that they could not see directly?

• Retrieval inherits the caller's permissions; the model never sees a document the user couldn't open.
• Each use case is registered with a classification ceiling, an owner and an approved model list.
• Redaction is configurable per use case, not one global rule that is either too loose or too strict.
• Tool access for agent-style use cases is explicit and minimal. See prompt injection defence for agents.

How the engagement ran

The governance loop followed the NIST AI RMF functions of govern, map, measure and manage⁵, adapted to the entity's own approval structure.

Key decisions and trade-offs

Routing and evaluation patterns are covered in LLM model routing and evaluation sets that predict production.

What I'd do again, and differently

Again: put the gateway and the audit log in before the first use case, and make departments bring their own evaluation examples. Differently: start capacity planning earlier. In-country GPU capacity is finite and slow to expand, so demand forecasting deserves the same attention as the architecture.

If you're planning sovereign AI in the region, the UAE page describes how I work there, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A B2B SaaS platform had grown on a single shared architecture: one application deployment, one database cluster, every tenant in the same collections. That served it well while tenants were similar in size. As larger customers arrived, two problems appeared together. A heavy import or reporting job from one tenant could slow the service for everyone. And enterprise prospects began asking, in security reviews, how their data was isolated and whether they could have dedicated infrastructure.

The team's instinct was to give big customers their own copy of everything. That answers the sales question and creates an operations problem: many stacks to upgrade, monitor and pay for.

What had to be true

• Isolation as a choice per tenant, not one model forced on all.
• One codebase and one release process across every tier.
• Per-tenant limits at every shared resource.
• Tenant context everywhere: in routing, data access, logs, traces and cost.
• Isolation proven by tests, so the answer to a security review is evidence.

Architecture

AWS's tenant isolation guidance frames the options as silo (dedicated resources), pool (shared resources) and bridge (a mix chosen per layer)¹. The design adopted that vocabulary directly: a pooled tier for most tenants and a dedicated tier for those whose requirements justify it, running the same services.

The data flow, end to end

On the data side, MongoDB's guidance compares database-per-tenant with shared collections keyed by a tenant field, and notes that the shared model needs every query to filter by tenant³. That requirement was moved out of individual queries and into one data access layer. Detail in MongoDB multi-tenant SaaS patterns.

The hard part: noisy neighbours at every layer

Noisy neighbours are rarely one bottleneck. Limits had to exist at each shared resource, or load simply moved to the next one. The Kubernetes documentation treats noisy neighbours as a core multi-tenancy concern and points to resource quotas and node isolation as controls²; quotas cap aggregate consumption per namespace⁵. The layers addressed were:

• Requests: per-tenant rate and concurrency limits at the gateway.
• Background work: per-tenant queues or fair scheduling, so one tenant's bulk import cannot starve others.
• Compute: dedicated namespaces with quotas for the dedicated tier; resource requests and limits in the pool.
• Database: indexes that start with the tenant key, query timeouts, and heavy reporting moved off the primary path.

None of that is manageable without seeing it per tenant. OpenTelemetry baggage carries key-value context such as a tenant identifier across service boundaries⁴, which let traces and metrics be broken down by tenant. More in Kubernetes multi-tenancy.

How the engagement ran

Key decisions and trade-offs

What I'd do again, and differently

Again: get tenant context right before anything else, and make isolation a test suite rather than a document. Differently: agree the tier definitions with product and sales earlier. The engineering was straightforward once everyone agreed what "dedicated" promised and who could buy it.

If your platform is facing the same pressures, the fractional CTO page describes how I work with teams on it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A logistics operator's customer service and operations teams handled shipment exceptions largely by email. A late container, a failed delivery, a missing customs document or an address query each arrived as free text, often with attachments, sometimes in a long thread. Staff read each message, worked out which shipment it concerned, looked it up in the transport management system (TMS), decided what to do and then did it, across several screens.

The work was repetitive but not trivial. The hard part of each email was usually the same: identify the shipment, understand the exception, gather the facts. The decision itself was often quick once the facts were on one screen.

What had to be true

• Structured, not chatty: every email becomes a typed exception record, or is routed to a person as unclassifiable.
• Governed tools: the agent reaches systems only through a defined, permissioned tool layer.
• Read freely, write with approval: lookups are automatic; anything with side effects needs a person.
• Inbound email is untrusted and never treated as instructions.
• Measured and traceable: behaviour evaluated before release, every run inspectable afterwards.

Architecture

The design favoured a defined workflow with agentic steps over an open-ended autonomous agent. Anthropic's guidance on building agents makes the same distinction, recommending the simplest pattern that works and adding autonomy only where it earns its keep¹.

The data flow, end to end

Classification and extraction used structured outputs, where the model's response is constrained to a supplied JSON schema⁵, so downstream steps receive valid records rather than prose. Approval pauses used LangGraph interrupts, which persist graph state while waiting for human input and resume with that input³. More in structured outputs for reliable LLM integration.

The hard part: tools you can trust an agent with

The model was rarely the weak point. The tool layer was where safety lived. The MCP specification describes tools as model-invoked and recommends that a human can always deny a tool invocation². The design took that literally:

• Separate read and write servers, with write tools reachable only from the post-approval step of the workflow.
• Narrow tools: "get shipment milestones" and "request redelivery", not a general API passthrough.
• Service identity per tool, scoped to the minimum permissions in the TMS.
• Email as data: inbound text is passed as content to classify, never as instructions, because prompt injection and excessive agency both appear in the OWASP Top 10 for LLM applications⁴.

The checklist used is in securing MCP servers and prompt injection defence for agents.

How it ran

Evaluation followed evaluating AI agents for reliability, scoring classification, shipment matching and proposed actions separately, so a regression could be located rather than just noticed.

Key decisions and trade-offs

What I'd do again, and differently

Again: start read-only, build the labelled set from real emails, and keep write tools behind approval. Differently: invest earlier in shipment matching. Ambiguous references caused more escalations than misclassification did, and better matching would have improved every downstream step.

If you are designing agents that touch operational systems, the AI architect page describes how I approach it, or analyse your challenge and bring the result to a call.

Read on asksumanta.com for diagrams, references and related reading.

The situation

A retailer sold through physical stores, its own website and several marketplaces. Stock levels lived in a warehouse system and store systems, orders in an order management database, and each channel received a stock file once a night. During the day, the same last unit could be sold in a store, online and on a marketplace before anyone knew.

The workarounds made things worse: large safety buffers that hid sellable stock, manual cancellations, and ad hoc point-to-point integrations that each team had built to get fresher data for its own channel.

What had to be true

• Every committed change is captured, without polling or adding load to operational databases.
• No dual writes: an order and its event either both happen or neither does.
• Channels read a projection, not the source systems.
• History is replayable, so a bug in a projection can be fixed and rebuilt.
• Contracts are explicit, so source teams can change their schemas without breaking consumers.

Architecture

Log-based change data capture reads committed row-level changes from the database's transaction log, so applications consume changes rather than querying the source¹. That became the backbone, with a stream processing layer building inventory views for each consumer.

The data flow, end to end

More on the capture side in change data capture and event streams, and on duplicate-safe consumers in idempotent order pipelines.

The hard part: events that mean something

Streaming raw table changes was quick to demonstrate and the wrong long-term contract. Consumers ended up decoding internal columns and status codes, and any schema change in the source broke them. The transactional outbox pattern addresses this: the service writes a domain event to an outbox table in the same transaction as the business change, and a relay publishes it³. Debezium's outbox event router does that relay from the log, routing events to topics by aggregate type and keying them by aggregate identifier².

• Events were designed with the consuming teams: "stock adjusted", "order allocated", "reservation released", not "row updated".
• Keys were chosen for ordering: everything about one SKU at one location lands in one partition, in order.
• Contracts went into the schema registry with compatibility rules, following a data contracts approach.

For current state, compacted topics keep the latest record per key and support rebuilding state after a failure⁴, which let new projections bootstrap without a bulk extract from the source.

How it ran