Safe by design: preparing a Canadian learning app's AI study companion for chatbot safety duties

The situation

A Canadian education company ran a learning app used by secondary school students and young adults. Its most popular feature was an AI study companion: a chatbot that explained concepts, quizzed students and helped plan revision. It had a friendly persona, remembered past sessions and was tied into the app's streaks and reminders. Most conversations were about schoolwork. Some were not: exam stress, trouble at home, and occasionally something more serious. The chatbot answered those with whatever the model produced in the moment.

In June 2026 the Government of Canada introduced Bill C-34, the Safe Social Media Act, which would create duties for social media and AI chatbot services and a Digital Safety Commission to oversee them¹. For chatbots, the proposed duty to act responsibly includes mitigating the risk of harmful content, putting emergency measures in place for crisis situations and mitigating harmful chatbot behaviour². Osler notes the proposals also address manipulative engagement techniques that encourage emotional attachment³. The bill was not yet law, but the company's school and parent customers were already asking the same questions.

Why now. Canada's Budget 2026, released on 5 November, proposed $176.3 million over five years for the new commission, a sign that enforcement would be funded, not just legislated⁴. Building safety in before the rules settled was cheaper than retrofitting it under a deadline.

What had to be true

• Crisis handled every time: signals detected on every message, with a fixed, tested response and a route to people.
• No manipulative design: no feature that kept students engaged against their interests.
• Age-appropriate by default, with stricter settings for younger users.
• Tested continuously: safety behaviour checked whenever the model, prompts or features change.
• Evidence on hand: a written safety plan and records that show it works.
• Privacy respected: conversation data kept only as far as safety needs, under Canada's privacy laws.

Architecture

All model calls went through a gateway that applied policy before and after the model. A dedicated crisis detector ran on every user message, separate from the general content checks, because the cost of missing a crisis is different from the cost of missing an off-topic question. When it fired, the gateway did not call the study model at all. It returned a fixed response, reviewed by clinicians the company worked with, that acknowledged the student and gave crisis resources, including Canada's 9-8-8 helpline, then offered to continue only if the student wanted to.

The data flow, end to end

The hard part: crisis handling and engagement design

Not missing, not over-blocking. A detector tuned too loosely misses students who need help; tuned too tightly, it interrupts a student revising a history essay about war or a novel about grief. We built evaluation sets from reviewed conversations, written with clinicians and teachers, covering clear crises, ambiguous messages and schoolwork that only looks alarming. Every change to the detector, model or prompts ran against them, and misses were treated as serious defects. See evaluation sets that predict production.

Reviewing conversations without over-collecting. Reviewers needed to see what the student said and what the companion replied, as a trace with prompts and outputs, which OpenTelemetry's generative AI conventions make possible to capture as opt-in content⁵. Conversation content was captured only for flagged cases and samples for evaluation, stored separately with short retention and access logged, rather than logging everything by default; see LLM observability with OpenTelemetry.

Taking manipulation out. The harder conversation was about engagement. The companion had been tuned to keep sessions going, and some streak reminders used guilt ("your companion misses you"). We reviewed every engagement feature against one question: does this serve the student's learning, or only our retention numbers? Guilt-based messages went, the persona stopped expressing feelings about the student's absence, and late-night nudges were switched off for younger users.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: put the crisis path outside the model. It meant model upgrades never put the most important behaviour at risk. Differently: involve teachers in building evaluation sets from the start; they knew exactly which schoolwork topics would trip a detector. Related: New York's youth chatbot law, prompt injection defence, the curriculum-grounded tutor case study, the 7 November tech news and where I work.

If you run an AI chatbot used by young people, in Canada or elsewhere, see my AI architecture work, or analyse your challenge and bring the result to a call.