Sonde - LLM Evaluation and Observability Platform with Release Gates
Quality, Cost and Drift Monitoring for a French Health Mutual
Designed and built a clickable LLM evaluation and observability platform for the AI team of a French health mutual: redacted production traces span by span, release gates that block a prompt or model change when quality regresses, alerts with owners and ageing, and the monitoring record its risk and compliance teams can show a supervisor.
Most of our work is under NDA. Per agreement, we can share the work but not the company details - all names and identifying information shown here have been changed.
Impact
Dashboard, traces, trace detail with span waterfall and 5 tabs, release gates with the gate run detail, and alerts with the alert panel. Every one of the 28 demo traces opens its own detail page.
Any personal data leak in outputs and a red-team pass rate below the minimum block a release outright. Other failed checks can be overridden only with a reason, risk accepted, an expiry and a second approver.
Conversation content stays redacted. Revealing it needs a reason and permission, shows an "Unredacted view - logged" banner and re-locks after 15 minutes.
Member assistant, claims triage agent, quote reader and adviser assistant on Mistral, Azure OpenAI and Bedrock models, with 28 traces, 10 gate runs, 12 alerts and 8 staff users.
The Problem
Serpolet's AI team shipped its first assistant with a spreadsheet of 60 test questions that one engineer ran by hand before each release. A prompt change made the assistant quote the wrong reimbursement ceiling for dental implants, and nobody noticed for eleven days. Logs lived in three places, token spend only showed on the provider invoice, and when the risk function asked how the claims triage agent had been validated, the answer was a slide deck.
Our Solution
Sonde ingests the traces of every LLM system Serpolet runs, redacts them at ingestion, scores them with online evaluators and tracks cost and latency per system. Each prompt, model or retrieval change runs through a release gate in CI that compares it with production, stops on hard rules and waits for the product owner's approval, and every breach becomes an alert with an owner.
The Challenge: Proving an LLM Still Works
A health mutual runs several LLM systems that answer members, triage claims and read quotes. Each one can drift silently, each release can break answers that used to be right, and the conversations they handle contain health data.
Regressions Found by Members
Testing was 60 questions run by hand. A prompt change could break dental, optical or hospital answers that passed before, and the first signal was a complaint. Quality had to be measured on golden sets before release and on sampled traffic after it.
Health Data in Every Trace
Member conversations carry personal data and often health data, a special category under Article 9 GDPR. Identifiers have to be masked at ingestion, member ids pseudonymised, retention set per system and every unredacted view recorded with who, when and why.
Evidence a Supervisor Can Read
The EU AI Act expects logging, human oversight and monitoring, and lists risk assessment and pricing in life and health insurance as high-risk. The ACPR is generally expected to act as market surveillance authority for AI used by the institutions it supervises. Classification per system and its dates must be settings compliance can change, not code.
Providers as ICT Dependencies
Under DORA, LLM API providers are ICT third-party providers. Sonde has to record which provider and model each system depends on and feed outage and latency data to the ICT incident process, while every pass mark stays Serpolet's own policy, not a legal rule.
Our Approach: From AI Governance Rules to a Clickable Platform
Spec and Rules Research
Mapped 9 user roles from AI engineer and product owner to model risk officer, DPO and auditor, 39 screens including 2 for domain reviewers, and the data model for systems, traces and spans, datasets, versioned evaluators, gate runs, threshold rules, monitors, alerts, model cards and a hash-chained audit log. Wrote down the rules the design follows: EU AI Act, GDPR and CNIL guidance, HDS hosting, DORA and the OpenTelemetry GenAI conventions.
Clickable Demo on a Production UI Kit
Built 5 key screens on a production dashboard kit: the dashboard with KPIs, a health card per system, quality over time and spend by week, traces with search and filters, the trace detail with span waterfall, span inspector, evaluations and retrieval, release gates with checks against baseline, hard stops, flipped items and approve, reject or override, and alerts with acknowledge, resolve, escalate and mute. All actions run on local state.
What a Production Build Adds
OTLP trace ingestion through the OpenTelemetry Collector, Presidio redaction before storage, ClickHouse for traces, LLM judges on Mistral, Azure OpenAI and Bedrock, garak and PyRIT red-team runs, gate status posted back to GitLab CI or GitHub Actions, PagerDuty or Opsgenie and Teams alerting, Entra ID single sign-on, HDS-certified hosting, and the remaining screens: monitors, datasets, evaluators, experiments, annotation queues, model cards, incidents and the audit log.
Why It Works: Evidence Before Every Release
Every Check Against Production
A gate run lists each evaluator with its dataset version, the candidate score against the current production score, the allowed regression and the minimum, such as groundedness 0.93 vs 0.94 with -0.02 allowed and a 0.90 floor. Items whose verdict flipped from pass to fail are listed one by one.
Approval Limited to the Owner
The signed-in product owner can approve gates only for her own system. Overrides need a reason, risk accepted, an expiry and a second approver, and hard stops cannot be overridden at all.
Redacted by Default
Trace pages open with a Content redacted banner naming the masking rules applied. Reveal content needs a reason, runs a 15-minute countdown and marks the view as logged, so engineers can debug without browsing health data.
One Trace, Span by Span
The waterfall colours agent steps, LLM calls, retrieval, tool calls, embeddings and guardrails, with errors in red. A span opens model, parameters, tokens, cost and the raw OpenTelemetry attributes, and the evaluations tab shows why a groundedness check failed.
Related case studies
View all: AI agents & automation
Tallyman - Intelligent Document Processing for Trade and Customs Documents
Designed and built a clickable document processing platform for a Rotterdam freight forwarder and customs representative: it classifies commercial invoices, packing lists, bills of lading, air waybills and certificates of origin, extracts each field with a confidence score, checks it against business rules and sends only the uncertain fields to a person in a side-by-side review workspace.
Sanbuk - Accounts Payable Agent with an Operator Console and Approval Gates
Designed and built a clickable back-office agent for a Dubai mainland distributor: it reads the AP and vendor mailboxes, matches invoices to POs and goods receipts, checks UAE tax invoice fields, drafts replies and posts to the ERP, while people approve by amount, see every tool call and handle every bank detail change themselves.
Enterprise Software Engineering - Human + AI
Contact Us
© 2026 - Copyright. All Rights reserved Oceanlab
Location
Global Headquarters
21 Merchant Road, 3rd Floor, Singapore 058267
Main corporate office and leadership operations
North America Office
548 Market Street, PMB 12345, San Francisco, CA 94104
Client operations and business development
India Office
402 Shivalik Shilp II, SG Highway, Ahmedabad 380015
Engineering and delivery center
Enterprise Software Engineering - Human + AI
Contact Us
Locations
Global Headquarters
21 Merchant Road, 3rd Floor, Singapore 058267
Main corporate office and leadership operations
North America Office
548 Market Street, PMB 12345, San Francisco, CA 94104
Client operations and business development
India Office
402 Shivalik Shilp II, SG Highway, Ahmedabad 380015
Engineering and delivery center
© 2026 - Copyright. All Rights reserved Oceanlab