Your AI agents are silently burning money and breaking production.
Sentinel cross-queries 7 observability sources via Coral SQL to detect, diagnose, and govern AI system failures — and routes high-risk actions through human approval before acting.
SELECT l.trace_id, l.cost_usd, s.title AS error,
g.sha, g.author, m.text AS slack_msg
FROM langfuse.traces l
JOIN sentry.issues s USING (trace_id)
JOIN github.commits g ON g.ts BETWEEN l.ts - '15m' AND l.ts
LEFT JOIN slack.messages m ON m.ts BETWEEN l.ts - '10m' AND l.ts + '10m'
WHERE l.cost_usd > 50
AND l.retry_depth >= 4
ORDER BY l.cost_usd DESC;Production AI fails in ways your existing tooling can't see.
Logs, metrics, and traces in isolation miss the cross-source patterns that actually cause incidents.
Agent loop cost explosions
Multi-agent retry loops burn hundreds of dollars before anyone notices. Sentinel detects loop patterns in Langfuse traces, fires a kill signal, and posts an interactive Slack alert with a Kill button.
Prompt drift & fragility
Minor prompt edits silently break downstream JSON parsing. Sentinel maintains schema snapshots per feature, validates every trace, and blames the exact GitHub commit that caused the regression.
Silent tool call failures
Tools return 200 OK with semantically wrong output. Sentinel cross-references tool outputs against Sentry errors in the same trace window to surface failures that look like successes.
No human-in-the-loop governance
Agents execute high-risk actions without checkpoints. Sentinel routes anomalies through an approval gate — Slack buttons or a web queue, full audit trail, state written back.
Telemetry data saturation
AI systems generate overwhelming trace volume. Sentinel scores every trace and drops routine noise — ~85% volume reduction while retaining 100% of actionable data.
Social context, fused via SQL.
Every detection JOINs slack.messages on the incident window — surfacing what the team was saying during the event. No other observability tool fuses social and technical signal this way.
Three activation modes. One agent. Seven sources.
Each mode runs cross-source SQL, lets Claude narrate the finding, and routes risky actions through human approval.
On-Call Brain
Trigger: Datadog/Sentry spike or PagerDuty incident
JOINs commits + errors + costs + Slack chatter. Claude writes a root-cause analysis, opens a GitHub issue, posts the summary to Slack.
PR Risk Scorer
Trigger: GitHub pull_request opened
Scores based on historical correlation of touched files to past cost and error regressions. Posts the risk score as a PR comment. Requests changes when risk > 70.
Weekly Digest
Trigger: Monday 09:00 cron
JOINs merged PRs + error-rate changes + Langfuse cost trends + Slack thread context. Output: What shipped · What broke · What to watch.
Dashboard · Incidents · Forensics · Risk · Quality · Approvals · Digest · Settings
Next.js + FastAPI, deployed on Vercel and Railway. React Flow forensics graph. Slack Block Kit write-back, HMAC verified.
Cross-source SQL
Coral SQL JOINs across 7 sources — the only way to ask 'what did the team say while costs spiked after this commit?'
Forensics graph
React Flow dependency graph: commits → errors → cost spikes → Slack messages. Causal chains for any window.
HITL approval gate
Interactive Slack Block Kit (HMAC-verified) and a web approval queue with full audit trail.
Schema drift detection
Per-feature output snapshots, validated on every trace. Drift events blame the commit.
Loop kill switch
Detects retry-loop signatures in Langfuse. One Slack button stops the bleeding before the bill arrives.
Smart sampling
Scores every trace — errors, cost spikes, novel patterns kept; routine noise dropped. ~85% volume reduction.
All routed through Coral SQL.
The Langfuse spec is an original contribution — first community Langfuse source for Coral. Basic Auth + page-number pagination, 4 tables.
Traces · Generations · Scores
Custom Coral sourceIssues · Events · Errors
Commits · PRs · File diffs
Messages · Threads · Channels
Monitors · Metrics · Events
Incidents · Urgency · Timelines
Issues · Sprints · Assignees
Coral source specs are pluggable. Drop one in, JOIN it with the rest.
Stop diagnosing AI failures one dashboard at a time.
Sentinel runs in your stack, joins the signal your team already produces, and asks before it acts. Built for the Pirates of the Coral-bean hackathon by WeMakeDevs × Coral OSS.