BLACKBOX

what they built

Black Box makes hallucinations in AI coding-agent workflows visible and inspectable. Instead of evaluating only an agent’s final answer, it checks individual steps as events arrive: invented issue references, nonexistent packages or functions, unsupported implementation summaries, and claims that tests passed when the recorded results show failures. The central idea is step-level traceability. Black Box connects actions, claims, and evidence in an interactive graph, letting developers inspect where a suspicious statement appeared and what earlier observations support—or contradict—it. The graph explicitly distinguishes evidence support from event order, avoiding the assumption that consecutive steps prove causation. We built the backend with FastAPI, deterministic detection rules, and Laya, an open-source semantic verifier running locally. Score fusion combines findings into an explainable verdict. Each check uses only earlier evidence, preventing later successful results from retroactively justifying an earlier claim. SQLite persists results before Server-Sent Events publish updates to the Vite/React dashboard. Reconnecting clients resume without duplicating steps, and stored verdicts remain available for replay and inspection. Our detailed demo processes 76 recorded steps across four agent roles, covering inventory repairs, payment changes, failed tests, recovery, and release review. It generates 137 graph nodes and 155 connections, navigable through phase filters, search, and neighbourhood views. The real detector caught all nine required planted failures; false positives and a missed claim remain visible and documented. To support development with Devin, we prepared two standalone task briefs: one for the backend detector and another for the replay client and frontend. A shared architecture document establishes interface contracts, integration milestones, and non-overlapping ownership, giving separate Devin sessions concrete tasks and a clear path to integration.