What the LightBI Beta Gate Actually Revealed
A development journal from the LightBI MVP proof gate: strong mapping precision, incomplete recall, and why a release gate must record failure instead of hiding it.
A beta gate is not a celebration dashboard
One of the most useful LightBI development milestones was also one of the least flattering. The MVP proof gate evaluated the canonical path across a frozen corpus instead of tuning the product while the measurement was running.
The evaluation covered physical profiling, semantic resolution, grain resolution, domain activation, metric preflight, question and action generation, runtime preflight, query planning, local DuckDB execution, and evidence production.
The numbers were mixed
- Confirmed and high-confidence mapping precision: 99.02%.
- Held-out core MVP signal recall: 59.09%, below the 90% gate.
- Domain activation precision: 100%.
- Runnable-action precision: 100%.
- Advertised-action execution success: 66.67%, below the 90% gate.
- Verified metric correctness: 50% (6 of 12).
- False executable actions: 0.
- Blocked explanation completeness: 100% (356 of 356).
These figures do not describe a simple “works” or “does not work” state. They show that the system can be conservative and precise in several places while still missing too many valid signals and producing too few verified metrics to pass the release gate.
Precision is not coverage
A 100% domain activation precision result sounds excellent until it is read alongside the coverage result. The engine avoided false domain claims, but it activated only 15 of 25 in-scope cases. That is a deliberate distinction: refusing to make an unsupported claim is good governance, but it is not the same as understanding enough of the data.
The same lesson appeared in semantic mapping. High-confidence selections were usually precise, while held-out recall remained incomplete. A system can be careful and still not be useful enough yet.
Execution must be measured against advertised actions
The gate also prevented a convenient denominator. Fourteen runtime-allowed actions executed in Node DuckDB, but seven additional generated candidates failed governed runtime planning. Since those candidates were advertised, the correct execution denominator was 21 rather than 14.
This is an important product rule: the system is evaluated on what it offers to users, not only on the subset that happens to reach the executor.
What the gate allowed us to say
Revenue grouped by product matched the verified May and June full-file totals. At the same time, revenue-over-time did not match full-file truth in six comparisons, and inventory, delivery/operations, and conditional finance did not obtain executable verified comparisons in that gate.
The honest conclusion was that the release gate did not pass. The blockers were signal recall, advertised-action execution coverage, metric correctness, and reproducibility of the exact candidate from a clean checkout. The report recorded those defects without turning them into marketing language.
Why this is part of the product design
LightBI is intended for operational data, so acceptance cannot stop at a green unit-test suite or a good-looking screenshot. The evidence chain must reach runtime behavior, real data, visible results, and machine-readable artifacts.
This is still a Public Beta. The work after the gate is not about hiding the weak numbers. It is about improving coverage while preserving the zero-false-action and fail-closed properties that make the results worth trusting.
LightBI can be evaluated at lightbi.thaiduy.digital/app and followed in the project repository.
Technical references
The evaluation method and release thresholds are recorded in the Phase 7 MVP proof and release gate, the MVP decision record, and the real-data quality gate.