← WRITING

Inside LightBI’s 5.6 MB Micro Brain

No embedding API and no runtime LLM call. MB combines BM25, TF-IDF/LSA and RRF, then hands retrieval evidence back to the governed pipeline.

Written byTDThái Duy
Published on19/09/2026
Updated19/09/2026
Reading time2 min
LightBI public Deep BA and evidence-governed analysis surface.

Micro Brain sounds like a tiny model. V1 is closer to a deterministic semantic retrieval library.

It runs locally, requires no embedding API, and identical source plus compiler settings must reproduce the same logical index.

248 cards become 1,118 retrieval units

The current source contains 248 cards: 225 canonical bridges, 23 open concepts, 25 guarded formulas, 159 relations and 56 confusion pairs.

The compiler splits cards into definition, terminology, positive, negative, relation and formula/evidence units. The current index contains 1,118 units and 2,048 sparse features. The compiled JSON is about 5.63 MB and roughly 1.73 MB gzipped.

Each query is retrieved twice

BM25 preserves exact business terminology. A TF-IDF/LSA projection provides a deterministic dense view for semantic interpolation.

RRF merges ranks, not confidence

BM25 and cosine similarity are not pretending to share one confidence scale. Reciprocal Rank Fusion only decides which concepts should be inspected first.

RRF = 1 / (k0 + sparse_rank)
    + 1 / (k0 + dense_rank)

Retrieval similarity remains provenance, never semantic authority.

The query is not allowed to know the answer

File names, corpus IDs, expected answers and hidden customer dictionaries are forbidden from the query signature. Bounded representative values remain evidence witnesses, not full-column truth.

Determinism is part of the product

The compiler records corpus, tokenizer, vector algorithm, dimensions and hashes. Reproducibility matters more here than mysterious “smartness.”

The value of this 5.6 MB brain is not that it can decide everything. It is that it can recall more while still handing authority back to the governed pipeline.

DISCUSSION

Join the discussion

Your Google account provides your name and avatar.
0APPROVED COMMENTS

No approved comments yet.