Notaknowledgebase.A Cognitive OS.
A three-layer architecture: a data layer that keeps the material clean, a search engine that finds it all and answers with evidence, and a feedback loop that makes it more accurate the more it's used.
Three layers, one loop
These layers are not one-way. Feedback flows back into the data layer — a closed loop, and the biggest difference between this and a system you build once and never touch again.
Defines the knowledge and storage structure so retrieval can be accurate — where data lives, how it's classified, processed, and stored.
The brain. Its core loop is Collect → Act → Verify: it searches on its own, and it verifies its own evidence before answering.
Collects real usage signals — ratings, conflicts, evaluations — and feeds them back into the data layer.
Good retrieval starts with good data
Tens of terabytes across legacy Notes archives, SharePoint, file servers, CMS, and engineering & records systems. The point is not how big it is — the point is how the engine can pinpoint things inside it.
Dump everything into one pool and every search digs through the whole warehouse. So we split it: top-level KBs by business area, sub-KBs by topic — and every KB states a name + description, written for the engine. A question comes in, the engine reads the descriptions, picks the right KB, and searches only there. Like a library: read the section signs first, then walk to the right shelf.
Every piece of data goes through the same line: define the type → tag it → clean it → decide how to store it. Tagging is the key step— department, project, year, status, access rights. With good tags the engine narrows the search automatically, and access control runs on the same tags: no permission means the file doesn’t even show up.
We don’t dump raw full text in. Each file type gets the structure that fits it — so a question about one clause returns that clause, not everything that sounds similar. Cleaning handles duplicates, format issues, version relationships, and removes outdated content.
Structured JSON — sender, recipient, subject, body, attachments as separate fields. Find a mail by person or topic, and locate its attachments.
Clause-level JSON, layered document → chapter → clause. Ask about eligibility criteria and get clauses from that chapter only — nothing similar-sounding leaks in.
Relational records — queryable rows and columns, not flattened text.
Vector JSON — geometry and annotations preserved for spatial questions.
Markdown with position info — every passage knows exactly where it came from.
Six tools, one toolbox
On top of the clean data sit six standard retrieval capabilities. The engine above picks whatever tool the question needs.
Vector + BM25 + knowledge graph, with metadata and permission filtering built in.
Structured questions answered from live business tables.
Character-exact lookup across the whole corpus.
Pull answers from your existing systems in real time.
Standard tool protocol — new capabilities plug in without rework.
Whitelisted external sources only — the open web on your terms.
Layer 2: the engine — Collect, Act, Verify
The first stages find more — that's recall. The gate and the filtering choose better — that's precision. Every answer is traceable, and the system would rather say 'not found' than invent one.
Understand & Rewrite
The query goes through a glossary first — casual phrasing, jargon, abbreviations rewritten into something the engine can use. If the question is unclear, it asks the user. It doesn't guess.
Intent detection + term normalization; clarifying questions instead of low-confidence guesses.
Plan the Retrieval
The engine picks the best target KBs by intent — reading the names and descriptions the data layer defined. It never broadcasts the question to every KB.
Like a library: read the section signs first, walk to the right shelf. KB descriptions are written for the engine.
Isolate at the Entrance
Before the KB is even entered, ACL and metadata lock the scope down. Content the user can't see never enters the candidate pool — security is isolation at the entrance, not filtering the answer at the end.
Permission tags + metadata (year, region, status) narrow the search space before retrieval starts.
Hybrid Recall
Three paths in parallel: BM25 for exact matches, vectors for meaning, the graph for relationships. Results merge and dedupe by chunk ID — say 30 + 50 with 10 overlapping = 70 unique candidates.
Each path owns one job the other two can't do; each blind spot is covered by another. Recall: find everything, miss nothing.
Rerank + Dynamic Top-K
Candidates are rescored against the question one by one; then the scores decide the cutoff. 0.92, 0.88, 0.85, then a cliff to 0.41? The line is drawn at the cliff — above it goes to the LLM, below is thrown away.
K is not hard-coded: a simple question keeps 3, a complex one keeps 8. Irrelevant context is noise that breeds hallucination.
Sufficiency Gate → Generate
Evidence is checked before any answer: conflicts resolved toward the fresher, more authoritative version; expiry, permission, completeness verified. Enough evidence → generate with citations. Not enough → re-route, re-retrieve, or pull the full text and go again.
Max-rounds and token caps prevent endless loops. Dead end → “I can't find it.” It never makes things up.
Three paths, zero gaps
Each path is the best at one thing — and blind at another. Use any single one and you lose a whole class of questions. Run all three in parallel — no gaps.
| Retrieval path | Its value | Its blind spot | Covered by |
|---|---|---|---|
| BM25 keyword | Never misses an exact match — invoice numbers, clause numbers, legal terms. | Change the wording and it's lost: ask “when does the customer pay”, the document says “payment terms” — no match. | Vector search catches the rephrasing. |
| Vector semantic | Understands meaning, not words — casual, formal, or another language; same idea, found. | Unreliable on precise items: INV-12345 and INV-12346 look almost identical to a vector model. | BM25 nails exact IDs; the graph separates confusable entities. |
| Knowledge graph | Follows relationships — “the partner's IT staff” lives in connections, not in any single passage. Multi-hop: partner → IT staff → operations → outsourced. | Only does relationships — regular content questions aren't its job. | BM25 and vector handle the content. |
The Sufficiency Gate
Evidence is checked before we answer — not after.
Two documents disagree? Trust the fresher, more authoritative version — and log the conflict for governance.
Is it expired? Does this user have permission? Is it complete? Fails here never reach the answer.
Every piece of evidence keeps its origin and link — ready for citation in the final answer.
Two exits. Evidence is good — generate the answer, with citations. Not enough — loop: change direction, switch tools, pull the full text, try again (with max-rounds and token caps, so it never loops forever). And if it’s truly a dead end — the system says “I can’t find it.” It never makes things up.
Cited answers, live graph
Ask in plain language — get the answer, the sources it came from, and the knowledge graph behind it.
Search is a capability. Many scenarios are workflows
Not everything is Q&A. The AI Workflow Layer builds flows from your own SOPs — calling the LLM, intent detection, and knowledge retrieval at exactly the right steps — to deliver the process the business needs, not just an answer box.
In one national-scale deployment we mapped 33 user stories — every single one:
The more it’s used, the more accurate it gets
Three feedback streams — two active, one passive — flow back into data quality, labels, knowledge structure, and retrieval strategy. This system is not static.
Likes and ratings flow straight back into data quality and labels. What gets praised and what gets thumbs-down becomes governance input.
During retrieval the system catches documents that contradict each other — an old and a new policy version — flags the fragments, and hands them to a human to confirm which to trust. Users do nothing.
Real user questions re-run regularly against KB v1.0 vs the improved v2.0. Results prove whether each change actually helped — and keep tuning the retrieval strategy.
From process-driven to cognition-driven
One deployment: 40TB across legacy archives, SharePoint, file servers, CMS, and records systems.
Specialists stop being retrievers and drafters — they become decision makers.
Every response is grounded in retrievable evidence with source links.
The Sufficiency Gate would rather say 'not found' than guess — grounding is enforced.
Bring your messiest knowledge problem
Policies in five systems, answers in none? We'll show trusted, cited answers on your own documents.