ArchRAG

The problem

Engineering teams collect architecture guidance across documentation, ADRs, standards, and design documents. As that grows, simple questions get harder to answer: which pattern should we use, what are the tradeoffs, and where is that written down?

Keyword search struggles when the person asking doesn't use the same words as the document. ArchRAG tests whether a RAG layer can close that gap while keeping every answer tied to its source.

The knowledge base is deliberately small: the Microsoft Azure Architecture Center's guide to event-driven architecture.

How it works

There are two pipelines. One builds the knowledge base, and the other answers questions.

Ingestion:  Document → Clean → Chunk (512 chars, 50 overlap)
                     → Embed (text-embedding-3-small) → Pinecone

Query:      Question → Embed → Pinecone search (top 5)
                     → GPT-5-mini + retrieved context
                     → Answer + sources + retrieval details

Three behaviors matter more than the pipeline itself:

• It's grounded. The model is told to answer only from the retrieved context.
• It can refuse. Without enough evidence, it's told to say "I don't know based on the available documentation."
• It's inspectable. The UI shows the similarity score, source, and text of each retrieved chunk, so you can tell whether a bad answer came from retrieval or from generation.

What the evaluation showed

I wrote a test set of 15 questions: 10 the document can answer and 5 deliberately out of scope, such as how to configure PostgreSQL.

Both failures were answerable questions. The evidence was in Pinecone, but it ranked outside the top three chunks, so the model never saw it.

The tempting fix would have been a better model or a longer prompt. Inspecting the retrieved chunks pointed to the real cause, and changing that one setting fixed both failures.

A model can't answer from evidence it never receives.

15 of 15 isn't a general accuracy claim. It's a small test set written for this one document, and it shows the method more than the score.

Limits

• One document. It produces about 52 chunks, far simpler than a real documentation estate.
• Vector search only. There's no keyword retrieval and no reranking.
• An uncalibrated similarity threshold. In this run, top scores were 0.51–0.80 for answerable questions and 0.05–0.32 for out-of-scope ones. That gap is encouraging, but 15 questions can't justify a cutoff.
• Sources shown on refusals. The UI can still list sources when the assistant says it doesn't know, which is confusing.

The extension I find most interesting is conflict detection: when an older document recommends synchronous REST and a newer decision recommends asynchronous events, retrieve both, compare their dates, and explain that the guidance changed.

Related writing

• The Hardest Part of Enterprise AI Isn't the Model — It's the Data
• ArchAgent, the agent I built on top of this knowledge base