The Hasan Dispatch

The Hasan Dispatch

Archives
Log in
Subscribe
November 10, 2026

RAG in Software Development: Giving AI the Context It Actually Needs

Retrieval-Augmented Generation (RAG) is the most practical fix we have. This post explains what RAG is, why it works, where it fits in the software development lifecycle, and where it quietly fails.

#DNS #WebDev #DevOps #Architecture

What RAG Is, Without the Hype

A large language model only knows what was in its training data, frozen at a point in time. It knows nothing about your private repositories, your internal API docs, last week's incident postmortem, or the library version released after its cutoff. RAG solves this by splitting the job in two:

  1. Retrieve: when a question arrives, search your own knowledge sources for the most relevant pieces.

  2. Generate: place those pieces into the model's prompt, then let the model answer using them. The model's weights never change. You change what it sees at question time. That is the whole idea.

Why RAG Instead of Fine-Tuning or Just a Bigger Context Window?

This is the question worth understanding, because the wrong choice wastes months. Fine-tuning changes the model's weights. It's good at teaching style and behavior, but poor at teaching facts. Facts baked into weights are hard to update, hard to verify, and impossible to cite. When your API changes on Tuesday, you would have to retrain. Large context windows let you paste in more text, but pasting an entire monorepo is slow, expensive, and often counterproductive. Models tend to use information less reliably when it is buried in a very long prompt, and you pay for every token on every request. RAG keeps knowledge outside the model, in an index you control. Updating knowledge means re-indexing a file, not retraining a model. Every answer can point back to the source it used. That combination of freshness, traceability, and low cost is why RAG became the default pattern for grounding AI in private data.

How a RAG Pipeline Works for Code

The basic pipeline has four stages.

  1. Ingestion and chunking

You split your sources (code, docs, tickets, pull request discussions) into chunks. For prose, fixed-size chunks with overlap are a decent start. For code, that approach is a trap: cutting a function in half destroys its meaning. Better code pipelines chunk along syntax boundaries (functions, classes, modules) using a parser such as Tree-sitter, and attach metadata like file path, language, and symbol name.

  1. Embedding

Each chunk is converted into a vector, a list of numbers that represents its meaning. Chunks with similar meaning end up close together in vector space. This is why a query like "where do we validate user sessions?" can find a function named checkAuthToken even though the words don't match.

  1. Retrieval

When a developer asks something, the question is embedded the same way and the system fetches the nearest chunks from a vector database. In practice, pure vector search is not enough for code. Identifiers, error messages, and exact function names are better matched by keyword search (BM25). The strongest systems use hybrid retrieval, combining both, and often add a re-ranking step where a smaller model reorders the top results by true relevance.

  1. Augmented generation

The top chunks are inserted into the prompt along with the question and instructions such as "answer only from the provided context, and cite file paths." The model then produces a grounded answer.

Where RAG Fits in the Development Lifecycle

Codebase Q&A and onboarding. A new engineer joining a large system, say an ERP with many modules and years of history, can ask "how does approval routing work in the purchase module?" and get an answer with links to the relevant files. This shortens onboarding from weeks of reading to days of guided exploration.

Context-aware code generation. Instead of generating generic code, the assistant retrieves your existing components, types, and conventions first. The output matches your patterns because it was shown your patterns.

Documentation assistants. Internal docs are usually scattered and stale. RAG over docs, runbooks, and architecture decision records makes them searchable by meaning rather than by guessing the right keyword.

Debugging and incident response. Feed an error trace in, retrieve similar past incidents, relevant code paths, and the fix that worked last time.

Code review support. Retrieve the team's style guide, related past pull requests, and the modules a change touches, so the reviewer (human or AI) judges the change against real context.

API and library help. Index the current docs of the exact library version you use, which avoids the classic problem of an assistant confidently suggesting a deprecated API.

A Minimal Example

Here is the shape of a RAG flow in pseudocode. The point is the structure, not a specific library.

```for file in repo.files(): for chunk in split_by_symbols(file): //syntax-aware, not fixed-size vector = embed(chunk.text) index.add(vector, chunk.text, meta={"path": file.path, "symbol": chunk.name})

Answering

def answer(question): candidates = hybrid_search(index, question, top_k=30) # vector + keyword best = rerank(question, candidates)[:6] prompt = build_prompt(question, best)

includes file paths

return llm(prompt)```

Three lines matter most: syntax-aware chunking, hybrid search, and re-ranking. Teams that skip them usually blame the model when the real problem is retrieval.

Where RAG Goes Wrong

RAG is not magic. Most failures are retrieval failures, not generation failures.

  1. Bad chunking. Split code mid-function and the model receives half an idea. Garbage in, confident garbage out.
  2. list itemMissing the relationships. Code is a graph: functions call other functions, types are imported across files. Similarity search finds related text but not call chains. Newer approaches add code-graph or dependency information so retrieval can follow imports and references.
  3. Stale indexes. If the index lags behind the repo, the assistant cites code that no longer exists. Re-index on every merge.
  4. Too much context. Retrieving 40 chunks "to be safe" dilutes the signal. More context is not better context.
  5. Irrelevant retrieval treated as truth. If the top result is only loosely related, the model may still build an answer on it. Instruct the model to say "not found in the provided context" when that's the case.
  6. Security and access. If your index mixes repositories with different permissions, a developer might retrieve code they should not see. Enforce access control at retrieval time, not afterward.

How to Evaluate It

You cannot improve what you do not measure. Evaluate the two halves separately:

  1. Retrieval quality: for a set of real questions with known correct files, does the right chunk appear in the top results? (Recall@k and mean reciprocal rank are common measures.)
  2. Answer quality: is the answer faithful to the retrieved context, and does it actually solve the question? Build a small test set from real developer questions. Even 50 good questions will reveal more than any benchmark, because they reflect your codebase.

RAG vs. Agents: Not Either/Or

Modern coding assistants increasingly act as agents: they decide to search the repo, read files, run tests, and loop. That is RAG with a decision-maker on top. The agent chooses what to retrieve, when, and whether to retrieve again. Understanding RAG gives you the foundation to understand agents, because retrieval quality still sets the ceiling on what an agent can do.

Practical Advice for Getting Started

Start small. Index one repo or one docs folder, not everything. Chunk by code structure from day one. Use hybrid search and add a re-ranker before reaching for a bigger model. Return citations (file path and line range) so developers can verify answers quickly. Build an evaluation set before tuning anything. Keep the index fresh with automatic updates on merge. Respect permissions at query time.

Conclusion

The quality of AI in software development depends less on how clever the model is and more on how well it is given the right context at the right moment. RAG is how you do that: it turns a general-purpose model into one that understands your code, your conventions, and your current reality, while keeping the knowledge in a place you can update, audit, and secure. The teams getting the most value are not the ones with the biggest models. They are the ones who treat retrieval as an engineering problem worth getting right.

Don't miss what's next. Subscribe to The Hasan Dispatch:
Share this email:
Share on LinkedIn
GitHub
🙃
LinkedIn
Powered by Buttondown, the easiest way to start and grow your newsletter.