All Posts

How AI Assistants Find the Right Answer

Engineering
Author: William Ngeow
23 February 2026·5 min read

RAG
RAG

So your company's chatbot just told a customer their order shipped yesterday. It didn't. The AI made up a tracking number that doesn't even exist. This is what happens when AI "hallucinates" - it fills gaps in its knowledge with confident-sounding fiction.

There's a technique designed to fix this called RAG (Retrieval-Augmented Generation). But honestly, I think most people reach for it way before they need to.

The Problem: AI Makes Stuff Up

AI models like ChatGPT have three fundamental limitations:

  • They stop learning after training. Whatever happened after their training cutoff, they don't know about. Ask about last week's news and you'll get either silence or fiction.
  • They hallucinate. When they don't know something, they don't say "I don't know." They make something up and present it with full confidence.
  • They can't see your data. Your company's internal documents, your product database, your customer records - the AI has never seen any of it.

How RAG Fixes This

The idea is straightforward: before the AI answers, it searches your data first. Then it uses what it found as a cheat sheet.

RAG Pipeline
RAG Pipeline

Think of it like an open-book exam. Instead of the AI answering from memory (where it might make things up), it gets to look up the answer in your actual documents first. The "retrieval" happens before the "generation" - hence the name.

An open-book exam - the AI gets to look up answers instead of guessing from memory
An open-book exam - the AI gets to look up answers instead of guessing from memory

What Makes It Smart: Meaning-Based Search

Here's what makes RAG interesting. The search isn't just matching keywords - it understands meaning. If a customer asks about "cloud credentials," a RAG system can find your article about Azure certifications, even though the word "credentials" never appears in that article. It understands that those concepts are related.

Keyword Matching vs Semantic Search
Keyword Matching vs Semantic Search

This is different from how a normal search bar works. Regular search matches exact words. RAG's search matches ideas.

When You Actually Need It

RAG is worth it when:

  • You have a large or growing collection of documents
  • People ask unpredictable questions that you can't anticipate
  • Your content changes frequently
  • Getting the answer right really matters (healthcare, finance, legal)
  • You need to show your sources ("here's where I found that")

RAG is overkill when:

  • You have a small, well-defined set of information
  • Questions are predictable and fall into clear categories
  • Your content rarely changes

When I built the chatbot for this portfolio, I didn't use RAG at all. My site has maybe a dozen pages of content. I just set up keyword matching: if someone asks about my skills, the system grabs my skills page. If they ask about my experience, it grabs my work history. That gets me 90% of the way there with zero extra infrastructure.

If your information fits in a cheat sheet, just use a cheat sheet. You don't need a research library.

Cheat sheet vs research library - pick the right tool for the size of your data
Cheat sheet vs research library - pick the right tool for the size of your data

The Trade-offs Nobody Talks About

  • Organizing your data is deceptively hard. You need to break documents into chunks, and the size of those chunks massively affects quality. Too small and you lose context. Too large and irrelevant information dilutes the good stuff. I think this is actually where most RAG projects fall apart - everyone's tuning the AI when they should be thinking about how they organize their documents.
  • Search isn't perfect. Sometimes irrelevant information sneaks in, and sometimes the right answer gets missed. The AI's response is only as good as what the search returns.
  • It's slower and more expensive. Every question now requires a search step and an AI step. That adds latency and cost.
  • It's another system to maintain. The search database needs hosting, monitoring, and keeping in sync with your actual documents. That's real, ongoing work.

The Bottom Line

RAG is a powerful tool, but it's not a starting point. Before building a search-powered AI system, ask yourself: does this actually need to search through a large, changing collection of information? Or can I just give the AI a well-organized cheat sheet?

Key Takeaways

  • RAG = search first, then answer. The AI looks up relevant information in your data before responding, instead of answering from memory.
  • It uses meaning-based search, not just keyword matching. "Cloud credentials" can match an article about Azure certifications.
  • Start simple. If a well-organized set of instructions gets you 90% there, you don't need a search database. I'd always try this first.
  • Organizing your data is the hardest part. How you break up and structure your documents matters more than which AI model you pick.
  • RAG reduces hallucination but doesn't eliminate it. The quality of your search results is the ceiling on your answer quality.

Dig Deeper