Does RAG Need A Vector Database? Vectorless RAG Explained For Growing Businesses
RAG Is A Pattern, Not A Database
No. Retrieval-augmented generation (RAG) needs a reliable way to find the right information before the AI model answers, but that does not have to be a vector database. Vectorless RAG finds it another way: keyword search, a database lookup, navigating a document's structure, or simply placing a short document in the prompt.
Vector databases became the default way to build RAG, partly because the original research and most early tutorials used one. For many business projects that is still a sound choice. For others, such as an assistant that answers questions from a handful of long contracts, manuals or financial reports, a vector database can add setup and running cost without making answers better, and can sometimes make them worse, for example when splitting a document into chunks separates a figure from the heading that explains it.
This guide explains what vectorless RAG is, five ways to build RAG without a vector database, where vector search still wins, and how to decide which fits your documents. It is written for business owners, so technical terms are explained as they come up.
What RAG Actually Is, And Why Vector Databases Became The Default
RAG is a pattern with three steps. When someone asks a question, the system first retrieves relevant information from your own content, then adds that information to the prompt, and then asks the model to answer using it, ideally with a reference to where each fact came from. It lets an AI assistant answer from your policies, product data or documents instead of only from what the model learned in training.
The term comes from a 2020 research paper, in which the retrieval step searched "a dense vector index of Wikipedia" using an AI search model (Lewis et al., 2020 (opens in a new tab)). That design shaped how most people learned RAG: split documents into small chunks, turn each chunk into an embedding, store the embeddings in a vector database, and search them for the chunks closest in meaning to the question.
An embedding is a list of numbers that represents the meaning of a piece of text, so that passages about similar things end up close together even when they use different words. A vector database stores those lists and finds the nearest ones quickly. That is useful when people describe the same thing in many different ways. But the pattern itself only asks for retrieval. How you retrieve is a design choice.
RAG Vs. Vector Database: Not The Same Thing
A lot of confusion comes from treating the two terms as one. RAG is the overall approach of retrieving information and giving it to a model. A vector database is one tool that can do the retrieving. You can build RAG without a vector database, and you can use a vector database for things that have nothing to do with RAG, such as product recommendations or finding duplicate records.
| RAG | Vector database | |
|---|---|---|
| What it is | A pattern for answering questions from your own information | A type of database that stores embeddings and finds similar ones |
| Its job | Find the right information, then have the model answer from it | Return the stored items closest in meaning to a query |
| Needs the other? | No. Retrieval can use keyword search, a database, document structure or the full text | No. It is also used for recommendations, deduplication and image search |
| What decides quality | Whether the right information is found and the answer stays faithful to it | Embedding quality, how content was split into chunks, and filters |
Fig. 2: The RAG vs. vector database distinction in one table.
Seen this way, the useful question is not which vector database to buy. It is how the system should find the information your users actually ask about. Sometimes the answer is a vector database. Often, at least to start with, it is something simpler.
What Is Vectorless RAG?
Vectorless RAG is retrieval-augmented generation that finds information without embeddings or a vector database. Instead of searching for text that is similar in meaning, it retrieves by exact words, by querying structured data, by having the model navigate a document's sections the way a person uses a table of contents, or by giving the model the whole document.
The term is used in two ways. In the broad sense, any RAG system without a vector search step is vectorless, including ones built on ordinary keyword search or a database query. In the narrower sense that has become popular more recently, vectorless RAG means reasoning-based retrieval over a document's structure, an approach associated with the open-source PageIndex project, which describes itself as "Vectorless, Reasoning-based RAG" with "No Vector DB, No Chunking" (PageIndex on GitHub (opens in a new tab)). This guide covers both, because a business choosing an approach should compare all the options, not only the newest one.
The case for vectorless RAG rests on a simple point that PageIndex puts as "similarity ≠ relevance." The passage most similar in wording to a question is not always the one that answers it. In a 300-page annual report, for example, the sentence that sounds most like "What was operating margin in 2025?" might sit in the chairman's letter, while the actual figure is in a table several sections later.
Five Ways To Do RAG Without A Vector Database
Each of the methods below can work on its own, and many good systems combine two or three. They are ordered roughly from simplest to most involved.
1. Put The Whole Document In The Prompt
The simplest alternative skips the search step. If the material is small enough, you give the model all of it along with the question. In September 2024, Anthropic, which makes the Claude models, noted that a knowledge base smaller than 200,000 tokens, "about 500 pages of material," can go straight into the prompt with no need for RAG (Anthropic, Introducing Contextual Retrieval (opens in a new tab)). A token is a small piece of text, roughly three quarters of an English word on average. Limits vary by model and have grown since then. Strictly, this skips retrieval altogether, but for small amounts of material it is often the best alternative to a vector database.
The trade-offs are cost and attention. Sending hundreds of pages with every question costs more per question than sending a few relevant passages, although prompt caching, where the provider reuses the document between questions, can reduce this considerably when the same document is sent repeatedly within a short window. Research has also found that models use long inputs unevenly: performance is often highest when the relevant information sits at the start or end and "significantly degrades" when it is in the middle (Liu et al., Lost in the Middle (opens in a new tab)). Newer models handle long inputs better, but it is worth testing rather than assuming.
This works well for a single policy handbook, a product specification or a contract that people ask about repeatedly. It does not scale to a document library.
2. Keyword Search
Keyword search finds passages containing the words in the question. The standard method, called BM25, ranks results by exact word matches while adjusting for document length and for how common each word is (Anthropic (opens in a new tab)). It has been used in search engines for decades and runs on tools many businesses already have, from database full-text search to dedicated search engines.
Keyword search is strong where vector search is weak: part numbers, error codes, product names, clause numbers and anything else where the exact string matters. Its weakness is the reverse. If a customer asks about "getting my money back" and the policy says "refund," a plain keyword search can miss it unless synonyms are handled.
3. Database And API Lookups
Much of what people ask an AI assistant is not in documents at all. "Where is my order?", "How many units are in stock?" and "When does this contract renew?" are answered by records in a database or another system. For these, the right retrieval step is a query to that system, through an API (a standard way for software to request data from another system) or a carefully limited database query, not a search through text.
For questions about records, this is usually the most reliable route, because the answer comes from the system that holds the official, current data rather than from a passage that might be out of date. The work is in access control: the assistant should only see the records the person asking is allowed to see, and generated database queries need tight limits and testing.
4. Reasoning Over A Document's Structure
This is what the term vectorless RAG most often refers to in recent discussion. The document is first turned into a tree that mirrors its structure: chapters, sections and subsections, each with a short summary and its page range. When a question arrives, the model reads the top of the tree, decides which branch is most likely to hold the answer, opens it, and keeps going until it reaches the right section. It then answers from that section and can point to the section and pages it used.
PageIndex describes this as working the way "a human expert turns to and reads the right section of a long report," and says it is designed for long, complex professional documents such as financial reports, legal documents, regulatory filings and technical manuals (PageIndex on GitHub (opens in a new tab)). Because the document is not cut into small chunks, a table stays with its heading and each section is read as a whole rather than in fragments.
How Tree-Based Vectorless RAG Finds An Answer
The headline result for this approach is a reported 98.7% accuracy on FinanceBench, a benchmark of questions about company financial filings, by Mafin 2.5, a system built on PageIndex (Mafin 2.5 evaluation (opens in a new tab)). For context, the researchers who created FinanceBench found in 2023 that GPT-4-Turbo used with a retrieval system "incorrectly answered or refused to answer 81% of questions" (Islam et al., FinanceBench (opens in a new tab)). Those two figures come from different systems, models and years, so they show how hard the task is rather than a like-for-like comparison.
The 98.7% figure is reported by the company behind PageIndex, not by an independent evaluation, and its own write-up notes that the benchmark "primarily focuses on simple retrieval tasks based on a single document." That matters for a business decision. Tree-based retrieval is promising for questions about one long, well-structured document. It is less proven for questions that span hundreds of documents, and each question can take several model calls as the model works through the tree, so answers are usually slower and can cost more per question than a single vector lookup.
Tree navigation is close to what is called agentic search, where the model decides each next retrieval step for itself. SensViz's comparison of agentic search and RAG (opens in a new tab) explains when letting the model choose its own search steps is worth the extra time and cost.
5. Knowledge Graphs
A knowledge graph stores information as things and the relationships between them: this supplier provides these parts, which are used in these products, which went to these customers. Graph-based RAG, such as Microsoft's open-source GraphRAG, builds that graph from your documents and uses it to answer questions that need connections across many sources or a summary of a whole collection (Microsoft GraphRAG (opens in a new tab)). A graph can be queried directly, the way a database is, with no vectors involved, although GraphRAG itself also uses embeddings for some kinds of search.
It is the most involved option here. Microsoft's own documentation warns that "GraphRAG indexing can be an expensive operation" and advises users to "start small." It is worth considering when your most important questions are about relationships and patterns, not when people mostly need a specific fact from a specific document.
Where Vector Search Still Wins
Vectorless RAG is not a replacement for vector search in every case. So does RAG need a vector database? Not always, but sometimes it is the best tool. It earns its place when several of these are true:
The collection is large and varied. Thousands of documents, tickets, emails or articles, where no single structure or table of contents covers everything.
People describe the same thing in many ways. Customers and staff rarely use the exact words in your documents, so meaning-based matching finds what keyword search misses.
Answers need to be fast at volume. A vector lookup typically returns results in a fraction of a second, without the multiple model calls that tree navigation needs.
Large volumes of new content arrive all the time. Where hundreds of new documents appear each week, building a section tree for each one can be slow or costly.
In practice, the best results usually come from combining methods rather than picking one. In Anthropic's testing, adding a short note of context to each chunk before indexing it and combining embeddings with keyword search cut failed retrievals by 49%, and adding a reranking step (a second pass that reorders results by how useful they are) brought the reduction to 67% (Anthropic, Introducing Contextual Retrieval (opens in a new tab)). Those figures come from Anthropic's own test sets, but the direction matches common experience: hybrid retrieval tends to beat any single method.
How Hybrid Retrieval Combines Methods
You may not need a separate product to get vector search. If your business already runs PostgreSQL, the open-source pgvector extension lets you "store your vectors with the rest of your data" in the same database (pgvector on GitHub (opens in a new tab)). Our guide to choosing a vector database for RAG covers how to decide whether that is enough or whether a dedicated vector database is worth it.
Vectorless RAG Vs. Vector RAG: Side By Side
The table below compares tree-based vectorless RAG, the version most comparisons focus on, with standard vector RAG. Keyword search and database lookups sit alongside both and are often used with either.
| Vectorless RAG (tree-based) | Vector RAG | |
|---|---|---|
| Best suited to | A small number of long, well-structured documents | Large, varied collections of shorter content |
| How it finds information | The model reads a table-of-contents tree and chooses where to look | Matches the question's meaning against stored embeddings |
| Setup | Build a tree for each document, using model calls to summarise sections; no vector database | Split content into chunks, create embeddings, run a vector database |
| Speed per question | Slower: several model calls per question | Fast: one lookup, then one answer |
| Cost pattern | Usually higher cost per question; no vector database to run | Usually lower cost per question; ongoing database and indexing cost |
| Explaining an answer | Shows which sections it opened and its stated reason | Shows which chunks matched, but not why they were chosen |
| Weak spot | Questions spanning many documents; documents with poor structure | Exact terms, numbers and context split across chunks |
Fig. 5: Vectorless RAG vs. vector RAG. Costs and speeds depend on the model, the documents and the volume of questions.
Neither column is better in general. The right choice depends on what your documents look like and what people ask about them, which is why the decision below starts with the questions, not the technology.
How To Decide Which Retrieval Approach You Need
Start with the questions people will actually ask, then look at where the answers live. Most business assistants end up with more than one route, for example a database lookup for order status and document retrieval for policy questions. The flow below is a starting point for each type of question.
Which Retrieval Approach Fits The Question?
Three checks apply whichever route you choose. First, permissions: the assistant must only retrieve information the person asking is allowed to see, which is easier to enforce in a database than in a pile of loose documents. Second, freshness: if the source changes, the retrieval step needs to change with it, whether that means rebuilding a document tree or updating a search index. Third, data handling: sending whole documents to a model provider means checking what that provider stores and for how long.
Common Mistakes When Choosing A RAG Approach
Choosing the database before the questions. Picking a vector database on day one fixes the design before anyone has checked what users ask and where the answers live.
Blaming retrieval for a content problem. Out-of-date, duplicated or contradictory source documents produce poor answers with any retrieval method.
Testing with easy questions. A demo that answers ten friendly questions says little about the questions real users ask, including the vague and badly worded ones.
Taking benchmark scores as a forecast. A published accuracy figure applies to that benchmark's documents and questions, not necessarily to yours.
Ignoring cost per question. An approach that is affordable at 50 questions a day can look very different at 5,000.
Overlooking document quality. Scanned PDFs, missing headings and complex tables hurt every retrieval method, because the text has to be extracted cleanly before anything can search it.
Our guide to building production RAG that reduces hallucinations covers the source content, citation and testing side in more depth.
How To Test Before You Commit
The most reliable way to choose is a short, structured comparison using your own documents.
Collect real questions. Gather a few dozen questions people have actually asked, including awkward ones, and write down the correct answer and where it comes from.
Start with the simplest approach that could work. For a few documents that might be the whole document in the prompt; for records, a database lookup.
Check retrieval separately from the answer. For each question, did the system find the right section or record? A good answer from the wrong source is luck.
Try one alternative on the same questions. For example, tree-based vectorless RAG against vector or hybrid search.
Compare accuracy, time per answer and cost per answer. Estimate cost at the question volume you expect, not just the test volume.
Check permissions and failure cases. Confirm the assistant says it doesn't know when the answer isn't there, rather than guessing.
If the simple approach passes, ship it and keep the test questions. They become the check you rerun when the documents, the model or the volume changes.
An Example Of Choosing A RAG Approach (Illustrative)
This example is hypothetical, to show how the options fit together. It is not a Fossilite client project.
An equipment servicing company with around 60 staff wants an assistant for its field engineers. Engineers ask two kinds of question: about a job ("What was replaced on this unit last visit?") and about equipment ("What is the torque setting for this valve?"). The equipment answers sit in about 40 manufacturer manuals of 200 to 600 pages each, with clear chapters and part numbers.
Job questions: database lookup. The job history already lives in the company's service system, so the assistant queries it directly, limited to the engineer's assigned jobs.
Manual questions: tree-based vectorless RAG plus keyword search. The job record names the equipment model, so the assistant first picks the right manual, then navigates that manual's section tree to the right chapter. Keyword search catches exact part numbers and model codes.
No vector database yet. With 40 well-structured manuals and a reliable way to pick the right one, the team starts without one. It agrees to revisit the choice if the library grows to thousands of documents or engineers start asking broad questions across many manuals.
Before rolling the assistant out, the company tests the manual route against hybrid search on 50 questions collected from senior engineers, and keeps those questions as its ongoing check.
How Fossilite Approaches Retrieval
Fossilite starts from the work, not from a preselected technology. For an AI assistant, that means understanding who asks questions, what they ask, and where the answers actually live before choosing how retrieval should work. A vector database is one possible answer, not the starting assumption.
Engagements follow five stages: understand the work, define what needs to change, design with the business, build and connect, then hand over and improve. In retrieval terms, that means using the systems and data the business already relies on, testing the simplest approach with real questions, and adding complexity only when the tests show it is needed.
Frequently Asked Questions
Common questions from business teams deciding how an AI assistant should find information.
Does RAG Require A Vector Database?
No, RAG does not require a vector database. It requires a way to retrieve relevant information before the model answers, and a vector database is only one option. Keyword search, database or API lookups, navigating a document's structure, and placing a short document directly in the prompt all work, and many reliable systems combine two or more of these methods.
Is Vectorless RAG Better Than Vector RAG?
Neither vectorless nor vector RAG is better in every case. Vectorless RAG that navigates a document's structure tends to suit a small number of long, well-organized documents such as reports, contracts and manuals. Vector RAG tends to suit large, varied collections where people phrase questions in many ways. Testing both on your own questions is the reliable way to choose.
What Is The Difference Between RAG And A Vector Database?
RAG is an approach: find relevant information, give it to an AI model, and have the model answer from it. A vector database is a tool that stores embeddings, numerical representations of meaning, and finds similar ones. A vector database can power the retrieval step in RAG, but RAG can also retrieve information without one.
Is Vectorless RAG Slower Or More Expensive?
Tree-based vectorless RAG is usually slower per question, because the model makes several decisions while navigating a document, and each decision is a model call. It avoids running a vector database, though building each document's tree has a one-off model cost. Whether it costs more overall depends mainly on how many questions you expect.
Can I Use PostgreSQL Instead Of A Vector Database?
Often, yes: PostgreSQL can stand in for a separate vector database. If your business already runs PostgreSQL, the open-source pgvector extension adds vector search to the same database, and its built-in full-text search covers basic keyword search. A dedicated vector database becomes worth considering when volume, speed or filtering needs outgrow what testing shows PostgreSQL can handle.
Will Larger Context Windows Replace RAG?
Larger context windows, the amount of text a model can read at once, will not fully replace RAG. They let small knowledge bases skip the search step. But sending everything costs more per question, models can miss details buried in long inputs, and many businesses hold far more information than any prompt can fit.
When Should A Vectorless RAG System Move To Vector Search?
Consider adding vector or hybrid search when the document collection grows from a handful of long documents to hundreds or thousands, when users ask broad questions that span many documents, or when answers need to come back faster than tree navigation allows. Rerun your test questions before and after, so the change is justified by results.
If you are weighing the vector database route, our guide to choosing a vector database covers what to compare. If you would rather have outside help with the whole decision, see our guide on how to choose an AI consulting partner.