RAG, Embeddings, Vector Databases, and GraphRAG: How AI Systems Find and Use Your Data

Large language models are excellent at generating text, but out of the box they do not reliably know your private docs, your latest data, or the reasoning that connects the pieces. The architectures described in this article exist to close that gap.

Retrieval-Augmented Generation (RAG), embeddings, vector databases, and GraphRAG are the four building blocks of modern, grounded AI systems. Understand them and you can see how a chatbot answers in your customer’s product terminology, how a research assistant pulls together themes across hundreds of documents, and where each approach hits its limits.

This is not a vendor pitch. It is a working-level explanation for people who build or buy AI, tradeoffs included.

What is RAG (Retrieval-Augmented Generation)?

RAG is the pattern of pulling relevant evidence from a knowledge source before a language model generates an answer, then feeding that evidence into the model so its output is grounded in what it found.

The term comes from a 2020 paper by Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” The core idea was simple: a generative model is powerful but its knowledge is fixed at training time. A retriever, on the other hand, can look up current or private information. Combine them, and the model answers from retrieved context rather than only from parameters.

Paper: Lewis et al., 2020

The standard RAG pipeline

A typical RAG system runs in five steps:

1. Chunk your source documents into small, meaningful passages.
2. Embed each passage – convert it into a vector that represents its meaning.
3. Store those vectors in a vector database.
4. Retrieve embed the user’s question and find the closest passages.
5. Generate hand the retrieved passages plus the question to the LLM and ask for an answer.

The model never memorizes your knowledge. It reads the retrieved evidence each time. That makes RAG dramatically easier to update than retraining a model. Change a document, re-embed, and the answer follows.

Why RAG is useful

• Access to private or recent data. LLMs do not know your internal procedures, customer contracts, or last week’s incident notes. RAG gives them access at query time.
• Grounding reduces hallucination. An answer built from retrieved evidence is easier to trace and more verifiable than one the model invents.
• Citations. Because each passage is a real document segment, the answer can point back to the source.
• Composable. RAG can be layered with rules, filters, and tools; it is a building block, not a finished product.

RAG’s limits

RAG is only as good as its retrieval. If the right passage is chunked badly, indexed with the wrong embedding model, or filtered out by a tenant rule you forgot about, the model has no clean material to work with. It either gives up or fills the gap with a guess.

RAG is also weak at questions that require synthesizing information distributed across many documents. That is precisely the problem GraphRAG targets.

What are embeddings?

An embedding is a numerical representation of a piece of data, most often a sentence or a document, as a list of real numbers, called a vector. An embedding model assigns values so that the resulting vector encodes the meaning of the text. Vectors with similar meaning end up geometrically close to each other in the space.

IBM’s definition: embeddings “represent objects like text, images and audio as points in a continuous vector space where the locations of those points in space are semantically meaningful to machine learning algorithms.”

IBM: What is embedding?

Why embeddings exist

Machine-learning systems can do arithmetic quickly. They cannot reason about words directly. Embeddings solve that by turning meaning into numbers: similar concepts become vectors near each other, so “semantically similar” becomes “near in vector space,” which computers can compute with cosine similarity, dot products, or Euclidean distance.

Google’s ML Crash Course captures why one-hot encoding is not enough: it produces high-dimensional sparse vectors where no two different words are meaningfully related. Embeddings fix that by giving each concept a dense, low-dimensional vector that does carry relationships.

Google ML Crash Course: Embeddings

A concrete example

In a simple embedding model, “dog,” “wolf,” and “bison” might land nearby in vector space, all mammals. “Apple” and “banana” cluster somewhere else, on the fruit side. That layout is learned, not hand-coded.

Once “kitten” is embedded, a search for it returns vectors for “cat,” “kitty,” “feline,” “pet”, all near. Vector DBs use this geometry to answer “what documents are about this?” without ever requiring the exact word match.

Common similarity metrics

• Cosine similarity: angle between two vectors; the most common choice for text embeddings
• Dot product: useful when vectors are normalized; equivalent to cosine in that case
• Euclidean distance: straight-line distance; sensitive to vector length
• Hamming distance: bit differences; relevant for binary representations

Different metric choices are valid for different models. The practical rule: follow the embedding model provider’s recommendation, then validate on your own queries.

Weaviate: What is a Vector Database?

Embedding models worth knowing

• Word2Vec (Google, 2013) – the classic word-embedding method; each word becomes a fixed-size vector
• GloVe (Stanford, 2014) – word embeddings learned from term co-occurrence counts
• Sentence-BERT (Reimers & Gurevych, 2019) – BERT-based sentence embeddings
• SimCSE (Gao et al., 2021) – contrastive learning to improve sentence embeddings
• text-embedding-ada-002 (OpenAI) – 1536-dimensional sentence embedding
• text-embedding-3-small / -large (OpenAI, 2024) – 1536 and 3072 dimensions, improved retrieval

Different models produce different vector spaces, and you cannot mix them without care. Use one embedding consistently across your content and your queries, or the geometry you trained your vector DB around no longer holds.

What is a vector database?

A vector database is a data system optimized to store, index, and query high-dimensional vectors. Its core operation is: “given this vector, find the closest ones.” Everything else (persistence, metadata, filters, access control, multi-tenancy) is added on top of that.

A production vector database typically provides:

• Vector search: similarity query by nearest neighbor
• Hybrid search: combining vector search with keyword (BM25) search
• Metadata filtering: “only return documents from customer X, version Y, after date Z”
• Persistence, replication, backups: production operations
• Access control: RBAC, authentication, tenant isolation

Weaviate: What is a Vector Database?

Vector DBs vs SQL vs NoSQL

The difference is not about how much data you can store. It is about the query you are optimizing for.

	SQL (relational)	NoSQL (document / key-value / graph)	Vector DB
Data model	Rows and columns in tables	Documents, key-value pairs, or graphs	High-dimensional vectors
Typical query	Match, join, aggregate	Key lookup, flexible document query	Nearest neighbor to a vector
Schema	Predefined and fixed	Dynamic, per-document	None (vectors are just floats)
Scaling	Vertical (typically)	Horizontal (native)	Horizontal, via ANN indexing
Best for	Transactions, reporting, structured data	Flexible schemas, real-time apps	Semantic search, RAG, recommendations

Traditional SQL databases were built to answer “where is the row with key = X and value = Y?” NoSQL databases answered “where is the document matching this flexible shape?” Vector databases answer a fundamentally different question: “which of these hundreds of thousands of documents is closest to this one in meaning?”

IBM: SQL vs. NoSQL Databases

Oracle: What is NoSQL?

Red Gate: How to choose between SQL and NoSQL

That “nearest neighbor in meaning” query is expensive at scale. Brute-force comparison against every stored vector grows linearly with the corpus size and is not viable for millions of documents. Vector databases solve it with approximate nearest neighbor (ANN) indexes:

• HNSW (Hierarchical Navigable Small World) – a graph index; the most common default
• IVF (Inverted File Index) – partitions vectors into clusters
  • SCANN / PQ – quantization-based compression
• LSH (Locality-Sensitive Hashing) – hash-based partitioning

The trade is precision for speed. A well-tuned ANN index can find the top-5 most similar vectors in a few milliseconds, rather than a few seconds with brute force – at a small, usually acceptable, recall cost.

Do you need a separate vector database?

Not always. Many modern SQL databases (PostgreSQL with pgvector, SQL Server with vector types), NoSQL systems (MongoDB Atlas, Elasticsearch, OpenSearch), and purpose-built vector databases (Pinecone, Weaviate, Qdrant, Milvus) can run similarity search.

The practical question: is similarity search central to your workload, or incidental?

• If a small, static dataset is your only vector workload, an in-memory index or an extension to your existing database is often enough.
• If vector search is the primary product – chatbot over a large corpus, multimodal search, hybrid keyword + semantic retrieval – a purpose-built vector database usually makes sense. The boundaries between categories are blurring, so choose based on the actual workload and the operations you need, not on the product name.

What is GraphRAG?

GraphRAG is an approach to RAG that builds a knowledge graph from your documents before query time. Where a standard RAG index stores a flat list of embeddings, a GraphRAG index stores entities and the relationships between them (a person, a product, an incident, a date, a policy clause, and how they connect) grouped into communities.

The academic version comes from a 2024 Microsoft Research paper: Edge and colleagues, “From Local to Global: A Graph RAG Approach to Query-Focused Summarization.”

Paper: Edge et al., 2024

Microsoft also ships an open-source implementation: the graphrag Python package. It was released in July 2024 and is now in maintenance mode.

How the GraphRAG index is built

1. Chunk the corpus into passages.
2. Extract entities from each passage (using an LLM).
3. Extract relationships between those entities.
4. Build the graph – nodes are entities, edges are relationships.
5. Detect communities – cluster the graph (Leiden or similar).
6. Summarize communities – each community gets a written summary of what it is about.
7. Store everything.

This indexing phase is the expensive part. Every chunk is an LLM call to extract entities, every edge is an LLM call to describe the relationship, every community is an LLM call to summarize it. A corpus of 10,000 pages can require tens of thousands to hundreds of thousands of tokens of LLM output just to build the index.

How GraphRAG answers questions

GraphRAG distinguishes two query modes:

Local search – “What does the corpus say about X?” The system identifies the entities most relevant to X, retrieves the entity subgraph plus related community summaries, and asks the LLM to compose an answer. It is similar to standard RAG but uses the graph for better neighbor discovery.

Global search – “What are the main themes in this corpus?” The system iterates over every community summary, generates a partial answer per community (a “map” step), and then combines those partial answers into a final answer (a “reduce” step). This is the capability that sets GraphRAG apart: it can answer questions that require understanding the corpus as a whole, rather than finding one best-matching passage.

RAG vs GraphRAG, side by side

	RAG	GraphRAG
Index	Vectors of chunks	Knowledge graph + community summaries
Strength	Specific lookups, factual recall	Corpus-wide themes, cross-document reasoning
Multi-hop questions	Hard (chunk-level)	Strong (graph traversal)
Global question (“what themes?”)	Weak	Strong (“global search”)
Indexing cost	Low	High (many LLM calls)
Query cost	Low	Higher for global searches
Setup complexity	Simple	Significant
Typical use cases	Product Q&A, support, document search	Research summarization, corpus-wide analysis, audit reports

Where GraphRAG really shines

GraphRAG is not a faster RAG. It is a different RAG. It is well suited to:

• Corpus summarization : “what are the dominant themes across these 5,000 support tickets?”
• Audit and compliance reports : “which policy clauses are contradicted by recent procedure changes?”
• Cross-document reasoning : questions whose answer requires information distributed across many sources
• Long-form analysis : reports, due diligence, thematic analysis

It is less compelling, or simply overkill, when your use case is “give me the closest passage to this question.” In that case, well-tuned RAG with a good embedding model and solid chunking strategy is simpler, cheaper, and usually as good.

What about real production systems?

In practice, teams use these tools together rather than as an either-or:

• A vector DB as the primary retrieval layer for fast, semantic nearest-neighbor search over a large corpus
• Hybrid search (vector + keyword) for cases where the exact wording matters – error codes, product names, citations
• Metadata filters in the vector DB to scope search to a tenant, a version, a date range
• A GraphRAG layer on top for the questions that require synthesizing across the whole corpus
• A reranker (often LLM-based) that takes the top-50 candidates from vector retrieval and reorders them before handing them to the LLM

Production systems in regulated industries – healthcare, financial services, infrastructure – often combine several of these layers, because the accuracy bar there is higher than for a generic chatbot.

A practical rule of thumb

When scoping a project, ask:

1. Is my primary workload “find the closest passage?” – RAG with a vector DB.
2. Do I also need “summarize the whole corpus?” – Add GraphRAG for those specific questions.
3. Do I need exact keyword matches (codes, names)? – Layer in hybrid search or keyword search.
4. Do I need permission / tenant scoping? – Choose a vector DB with first-class metadata filtering and RBAC; or layer it with your identity system.
5. How often does the corpus change? – RAG is cheap to update (re-embed changed chunks); GraphRAG index rebuild is expensive, so plan accordingly.

None of these are “better” than the others. They solve different retrieval problems, and the production systems that work well use more than one.

Where to go from here

If you are starting from scratch:

• Read the original RAG paper. It is short, and it frames the problem clearly: https://arxiv.org/abs/2005.11401
• Read the GraphRAG paper. It explains the indexing pipeline and local/global search modes: https://arxiv.org/abs/2404.16130
• Try the graphrag package. It is open-source and ships an interactive app: https://github.com/microsoft/graphrag
• Pick a vector DB and run a small corpus through it. Weaviate, Qdrant, pgvector, Pinecone, or Chroma are common starting points.
• Measure retrieval quality on your own data before optimizing for speed. Recall, latency, and cost are all tunable; relevance on real queries is what ultimately matters.

The details matter. The shape of the problem: retrieval followed by generation, similarity in a learned vector space, and graph-based synthesis across a corpus – is a stable core. Once you understand that shape, the rest of the tooling is details.

References

1. Lewis, P. et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” arXiv:2005.11401
2. Edge, D. et al. (2024). “From Local to Global: A Graph RAG Approach to Query-Focused Summarization.” arXiv:2404.16130
3. Microsoft. graphrag GitHub repository
4. IBM (n.d.) “What is embedding?” IBM Think
5. Google for Developers. “Embeddings.” ML Crash Course
6. IBM (n.d.) “SQL vs. NoSQL Databases: What’s the Difference?” IBM Think
7. Oracle (n.d.) “What is NoSQL?” Oracle
8. Red Gate Simple Talk (2021). “How to choose between SQL and NoSQL databases.” Red Gate
9. Weaviate (2023, updated 2026). “What is a Vector Database? Similarity Search & Semantic Search for AI.” Weaviate Blog
10. Reimers, N. & Gurevych, I. (2019). “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” arXiv:1908.10084
11. Gao, L. et al. (2021). “SimCSE: Simple Contrastive Learning of Sentence Embeddings.” arXiv:2104.08821
12. Menlo Ventures (2024). Enterprise AI report (RAG appears in 51% of production workloads, up from 31% the prior year). Cited in Weaviate: What is a Vector Database?

At Jimatech we are your partner in Data and AI