AI Guides · 8 min read

What Is a Vector Database? How It Stores Embeddings for Search and RAG

A vector database stores embeddings and finds similar items with approximate nearest neighbor search. How it differs from keyword search and supports RAG.

Glowing points form clusters in a dark vector space, with thin lines linking a query spark to its nearest neighboring points, and no text or logos

A vector database stores high-dimensional embedding vectors alongside metadata and retrieves the items closest to a query vector using similarity metrics such as cosine similarity, inner product, or Euclidean distance. Unlike relational lookups that match exact keys, vector databases power semantic search and retrieval-augmented generation (RAG) over text, images, and other unstructured data.

What is a vector database?

A vector database is a database, or a specialized vector index inside a broader data platform, designed to store embedding vectors and retrieve the items closest to a query vector. Distance or closeness between two vectors is measured mathematically using cosine similarity, inner product (dot product), or Euclidean (L2) distance.

Traditional database queries look for exact attributes, such as returning the row where an ID equals 55 or filtering for rows where status equals open. A vector query asks a different question: given a numeric representation of a user prompt, document passage, or image, return the stored items that are closest in meaning or content.

In practice, this capability can live inside a standalone specialized vector database or inside a general-purpose relational database or search engine that has added vector indexing. Regardless of packaging, the core job stays the same: index high-dimensional vectors, support metadata constraints, and return nearest neighbors fast enough for interactive applications.

Relational indexes such as B-trees and hash indexes struggle with high-dimensional similarity search because they are engineered for exact key lookups, scalar ranges, and table joins rather than distance calculations across hundreds or thousands of dimensions.

As the number of dimensions grows, nearest-neighbor search runs into the curse of dimensionality. In high-dimensional spaces, pairwise distances between points tend to concentrate, which reduces the effectiveness of simple spatial partitioning. Running an exact k-nearest-neighbor (kNN) scan requires comparing a query vector against every stored vector, which becomes slow and compute-heavy once a collection reaches millions of embeddings.

That limitation reflects the access pattern rather than an inability of relational systems to hold arrays of numbers. Several general-purpose databases and search engines now support vector indexes alongside standard tables so developers can run similarity search next to transactional data. Even in those systems, similarity queries still rely on specialized vector index structures instead of standard B-trees.

How do embeddings, metadata, and vector indexes work together?

Embeddings, metadata, and vector indexes work together by combining numeric representations of unstructured content with structured attributes that govern filtering and access. An embedding model translates text, audio, or images into fixed-length numeric vectors so that semantically related items sit near one another in vector space, a process covered in detail in our guide to embeddings.

A vector database stores each embedding vector alongside useful metadata, which typically includes:

  • A document or record identifier linking back to the source file
  • Timestamps, categories, or document types for freshness and scoping
  • Tenant identifiers and access-control attributes for multi-user applications
  • The text chunk itself, or a pointer to fetch that text from primary storage

At query time, applications often need filtered search, such as finding the closest chunks for one specific customer within policies updated after a given date. Vector indexes handle metadata filters before, during, or after nearest-neighbor traversal, and each strategy carries tradeoffs between retrieval recall and query latency.

What is approximate nearest neighbor search, and how does HNSW work?

Approximate nearest neighbor (ANN) search accelerates vector retrieval by accepting a small chance of missing a true nearest neighbor in exchange for much faster query execution. Because exact kNN scans scale poorly on large high-dimensional datasets, production vector databases rely on ANN indexes to stay responsive.

HNSW and graph-based indexing

Hierarchical Navigable Small World (HNSW) graphs, introduced by Yu. A. Malkov and D. A. Yashunin in 2018 (arXiv:1603.09320), are one of the most common ANN index designs. HNSW organizes stored vectors into a multi-layer neighbor graph.

The upper layers of an HNSW graph are sparse and contain long-range links across the vector space, while the bottom layer contains all vectors with denser local connections. When a query arrives, search begins on a sparse upper layer to traverse broad regions quickly, then steps down through progressively denser layers, following neighbor links closer to the query vector until it settles on the nearest candidates.

Other ANN index families

Graph indexes are one branch of ANN design. Other widely used families documented in libraries such as Faiss take different approaches:

  • Inverted-file (IVF) clustering: Vectors are grouped into clusters, often using k-means. At query time, the index inspects only the clusters nearest to the query vector instead of scanning the entire collection.
  • Quantization, including product quantization (PQ): Vectors are compressed into compact codes so larger collections fit in memory, trading some distance accuracy for lower RAM usage.
  • Locality-sensitive hashing (LSH): Classical hashing functions map similar vectors into matching buckets with high probability so queries only examine items in nearby buckets.

No single ANN algorithm fits every workload. Choosing an index depends on recall requirements, available memory, how frequently records are inserted or deleted, and how heavily queries filter on metadata.

Vector search and keyword search rely on different matching signals, while hybrid search combines both methods to capture semantic meaning alongside exact terms.

Keyword or lexical search uses inverted indexes and term-weighting formulas such as BM25 to rank documents by shared words and term statistics. It performs reliably when a query includes exact tokens like error codes, product SKUs, legal article numbers, or proper names. Vector search compares dense embeddings, allowing it to retrieve passages that discuss the same concept in completely different phrasing. However, dense embeddings can overlook rare exact identifiers if the model does not separate those tokens cleanly.

Hybrid search runs both lexical and vector retrieval on the same query, merges or reranks the resulting candidate lists, and applies metadata filters.

Retrieval method Primary matching signal Key strengths Main limitations Common role in RAG
Keyword search Exact terms and term frequency via inverted indexes Strong on SKUs, error codes, names, and exact phrases Misses synonyms and paraphrased concepts Catching exact identifiers and technical codes
Vector search Distance between embedding vectors Finds conceptually related passages with different wording Can miss rare exact tokens; sensitive to chunking quality Retrieving semantic context for natural-language queries
Hybrid search Combined lexical and vector scores with metadata filters Balances exact precision with semantic recall Adds pipeline steps for score fusion or reranking Default retrieval pattern for varied user queries

How vector databases fit into RAG and where they stop

A vector database serves as the retrieval layer in retrieval-augmented generation (RAG), supplying relevant context chunks to a language model at inference time. Rather than relying solely on static knowledge baked into model weights, a RAG pipeline pulls external facts dynamically, an architectural distinction explored further in RAG vs fine-tuning.

A typical conceptual workflow follows six steps:

  1. Split source documents into focused text passages, following strategies from our guide on chunking documents for RAG.
  2. Generate an embedding vector for each chunk using an embedding model.
  3. Store the vectors, metadata, and chunk references in a vector database or vector index.
  4. Encode the user’s incoming question with the same embedding model or a compatible query encoder.
  5. Query the vector index for the nearest chunks, applying metadata filters and optional keyword matching or reranking.
  6. Pass the retrieved passages into the language model prompt as context, as outlined in our overview of RAG.

What a vector database is not

Understanding boundaries keeps architecture clean. A vector database does not replace the language model that generates answers, the embedding model that turns content into vectors, or the primary system of record that owns business transactions.

Storing embeddings is also distinct from maintaining an authoritative source of truth. Teams still need access to the underlying text, clear permission enforcement, and a pipeline to update or delete vectors whenever source documents change. If the source text changes or the team upgrades to a new embedding model, affected documents must be re-embedded. Furthermore, not every RAG deployment requires a standalone specialized vector database; a vector index inside an existing database or search engine can fill the same role when scale and filtering requirements align.

Limitations to plan for in production

Vector databases carry practical limitations around index approximation, upstream data preparation, and ongoing maintenance that cannot be solved by picking a different index algorithm.

  • Approximate misses: Because ANN indexes trade exactness for speed, a query can skip a true nearest neighbor unless recall settings are raised at higher compute or memory cost.
  • Upstream quality dominates: Poor chunk boundaries or an embedding model ill-suited to your domain will produce weak search results regardless of how fast the vector index runs.
  • Similarity is not truth: A high similarity score only means two vectors sit close together in embedding space. Retrieved chunks can still be factually wrong, outdated, or unauthorized if metadata filters fail.
  • Operational upkeep: Swapping embedding models requires full re-embedding and index rebuilds, while frequent edits, deletions, and strict multi-tenant isolation require careful consistency and permission design.

FAQ

Do I need a standalone vector database for RAG?

No, a RAG architecture does not always require a standalone vector database. While dedicated vector databases are built specifically for high-dimensional retrieval, many general-purpose relational databases and search engines now include vector index extensions. Small or moderate collections may fit comfortably inside an existing data system if it supports your filtering and update needs.

What distance metric does a vector database use?

Most vector databases support three common metrics: cosine similarity, inner product (dot product), and Euclidean (L2) distance. The right choice depends on how the embedding model was trained and whether its vectors are normalized. Cosine similarity compares vector angle regardless of magnitude, whereas Euclidean distance and inner product incorporate vector length.

Why is HNSW used so often in vector indexes?

HNSW, or Hierarchical Navigable Small World graphs, is widely used because its multi-layer graph design navigates large vector collections quickly without scanning every stored vector. Searches start on sparse upper layers to jump across the vector space and refine results on denser lower layers, offering a practical balance between query speed and recall.

A vector database rarely replaces keyword search outright because semantic embeddings and lexical terms solve different retrieval problems. Vector search excels when queries paraphrase ideas in different words, while keyword search excels on exact error codes, SKUs, and legal citations. Many production retrieval pipelines combine both methods in hybrid search.

Sources

Written by

Chunyu Cai

Chunyu Cai is the CEO of Lotook, LLC and an entrepreneur who writes about AI models, agents, and how they change everyday work. Chunyu holds a master's degree from Imperial College London.