Short answer: A vector database stores embeddings, which are lists of numbers that capture meaning, and finds the stored items nearest to a query. That lets search match ideas instead of exact words, which is the retrieval step behind most RAG systems.
You may not need a separate one. Exact search in memory or in Postgres covers many projects, and this guide ends with a rule for choosing.
How does a vector database work?
A vector database works in four steps: embed the data, store the vectors, compare them with a distance metric, and index them for speed.
- Embed. OpenAI defines an embedding as "a vector (list) of floating point numbers." Its text-embedding-3-small returns 1536 numbers by default, and text-embedding-3-large returns 3072. The dimensions parameter shortens them, which OpenAI calls "the suggested approach."
- Store. Each record holds the vector, an ID and metadata. Qdrant calls this a point, with a payload that is "a JSON object with additional data." Qdrant groups points into collections, "named sets of points that you search within", and you choose the distance metric when you create one. Pinecone allows 40 KB of metadata per record.
- Compare. A metric scores how close two vectors are. pgvector offers L2 distance, negative inner product and cosine distance, as the operators
<->,<#>and<=>. OpenAI embeddings are normalized to length 1, so cosine and Euclidean distance "will result in the identical rankings." - Index. Exact search compares the query with every stored vector. An approximate index such as HNSW checks far fewer and gives up some recall.

Each dot is one embedded item. The search returns the points closest to the query, because closeness stands for related meaning. If you are still choosing a model, see our guide to choosing an embedding model.
Why do results change after you add an index?
Results change because an approximate index trades some recall for speed, and pgvector documents this. By default, pgvector "performs exact nearest neighbor search, which provides perfect recall." Its README also warns that "you will see different results for queries after adding an approximate index."
HNSW is the common approximate index. The original paper compares its layered graph to a skip list and reports "logarithmic complexity scaling." Instead of scanning every vector, the search follows links between nearby points.

The exact scan finds every true neighbor but does the most work. The graph skips most comparisons and finds almost all of them. pgvector also offers IVFFlat, a second index type that divides vectors into lists and searches only the lists closest to the query. These are pgvector's defaults:
| Setting | What it controls | pgvector default |
|---|---|---|
m | Max connections per layer | 16 |
ef_construction | Candidate list size while building the graph | 64 |
hnsw.ef_search | Candidate list size at query time | 40 |
IVFFlat lists | Number of partitions | Start at rows / 1000 up to 1M rows, sqrt(rows) above |
If recall drops too far, raise hnsw.ef_search: pgvector says a higher value gives better recall at the cost of speed.
Filters are the common surprise. pgvector applies them "after the index is scanned." If 10% of rows match and ef_search is 40, you get about 4 matching rows on average. Iterative scans fix this: they "automatically scan more of the index until enough results are found." The strict_order mode keeps exact distance order, while relaxed_order allows "slightly out of order" results for better recall. Turn them on with SET hnsw.iterative_scan = strict_order; (or relaxed_order).
Which kind of vector store fits your stack?
Vector stores fall into three types, and the difference is who runs the index.
| Type | Example | Who runs it | What the docs say |
|---|---|---|---|
| Embedded library | Chroma | Your program | Apache 2.0; the in-memory client loses data when your program ends; Chroma Cloud is managed and serverless |
| Dedicated server | Pinecone, Qdrant | The vendor (Pinecone; Qdrant Cloud) | Pinecone indexes dense, sparse or both vector fields, plus BM25 full-text search; Qdrant's free tier is 0.5 vCPU, 1GB RAM and 4GB disk, without high availability |
| Postgres extension | pgvector | Your existing database | Postgres 13+; "Store your vectors with the rest of your data" |
Qdrant also offers Qdrant Edge, which runs inside your application process. It is in beta as of August 2026, and Qdrant says the API may change. For paid plans on any of these, check the vendor page.
Try it yourself with Chroma's quickstart
Chroma's quickstart is the fastest way to see search by meaning.
- Run
pip install chromadb. - Create a client, then a collection.
- Call
collection.addwith five short documents. - Call
collection.querywith a question that shares no words with the best answer.
Chroma "will store your text and handle embedding and indexing automatically." Check whether the top result matches by meaning rather than by shared words.
Do you need a vector database? A decision rule
Start with exact search, and add an index or a dedicated store only when you have a reason. This is Toolbit's rule, built from the vendor docs above.
- Exact tokens matter: for SKUs, error messages and names, use full-text search. Pinecone's indexing docs describe BM25 as the fit for these.
- Your data lives in Postgres: use pgvector with exact search first. For filtered queries, the README starts with an index on the filter column. For normalized vectors, it recommends inner product "for best performance."
- A prototype in one process: an exact scan in memory, or Chroma, is enough.
- Exact search is too slow, or you want it run for you: add HNSW, or move to Pinecone or Qdrant. OpenAI says: "For searching over many vectors quickly, we recommend using a vector database."
Retrieval is one part of the job. What you put in front of the model matters as much, which is the subject of context engineering.
Vector database vs similar terms: what differs
| Term | What it is | How it differs |
|---|---|---|
| Embedding model | Turns text into a vector | Makes vectors; does not store or search them |
| Vector database | Stores vectors with IDs and metadata | Finds the nearest vectors to a query |
| Full-text search | Token matching, such as BM25 | Matches words, not meaning |
| pgvector | A Postgres extension | Keeps vectors next to your other columns |
| RAG | Retrieve facts, then generate an answer | Uses vector search as one step |
For the full pattern, read what RAG is.
Vector database FAQ
Do I need a vector database for RAG? Not always. RAG needs retrieval, and exact search can handle a small corpus. OpenAI recommends a vector database for "searching over many vectors quickly."
Is pgvector enough? Often, if your data is already in Postgres. It does exact search, HNSW and IVFFlat indexes, and hybrid search. The README says "You can use Reciprocal Rank Fusion or a cross-encoder to combine results."
Why did my results change after indexing?
An approximate index trades some recall for speed. pgvector says you "will see different results" after adding one. If recall drops too far, raise hnsw.ef_search; a higher value gives better recall at the cost of speed.
Cosine or dot product for OpenAI embeddings? OpenAI embeddings have length 1, so cosine and dot product give the same score. pgvector suggests inner product for normalized vectors.




