Vector Databases Explained: When You Need One (and When You Don't)

Vector databases became the "must-have" infrastructure for AI applications in 2024. Every RAG tutorial starts with "first, set up your vector database." But here's what most tutorials won't tell you: many AI applications don't need a dedicated vector database at all.

This guide explains what vector databases actually do, compares the major options with real pricing and performance data, and gives you a clear framework for deciding whether you need one.

What Vector Databases Actually Do

The Problem They Solve

Traditional databases are built for exact matches. You search for a customer ID, a product name, or a date range, and the database returns rows that match precisely. But AI applications need similarity search: "find the documents most similar to this question."

Similarity search works on embeddings: numerical representations of text (or images, or any data) where similar items are close together in high-dimensional space. The sentence "How do I reset my password?" and "I forgot my login credentials" have different words but similar embeddings because they mean similar things.

A vector database stores these embeddings and finds the closest ones to a query vector quickly, even across millions of items. That's the core functionality. Everything else is an optimization on top.

How Embeddings Work

An embedding model (like OpenAI's text-embedding-3-small or Cohere's embed-v3) converts text into a fixed-length array of numbers (typically 384-1536 dimensions). Two pieces of text with similar meaning produce vectors that are close together when you measure distance between them.

The most common distance metrics are:

• Cosine similarity: Measures the angle between vectors. Most popular for text similarity. Score ranges from -1 (opposite) to 1 (identical).
• Euclidean distance: Straight-line distance between points. Works well when vector magnitude matters.
• Dot product: Computationally fastest. Works well with normalized vectors (which most embedding models produce).

For most text-based AI applications, cosine similarity with a standard embedding model works well. Don't overthink the distance metric choice unless you're seeing specific retrieval quality issues.

The Major Vector Databases Compared

Pinecone

The most popular managed vector database. Fully hosted, no infrastructure to manage.

Weaviate

Open-source vector database with both self-hosted and managed cloud options.

Chroma

Lightweight, developer-friendly vector database designed for rapid prototyping.

pgvector (PostgreSQL Extension)

A PostgreSQL extension that adds vector similarity search to your existing Postgres database.

Performance Comparison

Here are realistic numbers for a common use case: 1M document chunks, 1536-dimension embeddings, returning top 10 results.

When You Need a Dedicated Vector Database

You actually need a dedicated vector database when:

You have more than 5 million vectors

At this scale, pgvector performance degrades noticeably, and dedicated vector databases maintain consistent latency through purpose-built indexing. If your document corpus is large (millions of pages), a dedicated solution makes sense.

You need sub-10ms query latency

Real-time applications like autocomplete or live recommendations need the fastest possible retrieval. Dedicated vector databases with in-memory indexing deliver consistent single-digit millisecond latency that pgvector can't match.

You're doing heavy vector operations

If your workload is primarily vector similarity search with high query volume (thousands of queries per second), a purpose-built database handles the load more efficiently. Postgres is general-purpose, which means it's adequate at vector operations but not optimal.

When You Don't Need One

This is the section most vector database marketing materials skip. You probably don't need a dedicated vector database when:

Your corpus is under 100K documents

For small to medium document sets, pgvector works perfectly well. Query latency at 100K vectors is under 20ms with a properly configured HNSW index. You avoid adding a new piece of infrastructure to your stack, which means less to monitor, maintain, and pay for.

You already use Postgres

If your application already runs on Postgres (and most web applications do), pgvector keeps everything in one database. Your vectors join with your relational data in a single query. You don't need a separate data pipeline to sync between systems. For most startups and early-stage products, this simplicity is worth more than the performance gains of a dedicated solution.

You're prototyping

During prototyping and early development, use Chroma (in-process, zero setup) or pgvector (if you already have Postgres). Don't add infrastructure complexity before you've validated that your RAG approach works. You can migrate to a dedicated vector database later if you need the scale.

Your retrieval quality issues aren't about the database

This is the most common mistake. Teams switch vector databases hoping to improve RAG quality. But poor retrieval quality is almost always a chunking problem, an embedding model problem, or a query formulation problem. The database just stores and retrieves vectors. Switching from pgvector to Pinecone won't fix a bad chunking strategy.

Migration Considerations

Starting with pgvector and migrating later is a valid strategy. Here's what the migration involves:

• Re-embed your documents: You might want to use a different embedding model anyway. Budget a few dollars for the embedding API calls.
• Update your retrieval code: Swap the query logic from SQL to the new database's SDK. Typically 50-200 lines of code.
• Set up the new infrastructure: Managed services (Pinecone, Weaviate Cloud) take 30 minutes. Self-hosted takes longer.
• Test retrieval quality: Ensure results are equivalent. Run your eval suite against both systems.

Total migration time: 1-3 days for a competent engineer. It's not a major undertaking, which is why starting simple and scaling up is usually the right call.

Practical Recommendations

The best vector database is the one that fits your current needs without adding unnecessary complexity. Start simple. Scale when you have data showing you need to.

For more on building AI applications with retrieval, see our RAG architecture guide and the vector database glossary entry.

Related: RAG Architecture Guide | RAG vs Fine-Tuning Cost Comparison | Vector Database Glossary Entry | AI Tools for Developers | Best Vector Databases | Pinecone vs Weaviate | Chroma vs pgvector | Pinecone Pricing