Do I need a vector database for my AI project?

What it is

Vector databases are often treated as mandatory for AI. They’re one option for retrieval, not a requirement.

How it works

1. Estimate the size of your collection in tokens.
2. If it fits in the context window with room to spare, test sending it directly (see RAG or full context).
3. If it doesn’t fit, try the managed file-search or knowledge feature in the platform you already use.
4. Move to a dedicated vector database when scale, filters or speed require it.

Rough thresholds (IonBin guidance, not a vendor rule):

• A few hundred short documents, or anything well under the model’s context limit, usually doesn’t need one.
• Large or fast-changing collections, or a need for metadata filters and quick answers, usually do.

Benefits

• Avoids building infrastructure you don’t need.
• Faster prototypes.
• A clear trigger for when to invest.

Limits and risks

• Long context costs more per request, and can reduce accuracy on details buried mid-document (see Lost in the Middle and context rot).
• The case for full context. Andrej Karpathy has argued for putting all the material in the context window when it fits, rather than trusting retrieval to choose. That’s a practitioner view.
• Retrieval quality depends on chunking and embeddings.
• More systems mean more security work and cost.