Vector Poisoning Detection: How to Catch Retrieval Corruption Before Agents Act

A poisoned vector store is not just bad search quality. It is persistent memory corruption. Once malicious chunks dominate retrieval, an agent can start making bad decisions long after the original ingest event is forgotten.

Why this matters

Most teams still talk about prompt injection as if every attack starts and ends inside one model interaction. That is too narrow. If bad embeddings make it into the retrieval layer, the system can keep serving attacker-controlled context over and over again. The blast radius is wider because the bad state becomes durable.

That is why vector-store poisoning belongs under the same threat family as memory poisoning. The retrieval layer is memory, just in dense form instead of markdown or JSON.

The fastest detector that is still honest

The cheap version is also the right starting point. Watch two explainable signals before a batch is promoted into production memory: centroid drift and hubness anomalies.

Those two checks catch the common ugly cases without pretending you need model retraining or a giant anomaly platform on day one.

What centroid drift catches

If a namespace is supposed to contain narrow, trusted material, a malicious or off-topic batch should not move the center of that embedding cloud very far. Large shifts often mean one of three things: attacker-controlled chunks were inserted in bulk, random junk was embedded, or the namespace quietly absorbed a source that does not belong there.

That signal gets stronger when you compare shift size against the normal radius of the trusted corpus instead of using a raw distance threshold. The point is not mathematical elegance. The point is asking whether this batch looks like it belongs in the room.

What hubness catches

Hubness is the retrieval version of an attention hijack. If a small number of chunks suddenly show up as neighbors for everything else, something is wrong. Sometimes it is duplication. Sometimes it is paraphrased poison engineered to look broadly relevant. Either way, it means the store is being tilted so the same attacker-controlled material keeps winning retrieval.

That is exactly the failure mode defenders should care about, because the model may appear normal while the context selection layer is already compromised.

The operational rule that matters most

Do not let suspicious batches define their own baseline. Keep trusted rolling baselines per namespace, reset them on embedding-model changes, and update them only from approved or consistently low-risk batches. Otherwise the store slowly learns that poisoned is normal.

What good looks like

A sane ingest path measures anomaly signals before commit, keeps the audit trail append-only, and gives operators a clean override path when they intentionally run backfills or model migrations. Static checks can verify those defenses exist, but runtime measurement has to happen near the vector write path.

The important part is staying explainable. Security teams need to know why a batch was blocked, not just that a black box disliked it.

KENSAI takeaway

The real lesson is boring in a good way. If your retrieval layer can be changed silently, then your agent memory can be changed silently. Start with cheap math, strong logging, and quarantine controls. Fancy modeling can wait.