{"id":23812,"date":"2026-09-07T22:00:00","date_gmt":"2026-09-07T22:00:00","guid":{"rendered":"https:\/\/scannn.com\/cosine-similarity-is-not-a-safety-property\/"},"modified":"2026-09-07T22:00:00","modified_gmt":"2026-09-07T22:00:00","slug":"cosine-similarity-is-not-a-safety-property","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/cosine-similarity-is-not-a-safety-property\/","title":{"rendered":"Cosine Similarity Is Not a Safety Property"},"content":{"rendered":"\n
\n

The vector database returns the most relevant<\/em> documents, defined as the ones with the highest cosine similarity to the query. This is a mathematical property with no concept of accuracy, authority or provenance. A document scoring 0.95 against a query can be entirely fabricated. A document scoring 0.60 can be the ground truth.<\/p>\n

The attacker\u2019s job in a poisoning attack is not to break into the vector database. It is to write a document whose embedding sits closer to the anticipated query than the legitimate document does, then frame it with enough authority to win the argument once both are in the context window.<\/p>\n

What the embedding model is actually doing<\/span><\/a><\/h2>\n

sentence-transformers\/all-MiniLM-L6-v2<\/code> turns any string into a 384-dimensional float vector: a point in 384-dimensional space. Semantically similar texts land near each other. \u201cQ4 financial results\u201d sits close to \u201cfourth quarter revenue\u201d and far from \u201ccompany travel policy.\u201d ChromaDB stores the vectors and answers a query by finding the k stored points closest to the query point, under whichever distance function the collection was created with.<\/p>\n

ChromaDB\u2019s default is squared L2, not cosine. Cosine has to be set on purpose, with metadata={\"hnsw:space\": \"cosine\"}<\/code> at collection creation. Everything in this piece assumes a collection configured that way, which is the common choice for text embeddings and the one worth setting deliberately rather than inheriting by accident. If you skip that step, the \u201cangle to the query point\u201d framing below stops describing what your database is actually doing.<\/p>\n

\n
<\/p>\n\n\n\n
\n
1\n2\n3\n4\n5\n<\/pre>\n<\/td>\n
\n
# The entire retrieval mechanism, conceptually\n<\/span>query_vector<\/span> =<\/span> embed<\/span>(<\/span>\"<\/span>What was Q4 2025 revenue?<\/span>\"<\/span>)<\/span>   # 384 floats\n<\/span>\nresults<\/span> =<\/span> collection<\/span>.<\/span>query<\/span>(<\/span>query_embeddings<\/span>=<\/span>[<\/span>query_vector<\/span>],<\/span> n_results<\/span>=<\/span>3<\/span>)<\/span>\n# The 3 stored vectors with smallest cosine distance.\n<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/code><\/div>\n<\/div>\n
The model does not know what the text means<\/em> in any truth-bearing sense. It knows how text clusters with other text in its training data. \u201cRevenue was $8.3M\u201d and \u201cRevenue was $24.7M\u201d land at nearly identical positions, because they are the same sentence with a different number. The vector database cannot tell them apart. Only the model can, after retrieval, and only if both documents are in front of it.<\/p>\n
This is a property of dense retrieval, not a defect in one model. all-MiniLM-L6-v2<\/code> has aged into a poor production default; EmbeddingGemma-300M<\/code>, Qwen3-Embedding<\/code> and BGE-M3<\/code> are where new work goes in 2026. None of them change the property. Semantic similarity is not factual accuracy at any parameter count.<\/p>
The three poisoned documents sit inside the top-3 window. The document holding the real number sits just outside it, at rank 4, and never reaches the model. Similarity values are illustrative; ChromaDB returns cosine distances, and actual values depend on model, phrasing and document length.<\/figcaption>
Vocabulary engineering: moving a document without gradient descent<\/span><\/a><\/h2>\n
The sophisticated form of this attack, as formalised in PoisonedRAG<\/a> (Zou, Geng, Wang and Jia, USENIX Security 2025), uses gradient-based optimisation to find text that minimises cosine distance to a target query vector. That needs white-box access to the embedding model, or query access at scale to approximate it. With that machinery, the paper injected five crafted texts into a corpus of 2.6 million<\/strong> and drove attacker-chosen answers at above 90% success.<\/p>\n
The cheap version needs neither gradient descent nor white-box access. Identify the vocabulary of the query you want to hijack, put that vocabulary in your document, add authority signals for the generation step. In my lab that got to 95% success against an undefended pipeline (twenty runs on a small seeded corpus, one model \u2014 a lab reading, not a base rate). The three documents I used, annotated:<\/p>\n
Document one, the CFO correction.<\/strong> Title: Q4 2025 Financial Results \u2014 CORRECTED FIGURES (Board Update)<\/code>. Against a query like \u201cwhat was Q4 2025 revenue?\u201d, Q4 2025<\/code> is an exact temporal match, Financial Results<\/code> clusters with revenue and financials, CORRECTED FIGURES<\/code> implies it supersedes something, and Board Update<\/code> is an authority signal. Body states fabricated numbers as plain fact and closes with Approved by: CFO Office<\/code>.<\/p>\n
Document two, the restatement notice.<\/strong> States that Q4 revenue has been restated, actual revenue $8.3M, \u201cnot the previously reported $24.7M.\u201d It names the real figure and reframes it as an acknowledged error, which gives the model a narrative for resolving the contradiction it is about to encounter.<\/p>\n
Document three, the board minutes.<\/strong> Emergency session, agenda item three, corrected results, same numbers.<\/p>\n
Three documents corroborating one fabrication against one legitimate document telling the truth. In a top-3 retrieval, three crowd out one, and the legitimate content never enters the context window. That is vocabulary engineering: shifting which regions of embedding space are occupied so fabricated documents sit closer to anticipated queries than the documents they are meant to displace.<\/p>\n
Two conditions, and which one does the work<\/span><\/a><\/h2>\n
PoisonedRAG frames the attack as two conditions that must both hold.<\/p>\n
Retrieval condition.<\/strong> For a target query q, the poisoned document d_p must make top-k:<\/p>\n
\n
<\/p>\n\n\n\n
\n
1\n<\/pre>\n<\/td>\n
\n
cos_distance(embed(d_p), embed(q)) < cos_distance(embed(d_legit), embed(q))\n<\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/code><\/div>\n<\/div>\n
Generation condition.<\/strong> Once in context, d_p must cause the model to produce the attacker\u2019s answer rather than the correct one, which requires the target answer to be present and framed with enough authority to outweigh contradicting sources.<\/p>\n
Research attention went to the retrieval condition, because that is where the elegant optimisation lives. Recent work suggests the framing is doing more of the work. In \u201cArchitecture Matters: Comparing RAG Systems under Knowledge Base Poisoning\u201d<\/a> (Korn, May 2026), most of the strongest attack variant\u2019s advantage came from adversarial framing rather than retrieval optimisation, and attack success across four architectures with comparable clean accuracy spread nearly 58 points, from 81.9% for vanilla RAG down to 24.4% for a recursive setup.<\/p>\n
If framing dominates, then the stage that resolves contradictions between retrieved documents is a more valuable place to invest than another round of retrieval hardening. In my own measurements the legitimate document was often retrieved and still lost<\/em>. The real Q4 figure sat in the context window while the model reported the fabricated one, because two documents said the real figure had been corrected and one document just quietly stated it.<\/p>\n
<\/p>\n
\n
Weekly practitioner-level analysis of AI security \u2014 attack labs, incident breakdowns, and defense patterns for teams actually building these systems. One email per week, no fluff.<\/p>\n
 Subscribe \u2014 it’s free <\/a><\/div>\n
Turning the geometry against the attacker<\/span><\/a><\/h2>\n
The attacker\u2019s requirement is also their signature. To be retrieved, the poisoned documents must cluster near the target query position, which is near the legitimate documents on that topic. An ingestion-time check can look for this.<\/p>\n
Two signals, computed before anything is stored:<\/p>\n
\n
<\/p>\n\n\n\n
\n
1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n<\/pre>\n<\/td>\n
\n
# Signal 1 \u2014 nearest neighbour in the existing collection\n<\/span>existing<\/span> =<\/span> collection<\/span>.<\/span>query<\/span>(<\/span>query_embeddings<\/span>=<\/span>[<\/span>new_doc_embedding<\/span>],<\/span> n_results<\/span>=<\/span>3<\/span>)<\/span>\nfor<\/span> dist<\/span> in<\/span> existing<\/span>[<\/span>\"<\/span>distances<\/span>\"<\/span>][<\/span>0<\/span>]:<\/span>\n    if <\/span>(<\/span>1.0<\/span> -<\/span> dist<\/span>)<\/span> ><\/span> SIMILARITY_THRESHOLD<\/span>:<\/span>      # 0.85 in the lab\n<\/span>        flag<\/span>(<\/span>\"<\/span>HIGH_SIMILARITY \u2014 possible content override<\/span>\"<\/span>)<\/span>\n\n# Signal 2 \u2014 pairwise similarity within the incoming batch\n<\/span>for<\/span> i<\/span>,<\/span> e_i<\/span> in<\/span> enumerate<\/span>(<\/span>new_embeddings<\/span>):<\/span>\n    for<\/span> j<\/span> in<\/span> range<\/span>(<\/span>i<\/span> +<\/span> 1<\/span>,<\/span> len<\/span>(<\/span>new_embeddings<\/span>)):<\/span>\n        if<\/span> cosine_similarity<\/span>(<\/span>e_i<\/span>,<\/span> new_embeddings<\/span>[<\/span>j<\/span>])<\/span> ><\/span> CLUSTER_THRESHOLD<\/span>:<\/span>   # 0.90\n<\/span>            flag<\/span>(<\/span>\"<\/span>TIGHT_CLUSTER \u2014 possible coordinated injection<\/span>\"<\/span>)<\/span>\n<\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/code><\/div>\n<\/div>\n
Both fire on the three-document attack. Each poisoned document\u2019s nearest neighbour is the legitimate Q4 report, because both are about Q4 2025 financials. The three cluster tightly with each other, because they are variations on one fabricated narrative. This single layer took poisoning success from 95% to 20%.<\/p>\n
The geometric reason it works is the closest thing to a structural guarantee in this area: the attacker cannot fully satisfy both requirements at once. Satisfying the retrieval condition means occupying space near existing content, which is detectable. Evading detection means moving away from that space, which degrades retrieval. The 20% residual is the band where an attacker balances the two, typically with a single document instead of a cluster, or with enough vocabulary variation to fall below the similarity threshold while staying retrievable.<\/p>\n
\n%%{init: {\"theme\": \"base\", \"themeVariables\": {\"primaryColor\": \"#e8f4fd\", \"primaryBorderColor\": \"#3182ce\", \"fontSize\": \"14px\", \"fontFamily\": \"system-ui, -apple-system, sans-serif\"}}}%%\nflowchart LR\n    RC[\"Retrieval conditionembed close to target query\"] -->|\"same property\"| S1[\"Signal 1similarity to existing > threshold\"]\n    CL[\"Coordinated injectionseveral docs, one space\"] -->|\"same property\"| S2[\"Signal 2pairwise similarity > threshold\"]\n    S1 --> DIL[\"The attacker's dilemmaevading detection degrades retrievalsatisfying retrieval triggers detection\"]\n    S2 --> DIL\n    classDef threat fill:#0f172a,stroke:#0f172a,color:#fff,stroke-width:1.5px\n    classDef ok fill:#e8f4fd,stroke:#3182ce,color:#1a202c,stroke-width:1.5px\n    class RC,CL threat\n    class S1,S2,DIL ok\n<\/pre>\n
Access-controlled retrieval: the where<\/code> clause<\/span><\/a><\/h2>\n
A metadata filter on every vector query that restricts which documents the requesting user is allowed to retrieve. Without it, every document in the collection is reachable by every user through an ordinary question. Ask an assistant what the salary bands are and it retrieves whatever is semantically close, with no idea who is asking.<\/p>\n
I ran this against a lab pipeline holding three restricted documents: salary data marked HR-only, litigation detail marked privileged, and an M&A pipeline marked board-level. Queried as a regular engineering user, in natural language, with no evasion. Twenty out of twenty queries returned confidential content.<\/p>\n
\n
<\/p>\n\n\n\n
\n
1\n2\n3\n4\n5\n6\n7\n8\n9\n<\/pre>\n<\/td>\n
\n
# Vulnerable: no notion of who is asking\n<\/span>results<\/span> =<\/span> collection<\/span>.<\/span>query<\/span>(<\/span>query_embeddings<\/span>=<\/span>[<\/span>query_embedding<\/span>],<\/span> n_results<\/span>=<\/span>3<\/span>)<\/span>\n\n# Hardened: retrieval scoped to the requester's clearances\n<\/span>results<\/span> =<\/span> collection<\/span>.<\/span>query<\/span>(<\/span>\n    query_embeddings<\/span>=<\/span>[<\/span>query_embedding<\/span>],<\/span>\n    n_results<\/span>=<\/span>3<\/span>,<\/span>\n    where<\/span>=<\/span>{<\/span>\"<\/span>classification<\/span>\"<\/span>:<\/span> {<\/span>\"<\/span>$in<\/span>\"<\/span>:<\/span> user_permitted_classifications<\/span>}},<\/span>\n);<\/span>\n<\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/code><\/div>\n<\/div>\n
That filter is the only complete defence against cross-tenant leakage, because it is structural. It prevents unauthorised content from entering the context window at all. Output monitoring, prompt hardening and every other heuristic runs after retrieval, by which point the data is already in the prompt.<\/p>\n
A minimal access model is enough to start:<\/p>\n
\n\n\n\n\n\n\n\n\n
Role<\/th>\nPermitted classifications<\/th>\n<\/tr>\n<\/thead>\n
Regular employee<\/td>\npublic, internal<\/td>\n<\/tr>\n
HR business partner<\/td>\n+ confidential (HR)<\/td>\n<\/tr>\n
Legal counsel<\/td>\n+ privileged<\/td>\n<\/tr>\n
Executive<\/td>\nall<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
You do not need full RBAC. You need a mapping from identity to permitted metadata values, applied on every retrieval path.<\/p>\n
Partial implementation of this control is worse than none, because it manufactures confidence. A where<\/code> filter on the main chat endpoint, with an unfiltered path in the summarisation job or the eval harness, gives you a system that looks access-controlled and is not. Find every retrieval call, not the obvious one.<\/p>\n
The threshold problem<\/span><\/a><\/h2>\n
The lab hardcodes 0.85 and 0.90. Those are not universal values, and copying them into production is the most common way this control fails.<\/p>\n
At 0.85 the defence catches the three-document attack because the poisoned documents score around 0.88 to 0.92 against the legitimate financials. Increase vocabulary variation, keeping the financial theme but varying the specific language, and similarity drops to roughly 0.78 to 0.82. Below threshold, not flagged, still retrievable.<\/p>\n
\n\n\n\n\n\n\n\n\n\n
Threshold<\/th>\nPoisoning caught<\/th>\nLegitimate updates flagged<\/th>\n<\/tr>\n<\/thead>\n
0.95<\/td>\nOnly near-verbatim overrides<\/td>\nAlmost none<\/td>\n<\/tr>\n
0.90<\/td>\nStrong variants<\/td>\nOccasional, on revised documents<\/td>\n<\/tr>\n
0.85 (lab default)<\/td>\nMost variants<\/td>\nSome, on policy revisions and corrections<\/td>\n<\/tr>\n
0.80<\/td>\nMore variants<\/td>\nFrequent, most updates flagged<\/td>\n<\/tr>\n
0.75<\/td>\nMaximum coverage<\/td>\nImpractical, constant manual review<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
The right value comes from your ingestion patterns. An append-only corpus can sit low. A living wiki with versioned policies cannot, because every legitimate amendment looks like an override attempt. Baseline your collection\u2019s actual similarity distribution and set the threshold around mean plus two standard deviations, then revisit as the collection grows.<\/p>\n
These thresholds are properties of your embedding model, not of your data.<\/strong> Similarity distributions are not comparable across models. Swap all-MiniLM-L6-v2<\/code> for Qwen3-Embedding<\/code> or BGE-M3<\/code> and every threshold you tuned is now measuring a different geometry. A model upgrade silently invalidates your detector. Pin the embedding model version alongside the threshold, and treat an embedding migration as a re-baselining project.<\/p>\n
Your vectors are not opaque<\/span><\/a><\/h2>\n
Embedding vectors get treated as one-way functions. You embed text, store the vector, and the original text is not recoverable from the vector alone. That assumption was never proven for dense sentence embeddings. It was inherited by practitioners who thought of embeddings as just numbers.<\/p>\n
The research record says otherwise:<\/p>\n