Embeddings¶

The default provider is not semantic

VENKAI_EMBEDDING_PROVIDER defaults to hashing, which measures lexical overlap, not meaning. If you expect a query about "the database choice" to match a memory saying "we picked Postgres", you need the semantic provider. This is the single most consequential configuration decision in Venkai.

The two providers¶

	`hashing` (default)	`semantic`
Class	`HashingEmbeddingProvider`	`SemanticEmbeddingProvider`
Model	none — SHA-256 buckets	`all-MiniLM-L6-v2`
Dimensions	128	384
Dependencies	none	`sentence-transformers` + CPU `torch`
Model size	0	~80 MB
Speed	very fast	a forward pass per text
Understands paraphrase	no	yes
Licence	—	MIT

How `hashing` works¶

Each token is hashed; the hash picks one of 128 buckets and its parity picks a sign. The vector is then normalised.

# conceptually
for token in tokens(text):
    h = sha256(token)
    vec[h % 128] += +1.0 if (h // 128) % 2 == 0 else -1.0

Two texts are similar to it exactly insofar as they share tokens. It is a bag-of-words model with collisions, expressed as a vector. It has no notion that "Postgres" and "database" are related — they land in unrelated buckets.

You saw this in the Quickstart: a clearly on-topic decision scored similarity: 0.1348 against the query "which database did we pick". Not a bug — the correct output of a lexical matcher on two sentences that share almost no words.

It is still useful for exact-terminology domains: error codes, function names, ticket ids, product SKUs. And because similarity is only 40 % of the score, ranking still works — carried by importance and recency (Ranking).

How `semantic` works¶

all-MiniLM-L6-v2 via sentence-transformers: a 384-dimensional sentence embedding that puts paraphrases near each other. Runs locally on CPU, no API call, no data leaving your infrastructure.

pip install sentence-transformers
export VENKAI_EMBEDDING_PROVIDER=semantic

It fails loudly on purpose¶

If sentence-transformers is missing, startup raises:

VENKAI_EMBEDDING_PROVIDER=semantic, but sentence-transformers is not installed…
  · set VENKAI_EMBEDDING_PROVIDER=hashing to ask for hashing on purpose.

A silent downgrade would be the worst outcome available: the operator asked for semantic matching, gets hash buckets, and nothing in any API response says so. Every answer would still look plausible.

The docstring used to lie about this

The retriever's documentation claimed semantic was the default until 2026-08-14. The code always defaulted to hashing. It is recorded here because if you read older material about Venkai, that is where the discrepancy came from.

Bake the model into your image¶

RUN python -c "from sentence_transformers import SentenceTransformer; \
               SentenceTransformer('all-MiniLM-L6-v2')"

The production Dockerfile does this. Without it, the first search after every deploy downloads ~80 MB from Hugging Face — meaning the container needs outbound internet at runtime, the first user pays the latency, and a Hugging Face outage becomes a Venkai outage. Baking it also pins which weights ship.

Choosing¶

flowchart TD
  A{"Will queries use different<br/>words than the memories?"} -->|no — exact terms| B["hashing"]
  A -->|yes — natural language| C{"Can you install<br/>~80 MB + torch?"}
  C -->|yes| D["semantic"]
  C -->|no| E["hashing, and raise importance<br/>to compensate"]

Use `hashing` when	Use `semantic` when
Queries reuse the memories' exact terminology	Queries are natural-language paraphrases
Dependency footprint must stay minimal	Retrieval quality is the point
Latency is critical and the corpus is large	You can afford a forward pass per query

Decide before loading data if you can — the migration is real work.

Switching providers¶

The dimension changes, so every stored embedding becomes incompatible.

flowchart LR
  A["switch provider"] --> B["stored vectors:<br/>wrong dimension"]
  B --> C["recomputed per read,<br/>not written back"]
  C --> D["✅ correct results<br/>❌ degraded latency, forever"]
  D --> E["backfill to recover"]

Nothing breaks and nothing is lost — an incompatible embedding is recomputed on the fly rather than dropping the memory from results. But it is recomputed on every query, for every stale candidate, until backfilled.

A backfill is a PATCH of each memory with its own content, which recomputes and persists the new vector:

# ponytail: N+1 by design — it runs once, and a bulk endpoint for a one-off
# migration is a maintenance burden that outlives the migration.
for page in paginate(f"/api/context/{PROJECT}", limit=200):
    for m in page["context"]:
        patch(f"/api/context/{m['id']}", {"content": m["content"]})

Planned — a built-in backfill command. There is none today.

Where embeddings live¶

JSON float arrays in contexts.embedding_json. No pgvector, no ANN index — cosine is computed in Python over the candidate set. See Storage.

Compatibility rule¶

A stored embedding is used when its length equals the provider's dimension. That is the whole check — length, not provenance. Two providers of the same dimension would be silently interchangeable, so do not add one.

Persistence — write, update, recompute
Ranking — where similarity fits in the score
Evaluation — why the provider choice makes benchmarks hard