Skip to content

Evaluation

The short version

Venkai has no published benchmark.

There is no recall@k figure, no token-savings figure, no latency figure and no head-to-head comparison that this project is prepared to stand behind. Nothing in this documentation quotes one as a product claim.

If you have seen a Venkai number that is not on this page, it did not come from here. Ask for the dataset, the run count and the baseline before believing it.

Why the page exists anyway

Because "no benchmark" is a fact a technical evaluator needs, and because the honest account of why is more useful than a number would be.

There are three separate reasons, and they are worth separating.

1. The default embedding provider is not semantic

The out-of-the-box hashing provider maps tokens into 128 buckets by SHA-256. It measures lexical overlap wearing a vector's clothes. Any recall figure measured on it describes a keyword matcher, not a semantic memory — and publishing it would set an expectation the semantic provider would then have to be compared against separately.

See Embeddings.

2. One measurement we do have is a negative result

The source records this, and it is genuinely informative:

Incrementing access_count on every retrieval collapsed recall@5 from 0.54 to 0.12 after roughly 50 queries, because the top-5 lock in permanently.

That is a real, reproducible finding on an internal fixture, and it is why retrieval does not count as an access today (Ranking).

But note what it is: a design rationale from an ablation, on a fixture built for the purpose. 0.54 is the number the retriever reached with that feedback loop disabled, on that fixture. It is not a benchmark result, the fixture is not public, and there is no baseline alongside it. Quoting "recall@5 = 0.54" as a product claim would be dishonest, so this documentation does not.

3. Evaluating agent memory is genuinely hard

There is no accepted benchmark for "did the agent get the context it needed". The honest constructions are workload-specific:

  • Retrieval quality needs a labelled set of (query → memories that should come back) for your domain.
  • Task quality needs the same agent run with and against memory, on the same tasks, graded by something you trust.
  • Token economics needs the counterfactual: what your prompt would have contained without Venkai. That is not something Venkai can measure from inside — it never sees your prompt.

A generic number across all three would mean nothing about your workload.

What is not evidence

Named explicitly, because these are the figures most likely to be mistaken for benchmarks:

Figure What it actually is
tokens_stored from GET /api/impact An estimate of stored volume derived from character counts. Not a measurement, and not a saving.
total_recalls Usage counter. Says nothing about quality.
most_accessed Ranked by access_count, which retrieval does not increment — near-meaningless until explicit feedback is wired up.
Any number in an internal report deck Not reproducible from this repository. Do not forward them.

If an internal document shows a continuity percentage, a quality score, or a token figure for Venkai, treat it as unverified until it is reproducible from a script in this repository against a stated dataset.

Evaluating it yourself

The only evaluation that will tell you anything is on your own data. Here is a workable protocol.

Step 1 — build a labelled set

Twenty to fifty realistic queries from your domain, each with the memories that should come back. This is the expensive part and there is no shortcut.

GOLD = [
    {
        "query": "which database did we choose and why",
        "should_return": ["ctx_cc55e4a7bbc5"],
    },
    {
        "query": "what are the approval rules for refunds",
        "should_return": ["ctx_f6f282cb1a2f", "ctx_a1b2c3d4e5f6"],
    },
]

Step 2 — measure recall@k and precision@k

def evaluate(gold, k=5):
    """recall@k  = fraction of expected memories that came back
       precision@k = fraction of returned memories that were expected"""
    recalls, precisions = [], []
    for case in gold:
        got = [m["id"] for m in recall(case["query"], limit=k)]
        want = set(case["should_return"])
        hit = len(want & set(got))
        recalls.append(hit / len(want) if want else 0.0)
        precisions.append(hit / len(got) if got else 0.0)
    n = len(gold)
    return {"recall@k": sum(recalls) / n, "precision@k": sum(precisions) / n, "n": n}

Report n alongside the score, always. A recall figure over eight queries is an anecdote.

Step 3 — compare against the baselines that matter

Two baselines, not one:

Baseline How What it tells you
Everything GET /api/context/{project}?limit=200 Whether ranking beats not ranking. Recall will be ~1.0 by construction — the question is what precision and context cost you paid for it.
Recency only Same, sorted by created_at Whether the similarity term is earning its 40 %.

If ranked retrieval does not beat "the 20 most recent memories" on your data, that is worth knowing before you adopt it.

Step 4 — compare providers

Run the same gold set under both:

VENKAI_EMBEDDING_PROVIDER=hashing   # then re-run
VENKAI_EMBEDDING_PROVIDER=semantic

Remember that switching providers invalidates stored embeddings; give the store a chance to backfill or the second run measures on-the-fly recomputation (Persistence).

Step 5 — measure the thing you actually care about

Retrieval quality is a proxy. If what you want is better agent outcomes, run your agent on the same task set with and without load_context() and grade the outputs. That is the only measurement that answers the adoption question.

Reporting honestly

Whatever you measure, state:
Dataset Size, domain, how the labels were made
Runs How many, and the variance across them
Baseline Which one, exactly
Configuration Provider, limit, any score floor
Metric Defined, not just named
Limitations What the result does not cover

Roadmap

Planned — a public evaluation harness and a published dataset, so a figure can be quoted with its method attached. Until it exists, this page is the accurate statement of the evidence, and it is deliberately the least flattering page in this documentation.