Skip to content

Evaluation

Our rule

no measurement -> no claim

Venkai does not publish a "% improvement", a "token saving", a recall figure or a latency number unless it comes with a baseline measured on the same cases under the same controls, a stated dataset, and its limitations.

Today, no benchmark meets that bar for publication. That is why this documentation contains no performance figures. If you have seen a Venkai number that is not on this page, ask for the dataset, the run count, the controls and the baseline before believing it.

How we evaluate internally

Venkai is evaluated against defined retrieval baselines — including plain similarity search — on annotated cases, measuring retrieval quality (whether the right memories come back in the top results) and noise (how much of what comes back is irrelevant).

Internal results are used to steer the product. They are not quoted as product claims, because a fixture we designed is not your workload.

Planned — published results, with methodology and conditions, once they are representative enough to support a claim.

What is not evidence

Signal Why it is not a performance claim
tokens_stored from GET /api/impact An estimate of stored volume from character counts. Not a measurement, and not a saving.
most_accessed Ranked by access_count, which retrieval does not increment.
score values Relative within one response. Not comparable across projects or releases.

Evaluating Venkai on your own data

The evaluation that matters is yours. A practical protocol:

  1. Collect real tasks. 30–50 tasks your agents actually run, each with the memories a careful human would want in the prompt for it.
  2. Load the memories into a dedicated project, with the type and importance you would use in production.
  3. Define a baseline. Typically what you do today: the full history, the last N messages, or a plain similarity search over the same texts.
  4. Query both with the same budget — the same limit, or the same character budget applied client-side.
  5. Measure for each task: how many of the expected memories came back, how many returned items were irrelevant, and the prompt size.
  6. Check the outcome, not just retrieval: run the agent with each context and grade the result.

The score, similarity and justification fields on every result are there so you can see why something ranked where it did while you do this.

Tell us what you find

Especially where Venkai loses. Evaluation reports from real workloads shape the ranking roadmap directly.