Skip to content

Foundry Evaluation

Status

Planned

No Foundry integration exists. No experiment has been run. This page describes the experiment that evals/systems/foundry.py is shaped for, and what needs to happen before it can execute.

The question

Foundry has a Memory Engine that retrieves candidate memories for an agent. Venkai is a context-selection layer that scores, filters, deduplicates and budget-constrains candidates from any retrieval engine.

The question: does Venkai's selection improve the quality/cost tradeoff of Foundry's retrieval?

That question is answerable only by measurement. No claim about Venkai improving Foundry's output has been measured.

The experiment

Two arms, one variable:

flowchart LR
  A["Foundry Memory Engine"] --> B["candidate memories"]
  B --> C["baseline greedy packer"] --> E["LLM"]
  B --> D["Venkai selection"] --> F["LLM"]

Arm A (baseline): Foundry retrieves candidates, the harness's shared greedy packer packs them under a token budget. This is what an agent using Foundry today would get if it simply packed the top-k results.

Arm B (Venkai): Foundry retrieves the same candidates, Venkai applies its scoring pipeline (relevance 40% + recency 20% + importance 25% + frequency 15%), filters by a relevance floor, deduplicates, and packs under the same budget.

Variables held constant

  • Dataset and memory corpus
  • Query set
  • LLM model and prompt template
  • Temperature
  • Context token budget

Variable (the only one)

Selection layer: baseline greedy packer vs. Venkai selection pipeline.

If the embedding model cannot be held constant (because Foundry uses its own), that is a second uncontrolled variable and the comparison is downgraded from "selection layer" to "end-to-end stack". This must be declared in the report.

Metrics

Metric What it measures Direction
Context recall fraction of relevant memories in the selected context higher is better
Context precision fraction of selected memories that are relevant higher is better
Redundancy ratio duplicate information in the selected context lower is better
Irrelevance ratio irrelevant information in the selected context lower is better
Selected tokens total tokens shipped to the LLM lower is better (at same quality)
Latency end-to-end time, query to selected context lower is better
Task success whether the LLM can answer from the context higher is better

What needs to happen before this can run

  1. Written permission to benchmark under the terms of access.
  2. Documented retrieval interface — how a corpus is loaded, how a query is issued, what comes back, what the ranking parameters are.
  3. Embedding model alignment — a way to hold the embedding model constant, or an explicit declaration that it is a second uncontrolled variable.
  4. Cost and rate-limit figures so the benchmark run does not silently truncate.

These four prerequisites are listed in evals/systems/foundry.py. Until all four exist, the adapter raises SystemUnavailable and the rest of the harness runs without it.

What data is needed

  • Access to Foundry's retrieval API — direct API access or a batch export of candidate memories for a test corpus.
  • A test corpus — memories and queries with known correct answers. The reference dataset (reference_v1, 41 synthetic cases) is the fastest path; a real workload is the more informative one.
  • Permission to benchmark — written, not verbal.

Minimum evaluation time

2–3 days: day 1 for integration alignment, day 2 for dry run and control verification, day 3 for full benchmark and report. Assumes the interface is stable and documented.

What the result will and will not tell us

A Venkai win on context recall with comparable or fewer tokens would mean the selection pipeline adds value on top of Foundry's retrieval. A loss, or a win bought with significantly more tokens, would mean it does not.

Either way, the result is specific to this dataset, this budget, and these queries. Generalising from a single run is not supported by the data.