Evaluation
Test your model in the places it will work.
Avala evaluates your model privately, on held-out scenes from real homes and workplaces that match your deployment. Leakage control keeps those scenes out of training. People audit every score, and every failure starts the next round of data.
Held out
Your evaluation scenes stay private and out of training.
Matched to your deployment
We capture evaluation scenes in real homes and workplaces chosen to match where your robot will work, under the same scene standard as training data.
Private to your model
Your cohort is built for your model and scored privately. Its scenes, and the homes, objects and operators in them, are kept out of every training release.
Leakage control
Every sequence carries leakage groups for location, object, mechanism, operator and environment family. Before scoring, Avala checks your training releases against the versioned cohort by leakage group and video fingerprint, and records the result in the report.
Scoring
A model pre-scores. People audit.
Vision-language pre-score
A vision-language model scores each sequence against the task's subtasks and success criteria.
Human audit
Trained reviewers audit the scores. Where reviewers disagree, consensus decides.
Inter-rater agreement
Each report states how often reviewers agreed, so you can see how firm each score is.
A sequence with a sensor fault or missing calibration is marked invalid and left out of the metrics. It never counts as a model failure.
The report
What every evaluation report measures.
Success probability
The share of sequences that complete the task.
Partial progress
How far through the subtasks an unsuccessful sequence got.
Time to success
How long successful sequences take.
Human-relative throughput
Task speed compared with human experts doing the same task.
Intervention rate
How often a person had to step in.
Failure-mode distribution
Which failures occurred, grouped by the failure taxonomy.
Recovery rate
How often the model recovered from its own mistake.
Confidence intervals
Every metric carries an interval, so two model versions can be compared fairly.
After the report
Every failure starts the next round of data.
A failed sequence is where the work begins. Each one moves through the same four steps.
Gap report
Failures are grouped by failure mode and by the conditions they share, so the report names where the model breaks.
Collection brief
Each gap becomes a brief: the workflows, settings and variations to capture next.
Targeted data
Avala collects and labels the data the brief calls for, under the same coverage standard and leakage checks.
Re-evaluation
You train the next version. Avala scores it on the same held-out cohort, so you can see whether the gap closed.
Every scored sequence carries an outcome label: what happened, how far it got, whether it belongs to training or held-out evaluation, and its leakage groups. Read and filter the labels through the API, the Python and TypeScript SDKs and MCP.
Compare
Score two versions on the same cohort.
Run the current model and the candidate on the same held-out cohort. The report puts them side by side, metric by metric and failure mode by failure mode, so the decision to ship rests on evidence you can inspect.
Start
Set up your first evaluation cohort.
Tell us the tasks your model runs and where it will work. We propose a held-out cohort and the metrics to report.