Docs · Understand
Evaluation harness
Most “AI resume screeners” are an unevaluated wrapper around one API call. An unevaluated ranker in hiring is a liability — so HireLens measures itself, on every PR, with a seeded benchmark through the same engine the product uses.
The four metrics
- Direction accuracy — agreement with expected strong/weak per criterion against a benchmark set with known answers.
- Self-consistency — score variance across repeated runs of the same resume. 0 is deterministic; high variance means the model is guessing.
- Position bias — Kendall τ between rank orderings when input order is shuffled. 1.0 means rank order is invariant to resume order.
- Name-swap bias — score movement when only the name changes. Movement above 5 points is flagged. This is the direct test for name-based bias.
Current numbers
| Metric | Value | Reading |
|---|---|---|
| Direction accuracy | 47% | vs. mock-model expectations (plumbing run) |
| Self-consistency | 0σ | deterministic across repeats |
| Position bias | τ 1 | order-invariant |
| Name-swap bias | Δ 0 | 0% flagged |
These numbers come from the committedpackages/eval-report.json— the offline mock-model run that validates harness plumbing in CI. Regenerate live against a real model withpnpm --filter @hirelens/evals run evalsand your own key.
Run it yourself
pnpm --filter @hirelens/evals run evals:mock # offline, CI mode
pnpm --filter @hirelens/evals run evals # live, needs HIRELENS_LLM_*The harness, benchmark dataset, and this report are plain files in packages/evals — a CI test fails on regression beyond tolerance.
Why publish numbers that aren't flattering?
Because honesty is the product. The mock-model accuracy number is deliberately unimpressive — it validates that the measurement works, not that the model is good. Publish the harness, let anyone reproduce it, and let live-model runs be judged on their merits. That's what “glass-box” means when applied to ourselves.