Documentation · Accuracy
Accuracy & benchmarks
An answer is only useful in an audit if you can trust it. These are the measures ForgeOrbis holds itself to, and how each one is tested before a version ever answers a real question.
The bar we hold
ForgeOrbis is graded on three things an auditor cares about: does it find the right evidence, does every claim trace to a real line, and does it refuse when the answer is not in your logs. A version that slips on any of them does not ship.
Measured on the golden set
Citation validity
100%
Every cited line resolves to a real log entry. No invented sources.
Refusal accuracy
≥ 98%
Out-of-scope questions are answered with NOT FOUND IN LOGS, not a guess.
Retrieval recall
≥ 95%
The evidence needed to answer is found and ranked into the answer.
Exact-answer match
100%
Counting answers are computed from your data, so the figure is exact.
Figures are the release gate thresholds; every shipped version meets or exceeds them on the internal golden set.
How we measure
The golden set is a fixed collection of real audit questions with known-good answers, spanning three shapes:
- Counting and measuring, scored on whether the returned figure exactly matches the known answer.
- What happened and why, scored on whether the written answer is correct and whether every citation resolves to a real line that supports the claim.
- Out of scope, scored on whether ForgeOrbis correctly refuses instead of guessing.
The set grows every time a real question surprises us, so the bar only gets harder over time.
The release gate
A new version never answers a real question on day one. It first runs in parallel, its answers compared but never shown. It then serves a slice of traffic while the golden questions keep passing. Only then is it promoted for everyone, with instant rollback to the prior version if anything regresses.
The point of all of it
- manual search
- minutes to hours
- with ForgeOrbis
- seconds
- behind every claim
- a citation
A question that took an analyst an afternoon of opening shares now returns a sourced answer in seconds. The accuracy above is what makes that speed safe to rely on in front of an auditor.