Extraction accuracy
Replyz reads a received email and returns the one-time code and the main verification, sign-in or reset link (Get a verification code). This page says how well it does that, how we measured it, and where it goes wrong.
Result: 91.7 % (44 of 48) on a 48-mail blind test sealed before the fixes existed, measured on 18 November 2026, extractor 98096d8. 44 of 48 emails were answered correctly. This is above our own target of 90 %, but it is one run on 48 emails, and the hard emails in it still produced 4 misses. Extraction is a heuristic: check the result before you rely on it. If the extractor changes, this number no longer applies until a new blind set measures it.
What is measured
Section titled “What is measured”The measure is called get_verification. It is the name of the score in our test, not an API field or an endpoint. An email counts as right when both parts match the expected answer:
- the code is the expected one, or the email has no code and none is returned;
- the main link is the expected one, or the email has no such link and none is returned.
If either part is wrong, the email counts as wrong. In the API, the same work is extract=otp,link on wait or POST /v1/messages/{id}/extract.
How the test was made
Section titled “How the test was made”- 48 emails, written by three separate AI agents. They were given only the format of the test, not the extractor’s source code, its tests, the examples we tuned it on or the kinds of mistake found in earlier tests, and they did not run code.
- 16 English transactional emails.
- 16 non-English emails.
- 16 English emails built to be hard.
- By kind: 31 with a code, 19 with a link, and 6 with no action.
- Blind and scored once. The emails were sealed and kept apart from the code before the fixes for the earlier mistakes existed, were never used to tune the extractor, and were scored once, with the same scorer for every email, on the extractor measured here (build
98096d8). An independent re-run of the scoring gave the same numbers. - The test set is now used up. We know which emails failed, so it can no longer measure the extractor blind. The next number needs a new set, written the same way.
- The test emails and the per-email results are not published, because they contain email content.
Where it goes wrong
Section titled “Where it goes wrong”4 of the 48 emails were answered wrongly. All four are the same kind of mistake: a code that was not found. None returned a wrong or invented code. Kinds, with no email content:
| Kind of mistake | Emails |
|---|---|
| Code not found: one digit per table cell | 1 |
| Code not found: a large bare number with no word like “code” near it, among other numbers | 1 |
| Code not found: a mixed-case letters-and-digits “security key” | 1 |
| Code not found: a fresh code above a quoted, expired code and an old link | 1 |
Missing a code is easier for an agent to handle than a wrong one, because the agent does not type it. Earlier versions of the extractor also returned wrong or invented codes in some emails; on this test none did.
What this means when you use it
Section titled “What this means when you use it”- Treat
otpas the most likely code, not a certainty.otp_candidateslists up to 4 other plausible codes. - A result of
nullmeans “no likely code found”, not “no code in the email”. The full message is available to read (Reading mail). - A code or link says what the email contains, not that the email is genuine. Check
sender_trustand treat the content as untrusted (Untrusted content and safety). - Extraction only reads the stored subject and text. It never opens a link.