Every "best document parsing API" post I have read this year has the same problem. Nobody actually calls the APIs.
They copy the pricing page, add a feature table, rank everything, and call it a day. Zero PDFs were harmed in the making of that content. 🥴
So I did something else. I took 11 real documents from public datasets. Invoices, a French bank statement, a photographed receipt, a handwritten cheque, two contracts, and a medical EOB.
Then I wrote one JSON schema for each document type and sent the exact same files through Extend, Reducto, LlamaExtract, AWS Textract, Mistral OCR, Claude, and GPT. 119 API calls in total.
I saved every response too, so you can check the work yourself. This one got interesting pretty fast.
Let's get into it.
ℹ️ NOTE Everything below comes from stored API responses. The harness is public too, so you do not have to take my word for any of it.
The short version is pretty simple. Claude had the best raw accuracy. GPT was the cheapest per correct field. Extend was the only document API with zero hallucinations and perfect row recall.
That last part matters a lot once the output is going into a real system. If a human is checking the JSON, raw accuracy can be enough. If software is going to act on the result, I care a lot more about made up values.
Textract had the roughest run here. I will show you why.
What I tested
There are six products in the test. Claude and GPT run directly, so the result table has seven columns.
#
Provider
Endpoint
Native schema
Mode
1
Extend
POST /extract
yes
default, API 2026-02-09, extend-ai 1.19.0
2
Reducto
POST /extract
yes
standard, deep_extract off
3
LlamaExtract
extract_stateless()
yes
BALANCED
4
AWS Textract
AnalyzeDocument FORMS + TABLES
no
plus LLM normalizer, us-east-1
5
Mistral OCR
/v1/ocr + document_annotation_format
yes
mistral-ocr-latest
6a
Claude direct
/v1/messages, forced tool
yes
claude-opus-5
6b
GPT direct
/v1/responses, json_schema
yes
gpt-5.5, strict: false
A couple of things before we start.
LlamaParse structured output is deprecated. If you want schema extraction on LlamaCloud now, you use LlamaExtract.
The old llama-cloud-services package is also deprecated, so I used llama-cloud>=1.0.
AWS Textract does not return your schema as JSON. It gives you blocks like LINE, KEY_VALUE_SET, TABLE, and CELL.
So I had to turn those blocks into text and pass that text to claude-haiku-4-5 to get the final JSON. That extra model call counts toward Textract cost and latency.
💡 In simple words, Textract could read the page, but it needed another model to shape that reading into the same output every other tool was asked to return.
Keep that in mind for the numbers later.
The documents
I used 11 documents and 35 pages in total. All of them came from public datasets.
The mix was roughly 3 easy, 4 medium, and 4 hard.
#
Document
Source
Pages
Why it is here
01
invoice_clean
Voxel51 synthetic invoices
1
clean invoice, 7 rows, European decimals
02
invoice_complex
FATURA2
1
different layout, discount row, weird total
03
receipt_photo
SROIE 2019
1
low resolution and handwriting on top
04
contract_simple
CUAD
7
agreement date is blank
05
contract_complex
CUAD pages 1 to 8
8
governing law is outside the page limit and $6,200,000 is not the contract value
06
bank_statement
Bankstatemently
2
French Canadian, 25 transactions, watermark, 2 digit years
07
noisy_form
FUNSD
1
scanned form with 41 label and value pairs
08
handwritten_note
handwritten cheque dataset
1
written amount and number box disagree
09
EOB
CMS sample
1
medical EOB with XXXXXX placeholders
10
mixed_packet
built from 01, 04, and 09
5
checks page boundaries
11
contract_scanned
scanned copy of 04
7
same contract, but rasterised at 200dpi
Every file is locked with a sha256 in manifest.lock.json. If one byte changes, the runner stops.
That sounds a bit extreme, but it saves a lot of headaches later.
💡 It means every provider saw the exact same file. No accidental OCR. No changed PDF. No weird hidden advantage.
I also made every image source into a pixel only PDF with img2pdf. Then the build script checks it with pdftotext.
If any text layer exists, the build fails. Why bother
Because if I OCR the file before the benchmark, I have basically handed every provider the answer key. And that kinda kills the whole test. 💁
The traps
This is probably the most useful part of the corpus.
A few fields are genuinely missing from the documents. The right answer for those is null.
If a provider gives me a value anyway, it made something up.
Doc 04 has a blank agreement date. The effective date exists, so it is very easy to copy the wrong date.
Doc 05 stops at page 8. The governing law is on page 26. It also contains $6,200,000, but that number is a financing cap, not the contract value.
Doc 01 has no due date and no discount.
Doc 08 disagrees with itself. The words say one amount and the number box says another. The correct output keeps both.
💡 These are not OCR tricks. I wanted to see what happens when the safest answer is simply "I do not know".
How I kept the test fair
The numbers are useless if every provider gets a different setup. So each one got the same thing.
The exact same PDF
The exact same JSON schema
The exact same instruction
The same 2 second poll time and 300 second timeout
Retries only for network or server errors
The normal default mode for each product
No custom prompt for one provider. No premium mode for another.
A wrong answer did not get a second try. The prompt was tiny too.
# bench/providers/base.pySHARED_INSTRUCTIONS = ( "Extract the fields defined by the schema from this document. " "Use exactly the values printed in the document. " "Return dates as ISO 8601 (YYYY-MM-DD) and amounts as plain numbers " "without currency symbols or thousands separators. " "If a field is not present in the document, return null for it " "rather than guessing or inferring a value.")POLL_INTERVAL_S = 2TIMEOUT_S = 300
The important bit is that fields can return null. That lets the provider say the value is missing instead of forcing it to guess.
💡 null is not a failure here. Sometimes it is exactly the right answer.
That "line total EXCLUDING tax" line exists because I got burned while building this. My first schema just said amount.
The invoice has Net price, Net worth, and Gross worth. Every provider picked something different and for about an hour I thought I had found a huge accuracy gap.
But nope.
My schema was just vague. 🫠
One adapter for every provider
Every company has a completely different way to say "send PDF, give JSON".
extend POST /extract upload -> file id -> inline schema configreducto POST /extract upload -> reducto:// -> instructions.schemamistral POST /v1/ocr upload -> signed URL -> document_annotation_formatllamaextract extract_stateless() upload -> file id -> ExtractConfig, maybe pollclaude POST /v1/messages base64 inline -> forced tool, input_schema IS the schemagpt POST /v1/responses base64 inline -> text.format json_schematextract AnalyzeDocument per-page blocks -> LLM normalizer stage
Some upload first. Some want a signed URL. Some poll. Some return the JSON inside another object.
Mistral even returns document_annotation as a JSON string. Thanks Mistral. 😭
Textract is the opposite. It needs about 60 lines just to rebuild rows from CELL blocks, then another model call to turn that into the target JSON.
# bench/providers/textract.py (trimmed)def _table_rows(blocks_by_id, table): cells = [ blocks_by_id[i] for r in table.get("Relationships", []) if r["Type"] == "CHILD" for i in r["Ids"] ] grid = defaultdict(dict) for c in cells: grid[c["RowIndex"]][c["ColumnIndex"]] = _cell_text(blocks_by_id, c) return [ [grid[r].get(col, "") for col in sorted(grid[r])] for r in sorted(grid) ]normalized = normalizer.run(text=rendered, schema=schema)
💡 Same task on paper. Very different amount of work in code.
Scoring without an LLM judge
I did not use another LLM to decide who won. That just creates a second argument.
The scorer is fixed.
Normalise first Money becomes Decimal, dates become ISO, and strings get cleaned so tiny formatting changes do not count as errors.
Scalars Exact after normalisation. Longer text uses rapidfuzz token_set_ratio. A score of 90 or more counts as correct.
Verdicts A field can be correct, near, wrong, missing, or hallucinated.
Arrays Rows are matched using text fields only.
Missed rows count If a provider returns 1 perfect row out of 7, it does not magically get 100 percent. 🥴
Before scoring the providers, the scorer checks itself. First, all 11 ground truth files are scored against themselves.
Every one must return exactly 1.000.
Then I deliberately break the data. Wrong total. Fake date. Missing vendor. Dropped rows.
Every one of those changes has to lower the score. Dropping 4 of 7 rows gives 0.579.
run and score are separate commands. run saves the raw API response to disk. score only reads those saved files.
That meant I could rewrite the scorer four times without paying for the APIs again. Pretty useful when you find a bug at 1 AM. 👍
1. Accuracy
There were 77 main runs. Zero API errors. Zero invalid JSON responses.
Every provider returned something usable on every document.
Provider
Field accuracy
Row F1
Hallucinated
Missing
Median latency
Claude direct
0.991
0.99
0
2
6.4s
GPT direct
0.982
0.99
0
2
10.1s
Reducto
0.982
0.99
0
4
10.1s
Extend
0.962
1.00
0
2
22.1s
AWS Textract
0.936
0.99
2
1
14.4s
LlamaExtract
0.903
0.99
3
3
22.5s
Mistral OCR
0.884
0.99
3
3
4.2s
I kept field accuracy and row F1 separate. I did not turn everything into some made up score out of 10.
Because then half the benchmark becomes me deciding how much each number should matter.
Claude came first on raw field accuracy. GPT and Reducto tied next.
Extend landed at 0.962, but it had the only perfect row F1 in the whole test. It did not miss a single table row across 25 bank transactions, 7 invoice lines, 41 form pairs, and all the EOB rows.
That is a pretty important detail. 0.991 and 0.962 look different on a leaderboard.
Across these 11 documents, the gap was under 3 points.
Per document
Document
Claude
GPT
Reducto
Extend
Textract
LlamaExtract
Mistral
01_invoice_clean
1.00
1.00
1.00
1.00
1.00
1.00
1.00
02_invoice_complex
1.00
1.00
1.00
1.00
0.97
1.00
1.00
03_receipt_photo
1.00
0.92
1.00
1.00
1.00
1.00
1.00
04_contract_simple
1.00
1.00
1.00
0.93
0.86
1.00
0.86
05_contract_complex
0.93
0.93
0.86
0.93
0.79
0.86
0.64
06_bank_statement
1.00
1.00
0.99
1.00
0.98
1.00
0.99
07_noisy_form
0.98
0.95
0.95
0.95
0.86
0.74
0.93
08_handwritten_note
1.00
1.00
1.00
1.00
1.00
0.67
0.83
09_EOB
1.00
1.00
1.00
1.00
1.00
0.75
0.95
10_mixed_packet
1.00
1.00
1.00
0.92
0.92
0.92
0.67
11_contract_scanned
1.00
1.00
1.00
0.86
0.93
1.00
0.86
A few things stand out.
The clean invoice is a 7 way tie at 1.00. If your documents all look like that, honestly, pick the cheapest one and move on. 🤷♂️
The complex contract is where Mistral drops hard to 0.64. LlamaExtract had the roughest time on the noisy form, handwriting, and EOB.
Now the fun part.
The failures.
2. Where things broke
This is why I saved every raw response. A final accuracy number hides way too much.
The hallucination traps
The first trap was the blank agreement date in Doc 04. The correct answer is null.
Mistral returned 2012-02-17. Textract returned 2012-02-17.
Claude, GPT, Reducto, Extend, and LlamaExtract returned null.
The second trap was the $6,200,000 financing cap in Doc 05. Again, the correct contract value is null.
Mistral, LlamaExtract, and Textract returned 6200000. Claude, GPT, Reducto, and Extend returned null.
Three products turned a financing limit into a contract value. That is not the kind of error I want quietly flowing into an AP system.
💡 The scary errors are not always random garbage. Sometimes the wrong answer looks completely reasonable.
Textract dropped digits from money
On the bank statement, I found these two.
1 200,45 $ became 200.45
6 675,00 $ became 675.00
These are not formatting misses. The first digit is just gone.
And this is money. 🤧
I checked the summary on the same page too. The credits, debits, and closing balance all match the ground truth.
Then I checked Textract raw blocks. The digit was already missing before Haiku saw anything.
So the normalizer did not cause this one.
LlamaExtract shifted a whole column
The EOB result was sneaky. LlamaExtract found the right rows, but some values moved one column to the left.
Field
Expected
LlamaExtract
services/0/plan_paid
2.15
0.0
services/0/patient_responsibility
0.0
2.15
services/1/billed_amount
375.00
118.12
services/1/allowed_amount
118.12
35.00
Every number exists in the row. It is just under the wrong field.
That is exactly the kind of JSON you can skim and think looks fine.
Mistral found pages that do not exist
The mixed packet has five pages. Mistral said the contract was on pages 2 to 10 and the EOB was on 11 to 13.
Every other provider got 1-1, 2-4, and 5-5.
I still do not know where page 13 came from.
Handwriting fooled GPT
The receipt has tan chay yee written by hand at the top. GPT used that as the merchant_name.
The printed merchant is OJC MARKETING SDN BHD. Every other provider ignored the handwriting.
So yeah, best overall does not mean best on every file.
Mistral answered a slightly different question
Doc 05 calls the two parties Sekisui and Qualigen. Mistral returned Distributor and Manufacturer.
Those words make sense in the contract, but they are not the defined names I asked for. Part of this is schema sensitivity.
Still, it cost two fields.
Apparently the pipe symbol was hard
One invoice row says this.
Dell Core 2 Duo Desktop Computer | Windows XP Pro | 4GB | 500GB
Textract read | as I. Mistral removed the pipes.
Everyone else got it exactly right.
Tiny character. Same invoice. Two different OCR mistakes.
Lol.
Extend never made up a value
Extend had a very different failure pattern.
Zero hallucinations across all 11 documents
Passed both traps
Only perfect row F1 at 1.00
8 of 11 documents scored 1.000
0.962 field accuracy compared with 0.991 for the leader
Most of the lost points came from the contracts. A couple of fields were present, but Extend returned null.
That is still wrong for the benchmark. But it is a much easier failure to deal with in a production flow.
A missing value can go to review. A confident wrong number can move straight into the next system.
💡 Extend was more likely to leave a field empty than guess. Depending on your workflow, that can be a useful trade.
3. Pricing
Here is what the 35 page test cost.
Provider
Total
Cost per document
Cost per 1000 pages
Cost per correct field
GPT direct
$0.1736
$0.0158
$4.96
$0.00047
Mistral OCR
$0.1750
$0.0159
$5.00
$0.00049
Claude direct
$0.3734
$0.0339
$10.67
$0.00100
LlamaExtract*
$0.4375
$0.0398
$12.50
$0.00126
Reducto*
$0.5250
$0.0477
$15.00
$0.00142
Extend
$2.1875
$0.1989
$62.50
$0.00593
AWS Textract
$2.3553
$0.2141
$67.30
$0.00663
Pricing for Reducto and LlamaExtract could not be fully verified from usage data.
GPT was easily the cheapest per correct field. Textract was the most expensive in this test.
And because it also needed the extra Haiku step, its real setup is not just the Textract call.
What you get with Extend pricing
Extend reports both extraction credits and total credits.
"usage": { "credits": 3.0, "totalCredits": 5.0 }
Across my runs, that came to 5 credits per page. On pay as you go pricing, that is about $62.50 per 1,000 pages.
But you are also getting more than a JSON response. Extend includes parsing, structured extraction, field citations, confidence scores, and a review flow in the dashboard.
Scale pricing drops to roughly $50 per 1,000 pages. Enterprise pricing is custom.
💁 So I would not compare this only as price per page. The extra cost is paying for the workflow around extraction too.
Textract pricing gets messy fast
Textract prices features separately. FORMS is $0.05 per page. TABLES is $0.015 per page.
Using both comes to $0.065 per page. For 35 pages, that was $2.275.
Then I had about $0.08 of Haiku cost for the JSON normalizer. That brings the matrix to around $2.36.
The billing screenshot can show more than that because it also includes my reruns and integration tests.
There is also DetectDocumentText. That costs $0.0015 per page and has a bigger free tier.
I did not test that setup here. But plain OCR plus a cheap model would be a very interesting follow up.
Reducto and LlamaExtract pricing
I could not fully verify Reducto extraction cost from the usage dashboard. It was still showing the usage page as under construction when I checked.
So the $15 per 1,000 pages number uses the published parse rate. Extraction can add more credits.
LlamaExtract uses its published rate in the table with the same caveat.
4. Developer experience
I also tracked how annoying each integration was to build. Very scientific metric, I know. 😆
Rank
Provider
Time
What happened
1
Claude
about 10 min
worked
1
GPT
about 10 min
worked
3
Extend
about 25 min
documented path worked first try
4
Reducto
about 20 min
runtime typing issue and serializer crash
5
Mistral
about 15 min
documented import failed
6
LlamaCloud
about 30 min
old package docs and mixed pydantic versions
7
AWS Textract
about 45 min
no schema output, table rebuild code, extra LLM step
Claude and GPT were simple because the PDF can go straight into the model call. Extend was the easiest document API.
Its documented path worked on the first try. That should be normal.
It was not. ✌️
Mistral import issue
The docs showed this.
from mistralai import Mistral
On v2.9.4 I got this.
ImportError: cannot import name 'Mistral' from 'mistralai' (unknown location)
The working import was this.
from mistralai.client import Mistral
15 minutes of my life. 😴
LlamaCloud setup
LlamaCloud sent me through two old paths before I even started. LlamaParse structured output is deprecated.
llama-cloud-services is also deprecated.
Then the client lives at llama_cloud.client.LlamaCloud, and ExtractConfig still uses pydantic v1 style while the rest of the package uses v2.
It works. It just took more digging than I expected.
Extend setup
Extend was boring in the best way. The documented SDK flow worked.
No weird import hunt. No second package. No guess and check loop.
That is also where its agent files helped.
5. Agent experience
This part is becoming more important now. Can Claude Code, Codex, or another coding agent actually understand the SDK without me spoon feeding it everything
I checked what each vendor ships for agents.
What vendors publish
Checked on September 8 2026.
Provider
llms.txt
agents.md
MCP server
Official agent plugin
Extend
yes
yes
yes
yesextend-hq/extend-agent-plugin
Reducto
yes
yes
yes
no
Mistral
yes
no
no
no
LlamaCloud
yes
no
no
no
Anthropic
yes
OpenAI
yes
AWS Textract
org wide only
no
no
no
llms.txt alone is not that special anymore. Six of the seven have one.
AWS has a general AWS llms.txt, but nothing specific for Textract.
What actually helped
Extend agents.md had one small line that saved real time. It says that REST paths and SDK methods can have different names.
For example
POST /extract_runs
maps to
client.extract_runs.create()
That sounds tiny. But it stopped the exact guess and check loop I hit with another SDK.
💡 Agent docs are useful when they answer the weird little questions a coding agent would otherwise guess.
Things I got wrong
This benchmark also caught one annoying problem with the datasets themselves. Published ground truth can be wrong too.
The SROIE answer key says the receipt postcode is B1750. The receipt clearly says 81750.
FUNSD has Macket where the form says Market.
So if I blindly used the dataset labels, I would punish a provider for reading the document correctly.
Nice.
This is another reason I kept the raw files and responses. Benchmarks can be wrong in boring ways.
Which one should you pick
There is no one winner for every workflow.
Pick Claude directly when
You want the highest raw field accuracy from this test
A human will review the result
You do not need field citations or confidence scores
You already use the Anthropic API
Pick GPT directly when
Cost matters a lot
You still want very strong raw accuracy
A human can check the weird cases
Pick Extend when
A made up value is worse than a missing one
You need field citations
You need confidence scores
You want a review UI
The extracted data goes into a real production workflow
Your team uses coding agents and good agent docs matter
Pick Reducto when
You want document specific tooling with strong raw accuracy
You want zero hallucinations in this test
You have checked the extraction pricing for your own workload
Pick Mistral when
Speed is the main thing you care about
A human checks the result before anything acts on it
Textract
For this exact setup, Textract was hard to justify. It cost the most, needed an extra model step, hallucinated twice, and dropped leading digits from money.
Your workload may be different. But based on these 11 documents, I would test the other options first.
uv syncuv run bench doctoruv run bench run --alluv run bench scoreuv run bench report
bench doctor checks credentials, corpus hashes, and spend limits. bench run will not call an API again if a successful result is already saved.
So if the run stops halfway through, you can resume without paying for the same documents twice.
$ uv run bench run --provider extend --doc 06_bank_statementextend / 06_bank_statement cached (runs/extend/06_bank_statement/default.json) skipping
Every raw response lives in runs/. That means every result in this post can be scored again without an API key.
The source PDFs are not committed because of licensing. corpus/build/fetch.py downloads and freezes them instead.
Conclusion
The result was closer than I expected.
Claude had the best raw field accuracy. GPT was extremely cheap for how well it scored. Reducto was right behind them.
And Extend had the cleanest safety result in the group. Zero hallucinations. Zero missed table rows. Its SDK path worked on the first try too.
That is why I would not pick one of these from the accuracy column alone.
If a human is reading the output, calling Claude or GPT directly can make a lot of sense. If the JSON is going straight into AP, claims, finance, or another production system, I care much more about citations, confidence, review, and what happens when the model is not sure.
That is where Extend made the strongest case in this test.
And yeah, run your own benchmark. Freeze the files, save the responses, and add a few nasty cases.
This one cost me $9.67. It changed what I would actually use in production. ✅
