I've got a scan from a book that I OCR with new releases. Ligatures, critical sigla, Fraktur letterforms, subscripts, superscripts, etc.
Nothing special about this model for overly-detailed work like mine.
It's been a while since I last tested (and discontinued my subscription), but the "pro" models from OpenAI dominate. Not surprising, given the price difference, but it would be nice if an OCR-specific model could perform better. It's worth mentioning that even the highest-end models do a pretty poor job with intricate text like mine.
While I haven't tried OpenAI for OCR, I've put my small scale OCR work through both Claude and Mistral OCR. Claude is absolutely better - even in OCR work I did last week and compared with Mistral OCR 4.0.
Mistral's one advantage is that Anthropic now flags OCR, because they don't allow anything that could be considered "reproduction", even of work for which you own the copyright. So my new workflow is Mistral OCR for the actual OCR, followed by a proofreading pass by Claude (which is allowed). Claude is obviously more expensive, but it caught entirely hallucinated sentences created by Mistral OCR 4.0, so I was glad for the backup check.
Same company that OCRed millions of books, the irony.
I feel Anthropic is destroying itself with all these restriction. They got away because their models were the best for coding, but that is not an advantage anymore as OpenAI and other open source are already better.
Claude refuses to translate lyrics of songs even if it's just for your own understanding. ChatGPT allows it.
Also, Suno is so restrictive that it won't make songs of public domain 19th century poems because someone else already made a song and the lyrics are in their song database. And it won't do covers of your own uploaded humming of a novel melody because it matches "humming in an empty room" to some song fingerprint in the database.
They are all trying to avoid these lawsuits by being ridiculously overly cautious. "Your honor, look how much we went above and beyond, even when it made the product worse for legitimate use cases"
Yep, my understanding is that many guardrails like this are actually the result of government legislation (eg the Fable bans) or terms of settling copyright lawsuits over reproducing copyrighted text and lyrics.
I mentioned it in a sibling reply, but here's Anthropic's support document about not using Claude to reproduce content verbatim that already exists, regardless of copyright.
> Claude is obviously more expensive, but it caught entirely hallucinated sentences created by Mistral OCR 4.0, so I was glad for the backup check.
What does this entail? What does Claude do to decide that the text it was provided was hallucinated? Are you telling Claude that the source was OCR'd by another LLM?
I'm basically doing the OCR twice, except in the Claude proofreading pass, it is not being asked to transcribe the document to a Markdown file. I'm pointing it to the same image input files, and to the Mistral OCR transcript Markdown file (it knows it's a Mistral OCR output), and ask Claude to check that the text is correct and point out the errors - and then make the necessary edits.
I can't speak for Mistral OCR 4.1, but the hallucinations in 4.0 were so egregious (just completely making up new sentences in the middle of a page) that I knew I can't trust Mistral OCR on its own.
How bad is the first OCR pass allowed to be to still count as proofreading? Can you let Claude compare the images with /dev/random and make the necessary edits to correct differences?
Hmm, that's an interesting idea. But it's the classifier that is triggering, and it triggers specifically on Claude's output. So I think the /dev/random case wouldn't work, because that gets Claude into the state of just reproducing the entire text from the original again.
It doesn't always get flagged. Single pages are almost always okay. Running a program that sequentially runs single pages through the API is often not okay - I wrote a program in the early 4.x days before the rule came in, that's how I hit it first. But I've also had entire articles go through just fine recently in a Claude Code session (I'd forgotten about Anthropic's rules!), and then others where I get classifier errors by page 4.
The Mistral OCR errors were small in size. Single sentences, formatting errors, paragraphs with newlines. So this was a genuine proofreading job with small changes. For the most part Mistral is actually good, but I can't have it just inventing sentences in the middle of a document. That's where the Claude proofreading pass was most helpful.
I read that as “hallucinated” in the practical sense: the OCR output contained text that wasn't actually present in the scan, rather than just misreading a character.
Yes, that's how I read it as well, but the question remains in how did Claude know this? It was not clear to me that Claude was being presented the original scan. The entire exercise is farcical, but it wasn't as clear exactly how farcical it was that the person was essentially running OCR wastefully on a different model just to convince Claude to do the OCR as desired because of Anthropic's rules. Here's this cool tool that is supposed to make you do less work and be able to do it faster and be able to do it more accurately, except you first have to do all of this other unnecessary work with a totally different company first. <face-palm>
Not the person you’re responding to, but I’ve had Claude refuse to OCR pages from in-copyright books. I was sometimes (but not always) able to get around that by changing models, by telling it that I was doing the text conversion only for personal use, or by first telling it to use Tesseract or another OCR engine to do the initial pass and then having a Claude subagent proofread and clean up the OCR output.
I’ve also had it refuse to OCR public-domain books that included content that it didn’t like, such as references to prostitution in 19th-century books about Japan.
I had one session where Claude refused to continue after it hit some kind of guiderail restriction. I couldn’t see what the trigger was, so I started a new session, gave Claude the link to the previous session, and asked it to diagnose the problem. This new Claude said it couldn’t view the exact guardrail issue, but it did suggest a workaround that turned out to be effective.
Are you familiar with the Thesaurus Linguae Graecae? It's a project that may meet your needs. Paywalled, unfortunately, but I assume it would be a one-time expense.
This is too funny considering they did that themselves. I’m pretty tired of these companies deciding what we can and can’t do while they act with impunity.
"it refuse to OCR public-domain books that included content that it didn’t like, such as references to prostitution in 19th-century books about Japan."
Thoughtcrime -like territory and self-sensorship. The AI safety lobby is such a vile influence on the freedom of expression and communication via technology (since AI is starting to eat up rest of technology).
I guess the main problem is positioning AI tools as "human-equivalent" creators by the big AI corps. If they were positioned simply as "better OCR and proofreading" people would attribute to them as much responsibility as they would to a - say - typewriter and we would not need to have this nonsense.
I do realize most of the valuation comes from the positioning of "our TAM is the global salary base of 50 trilion and we aim to supesede human workers in the near future" which implies they need to position this technology as "human equivalent" or that valuation is no longer as credible.
I also saw this when I made a (personal use only) book translation (agent learns how book 1-3 of a row is translated, then translates book 4 the same way because it is not available in my language).
Claude happily translated most of the book, except a few chapters which it denied. Tried several times always the same results (with no other context about the rest of the book).
The response / conversation gets blocked by the guardrails with "API Error: 400 Output blocked by content filtering policy"
At first I thought it was something in the scanned content that was being flagged, but it was the attempt to transcribe that was itself being flagged. Anthropic mention it on their pages:
"Anthropic takes these steps because Claude’s purpose is to generate new content and ideas, not to reproduce content that already exists."
Side note - Claude itself is not aware of this policy, and is unable to see the API responses - the turn just ends. Which turned into a really bizarre failure state where Claude thought I was gaslighting it and kept insisting it could do the work and even had the entire text in memory. Every time it would go to show me and prove it, it would hit API Error 400. I was only able to convince Claude by showing screenshots of my Claude Code screen output so it could see that I was seeing API errors. I've never seen Claude get into that angry & snarky state before, and I hope it doesn't happen again.
I have been quite happy with Mistral OCR for the documents I needed to process (typeset, but old, with questionable scan quality, sometimes elaborate typesetting or, much worse, typewriter-and-handwriting approximations of it). I do not test every new model when they are released, but I did a review shortly after Mistral OCR 3 was released and it was a very good compromise: cheap, fast, and good results without further processing. I found generalist models to be way too much faf to get them to avoid unnecessary modifications to the text and report accurate bounding boxes for figures and tables.
That said, models have sometimes surprising weaknesses and a model could be terrible overall but magically work for one type of document.
I got the opposite experience very recently : tried to OCR a bunch of handwritten emails addresses with chatGPT and I had to make so many corrections that I gave up. Whereas Mistral nailed it on first pass.
> the "pro" models from OpenAI dominate. Not surprising considering the price difference, but it would ne nice if an OCR-specific model could do better.
I haven't been impressed with any of Mistral's models. They obviously realized that they couldn't compete at the frontier so they decided to go for smaller focused models but even those have not been that good.
We moved away from Cursor but I was looking for a model that would help with FIM (fill-in-middle) multiline autocompletion and people were recommending Mistral's Codestral. We gave it a shot and it was lackluster at best.. Even Google's Gemini did a significantly better job than Codestral.
Ultimately Opus-class models got good enough and I don't do much manual coding anymore.
Yet: how is pricing?
Evaluating contents and routing appropriately isn't a new challenge in OCR, one of the oldest fields of applications in ML.
Thus, how do the smaller open models perform in tandem with relatively pricy $/pg models & APIs?
Your use case is remarkably rare relative to the volume and price sensitivity of enterprise data warehouse ops.
The big difference is traditional OCR used basic pattern matching to find text, whereas models like Mistral OCR (and GPT, etc) use computer vision instead and deep learning to parse text, math equations, and apparently in some cases extract images too.
I'd love to see some advancements in traditional OCR based on ideas and concepts we've learned from newer "OCR-like" models since traditional OCR is drastically cheaper.
I made a benchmark for handwriting recognition for a project while keeping line breaks and errors (grammar, spelling). Sonnet absolutely dominates it since a good half a year. 5.6 did not change that for me. This should also translate to better ocr.
You're the second person to mention handwriting. I think it might be handwriting-specific; perhaps Anthropic has a better corpus for this.
I really wouldn't know, though. Anthropic models barf out copyright issues for my use case, so I'm unable even to benchmark them. It's a common problem when you're scanning public domain books. Mine are reference texts often cited.
So is Mistral OCR the best one? Have any other OCR models caught some of what you describe? I've been kind of interested in how "OCR" type models work compared to old school OCR.
My use case isn't in the realm of old-school OCR, so it's not a good comparison, but anyway:
As another user pointed out, it's surprisingly random (task-specific). Llama Scout outperformed Gemini Flash 2.5 on a benchmark I built at the time. I didn't include an OCR models.
Mistral might indeed be the best OCR-specific model for my task, now that you ask. Funny. It's so bad at my work that I didn't register it might be the best in its category. This is just based on vibes from my single scan.
It's very common. PaddleOCR is enough to get extremely well-done bounding boxes, and it runs very fast on a $150.00 GPU.
There's always room for improvement, though. I suspect a tool will emerge for highly detailed OCR that implements a nested bounding-box-based multi-scale approach, effectively OCRing small sections at a time and then gradually compiling them by expanding the surface area using the bounding boxes.
I've thought a lot about implementing it anyway.
edit: I see you're asking about the block labels. Leaving the comment in case someone finds it interesting.
At this point I lost all hope for Europe playing any significant role in the AI race. If that’s a good or a bad thing I don’t know, but it seems to me like that’s the reality.
Commoditization is a beautiful thing. It seems Anthropic and OpenAI are really struggling to maintain much of a moat. Mistral might not be leading but it's not trailing by that much either. And of course the Chinese are doing their own thing quite successfully.
The reality is that the US is betting its economy on data centers at great expense and is exposing its economy to great risk.
Also while geographically a lot of the money and processing power is in the US, the US has been relying on immigration to power its universities and especially AI research has roots all over the globe. India, China, Russia, Europe, etc. AI related know how is finding its way back to all these places.
So, I'm not too worried about the long term here. It will be interesting to see if Anthropic and OpenAI survive their IPOs. Seems like a risky financial bet at this point given the apparent lack of a moat. But if it works out, it will result in a lot of that IPO money being invested in data centers abroad. Including in the EU. Because data residency is a thing here and the EU is too big of a market for companies with that kind of valuation to ignore. We also produce a lot of energy infrastructure (e.g. gas and wind turbines). Those data centers will need lots of power.
Nothing to worry about for europe. Open source and open weights will enable europe to pick up the slck when the party is over. Then the US markets will possibly regret the AI-bubble.
Much like spaceflight, aerospace or nuclear engineering, you need to retain local talent for national defense purposes. Being 70% as good is still way better than being 100% dependent and heavily leveraged by your opponents.
It gets a little weird when you start thinking of AI like that, because AI is very corporately based in the US and China, but not as heavily that when from a talent position.
Demis Hassabis and Google's Deep Mind are all British in terms of location/talent, but they certainly aren't from a coorporate perspective (Deep Mind work might happen in London, but it is owned by Google, as US company)
(I don't know a lot about nuclear/aerospace industries, I'm guessing there more location centered just in virtue of being based around physical things, but I could be wrong)
If your employees are only 70% as good, as skilled, as knowledgeable while not knowing it, then you can absolutely kill a reactor just like Chernobyl. The overconfidence is what allowed them to put the reactor in the state they did and not realize the implications.
Far worse than 70%. Chernobyl was a catastrophically flawed reactor design. Operators ran a badly planned safety test in which operators intentionally caused a dangerous state while key protections were disabled or bypassed.
? It absolutely is a race. Whether thats a positive thing or not is debatable but every lab is definitely in a race. What prize do you win? Imagine a world where only one country has AGI/ASI. Or a world where Europe only gets access to frontier models 6 months later. Far from ideal.
> What prize do you win? Imagine a world where only one country has AGI/ASI.
"Winning the race" doesn't give you that in any meaningful capacity. It gives you, at the absolute most, a temporary window where that's the case. See: nuclear weapons.
Citing nuclear weapons isn't the flex you think it is. There are only 9 countries that have nuclear weapons and they absolutely flex this power over non-nuclear powers (see Ukraine, Germany, SE Asia etc..)
Europe already has an innovation problem that's already causing structural economic instabilities which Germany has been struggling (and lately failing) to prop up.
As much as it pains me to say this, AI is already a tech revolution and it seems like Europe is just ignoring it. There's more innovation in 3 blocks in downtown San Francisco than the entire continent of Europe.
> There are only 9 countries that have nuclear weapons
QED being first didn't grant exclusivity.
The premise wasn't that AGI isn't useful. There are actually layers to the metaphor where first-mover advantage of AGI is even less meaningful than it was for nuclear weapons, but I leave those as an exercise to the reader to discover.
Lots of tech money is demonstrably wrong, at least in the general case. Microsoft's whole business model is to be the second mover. Patents exist because first mover advantage alone isn't enough to hold onto a win.
If AI is different, the onus is on AI folk to explain why, not the reverse.
The frontier labs are racing as hard as they can right now because they know as well as anyone else that they're at best 6 months, probably closer to 3 months, ahead of the generic commodity AI market. The majority of the tech money currently boosting them is a bet that either they can stay that far ahead without tripping, or that a moat can be engineered. It's not at all clear that it can - or, if it can, that it will before the money runs out.
> You should at least try to explain your contrarian position
FWIW I explained it fully and provided the argument, along with explicit grounding examples. Can you tell me what exactly you don't understand?
To quote your other post:
> This is the weakest part of the argument. Absolutely no reason to believe the second inventor will be 10x faster.
Nor is there any reason to believe that it will work at any usable speed. An LLM capable of AGI running at 0.000001tok/s isn't a very good head start. Nor is it a good moat if it can't actually realize anything material fast enough. Lord knows America has labor problems.
I'm demonstrating the context is a lot more complicated than "be first". There are innumerable specific examples. They're not guaranteed to happen, that's not the point being made.
Also "no reason" is curious when it's the explicit pattern this exact industry has exhibited. Chinese models lag a few months, and come in swinging with an order of magnitude more efficiency. Does it apply? Who can say. It's certainly on the table. Pretending like it's any more ridiculous than AGI itself is irrational.
> Does this ever happen? Even in traditional manufacturing, the second “inventor” starts behind and has to improve their own process to surpass the first.
To use the industry itself once again: Japan was the first to deep learning. Performance issues prevented them from capitalizing on it. Now they're barely even a player on the field. Interestingly and adjacent to the industry, Japan was second to symbolic AI, and the FGCS was lightyears ahead of America's crufty Lisp ecosystem. For a more modern example, Google was the first to transformers. Much of this revolution is thanks to them. A shame that Gemini is third-rate at best.
It's not unique to this industry. That second inventor more often than not does a better job than the original one. It's a pretty common pattern. Otherwise, we wouldn't need patents.
My "contrarian position" is anything but. There is nothing new under the sun. You don't just need to be first, you need to maintain exclusivity.
A reductive equation, economics isn't thermodynamics. Money is fictional and value is subjective and unstable. Within this context, being first to AGI means nothing if the second invention of it comes 2 months later and works an order of magnitude faster than what the first iteration had self-improved to at that point in time. First mover advantage isn't decisive, you have to actually be able to capitalize on it in a robust way.
This is the weakest part of the argument. Absolutely no reason to believe the second inventor will be 10x faster.
Does this ever happen? Even in traditional manufacturing, the second “inventor” starts behind and has to improve their own process to surpass the first.
> Imagine a world where only one country has AGI/ASI.
You've fallen for the hype. The only way these AI companies can justify their bullshit worth and burn of capital is by promising a magical AGI. It's just marketing and it is still unclear if LLM's will ever be profitable.
If they won't - then EU is doing the right thing laying low.
> Or a world where Europe only gets access to frontier models 6 months later.
> If they won't - then EU is doing the right thing laying low.
Exactly. People seem to forget that US is burning money at an accelerating rate without any promise of returns. This is a high risk situation. Time will tell.
This presupposes AGI or ASI are a real, reachable thing. My read is they might be, but not as LLMs. Until there's a fundamental rearchitecture, I'm AGI-agnostic and given that view, it doesn't seem rational to bet the house on it.
True, but that isn't EU/US/China, it's OpenAI/Anthropic/Grok/ …/DeepMind (based in UK)/… DeepSeek
With a lot of Chinese nationals in American companies, and an American corporation owning DeepMind, and a lot of people very upset with all of them at the same time, this is very messy.
The non-doom scenarios include "utopia for all", and "power flows to investors, not citizens of whichever nation the winning model's corp. was registered in".
Independently, "oh look all the investors went bankrupt" can happen in both "doom" and "normal technology" timelines.
And US lost the significant role in chip/pc manufacturing. But if a product becomes commodity or utility (which at least for now it seems is the direction), with little lockin, it's not a big deal.
I hope we (EU) don't waste money trying to train local models (which at least some people in Poland try to do), and tries to build our own chips - AI chips have different architecture than regular processor/GPU, and TSMC doesn't need to be winner in this new race.
And if not this, then smaller labs, harnesses and actual application.
Huh, this is the first I've heard of that one. Has anyone here tried it? I assume the main appeal for those who use it is the country of origin, rather than performance?
GigaChat is russo-centric and is completely useless outside Russia itself, performs absolutely worse than Mistral once you start using English. Reminder that Russia's economy is smaller than Italy's alone. They are not winning over Europe in any single metric, let alone against the poorest country in Europe, which is not even part of NATO nor the EU, on the battlefield.
I've heard that before though. I don't think you can compare economies like that since Russia has "near infinite potential" in natural resources, people, etc which Italy doesn't.
They can kick into a true war economy and then become significantly more of a threat. They haven't done so yet because the middle-class population likes their luxury lifestyle (for now).
I highly doubt Ukraine will "win" - in fact I believe Russia is ramping up to find an excuse at some point to attack a NATO country, but in a way and during a point in time it's difficult for NATO to retaliate.
> I've heard that before though. I don't think you can compare economies like that since Russia has "near infinite potential" in natural resources, people, etc which Italy doesn't.
Fair enough. But most of the time, the figure is not used to say that Italy is more advanced than Russia, for whatever definition of advanced. It’s a check in the other direction: even if Russia does have advantages, in the grand scheme of things they are not that big.
> They can kick into a true war economy and then become significantly more of a threat. They haven't done so yet because the middle-class population likes their luxury lifestyle (for now).
They did, just in the parts of Russia that are poorer and less-connected (and less powerful politically). They leveraged their huge geographical advantage to spread the pain as much as possible in the regions that don’t matter politically. It is very difficult to see their situation and argue that they are not in a war economy. The middle class in Moscow and St Petersburg are sheltered, true, but still.
> I believe Russia is ramping up to find an excuse at some point to attack a NATO country, but in a way and during a point in time it's difficult for NATO to retaliate.
I don’t think that is likely. But then, of course Trump is making it more and more likely by the day as he insists on getting bogged down in Iran.
Yeah? And here I've been a happy Transkribus customer for some time now. If there are better models or interfaces out there for analyzing historical handwriting, I'll definitely take a look.
For anyone interested, I have an ocr pipeline running on rented GPUs, doing around 1000pages for 0.05-01 usd with around 0.8 seconds per page with full bounding boxes support for grounding.
If you’re interested you can find contact to me via this profile.
Accuracy is truly what people die for in the OCR game. Price isn't the primary function here.. it's an equation of price, accuracy, speed, and in mayn cases regulation.
Tbf even with tesseract you already get shit ton of accuracy and you can probably do these 1000 pages for way less than 3.5€. For 3.5€ you can spin up a cloud instance with 8vCPU+32gb on gcloud for 11 hours (or 11 instances for an hour) which can do way more than 1000 pages per hour on tesseract. It takes you around 6 second per page +-4 seconds start/stop depending on what you are doing on that instance size without too much optimization (you can probably even run multiple processes on a single node)
Google documentai costs 1.5$ per 1000 which is probably better in quality and speed.
Tesseract is not a substitute for these models, which understand complex layouts and also extract bounding boxes for things like tables and pictures. They are also much better at making sense of cursive scripts.
I’ve been there, implementing a way to linearise text from a document with pages with 1, 2 or 3 columns, some of them in landscape is a nightmare. And that’s not even considering equations.
In the end it’s way easier to use a specialised model, trained by other people to do exactly what I need.
What was not so fun was to make it work reliably. I ended up with piles of ugly code to handle edge cases, and issues kept piling up. So in the end I was happy to use someone else’s solution.
It‘s also cheaper than the murican ones from Google, Amazon, …. And tesseract was an example. Heck you can go xberg and use paddleocr. Most often layout is less of a problem for ocr. Most often you need high accuracy, which tools like these are often worse in the 95 percentile.
Most use cases dont need that kind of accuracy, just doesnt justify the 3-4usd range. I build for that exact case (tender documents, we’re processing north of 100k pages per day), it doesnt need to recognize scanned written text from 1930s, its usually pdf/docs/scanned printed pages.
The accuracy is great, bounding boxes are must have for proper grounding for building answers by LLMs. Tesseract was too slow and not enough in some cases (for example tables or images which we also recognize and describe)
If you're getting inaccurate results from OCR what's the purpose of even doing it? Inaccuracy of text of any kind seems like a completely obvious failure of the entire purpose of scanning text into a computer.
It depends on what you need. For example a while ago I scanned and OCR'ed a bunch of receipts to get a timeline of my salary. I only cared about the gross and net figures, and nothing else mattered. Tesseract's output had a bunch of errors and misdetections, but the main figures always came out OK, and a local LLM was able to pick them out from the noise every time.
There's a big gulf between "it's as if a human being had transcribed it and reconstructed the original document" and "so completely broken it can't be used for anything".
Accuracy can have different dimensions, depends on what you can tolerate and whether you can detect it to apply more powerful methods.
Imagine you have a cheap and 99% accurate ocr. The other 1% you can detect and apply more powerful (more accurate but slower and more expensive) ocr method.
What would you use? At scale these things add up.
Can it produce accessible PDF files that will pass accessibility tests? Someone who can do that will make a killing laundering PDFs for academia: by April 26, every PDF, syllabus, and academic document needs to comply with WCAG 2.1 Level AA, which means structural tagging, alt text, and lots of other checklist items that AI could probably generate.
Anecdotal as I haven't had time for a full benchmark, but I have a dense tabular handwritten form that is pretty challenging: LLMs with vision don't do well because the content isn't English words. Textract did poorly because it was just too dense for their model (although I haven't tried recently) Whereas I tried this model and it was a champ: Reasonable Markdown format with accurate content.
faced something similar with Azure, try upscaling the page (2x or more) before sending to the OCR, it increased the accuracy a lot for tiny tables on landscape pages.
The VLM's are so good at complex document understanding now. But you just can't trust them not to invisibly censor sensitive clinical/legal docs, even at the maximally permissive settings.
And the deep learning OCR-only models won't censor, but can and do hallucinate. I've yet to see a 'scan with different approaches and reconcile and say you're not sure if they don't agree' system just work for generic complex documents.
The way my harness set it up is going through 2 or 3 providers, and cross-checking through them, also with plain text extracted if available.
I think we also had a layer that for any quote extracted tested it back if it exists within the original.
If you wanted 100% accuracy, I think it wouldn't be too difficult nowadays to guess the font&size&other text settings, and re render the crucial parts.
I think people misunderstand the utility of Mistral's OCR. It's not going to beat SOTA models for extraction on edge-case docs, but it's MUCH cheaper and faster and does an excellent job on simple ones. I've been working on converting PDFs to EPUBs and Mistral has been making steady improvements. On a chapter of Bleak House it was able to extract and tag the header, titles, and references at the bottom every time. The only thing it struggled on was line numbers in the right margin which it correctly tagged as "aside text" 3/5 times, but always separated from the core text each time. The important thing to keep in mind is that there's no prompting needed, just upload the PDF and voila!. There's even a batch mode with a 50% discount.
> The important thing to keep in mind is that there's no prompting needed, just upload the PDF and voila!
I'm sorry, noob here. I have a special book that I bought which I can open only inside the Kindle app (Windows/mobile). I have been meaning to screenshot the pages and convert them into a document/PDF. What do I have to do to make it fast? Just upload all the screenshots one by one and tell Mistral "Chat" to OCR them?
Does anyone know a site that lets you browse examples of input / output pairs?, particularly with layout analysis (bounding boxes of figures, tables, etc).
How does this compare to Baidu Unlimited OCR. I've been very impressed with Baidu and it's essentially free to run on a decent computer, other than electricity costs.
I've been experimenting with using NuExtract this week on locally OCRing bank statements that don't have a predefined document structure. It's way better than Tesseract or a generic vision-enabled model. It runs great on a single RTX 4090 at the modest throughput I need.
Their hosted, API-based service is something like a third of the cost of this model.
Good luck with that. Mistral is never going to produce a frontier model. Their business model is to rely on the EU regulations and cater to the bureaucracy. I won't be surprised if their next model is specialized on generating laws/regulations for EU lawmakers so they can sell it to EU. That way their business would survive for the next 5-10 years till EU implodes.
I've got a scan from a book that I OCR with new releases. Ligatures, critical sigla, Fraktur letterforms, subscripts, superscripts, etc.
Nothing special about this model for overly-detailed work like mine.
It's been a while since I last tested (and discontinued my subscription), but the "pro" models from OpenAI dominate. Not surprising, given the price difference, but it would be nice if an OCR-specific model could perform better. It's worth mentioning that even the highest-end models do a pretty poor job with intricate text like mine.
While I haven't tried OpenAI for OCR, I've put my small scale OCR work through both Claude and Mistral OCR. Claude is absolutely better - even in OCR work I did last week and compared with Mistral OCR 4.0.
Mistral's one advantage is that Anthropic now flags OCR, because they don't allow anything that could be considered "reproduction", even of work for which you own the copyright. So my new workflow is Mistral OCR for the actual OCR, followed by a proofreading pass by Claude (which is allowed). Claude is obviously more expensive, but it caught entirely hallucinated sentences created by Mistral OCR 4.0, so I was glad for the backup check.
Same company that OCRed millions of books, the irony.
I feel Anthropic is destroying itself with all these restriction. They got away because their models were the best for coding, but that is not an advantage anymore as OpenAI and other open source are already better.
Claude refuses to translate lyrics of songs even if it's just for your own understanding. ChatGPT allows it.
Also, Suno is so restrictive that it won't make songs of public domain 19th century poems because someone else already made a song and the lyrics are in their song database. And it won't do covers of your own uploaded humming of a novel melody because it matches "humming in an empty room" to some song fingerprint in the database.
They are all trying to avoid these lawsuits by being ridiculously overly cautious. "Your honor, look how much we went above and beyond, even when it made the product worse for legitimate use cases"
It is why they're pushing for regulatory capture. Let us do the bad stuff, make the money, then we'll regulate everybody else out of the market.
They were criticised/sued early on when people could reproduce copyright things. I dont think in this case it's something they'd prefer to do?
Yep, my understanding is that many guardrails like this are actually the result of government legislation (eg the Fable bans) or terms of settling copyright lawsuits over reproducing copyrighted text and lyrics.
I mentioned it in a sibling reply, but here's Anthropic's support document about not using Claude to reproduce content verbatim that already exists, regardless of copyright.
https://privacy.claude.com/en/articles/10023638-why-am-i-rec...
> Claude is obviously more expensive, but it caught entirely hallucinated sentences created by Mistral OCR 4.0, so I was glad for the backup check.
What does this entail? What does Claude do to decide that the text it was provided was hallucinated? Are you telling Claude that the source was OCR'd by another LLM?
I'm basically doing the OCR twice, except in the Claude proofreading pass, it is not being asked to transcribe the document to a Markdown file. I'm pointing it to the same image input files, and to the Mistral OCR transcript Markdown file (it knows it's a Mistral OCR output), and ask Claude to check that the text is correct and point out the errors - and then make the necessary edits.
I can't speak for Mistral OCR 4.1, but the hallucinations in 4.0 were so egregious (just completely making up new sentences in the middle of a page) that I knew I can't trust Mistral OCR on its own.
How bad is the first OCR pass allowed to be to still count as proofreading? Can you let Claude compare the images with /dev/random and make the necessary edits to correct differences?
Hmm, that's an interesting idea. But it's the classifier that is triggering, and it triggers specifically on Claude's output. So I think the /dev/random case wouldn't work, because that gets Claude into the state of just reproducing the entire text from the original again.
It doesn't always get flagged. Single pages are almost always okay. Running a program that sequentially runs single pages through the API is often not okay - I wrote a program in the early 4.x days before the rule came in, that's how I hit it first. But I've also had entire articles go through just fine recently in a Claude Code session (I'd forgotten about Anthropic's rules!), and then others where I get classifier errors by page 4.
The Mistral OCR errors were small in size. Single sentences, formatting errors, paragraphs with newlines. So this was a genuine proofreading job with small changes. For the most part Mistral is actually good, but I can't have it just inventing sentences in the middle of a document. That's where the Claude proofreading pass was most helpful.
I read that as “hallucinated” in the practical sense: the OCR output contained text that wasn't actually present in the scan, rather than just misreading a character.
Yes, that's how I read it as well, but the question remains in how did Claude know this? It was not clear to me that Claude was being presented the original scan. The entire exercise is farcical, but it wasn't as clear exactly how farcical it was that the person was essentially running OCR wastefully on a different model just to convince Claude to do the OCR as desired because of Anthropic's rules. Here's this cool tool that is supposed to make you do less work and be able to do it faster and be able to do it more accurately, except you first have to do all of this other unnecessary work with a totally different company first. <face-palm>
I assume they pass the image alongside the text to redo/recheck the work. Yes this is silly, but is apparently required to get around refusals.
>Anthropic now flags OCR
I haven't seen any difference in my ocr workflows, what do you mean by this?
Not the person you’re responding to, but I’ve had Claude refuse to OCR pages from in-copyright books. I was sometimes (but not always) able to get around that by changing models, by telling it that I was doing the text conversion only for personal use, or by first telling it to use Tesseract or another OCR engine to do the initial pass and then having a Claude subagent proofread and clean up the OCR output.
I’ve also had it refuse to OCR public-domain books that included content that it didn’t like, such as references to prostitution in 19th-century books about Japan.
I had one session where Claude refused to continue after it hit some kind of guiderail restriction. I couldn’t see what the trigger was, so I started a new session, gave Claude the link to the previous session, and asked it to diagnose the problem. This new Claude said it couldn’t view the exact guardrail issue, but it did suggest a workaround that turned out to be effective.
Yeah I do something similar and Claude (well OpenAI too) just refuses to transcribe anything related to slavery and pederasty in Ancient Greece.
Are you familiar with the Thesaurus Linguae Graecae? It's a project that may meet your needs. Paywalled, unfortunately, but I assume it would be a one-time expense.
This is too funny considering they did that themselves. I’m pretty tired of these companies deciding what we can and can’t do while they act with impunity.
"it refuse to OCR public-domain books that included content that it didn’t like, such as references to prostitution in 19th-century books about Japan."
Thoughtcrime -like territory and self-sensorship. The AI safety lobby is such a vile influence on the freedom of expression and communication via technology (since AI is starting to eat up rest of technology).
I guess the main problem is positioning AI tools as "human-equivalent" creators by the big AI corps. If they were positioned simply as "better OCR and proofreading" people would attribute to them as much responsibility as they would to a - say - typewriter and we would not need to have this nonsense.
I do realize most of the valuation comes from the positioning of "our TAM is the global salary base of 50 trilion and we aim to supesede human workers in the near future" which implies they need to position this technology as "human equivalent" or that valuation is no longer as credible.
I also saw this when I made a (personal use only) book translation (agent learns how book 1-3 of a row is translated, then translates book 4 the same way because it is not available in my language). Claude happily translated most of the book, except a few chapters which it denied. Tried several times always the same results (with no other context about the rest of the book).
The response / conversation gets blocked by the guardrails with "API Error: 400 Output blocked by content filtering policy"
At first I thought it was something in the scanned content that was being flagged, but it was the attempt to transcribe that was itself being flagged. Anthropic mention it on their pages:
"Anthropic takes these steps because Claude’s purpose is to generate new content and ideas, not to reproduce content that already exists."
https://privacy.claude.com/en/articles/10023638-why-am-i-rec...
Side note - Claude itself is not aware of this policy, and is unable to see the API responses - the turn just ends. Which turned into a really bizarre failure state where Claude thought I was gaslighting it and kept insisting it could do the work and even had the entire text in memory. Every time it would go to show me and prove it, it would hit API Error 400. I was only able to convince Claude by showing screenshots of my Claude Code screen output so it could see that I was seeing API errors. I've never seen Claude get into that angry & snarky state before, and I hope it doesn't happen again.
I have been quite happy with Mistral OCR for the documents I needed to process (typeset, but old, with questionable scan quality, sometimes elaborate typesetting or, much worse, typewriter-and-handwriting approximations of it). I do not test every new model when they are released, but I did a review shortly after Mistral OCR 3 was released and it was a very good compromise: cheap, fast, and good results without further processing. I found generalist models to be way too much faf to get them to avoid unnecessary modifications to the text and report accurate bounding boxes for figures and tables.
That said, models have sometimes surprising weaknesses and a model could be terrible overall but magically work for one type of document.
I got the opposite experience very recently : tried to OCR a bunch of handwritten emails addresses with chatGPT and I had to make so many corrections that I gave up. Whereas Mistral nailed it on first pass.
Which chatgpt model?
I tested different models of the 5.6 family with different settings, although I couldn't exactly remember precisely which ones.
> the "pro" models from OpenAI dominate. Not surprising considering the price difference, but it would ne nice if an OCR-specific model could do better.
I haven't been impressed with any of Mistral's models. They obviously realized that they couldn't compete at the frontier so they decided to go for smaller focused models but even those have not been that good.
This is what I found as well.
We moved away from Cursor but I was looking for a model that would help with FIM (fill-in-middle) multiline autocompletion and people were recommending Mistral's Codestral. We gave it a shot and it was lackluster at best.. Even Google's Gemini did a significantly better job than Codestral.
Ultimately Opus-class models got good enough and I don't do much manual coding anymore.
Yet: how is pricing? Evaluating contents and routing appropriately isn't a new challenge in OCR, one of the oldest fields of applications in ML. Thus, how do the smaller open models perform in tandem with relatively pricy $/pg models & APIs? Your use case is remarkably rare relative to the volume and price sensitivity of enterprise data warehouse ops.
The big difference is traditional OCR used basic pattern matching to find text, whereas models like Mistral OCR (and GPT, etc) use computer vision instead and deep learning to parse text, math equations, and apparently in some cases extract images too.
I'd love to see some advancements in traditional OCR based on ideas and concepts we've learned from newer "OCR-like" models since traditional OCR is drastically cheaper.
I made a benchmark for handwriting recognition for a project while keeping line breaks and errors (grammar, spelling). Sonnet absolutely dominates it since a good half a year. 5.6 did not change that for me. This should also translate to better ocr.
You're the second person to mention handwriting. I think it might be handwriting-specific; perhaps Anthropic has a better corpus for this.
I really wouldn't know, though. Anthropic models barf out copyright issues for my use case, so I'm unable even to benchmark them. It's a common problem when you're scanning public domain books. Mine are reference texts often cited.
Ah okay. Ofc handwriting is not copyrighted. Makes sense, that we do not have these issues.
So is Mistral OCR the best one? Have any other OCR models caught some of what you describe? I've been kind of interested in how "OCR" type models work compared to old school OCR.
My use case isn't in the realm of old-school OCR, so it's not a good comparison, but anyway:
As another user pointed out, it's surprisingly random (task-specific). Llama Scout outperformed Gemini Flash 2.5 on a benchmark I built at the time. I didn't include an OCR models.
Mistral might indeed be the best OCR-specific model for my task, now that you ask. Funny. It's so bad at my work that I didn't register it might be the best in its category. This is just based on vibes from my single scan.
The datalab models are the best ones.
Whats the best open OCR at the moment?
I’ve had good experiences with MinerU and PaddleOCR-VL
Do all the models give you the bounding boxes, block labels as this one (allegedly) do?
It's very common. PaddleOCR is enough to get extremely well-done bounding boxes, and it runs very fast on a $150.00 GPU.
There's always room for improvement, though. I suspect a tool will emerge for highly detailed OCR that implements a nested bounding-box-based multi-scale approach, effectively OCRing small sections at a time and then gradually compiling them by expanding the surface area using the bounding boxes.
I've thought a lot about implementing it anyway.
edit: I see you're asking about the block labels. Leaving the comment in case someone finds it interesting.
At this point I lost all hope for Europe playing any significant role in the AI race. If that’s a good or a bad thing I don’t know, but it seems to me like that’s the reality.
Commoditization is a beautiful thing. It seems Anthropic and OpenAI are really struggling to maintain much of a moat. Mistral might not be leading but it's not trailing by that much either. And of course the Chinese are doing their own thing quite successfully.
The reality is that the US is betting its economy on data centers at great expense and is exposing its economy to great risk.
Also while geographically a lot of the money and processing power is in the US, the US has been relying on immigration to power its universities and especially AI research has roots all over the globe. India, China, Russia, Europe, etc. AI related know how is finding its way back to all these places.
So, I'm not too worried about the long term here. It will be interesting to see if Anthropic and OpenAI survive their IPOs. Seems like a risky financial bet at this point given the apparent lack of a moat. But if it works out, it will result in a lot of that IPO money being invested in data centers abroad. Including in the EU. Because data residency is a thing here and the EU is too big of a market for companies with that kind of valuation to ignore. We also produce a lot of energy infrastructure (e.g. gas and wind turbines). Those data centers will need lots of power.
Nothing to worry about for europe. Open source and open weights will enable europe to pick up the slck when the party is over. Then the US markets will possibly regret the AI-bubble.
Much like spaceflight, aerospace or nuclear engineering, you need to retain local talent for national defense purposes. Being 70% as good is still way better than being 100% dependent and heavily leveraged by your opponents.
It gets a little weird when you start thinking of AI like that, because AI is very corporately based in the US and China, but not as heavily that when from a talent position.
Demis Hassabis and Google's Deep Mind are all British in terms of location/talent, but they certainly aren't from a coorporate perspective (Deep Mind work might happen in London, but it is owned by Google, as US company)
(I don't know a lot about nuclear/aerospace industries, I'm guessing there more location centered just in virtue of being based around physical things, but I could be wrong)
Being 70% as good in nuclear engineering sounds scary AF. How would you rank Chernobyl? Better or worse than 70%?
Being 70% as good does not mean causing accidents. Accidents are caused by systematic operational failures and fundamental design flaws.
If your employees are only 70% as good, as skilled, as knowledgeable while not knowing it, then you can absolutely kill a reactor just like Chernobyl. The overconfidence is what allowed them to put the reactor in the state they did and not realize the implications.
Far worse than 70%. Chernobyl was a catastrophically flawed reactor design. Operators ran a badly planned safety test in which operators intentionally caused a dangerous state while key protections were disabled or bypassed.
Exactly, the intelligence/competence of the engineers running the plant had no actual bearing on the incident.
Chernobyl was basically Russia then. The more honest comparison would be the numerous nuclear power plants in France.
70%… Not great, not terrible.
It's not a race. You don't get anything for winning.
Well, you get hundreds of billions in debt, and then you get open weight models distilling your proprietary models.
? It absolutely is a race. Whether thats a positive thing or not is debatable but every lab is definitely in a race. What prize do you win? Imagine a world where only one country has AGI/ASI. Or a world where Europe only gets access to frontier models 6 months later. Far from ideal.
> What prize do you win? Imagine a world where only one country has AGI/ASI.
"Winning the race" doesn't give you that in any meaningful capacity. It gives you, at the absolute most, a temporary window where that's the case. See: nuclear weapons.
Citing nuclear weapons isn't the flex you think it is. There are only 9 countries that have nuclear weapons and they absolutely flex this power over non-nuclear powers (see Ukraine, Germany, SE Asia etc..)
Europe already has an innovation problem that's already causing structural economic instabilities which Germany has been struggling (and lately failing) to prop up.
As much as it pains me to say this, AI is already a tech revolution and it seems like Europe is just ignoring it. There's more innovation in 3 blocks in downtown San Francisco than the entire continent of Europe.
> There are only 9 countries that have nuclear weapons
QED being first didn't grant exclusivity.
The premise wasn't that AGI isn't useful. There are actually layers to the metaphor where first-mover advantage of AGI is even less meaningful than it was for nuclear weapons, but I leave those as an exercise to the reader to discover.
Nobody argued exclusivity. But there have been clear advantages to having nukes first.
> Nobody argued exclusivity.
And I quote:
> Imagine a world where only one country has AGI/ASI.
> first-mover advantage of AGI is even less meaningful
This is your opinion. Lots of tech money appears to disagree.
You should at least try to explain your contrarian position, or cite your favorite source that makes an argument that you believe.
Lots of tech money is demonstrably wrong, at least in the general case. Microsoft's whole business model is to be the second mover. Patents exist because first mover advantage alone isn't enough to hold onto a win.
If AI is different, the onus is on AI folk to explain why, not the reverse.
The frontier labs are racing as hard as they can right now because they know as well as anyone else that they're at best 6 months, probably closer to 3 months, ahead of the generic commodity AI market. The majority of the tech money currently boosting them is a bet that either they can stay that far ahead without tripping, or that a moat can be engineered. It's not at all clear that it can - or, if it can, that it will before the money runs out.
> You should at least try to explain your contrarian position
FWIW I explained it fully and provided the argument, along with explicit grounding examples. Can you tell me what exactly you don't understand?
To quote your other post:
> This is the weakest part of the argument. Absolutely no reason to believe the second inventor will be 10x faster.
Nor is there any reason to believe that it will work at any usable speed. An LLM capable of AGI running at 0.000001tok/s isn't a very good head start. Nor is it a good moat if it can't actually realize anything material fast enough. Lord knows America has labor problems.
I'm demonstrating the context is a lot more complicated than "be first". There are innumerable specific examples. They're not guaranteed to happen, that's not the point being made.
Also "no reason" is curious when it's the explicit pattern this exact industry has exhibited. Chinese models lag a few months, and come in swinging with an order of magnitude more efficiency. Does it apply? Who can say. It's certainly on the table. Pretending like it's any more ridiculous than AGI itself is irrational.
> Does this ever happen? Even in traditional manufacturing, the second “inventor” starts behind and has to improve their own process to surpass the first.
To use the industry itself once again: Japan was the first to deep learning. Performance issues prevented them from capitalizing on it. Now they're barely even a player on the field. Interestingly and adjacent to the industry, Japan was second to symbolic AI, and the FGCS was lightyears ahead of America's crufty Lisp ecosystem. For a more modern example, Google was the first to transformers. Much of this revolution is thanks to them. A shame that Gemini is third-rate at best.
It's not unique to this industry. That second inventor more often than not does a better job than the original one. It's a pretty common pattern. Otherwise, we wouldn't need patents.
My "contrarian position" is anything but. There is nothing new under the sun. You don't just need to be first, you need to maintain exclusivity.
Russia invaded Ukraine specifically because they have nukes. Everyone is afraid to fight back.
As if time isn't money? Everything is temporary... Getting somewhere first has immense value.
A reductive equation, economics isn't thermodynamics. Money is fictional and value is subjective and unstable. Within this context, being first to AGI means nothing if the second invention of it comes 2 months later and works an order of magnitude faster than what the first iteration had self-improved to at that point in time. First mover advantage isn't decisive, you have to actually be able to capitalize on it in a robust way.
> works an order of magnitude faster
This is the weakest part of the argument. Absolutely no reason to believe the second inventor will be 10x faster.
Does this ever happen? Even in traditional manufacturing, the second “inventor” starts behind and has to improve their own process to surpass the first.
> Imagine a world where only one country has AGI/ASI.
You've fallen for the hype. The only way these AI companies can justify their bullshit worth and burn of capital is by promising a magical AGI. It's just marketing and it is still unclear if LLM's will ever be profitable.
If they won't - then EU is doing the right thing laying low.
> Or a world where Europe only gets access to frontier models 6 months later.
What a calamity!
> If they won't - then EU is doing the right thing laying low.
Exactly. People seem to forget that US is burning money at an accelerating rate without any promise of returns. This is a high risk situation. Time will tell.
> Imagine a world where only one country has AGI/ASI
My own bet is that this won't happen with the current race, but that's another discussion.
Anyways, bit of a doomer take but if I imagine a world with AGI/ASI, countries don't mean shit anymore. Humans are not the top dog anymore.
> Imagine a world where only one country has AGI/ASI
Who says the LLM race ends at an AGI/ASI finish line in the first place ?
This presupposes AGI or ASI are a real, reachable thing. My read is they might be, but not as LLMs. Until there's a fundamental rearchitecture, I'm AGI-agnostic and given that view, it doesn't seem rational to bet the house on it.
We'll see.
How about "being able to align ASI somewhat to your values"?
True, but that isn't EU/US/China, it's OpenAI/Anthropic/Grok/ …/DeepMind (based in UK)/… DeepSeek
With a lot of Chinese nationals in American companies, and an American corporation owning DeepMind, and a lot of people very upset with all of them at the same time, this is very messy.
A race to the bottom.
The only prize is control of the light cone.
It's red queen. You stay alive by winning, you lose everything by losing.
Not sure you get either outcome in either case.
Race dynamics increases p(doom) for everyone.
The non-doom scenarios include "utopia for all", and "power flows to investors, not citizens of whichever nation the winning model's corp. was registered in".
Independently, "oh look all the investors went bankrupt" can happen in both "doom" and "normal technology" timelines.
Unless you manage to build a god, and keep it under control... okay, we're all going to lose
Mistral is (wisely) are changing their strategy. They switched to hosting open models and they are investing in hosting inference within EU.
Fine tuning an open model to European values is significantly cheaper than making your own model.
Not being a rat in the rat race is the real win.
> hope for Europe playing any significant role in the AI race
Ha. It's a large market of the LLMs consumption. So it which will affect the AI race. Just from other perspective than you assumed.
And US lost the significant role in chip/pc manufacturing. But if a product becomes commodity or utility (which at least for now it seems is the direction), with little lockin, it's not a big deal.
I hope we (EU) don't waste money trying to train local models (which at least some people in Poland try to do), and tries to build our own chips - AI chips have different architecture than regular processor/GPU, and TSMC doesn't need to be winner in this new race.
And if not this, then smaller labs, harnesses and actual application.
The largest non-US non-China model is Russian which surprised me.
Largest by what metric? And which model is that?
Largest by total parameters. GigaChat 3.1 Ultra: 702B
Huh, this is the first I've heard of that one. Has anyone here tried it? I assume the main appeal for those who use it is the country of origin, rather than performance?
Are you saying Russia is winning over Europe in AI already? Dang, sanctions really don't work.
GigaChat is russo-centric and is completely useless outside Russia itself, performs absolutely worse than Mistral once you start using English. Reminder that Russia's economy is smaller than Italy's alone. They are not winning over Europe in any single metric, let alone against the poorest country in Europe, which is not even part of NATO nor the EU, on the battlefield.
I've heard that before though. I don't think you can compare economies like that since Russia has "near infinite potential" in natural resources, people, etc which Italy doesn't.
They can kick into a true war economy and then become significantly more of a threat. They haven't done so yet because the middle-class population likes their luxury lifestyle (for now).
I highly doubt Ukraine will "win" - in fact I believe Russia is ramping up to find an excuse at some point to attack a NATO country, but in a way and during a point in time it's difficult for NATO to retaliate.
> I've heard that before though. I don't think you can compare economies like that since Russia has "near infinite potential" in natural resources, people, etc which Italy doesn't.
Fair enough. But most of the time, the figure is not used to say that Italy is more advanced than Russia, for whatever definition of advanced. It’s a check in the other direction: even if Russia does have advantages, in the grand scheme of things they are not that big.
> They can kick into a true war economy and then become significantly more of a threat. They haven't done so yet because the middle-class population likes their luxury lifestyle (for now).
They did, just in the parts of Russia that are poorer and less-connected (and less powerful politically). They leveraged their huge geographical advantage to spread the pain as much as possible in the regions that don’t matter politically. It is very difficult to see their situation and argue that they are not in a war economy. The middle class in Moscow and St Petersburg are sheltered, true, but still.
> I believe Russia is ramping up to find an excuse at some point to attack a NATO country, but in a way and during a point in time it's difficult for NATO to retaliate.
I don’t think that is likely. But then, of course Trump is making it more and more likely by the day as he insists on getting bogged down in Iran.
They've been unfortunately hugged by the "little death" of working with the EU institutions, though I wouldn't bet on them not coming back.
Yeah? And here I've been a happy Transkribus customer for some time now. If there are better models or interfaces out there for analyzing historical handwriting, I'll definitely take a look.
Even Africa will surpass Europe with its massive data center upstarts breaking ground.
Huh? Mistral 7b was pioneering in its day and IMO they have been very on top of releasing niche useful models like moderation, OCR, etc.
I’m glad Mistral is working on useful solutions.
OpenAI/Anthropic is like a retarded little sibling chasing “AGI” and giving up on rich media and other modalities.
OpenAI/Anthropic is the worst of the mainstream AI.
It goes:
1. Gemini
2. Vidu
3. Le Chat (Mistral)
4. DeepAI
5. [insert MiniMax provider]
Totally fake post. Gemini in the top?? For what?
Gemini is best at OCR and document extraction so far. We had tested for insurance needs. None match Gemini even flash level yet.
Same here. Maybe Fable is better but in terms of cost effectiveness it wouldn't even make sense to test it
For anyone interested, I have an ocr pipeline running on rented GPUs, doing around 1000pages for 0.05-01 usd with around 0.8 seconds per page with full bounding boxes support for grounding.
If you’re interested you can find contact to me via this profile.
3.5 usd/1000 pages is just too expensive…
Accuracy is truly what people die for in the OCR game. Price isn't the primary function here.. it's an equation of price, accuracy, speed, and in mayn cases regulation.
Tbf even with tesseract you already get shit ton of accuracy and you can probably do these 1000 pages for way less than 3.5€. For 3.5€ you can spin up a cloud instance with 8vCPU+32gb on gcloud for 11 hours (or 11 instances for an hour) which can do way more than 1000 pages per hour on tesseract. It takes you around 6 second per page +-4 seconds start/stop depending on what you are doing on that instance size without too much optimization (you can probably even run multiple processes on a single node)
Google documentai costs 1.5$ per 1000 which is probably better in quality and speed.
Tesseract is not a substitute for these models, which understand complex layouts and also extract bounding boxes for things like tables and pictures. They are also much better at making sense of cursive scripts.
I’ve been there, implementing a way to linearise text from a document with pages with 1, 2 or 3 columns, some of them in landscape is a nightmare. And that’s not even considering equations.
In the end it’s way easier to use a specialised model, trained by other people to do exactly what I need.
> "I’ve been there, implementing a way to linearise text from a document with pages with 1, 2 or 3 columns, some of them in landscape is a nightmare."
I thought it's actually a fun problem! Some of the naive solutions (based on alternating horizontal/vertical recursion) are quite elegant and cool.
It was fun!
What was not so fun was to make it work reliably. I ended up with piles of ugly code to handle edge cases, and issues kept piling up. So in the end I was happy to use someone else’s solution.
Tesseract is super picky though often failing pixel-perfect screenshots...
So, maybe it can be tuned for your usecase but with that kind of investment €3.5 for 1000 pages is a bargain...
It‘s also cheaper than the murican ones from Google, Amazon, …. And tesseract was an example. Heck you can go xberg and use paddleocr. Most often layout is less of a problem for ocr. Most often you need high accuracy, which tools like these are often worse in the 95 percentile.
Are you serious? Tasserect is the worst compare to PaddleOCR , Surya/Marker
Most use cases dont need that kind of accuracy, just doesnt justify the 3-4usd range. I build for that exact case (tender documents, we’re processing north of 100k pages per day), it doesnt need to recognize scanned written text from 1930s, its usually pdf/docs/scanned printed pages. The accuracy is great, bounding boxes are must have for proper grounding for building answers by LLMs. Tesseract was too slow and not enough in some cases (for example tables or images which we also recognize and describe)
If you're getting inaccurate results from OCR what's the purpose of even doing it? Inaccuracy of text of any kind seems like a completely obvious failure of the entire purpose of scanning text into a computer.
It depends on what you need. For example a while ago I scanned and OCR'ed a bunch of receipts to get a timeline of my salary. I only cared about the gross and net figures, and nothing else mattered. Tesseract's output had a bunch of errors and misdetections, but the main figures always came out OK, and a local LLM was able to pick them out from the noise every time.
There's a big gulf between "it's as if a human being had transcribed it and reconstructed the original document" and "so completely broken it can't be used for anything".
Accuracy can have different dimensions, depends on what you can tolerate and whether you can detect it to apply more powerful methods.
Imagine you have a cheap and 99% accurate ocr. The other 1% you can detect and apply more powerful (more accurate but slower and more expensive) ocr method. What would you use? At scale these things add up.
Can it produce accessible PDF files that will pass accessibility tests? Someone who can do that will make a killing laundering PDFs for academia: by April 26, every PDF, syllabus, and academic document needs to comply with WCAG 2.1 Level AA, which means structural tagging, alt text, and lots of other checklist items that AI could probably generate.
You should put contact details in your profile :)
Is it European-hosted and fully outside of both CLOUD Act and CCP reach?
Because I'm assuming that's why they get to charge more for the right type of customer.
You can even run it on your desk if you want, a single gtx 4090 is enough. It can be fully air gapped.
1000 Pages / 3.5€ this is expensive as hell. If this is not fastly superior than something like tesseract it is not worth it.
Agreed. Does the GTM team there really sit together like "oh yeah, that sounds reasonable" while being totally beyond typical market prices?
Even comparing to AWS Textract or Azure Document Intelligence, this is very expensive (more than double)
How does it compare to those services? Are there any benchmarks yet?
Anecdotal as I haven't had time for a full benchmark, but I have a dense tabular handwritten form that is pretty challenging: LLMs with vision don't do well because the content isn't English words. Textract did poorly because it was just too dense for their model (although I haven't tried recently) Whereas I tried this model and it was a champ: Reasonable Markdown format with accurate content.
faced something similar with Azure, try upscaling the page (2x or more) before sending to the OCR, it increased the accuracy a lot for tiny tables on landscape pages.
The VLM's are so good at complex document understanding now. But you just can't trust them not to invisibly censor sensitive clinical/legal docs, even at the maximally permissive settings.
And the deep learning OCR-only models won't censor, but can and do hallucinate. I've yet to see a 'scan with different approaches and reconcile and say you're not sure if they don't agree' system just work for generic complex documents.
The way my harness set it up is going through 2 or 3 providers, and cross-checking through them, also with plain text extracted if available.
I think we also had a layer that for any quote extracted tested it back if it exists within the original.
If you wanted 100% accuracy, I think it wouldn't be too difficult nowadays to guess the font&size&other text settings, and re render the crucial parts.
> But you just can't trust them not to invisibly censor sensitive clinical/legal docs
What's an example of this?
Benchmarks? I'm currently using Tesseract (via OcrMyPDF) and would like to compare the difference.
I think people misunderstand the utility of Mistral's OCR. It's not going to beat SOTA models for extraction on edge-case docs, but it's MUCH cheaper and faster and does an excellent job on simple ones. I've been working on converting PDFs to EPUBs and Mistral has been making steady improvements. On a chapter of Bleak House it was able to extract and tag the header, titles, and references at the bottom every time. The only thing it struggled on was line numbers in the right margin which it correctly tagged as "aside text" 3/5 times, but always separated from the core text each time. The important thing to keep in mind is that there's no prompting needed, just upload the PDF and voila!. There's even a batch mode with a 50% discount.
> The important thing to keep in mind is that there's no prompting needed, just upload the PDF and voila!
I'm sorry, noob here. I have a special book that I bought which I can open only inside the Kindle app (Windows/mobile). I have been meaning to screenshot the pages and convert them into a document/PDF. What do I have to do to make it fast? Just upload all the screenshots one by one and tell Mistral "Chat" to OCR them?
> it's MUCH cheaper and faster and does an excellent job on simple ones.
OCR should be:
1. Privacy-respecting, i.e. running on your own machine without network communications. 2. Fully open-source. 3. Gratis.
The first one is a must, the second is very important for the public interest, and the third one is a nice-to-have.
Mistral does not appear to satisfy even the first-, let alone all three.
Does anyone know a site that lets you browse examples of input / output pairs?, particularly with layout analysis (bounding boxes of figures, tables, etc).
Mistral has been very good with handwritten ocr, expecting the new models to get better with that across languages.
I won't comment on accuracy, but in internal benchmarks, Mistral OCR is significantly faster than comparable APIs.
Given the latest vibe release's new "follow default" model option, we should see a new coding / general Mistral model, mistral 4, soon.
How does this compare to Baidu Unlimited OCR. I've been very impressed with Baidu and it's essentially free to run on a decent computer, other than electricity costs.
Where do your documents go?
They go to the decent computer hosting the model, which can be yours if you pay the electricity costs
I've been experimenting with using NuExtract this week on locally OCRing bank statements that don't have a predefined document structure. It's way better than Tesseract or a generic vision-enabled model. It runs great on a single RTX 4090 at the modest throughput I need.
Their hosted, API-based service is something like a third of the cost of this model.
I'm wondering if this model performs better on french (and other european languages) documents than others
I didn't quite understand.
Mistral is bumping the price of this thing every release. I think we're at 2x now?
How does this compare to 4?
The chinese did it better, mistral is alive thanks to regulations.
No it's the other way round : the Chinese do better thanks to regulations : massive amounts of money from Big tech and public money.
It's not regulations, it's European companies being (legitimately) worried about their data sovereignty if they use US or Chinese AI providers.
Yeah, I'm not sending personal bills etc to china. No thanks
It's still open-weight models that you can download and run locally. You don't need to send anything to China.
You can use from EU/US hosters. No need to send data to China.
and you trust the french?
I trust them more than the Americans and Chinese. Genuinely rooting for them to close the gap.
Good luck with that. Mistral is never going to produce a frontier model. Their business model is to rely on the EU regulations and cater to the bureaucracy. I won't be surprised if their next model is specialized on generating laws/regulations for EU lawmakers so they can sell it to EU. That way their business would survive for the next 5-10 years till EU implodes.
Your disdain for the EU is genuinely puzzling to me.
i.e. it's one AI company that's basically guaranteed to never fail since it has a market niche guaranteed by European companies and governments.
Which is also why their most recent model "Shieldstral" does nothing except monitor and moderate internet content.
After stuff like Chat Control I think they're obviously seeing a big demand for this kind of "internet safety" technology in Europe.
I don’t think any European AI model can reach the experience and productivity of average NSA/CIA analyst.
I've been a bit more careful about complaining about regulations broadly due to competitive advantage, e.g. ITAR
How does this perform with non-Latin and non-LTR scripts? Say, Chinese, Arabic, Devangari, Adlam, etc.?
Who the hell at Mistral thinks it is a good idea to register CMD + T as a shortcut for switching theme!?
it's control + T right?
Whoever is paying all that for OCR is being scammed.
* scanned