If you’ve ever tried to extract data from invoices automatically, you know the pain of dealing with different formats, poor quality scans, and the occasional occasional coffee-stained receipt. After spending a week struggling through different OCR solutions, I decided to put three popular tools to the test: PyTesseract, PaddleOCR, and Surya OCR.

I ran 212 real-world invoices through each system, and the results were eye-opening. Some tools handled slightly tilted smartphone photos surprisingly well, showing impressive resilience, while others revealed areas for improvement. Here’s what I found out about each tool’s real-world performance, costs, and processing times.

The Dataset: A Mix of Real-World Chaos

Let’s talk about what we’re actually working with here. The test set included 212 invoices that represent the kind of mess you’d deal with in a real business:

• Smartphone photos of paper invoices (lots of them wrinkled and poorly lit)

• PDF invoices exported from various accounting systems

• Screenshots from payment platforms and digital receipts

• Handwritten receipts

So I really went above and beyond to put the tools to a real test.

Why these tools?

Let me break down why I picked these contestants:

1. PyTesseract:

A reliable, open-source tool that, while not the most advanced, consistently gets the job done. And another solid perk, it’s free!

2. PaddleOCR:

A promising tool developed by PaddlePaddle, offering a well-rounded performance. It supports multiple languages, making it a versatile choice for a variety of OCR tasks.

3. Surya OCR:

A specialized document OCR toolkit designed for a range of document processing tasks. It supports OCR in over 90 languages and performs competitively against cloud services. Surya offers features such as line-level text detection in any language, layout analysis (including table, image, and header detection), reading order detection, and table recognition (identifying rows and columns). However, it requires a GPU for optimal performance.

What the Numbers Tell Us?

The accuracy of each tool was measured across eight key invoice fields: acccount number, date of invoice, invoice by, invoice number, invoice to, remarks, total amount, and vendor name. Below are the results for each field:

Looking at the accuracy graph above, a few things jump out:

• PyTesseract is surprisingly good with simple fields like account numbers (93.90%) but struggles with invoice numbers (77.40%). It’s free though, so there’s that, and it gave an overall accuracy of 87.74%.

• PaddleOCR is the steady performer, hitting above 90% on pretty much everything. It gave an overall accuracy of 96.58%!

• Surya OCR edges out the competition overall (97.70%), especially with complex fields like invoice details.

Speed vs. Accuracy: The Real Trade-off

Here’s what you actually need to know about processing times:

1. PyTesseract OCR

Pytesseract OCR on CPU-time

• Quick with clean PDFs (2 seconds average)

• Gets confused by rotated images

• Free, but you’ll pay with your time on complex documents

2. Paddle OCR

Paddle OCR on CPU (in sec)

• Gives an average of 3.15 seconds per image/pdf

• Handles rotated images like a champ

• Watch out for memory usage on large batches

3. Surya OCR

Surya OCR on T4 x 2 GPU (in sec)

Surya OCR on CPU (in sec)

• GPU: Blazing fast at 2.42 seconds

• CPU: Better grab a coffee (157.22 seconds)

• Best at handling messy inputs

Insights

Making the Choice

Here’s my practical take:

• Small operation with clean invoices? PyTesseract will do the job. It’s not perfect, but it’s free and good enough for basic needs.

• Processing lots of different invoice types? PaddleOCR is your friend. It’s reliable and won’t break the bank.

• Got GPU hardware and need top accuracy? Surya OCR is the best pick, especially for large-scale operations.

While these tools serve different needs well, we’re not stopping here. As part of this research, we’re also exploring transformer-based OCR solutions, which could potentially offer even better accuracy and flexibility. Stay tuned for those results!

- Anannya Chaudhary, AI intern @researchify.io