Docling Found 41 Images and Described Zero. Why I Eventually Replaced My PDF Pipeline With Base64 + Vision LLMs.
I was working on a document ingestion pipeline for an AI-powered financial research system when I encountered an interesting problem.
A PDF converted successfully. The Markdown file was generated. Headings and paragraphs appeared where they should.
But the output had lost some of the most important financial information.
Charts, graphs and image-heavy sections were either missing or reduced to placeholders.
One of my Docling experiments detected 41 pictures and generated zero useful picture descriptions.
Another experiment took 266.96 seconds to process a four-page PDF chunk.
For a production financial RAG system, those are not minor inconveniences.
Imagine an investor asking:
"What was the company's revenue growth over the last five years?"
The answer might exist only in a bar chart. The textual paragraph beside it might discuss the business strategy but never repeat the actual numbers.
If the ingestion pipeline stores only the surrounding paragraphs, the retrieval system has no way to recover those figures from its text index.
The document was processed, but the information was lost.
That failure eventually pushed me through three generations of document extraction architecture:
- Docling-based parsing: Conventional OCR, layout recognition, table extraction and document understanding.
- Hybrid Docling + GPT-4o: Docling handled structural extraction, while GPT-4o interpreted extracted images and charts.
- Direct Base64 + multimodal LLM: Complete PDF pages were rendered into images, Base64-encoded and sent directly to vision models for Markdown generation, removing Docling from the extraction path entirely.
The third architecture became the direction I preferred for visually complex financial documents.
And the most interesting part was not that a larger model suddenly became better at OCR.
It was that simplifying the pipeline removed several failure points simultaneously.
1. What Makes Financial Document Extraction So Difficult?
Financial documents combine multiple information formats on the same page.
A typical annual report or factsheet can contain:
- Ordinary text and headings.
- Financial tables with merged cells.
- Revenue, margin and profit charts.
- Footnotes explaining accounting assumptions.
- Scanned images.
- Multi-column layouts.
- Figures expressed in thousands, lakhs, crores or millions.
- Percentages that need to remain associated with particular years or business segments.
Consider a table:
| Financial year | Revenue (₹ Cr) | EBITDA margin |
|---|---|---|
| FY23 | 12,430 | 17.2% |
| FY24 | 14,850 | 18.1% |
| FY25 | 17,210 | 19.4% |
These are illustrative figures, not extracted company data.
A good document parser must preserve more than the raw strings.
It must know that 17,210 is associated with FY25 revenue, that the unit is crores, and that 19.4% belongs to a different column.
The problem becomes harder when the same data is presented as a chart.
A conventional OCR engine may recognize the axis labels and numbers but fail to reconstruct what each bar represents.
A multimodal model can reason about the visual relationships, although it can still misread numbers or hallucinate values.
For financial RAG, I think of document-extraction quality in three layers:
Text fidelity: Did we extract the words correctly?
Structural fidelity: Did we preserve tables, headings, units, relationships and reading order?
Semantic fidelity: Did we actually capture what a graph, illustration or diagram communicates?
The third dimension is what many OCR pipelines miss.
And if that information never reaches your vector database, better embeddings and rerankers cannot recover it.
2. Stage One: Docling-Based PDF Processing
My initial approach used Docling, the open-source document conversion framework associated with IBM Research.
Docling offers conventional parsing pipelines and vision-language-model-based pipelines. These should not be confused with one another.
The standard pipeline typically combines PDF parsing, layout understanding, OCR where needed, table recognition and document reconstruction.
The VLM pipeline instead uses a vision-language model to understand rendered document pages.
My ingestion system sat within a broader pipeline involving document storage, queue-based processing, Markdown generation, chunking, embeddings and retrieval.
The basic flow was:
Financial PDF
|
v
Azure Blob Storage
|
v
Queue / Ingestion Worker
|
v
Docling PDF Converter
|
+---- OCR / Layout Analysis
|
+---- Table Recognition
|
+---- Picture Extraction
|
v
Markdown Export
|
v
Chunking + Embeddings
|
v
Vector Database
|
v
Financial RAG Retrieval
Docling was attractive for several reasons.
It provided a structured document representation, understood more than just plain text, and supported Markdown export. It also offered a reasonable starting point for processing large collections of PDFs.
For text-heavy documents, this architecture made sense.
But enabling image understanding exposed multiple problems.
Problem A: Picture detection did not mean picture understanding
One of the biggest misconceptions I encountered was assuming that detecting an image meant its content would appear in the exported Markdown.
Those were separate operations.
Docling might detect a picture, generate a cropped image representation, and still export a placeholder rather than useful semantic content.
In my experiments, one particularly unsuccessful run detected 41 pictures but generated no descriptions.
The downstream Markdown could contain:
<!-- image -->
For a retrieval system, that placeholder is almost useless.
It tells me an image existed. It tells me nothing about what the image contained.
Problem B: Picture annotations and Markdown export were different problems
Even generating descriptions was not always sufficient.
The picture annotations could exist within the document representation without appearing correctly in the final Markdown output.
This distinction produced additional debugging work around image annotations, export options and serialization.
I also encountered an API compatibility issue where a version of export_to_markdown() rejected an image_ref_mode argument.
One test involving chunk_1_9-20.pdf spent 173.65 seconds on conversion before encountering that export error.
That is especially painful because the expensive document conversion has already happened, but the expected final artifact still does not exist.
Interestingly, this was not an entirely isolated class of issue. Public Docling GitHub discussions have documented picture annotations being omitted from Markdown exports and inconsistent behavior around picture-description metadata.
Problem C: Page-range processing and image enrichment
To handle larger documents, I experimented with breaking PDFs into page ranges.
However, some combinations of page-range processing and picture enrichment introduced indexing problems, including an IndexError involving conv_res.pages[page_ix].
The general engineering problem was straightforward: page indices in a converted subset must remain correctly mapped to their original positions.
When that assumption breaks, extracted image content can be lost or processing can fail.
Problem D: Startup and model-loading overhead
There were also operational problems unrelated to extraction accuracy.
On a Python 3.12 environment, I encountered slow startup and interrupted imports involving Docling's dependency chain.
The traces went through modules such as Pydantic, pandas, Transformers and metadata scanning.
That introduced another source of overhead before useful document extraction even started.
In a persistent worker, heavy imports may be amortized.
In a frequently restarted process or constrained virtual machine, startup overhead can become an important part of actual end-to-end latency.
The lesson from this first architecture was clear:
A library successfully detecting document elements does not guarantee that the downstream output preserves their meaning.
3. Stage Two: Hybrid Docling + GPT-4o
My next approach was a hybrid.
Instead of expecting Docling to understand every visual element, I separated document parsing from image interpretation.
Docling continued handling the structural extraction.
GPT-4o handled the visual content.
The pipeline looked roughly like this:
PDF
|
v
Docling Extraction
|
+---- Text + Tables ---------+
| |
+---- Extracted Images |
| |
v |
Base64 Encoding |
| |
v |
GPT-4o |
| |
v |
Image Descriptions |
| |
+-------------------+
|
v
Enriched Markdown
|
v
RAG Ingestion
Base64 was already part of this stage.
The important difference is that Base64-encoded images were being used to enrich Docling's extraction, not replace Docling itself.
GPT-4o was useful for tasks such as explaining charts, interpreting diagram relationships, and identifying numeric information represented visually.
That helped address one of the original problems: the text representation no longer had to ignore visual elements entirely.
But the architecture still had a serious dependency.
GPT-4o could only interpret the images that Docling successfully identified and passed to it.
If a chart was not extracted, was incorrectly cropped, or lost the surrounding heading and axis context, the vision model received an incomplete view of the document.
This is an important architectural limitation.
Imagine a chart whose title says "Segment Revenue Growth" and whose legend is located outside the chart crop.
An isolated image crop may omit information needed to associate individual bars with business segments.
A whole-page image contains more surrounding context.
The hybrid architecture also required merging outputs from two separate extraction systems.
That creates its own questions:
- Where should each image description be inserted?
- How do you preserve page and section references?
- How do you avoid duplicating text already extracted by Docling?
- What happens if one stage succeeds and the other fails?
- How do you reconcile conflicting numbers extracted from different representations?
The hybrid design improved visual coverage, but it did not eliminate the complexity of the original parsing pipeline.
Eventually, I started questioning whether that complexity was necessary for the documents I was processing.
4. Stage Three: Removing Docling Entirely With Base64 + Multimodal LLMs
The next evolution was more fundamental.
Instead of parsing a PDF with Docling and sending selected images to a vision model, I moved toward rendering complete PDF pages as images and asking a multimodal LLM to convert the pages directly into Markdown.
The core architecture became:
PDF
|
v
Render Each Page as Image
|
v
Base64 Encode Image
|
v
Multimodal LLM API
|
+---- Recognize Text
|
+---- Reconstruct Tables
|
+---- Interpret Charts
|
+---- Preserve Numbers
|
v
Page-Level Markdown
|
v
Validation + Page Metadata
|
v
Chunking + Embeddings
|
v
RAG Retrieval
This removed Docling from the extraction path.
There was no longer a separate document-understanding framework deciding which figures were important enough to send to another model.
The multimodal model received the page itself.
Why Base64?
Strictly speaking, Base64 does not improve OCR accuracy.
It is simply a way of encoding binary image data so it can be carried in an API request, often as a data URL.
The improvement comes from changing what the model sees.
Instead of giving the model isolated image crops and separately extracted text, we can give it a complete rendered page.
That preserves visual relationships between headings, tables, graphs, footnotes and nearby explanations.
It also simplifies the extraction architecture.
The PDF renderer still needs to work correctly, and the model can still make mistakes. However, an entire class of layout-to-image-selection and output-merging problems disappears.
A simplified implementation
Here is an illustrative Python implementation using PyMuPDF and the OpenAI Responses API.
This demonstrates the approach, not a verbatim copy of my original production code.
import base64
import fitz
from openai import OpenAI
client = OpenAI()
PROMPT = """
Convert this financial document page into
high-fidelity Markdown.
Requirements:
1. Preserve all visible text and headings.
2. Reconstruct tables with correct rows
and columns.
3. Preserve numbers, units, negatives,
percentages and financial years exactly.
4. Describe charts and their relationships.
5. Do not invent unreadable values.
6. Mark uncertain values as [UNCLEAR].
7. Keep footnotes and financial context.
8. Return Markdown only.
"""
def extract_page(page):
pixmap = page.get_pixmap(
matrix=fitz.Matrix(2, 2),
alpha=False
)
image_bytes = pixmap.tobytes("png")
encoded = base64.b64encode(
image_bytes
).decode("utf-8")
response = client.responses.create(
model="gpt-5.6-luna",
input=[{
"role": "user",
"content": [
{
"type": "input_image",
"image_url":
"data:image/png;base64,"
+ encoded,
"detail": "original"
},
{
"type": "input_text",
"text": PROMPT
}
]
}]
)
return response.output_text
def extract_pdf(path):
document = fitz.open(path)
extracted_pages = []
for page_no, page in enumerate(document):
markdown = extract_page(page)
extracted_pages.append({
"page": page_no + 1,
"markdown": markdown
})
return extracted_pages
In production, this basic implementation needs additional capabilities: asynchronous processing, retries, concurrency limits, idempotent queue jobs, error classification, observability and cost monitoring.
Most importantly, it needs extraction validation.
A clean-looking Markdown response is not proof that the extracted financial values are correct.
Why the prompt matters
Document extraction is a relatively constrained task.
I am not asking a model to write an investment thesis, make a forecast or creatively summarize a company.
I want it to faithfully transcribe the document into a usable representation.
That means prompts should prioritize fidelity over verbosity.
For example:
"Do not calculate missing totals. Preserve the numbers as printed."
That instruction is more useful for ingestion than:
"Provide a helpful and detailed financial analysis."
The second prompt encourages interpretation. The first emphasizes faithful extraction.
I also prefer keeping uncertain values explicit rather than letting an LLM silently make up an answer.
5. Where IBM Granite-Docling Fits
IBM's Granite-Docling deserves a separate discussion because it is easy to confuse the Docling framework with a specific Docling VLM.
Granite-Docling is a compact document-understanding vision-language model, with a documented 258-million-parameter version.
It can convert document-page images into structured representations using DocTags, a markup format designed to preserve document layout and hierarchy.
This is genuinely interesting.
A small, specialized document model can have significant advantages over a general-purpose LLM.
It may be easier to self-host, offers a more constrained document-conversion task, and can integrate with existing document-processing infrastructure.
However, model parameter count is not the same thing as production inference cost.
You still need to consider model startup, framework initialization, runtime dependencies, throughput and available hardware.
A compact document VLM may be a strong option for environments where local processing or data residency is essential.
For my situation, the more important comparison was total pipeline complexity rather than parameter count alone.
A small local model is not automatically cheaper if maintaining its runtime requires persistent GPU infrastructure and low document volume leaves that infrastructure underutilized.
My verdict: Granite-Docling is worth considering for self-hosted document extraction, particularly when structured output and infrastructure control matter. It is not proof that every document pipeline needs a self-hosted VLM.
6. Mistral OCR: A Serious Alternative to General-Purpose Vision Models
Mistral's Document AI OCR service is another interesting alternative.
Unlike general-purpose multimodal models, it is built specifically around extracting document information.
As of November 2026, Mistral OCR 4.1 provides features including:
- Structured document extraction.
- Paragraph-level bounding boxes.
- Structural block labels.
- Confidence scores at different granularities.
- Table formatting and page-level output.
- An API-based processing workflow.
Its published list pricing is 5 per 1,000 pages for annotated document processing.
These are published prices, not measurements from my production workload.
The pricing is attractive because it is predictable on a per-page basis.
For a pipeline ingesting large quantities of PDFs, knowing the approximate cost before parsing can be valuable.
There is also an important distinction between using Mistral's hosted OCR endpoint and self-hosting a document model.
The hosted API removes the need for my team to maintain inference hardware.
Mistral has also described self-hosted enterprise deployment options for its OCR technology, but those should not be confused with unrestricted open-weight model availability.
What I would test before switching
For financial documents, I would specifically compare Mistral OCR against a general vision model on:
- Exact table cell reconstruction.
- Recognition of negative values and decimal points.
- Retention of financial units.
- Chart interpretation.
- Footnote association.
- Missing-content rate.
A document-specialized model may achieve excellent transcription fidelity without necessarily producing the best semantic description of a financial chart.
Those are related but different tasks.
Also, extracted images can still appear as references or assets rather than complete semantic explanations in the Markdown output.
So the output format and downstream image handling matter almost as much as the model's underlying OCR quality.
My verdict: Mistral OCR is one of the strongest API-first alternatives to evaluate when a production pipeline requires structured text, bounding boxes and predictable per-page pricing.
7. DeepSeek-OCR: The Self-Hosting Trade-Off
DeepSeek took a different approach with DeepSeek-OCR and the subsequent DeepSeek-OCR 2.
The original model explored efficient visual-text representation through optical compression.
DeepSeek-OCR 2 introduced a more specialized visual processing architecture, including a vision encoder, hybrid attention and a mixture-of-experts language-model component.
These are technically compelling developments.
But model architecture is only one part of an ingestion system.
The official DeepSeek-OCR 2 inference examples use an NVIDIA CUDA environment with PyTorch, FlashAttention, BF16 execution and model-specific dependencies.
That means self-hosting is not simply a matter of installing an OCR package and calling a function.
The operational checklist includes:
- Compatible GPU infrastructure.
- CUDA and PyTorch versions.
- Attention implementation and dependency compatibility.
- Model initialization and memory capacity.
- Inference worker lifetime.
- Queue concurrency.
- Batch processing and throughput tuning.
There is an important cost distinction here.
Model weights fitting into memory is not the same as the whole inference service operating comfortably under load.
Memory consumption also depends on activation buffers, input resolution, batching, runtime implementation and concurrent requests.
My experience with heavy document-processing dependencies made me especially sensitive to model-loading and environment-maintenance overhead.
For a startup, managing another dedicated inference service can become engineering work that has nothing to do with improving document accuracy.
This does not make DeepSeek-OCR a bad choice.
If an organization has GPUs available, processes large document volumes and needs self-hosted inference, that trade-off can become attractive.
But for an API-first application where document ingestion is one component of a larger AI product, hosted vision models can reduce operational complexity dramatically.
My verdict: DeepSeek-OCR is worth benchmarking for self-hosted, high-volume extraction, but the deployment cost should include engineering effort, warm capacity and dependency maintenance, not just GPU inference time.
8. Gemini Document Understanding: Skip Page Rendering Too?
Google Gemini offers another architectural option.
Rather than rendering a PDF page into an image ourselves, the Gemini API supports direct PDF inputs.
According to Google's documentation, its document-understanding capabilities include interpreting text, images, diagrams, charts and tables, with support for PDFs up to 1,000 pages or 50 MB.
This creates a fourth possible architecture:
PDF
|
v
Gemini Files API / Inline PDF
|
v
Native PDF Understanding
|
v
Structured Extraction
|
v
Markdown / JSON
|
v
RAG Pipeline
This can eliminate the need to maintain our own page-to-image rendering stage.
That is a genuine simplification.
However, simpler input handling does not automatically imply more accurate financial extraction.
Long-context processing creates separate questions about consistency, omission and validation.
A model receiving a hundred-page report may have more context, but the ingestion system still needs to determine whether every important numeric value has actually been preserved.
For archival ingestion, I would prefer page-level or small-batch extraction with explicit page attribution, even when using a model capable of processing the complete PDF.
This makes debugging easier.
If a particular value is missing, I can identify the original page rather than inspect a large document-wide response.
Gemini is therefore an interesting candidate for both document-level understanding and direct extraction, but I would still validate its page-level fidelity on financial reports.
9. Why Smaller GPT Models Became Interesting
One of my biggest takeaways was that document extraction does not necessarily require the most powerful reasoning model available.
The task is mostly constrained transcription and visual interpretation.
That makes smaller multimodal models attractive.
OpenAI's GPT-5.6 family now includes Terra and Luna, both with image-input support.
Terra is positioned as a balance between capability and cost.
Luna targets cost-sensitive, high-volume workloads.
Here is the published standard API pricing:
| Model | Input / 1M tokens | Output / 1M tokens |
|---|---|---|
| GPT-5.6 Luna | $0.20 | $1.20 |
| GPT-5.6 Terra | $2.00 | $12.00 |
These are current model options for evaluating the architecture, not a claim that my earlier GPT-4o and Base64 migration used these exact 2026 model versions.
What could this mean per page?
Suppose we process a page rendered as a 1,024 × 1,024 image.
Under OpenAI's published patch-based image accounting for these models, this corresponds to approximately 1,229 image input tokens.
Now assume another 250 input tokens for instructions and 700 output tokens for the extracted Markdown.
That produces a hypothetical cost calculation:
| Model | Estimated cost / page | Per 1,000 pages |
|---|---|---|
| GPT-5.6 Luna | $0.00114 | $1.14 |
| GPT-5.6 Terra | $0.01136 | $11.36 |
| Mistral OCR 4.1 | $0.00400 | $4.00 |
This is an illustrative cost model, not a measured benchmark. Actual costs depend heavily on image dimensions, token consumption, output verbosity, caching, model settings and processing overhead.
The Mistral row is its published per-page OCR price, not a token-based estimate.
The comparison is nevertheless useful.
A smaller general-purpose vision model can be economically competitive with dedicated OCR services, depending on the output requirements.
But cost alone should not determine the model choice.
I would happily pay more for an extraction model that preserved financial values more reliably.
One incorrectly extracted decimal point can be far more expensive than the entire document processing bill.
10. Numerical Precision Was the Real Quality Problem
There are several ways a PDF extraction model can be wrong.
The obvious failure is failing to extract any text.
The more dangerous failure is extracting text that looks believable but is numerically incorrect.
Consider these illustrative examples:
| Original | Incorrect extraction | Failure |
|---|---|---|
| 12.35% | 12.85% | Decimal/digit substitution |
| (₹245 Cr) | ₹245 Cr | Negative value lost |
| 1,24,560 | 12,456 | Missing digit |
| FY24 | FY25 | Incorrect period |
| ₹ million | ₹ crore | Unit mismatch |
These are examples of errors to test for, not claimed observations from my historical production logs.
The risk is that an LLM can produce extremely clean Markdown containing one incorrect number.
If that value gets embedded, retrieved and cited by a downstream AI system, the mistake becomes much harder to identify.
This is why I distinguish semantic coverage from numerical fidelity.
The direct Base64 approach addressed the loss of visual context that motivated our architecture change.
But any claim of percentage improvements in numeric precision requires a controlled benchmark.
The evaluation should compare every pipeline against manually verified ground truth, using metrics such as:
- Exact-match accuracy for numeric values.
- Correct association of numbers with row and column labels.
- Preservation of units, signs and percentages.
- Table cell completeness.
- Chart information coverage.
- Missing or hallucinated content.
- End-to-end latency and processing cost.
A model that recognizes every chart but incorrectly copies a few numbers is not necessarily better for financial RAG than a more conservative parser.
11. What My Actual Experiments Established
These are the concrete observations I can point to from my earlier development work.
| Experiment / issue | Observed result |
|---|---|
| Docling picture-description run | 41 detected pictures, zero descriptions |
| Four-page Docling conversion chunk | 266.96 seconds |
| Another Docling conversion | 173.65 seconds before a Markdown export API error |
| Docling Markdown output | Image placeholders and missing descriptions |
| Page-range plus image processing | Indexing failure in the conversion result |
| Docling dependency initialization | Slow/interrupted imports on a Python VM |
| Hybrid Docling + GPT-4o | Introduced visual interpretation alongside structural parsing |
| Direct Base64 + multimodal LLM | Removed Docling from the extraction path and enabled whole-page visual interpretation |
These observations explain the architectural decisions.
They do not constitute a complete head-to-head accuracy benchmark between the latest versions of every OCR model.
And I think that distinction matters.
Many online comparisons take a few screenshots, look at the generated Markdown, and declare one model the winner.
For a financial research system, that is not enough.
12. The Benchmark I Would Run Today
If I were evaluating the latest models for this problem again, I would construct a fixed dataset of financial PDF pages.
The evaluation set would include born-digital reports, scanned statements, multi-column pages, tables with merged cells and graph-heavy investor presentations.
Every model would receive equivalent source content, with a documented resolution and extraction prompt.
I would compare:
| Candidate | Main evaluation objective |
|---|---|
| Docling Standard | Text and structural fidelity |
| Docling + GPT-4o | Visual enrichment quality |
| Granite-Docling | Self-hosted document conversion |
| Mistral OCR 4.1 | Structured OCR and numeric fidelity |
| DeepSeek-OCR 2 | Self-hosted throughput and fidelity |
| Gemini PDF understanding | Native PDF parsing and context preservation |
| GPT-5.6 Luna | Low-cost multimodal extraction |
| GPT-5.6 Terra | Accuracy/cost trade-off |
For each model, I would record cold-start and warm processing times separately.
For self-hosted models, I would additionally record peak GPU memory, throughput at several concurrency levels and deployment cost.
For API models, I would track tokens, per-page cost, rate limiting and retry overhead.
And for every model, numerical accuracy would be evaluated against the same manually verified values.
This would tell us something much more useful than which output looked the cleanest.
It would tell us which system we could trust enough to put in front of financial research users.
13. What I Would Deploy Today
I would not recommend replacing every document-processing system with a vision LLM.
That would be unnecessary.
For clean, born-digital PDFs where text and structure are already accessible, Docling's standard pipeline remains a reasonable option.
If the task requires dedicated OCR with structured blocks, bounding boxes and predictable page pricing, Mistral OCR is compelling.
For privacy-sensitive environments with suitable infrastructure, Granite-Docling and DeepSeek-OCR deserve serious consideration.
Gemini is attractive when native PDF understanding and broader document context are important.
For the image-heavy financial documents that motivated my architecture change, I prefer the operational simplicity of direct multimodal page extraction.
I would begin with a smaller model such as Luna, validate its extraction quality, and escalate difficult pages to a stronger model such as Terra when justified.
However, that routing policy should be guided by actual error rates rather than assumptions about model size.
And no model should be trusted merely because it generated attractive Markdown.
The extraction layer needs validation, provenance and the ability to flag ambiguous values.
Final Verdict: The Best Pipeline Was Not Necessarily the Most Sophisticated
My original pipeline tried to solve document ingestion by combining specialized components.
Docling handled parsing, OCR and structural extraction.
Then GPT-4o helped recover the meaning of extracted images.
Eventually, I moved toward direct Base64 page-image extraction through multimodal LLMs.
For my use case, the architectural simplification was as meaningful as the improvement in visual coverage.
I removed a dependency between detecting visual elements and interpreting them.
I reduced the amount of output-merging logic required.
And I could treat the PDF page itself as the primary source of truth for extraction.
The broader lesson is not that Docling is bad or that general-purpose LLMs always outperform specialized OCR.
It's that document extraction should be evaluated by what the downstream application can actually retrieve and trust, not by whether the parser successfully produced Markdown.
For financial RAG, that ultimately means preserving text, relationships, charts and numbers faithfully.
A missing chart can make an answer incomplete.
A wrong number can make an answer dangerous.
And a successful PDF conversion can still produce both problems.
Worth it? For my visually complex financial-document workload, moving toward direct multimodal extraction was worth the architectural change. But the next improvement I would prioritize is not a more powerful model. It is a more rigorous, numerical ground-truth evaluation layer.
That is what turns a convenient extraction pipeline into a trustworthy financial-data ingestion system.
References and Technical Documentation
- Docling official documentation
- IBM Granite-Docling model documentation
- Mistral OCR 4.1 documentation
- Mistral OCR processor and structured output
- DeepSeek-OCR GitHub repository
- DeepSeek-OCR 2 model card
- Gemini document understanding
- OpenAI image and vision processing
- GPT-5.6 Luna model and pricing
- GPT-5.6 Terra model and pricing
- Docling picture-annotation Markdown export issue
- Docling picture-description export inconsistency