← All projects

MiniVLMDocEval: Are Sub-1B VLMs Ready for Documents?

Six open vision–language models under one billion parameters, five document benchmarks, one matched protocol. Which tiny model should you build on, where exactly does it break, and can a targeted fine-tune fix it without breaking something else?

  • Independent research · 2026
  • VLMEvalKit
  • LoRA · PEFT · TRL
  • Hugging Face
6generalist VLMs, 0.45–0.9B parameters
5benchmarks, from reading text to answering over tables
70.3best mean score: Qwen3.5-0.8B, winning 3 of 5
N = 1000seed-42 subsets, greedy decoding, 16-bit, 128-token cap

A diagnostic ladder

The benchmarks are ordered so each rung adds one skill. A model that fails high on the ladder but passes lower rungs tells you which skill is missing.

  1. OCRBenchRecognition: can it read the text at all?
  2. DocVQAExtraction: find an answer on a scanned page.
  3. ChartQANumeric: read values and trends off charts.
  4. InfoVQALayout: reason over dense infographics.
  5. TableVQATabular: look up and aggregate over rendered tables.

Leaderboard

Six models, five benchmarks

Interactive

Click a column header to sort. Pick a benchmark below to chart it. Scores are each benchmark's native metric on a 0–100 scale.

Accuracy vs. speed

Architecture beats parameter count. The smallest model, LFM2.5-VL at 0.45B, outranks both 0.9B models, wins ChartQA, and is about five times faster than Qwen3.5. LFM2.5-VL and Qwen3.5 own the Pareto front; FastVLM is dominated on both axes.

Resolution strategy is the lever. How a model tiles or compresses the page image drives both accuracy and latency more than scale does. FastVLM's aggressive token compression is why it collapses on OCRBench (26.2) despite a high-resolution encoder.

Latency is mean seconds per sample, indicative only (mixed T4/L4 GPUs), so only the ordering is robust. Hover a point for details.

The gap, localised

No model passes 50.1 on TableVQA, so tables are the frontier. Splitting the benchmark into its four sub-domains shows the best model's weakness is plain visual lookup over Wikipedia-style tables, not numbers: Qwen3.5 scores 84.0 on financial tables but 27.8 on lookup. That corrects a common assumption in the table-QA literature, and since lookup is roughly half the benchmark, it is where the mean is lost.

Part 2: a fix that was rejected, on purpose

LoRA on table lookup vs. a pre-registered no-regression gate

Interactive

Before training, I fixed a rule: the adapter ships only if it improves lookup without hurting any protected benchmark. Same seed-42 subsets before and after.

The adapter failed the gate. TableVQA fell from 50.1 to 19.6 and DocVQA from 89.3 to 50.8: catastrophic forgetting. The diagnosis is an answer-style mismatch: verbose training answers versus short-answer scoring. The report gives a corrected recipe.

A gate that rejects a degraded model is the protocol working as designed. The training set and the failed adapter are published anyway (LoRA adapter), so the negative result can be checked rather than taken on trust.

Examples

What Qwen3.5-0.8B gets right, and wrong

Browse
Benchmark example

Reproducibility

Everything is seeded and pinned: VLMEvalKit at a fixed commit (cloned, never forked), per-model precision, fixed N and seed, and greedy decoding. Two incompatible transformers versions are handled as two phases of a single reproduce.sh run or Colab notebook, and results merge into one comparison table. I contributed wrappers for three models the harness lacked: FastVLM-0.5B, Qwen3.5-0.8B and LFM2.5-VL-450M.