Figures as Evidence: Multi-Image Scientific Generation

✅ Accepted at 20th International Conference on Document Analysis and Recognition (ICDAR 2026)

📄 Paper: Yet to publish
🏛 Affiliation: Apurba-NSU R&D Lab, Department of ECE, North South University, Dhaka, Bangladesh
🤝 Equal Contribution: Jawad Ibn Ahad*, Mritunjoy Chakraborty* (* Equal contribution)
💰 Grant: NSU CTRGC Research Grant [CTRG-24-SEPS-12]

Abstract

Scientific documents present a fundamental challenge for automated understanding: key findings are distributed across multiple figures, tables, and captions that must be interpreted collectively rather than independently. We propose a structured multimodal pipeline for section-level scientific content generation that jointly reasons over multiple figures within a document section — including Methodology, Exploratory Data Analysis, Results, and other visually grounded sections. Instead of processing figures individually, we introduce a deterministic visual canonicalization strategy that aggregates all figures in a section into a unified composite representation, enabling cross-figure reasoning while maintaining fixed computational complexity. A fixed-token compression mechanism transforms high-resolution visual inputs into a compact visual prefix, ensuring constant-length conditioning regardless of the number of figures. This design mitigates the quadratic computational growth typically associated with multi-figure inputs and allows scalable long-form generation using standard generative transformers. We evaluate the proposed framework on CoCoOne, a benchmark comprising 668 scientific papers across Agriculture, Environment, and Atmosphere domains with over 10,000 figure–text pairs. Across multiple vision–language backbones, the approach achieves consistent improvements in lexical overlap, semantic alignment, and structural completeness.

Introduction

Figure 1: Comparison between existing multimodal pipelines and the proposed framework. Left: conventional approaches process individual figures (Seq_1, Seq_2, Seq_3) independently and sequentially before aggregating into scientific content — leading to fragmented contextual understanding. Right: our proposed pipeline first unifies multiple figures into a structured composite representation, compresses it via DeepEncoder into a fixed-length visual prefix, and provides it to a generative VLM for globally coherent, long-form scientific content generation grounded jointly across all figures.

Existing multi-figure approaches face three key limitations:

• (a) Fixed token expansion per image causes visual token count to grow linearly with number of figures, leading to quadratic attention cost and high memory use.
• (b) Independently encoded images provide limited structured alignment across figures, forcing the language model to infer cross-figure relationships implicitly.
• (c) Independent tokenization can introduce redundant visual tokens and fragmented grounding, causing long-form outputs to miss cross-references or lose narrative consistency.

🎯 Key Contributions

• Structured Multi-Figure Canonicalization — A deterministic grid-based visual canonicalization strategy that transforms a variable number of figures and tables into a unified 1024×1024 composite representation, enabling joint cross-figure reasoning without modifying the vision–language architecture.
• Fixed-Length Visual Prefix Compression — A fixed-length visual compression framework converting high-resolution multi-figure inputs into a compact 256-token visual prefix — constant computational cost regardless of the number of input figures.
• Architecture-Agnostic Evaluation — Evaluated across multiple VLM backbones (Qwen2.5-VL-7B, InternVL-7B, LLaVA-1.5-7B), domain splits, ablation settings, and external multimodal summarization benchmarks (MMSS, MSMO, M3LS, OASum) without benchmark-specific fine-tuning.

Methodology

Figure 2: Overview of the proposed pipeline. A variable-sized set of N figures and tables is arranged into a deterministic 1024×⌈√N⌉×⌈√N⌉ composite grid and resized to 1024×1024. The image is processed by a frozen SAM-based vision encoder (ViTDet, 95M) to extract dense spatial features under local attention. A 16× convolutional downsampling module reduces 4096 patch tokens to 256 vision tokens. Tokens are projected via MLP + learnable positional embeddings to match the decoder embedding dimension. The resulting visual prefix is concatenated with tokenized prompts and passed to a decoder-only VLM with LoRA adapters, generating long-form scientific text autoregressively.

Problem Formulation

Let I = {Iᵢ}ᴺᵢ₌₁ be a variable-sized set of figures, each Iᵢ ∈ ℝ^{Hᵢ×Wᵢ×C}. Let Y = (y₁,…,yₜ) be a long-form textual sequence. The objective is to learn:

Multi-Figure Canonicalization

For N input figures, construct composite grid image I^grid = Φ_grid(I):

• Grid size: c = ⌈√N⌉ columns, r = ⌈N/c⌉ rows
• Each figure resized to s×s (s=1024) per cell
• Final composite resized to canonical 1024×1024

Optical Token Compression (DeepEncoder)

Achieved via adaptive average pooling — 16× token reduction, fixed regardless of N.

Modality Alignment & Prefix Construction

Alignment MLP: P: ℝ^256 → ℝ^{dℓ} (applied token-wise)

Training

• Backbone VLMs: Qwen2.5-VL-7B, InternVL-7B, LLaVA-1.5-7B
• LoRA: rank r=16, ~5.6×10⁸ trainable params (~7% of model)
• Optimizer: 8-bit AdamW — lr=2×10⁻⁵ (transformer), 5×10⁻⁵ (visual projection)
• Weight decay: 0.05 · Gradient clipping: 1.0 · Cosine scheduling · Warmup: 150 steps
• Inference: Nucleus sampling (p=0.9), temperature=0.7, repetition penalty=1.15, 1500–2500 words/document
• Hardware: Single NVIDIA RTX 5090 GPU

📊 Datasets

Table 1: CoCoOne Dataset Statistics (3 Domain Splits)

Content extracted exclusively from Results and Analysis sections. Tables represented as rendered images.

📊 Results

Table 2: Baseline Model Scale and Training Regime

Our model is trained only on CoCoOne and evaluated zero-shot on MMSS, MSMO, M3LS, OASum — no benchmark-specific fine-tuning.

Table 3: Performance Across Backbone VLMs and Domain Splits

⃝ = Qwen2.5-VL-7B · □ = InternVL-7B · △ = LLaVA-1.5-7B. Bold = best, underline = second best per split.

Qwen2.5-VL-7B achieves the highest average (0.564–0.576) across all splits. Largest gains are in BERTScore and Cosine Similarity — indicating stronger contextual grounding. Performance is stable across domain shifts, confirming architecture-agnostic robustness.

Table 4: Cross-Benchmark Generalization (Zero-Shot on External Datasets)

R1/R2/RL = ROUGE-1/2/L (×100) · BS = BERTScore F1. Existing baselines trained on their target benchmarks; ours is zero-shot.

Despite zero-shot evaluation, our model approaches SOTA ROUGE scores while achieving the highest BERTScore on MMSS and MSMO, demonstrating strong cross-dataset generalization.

Table 5: Long-Form Quality and Cross-Dataset Generalization

(a) LLM-based Evaluation (1–5 scale: Coherence / Factual Consistency / Visual Grounding)

(b) Cross-Dataset Generalization — ROUGE-1 / BERTScore F1 (Qwen2.5-VL-7B)

📊 Ablation Study

Table 6: Grid vs. Sequential Input and Reranker Selection

(a) Canonical Grid vs. Sequential Figure Processing

Grid maintains constant 256-token prefix and improves ROUGE-L from 23.3 → 24.5 with 5 figures while reducing latency from 3.4× → 1.0×.

(b) Reranker-Assisted Figure Selection (N > 6)

Effect of Input Figures and Visual Token Budget

Figure 3 (left): Precision, Recall, and F1 gradually decrease as input grows from 3 to 8 figures (F1: 0.76 → 0.72 → 0.68), showing that dense multi-figure settings make visual grounding more challenging.

Figure 4 (right): Increasing the visual prefix from 128 to 256 tokens substantially improves all metrics. 512 tokens yields only marginal gains — 256 tokens is the optimal performance–efficiency trade-off.

📚 Citation

@inproceedings{ahad2026figures,
  title={Figures as Evidence: Multi-Image Scientific Generation},
  author={Ahad, Jawad Ibn and Chakraborty, Mritunjoy and Rahman, Fuad and Momen, Sifat and Rahman, Shafin and Mohammed, Nabeel},
  booktitle={20th International Conference on Document Analysis and Recognition (ICDAR)},
  year={2026},
  note={Yet to publish}
}