Empowering Meta-analysis: Leveraging Large Language Models for Scientific Synthesis
Jawad Ibn Ahad, Rafeed Mohammad Sultan, Abraham Kaikobad, Fuad Rahman, Mohammad Ruhul Amin, Nabeel Mohammed, Shafin Rahman
✅ Accepted at IEEE International Conference on Big Data 2024
📄 Paper: IEEE Xplore
💻 Code & Dataset: github.com/EncryptedBinary/Meta_analysis
Abstract
This study investigates automating meta-analysis in scientific documents using large language models (LLMs). Meta-analysis synthesizes findings from multiple studies but is labor-intensive when done manually. Our approach fine-tunes LLMs for efficient, automated meta-analysis using Retrieval Augmented Generation (RAG) and a novel loss metric, Inverse Cosine Distance (ICD). The fine-tuned models achieved 87.6% relevant meta-analysis abstracts and reduced irrelevance from 4.56% to 1.9%, demonstrating efficiency in a low-resource environment.
Introduction
Scientific meta-analysis requires experts to read, compare, and synthesize hundreds of papers — a bottleneck that limits knowledge discovery at scale. LLMs offer a promising route to automation, but standard fine-tuning on context-length-restricted models produces incoherent or irrelevant summaries. Two core challenges: (1) input papers exceed LLM context windows, and (2) standard cross-entropy loss does not penalize semantically irrelevant outputs.
We address both via chunked RAG pipelines and a semantic-distance-aware loss function.
🎯 Key Contributions
- Inverse Cosine Distance (ICD) Loss — A novel training objective that minimizes semantic distance between generated and reference meta-analysis abstracts, replacing standard CE loss.
- RAG-based Context Management — Chunking and semantic retrieval to handle papers exceeding LLM context limits (up to 32K tokens → manageable 2K chunks).
- Meta-Analysis Dataset (MAD) — 625 annotated scientific document clusters with human-evaluated meta-analysis abstracts.
- Low-Resource Efficiency — Fine-tuned 7B models achieve better BLEU/ROUGE than GPT-4 baselines on domain-specific datasets.
Methodology
Figure: Three approaches compared — (a) Paraphraser-based, (b) Standard RAG, (c) Our approach: fine-tuned LLMs with ICD loss and RAG. Input studies are chunked and encoded; relevant chunks retrieved via semantic search; fine-tuned LLM synthesizes the meta-analysis abstract under ICD supervision.
Dataset Statistics
| Metric | Actual | Chunked |
|---|---|---|
| Min. input context length | 733 | 1,005 |
| Max. input context length | 32,767 | 2,000 |
| Avg. input context length | 16,890 | 1,542 |
| Total Instances | 625 | 7,447 |
- Train / Validation / Test: 400 / 75 / 50 documents
- Human Evaluators: 13 annotators (9M, 4F), avg. age 23
📊 Results — Full Model Comparison Across 3 Datasets
| Method | Model | Open-i BLEU ↑ | Open-i ROUGE ↑ | writer_sum. BLEU ↑ | writer_sum. ROUGE ↑ | CL-SciSumm BLEU ↑ | CL-SciSumm ROUGE ↑ |
|---|---|---|---|---|---|---|---|
| Established | GPT-4 w/ ICL | 46.0 | 68.2 | — | — | — | — |
| Established | InstructGPT davinci v2 | — | — | — | — | 48.0 | — |
| Established | GCN Hybrid | — | — | — | — | — | 33.88 |
| Pre-trained | Falcon 7B | 0.19 | 3.17 | 0.76 | 5.19 | 0.71 | 2.21 |
| Pre-trained | Gemma 7B | 2.13 | 8.81 | 4.47 | 30.28 | 2.44 | 20.78 |
| Pre-trained | Orca-2 7B | 3.53 | 8.36 | 4.29 | 22.51 | 2.86 | 15.55 |
| Pre-trained | StableLM-Base-Alpha 7B | 2.01 | 2.45 | 3.56 | 15.36 | 1.17 | 16.58 |
| Pre-trained | Llama-2 7B | 4.81 | 10.28 | 5.21 | 31.61 | 3.01 | 22.84 |
| Pre-trained | Mistral-v0.1 7B | 1.21 | 6.57 | 1.62 | 6.37 | 0.36 | 2.55 |
| Ours (FT+ICD) | Llama-2 7B | 10.14 | 27.39 | 12.66 | 31.36 | 7.15 | 25.22 |
| Ours (FT+ICD) | Mistral-v0.1 7B | 12.42 | 31.57 | 14.56 | 35.56 | 8.38 | 27.29 |
Fine-tuned models with ICD loss significantly outperform all pre-trained baselines. Mistral-v0.1 7B FT is the best open-source model across all three datasets.
📚 Citation
@inproceedings{ahad2024empowering,
title={Empowering Meta-analysis: Leveraging Large Language Models for Scientific Synthesis},
author={Ahad, Jawad Ibn and Sultan, Rafeed Mohammad and Kaikobad, Abraham and Rahman, Fuad and Amin, Mohammad Ruhul and Mohammed, Nabeel and Rahman, Shafin},
booktitle={IEEE International Conference on Big Data (BigData)},
pages={},
year={2024},
doi={10.1109/BigData62323.2024.10825310}
}