AutoDSPy: Automating Modular Prompt Design with Reinforcement Learning for Small and Large Language Models

✅ Accepted at EMNLP 2025 Industry Track · Pages 2881–2896 · November 4–9, 2025, Suzhou, China

📄 Paper: ACL Anthology · PDF
💻 Code: github.com/nafew-azim/AUTODSPy · DOI: 10.5281/zenodo.17276875

Abstract

Large Language Models (LLMs) excel at complex reasoning tasks, yet their performance hinges on the quality of their prompts and pipeline structures. Manual prompt design — as used in frameworks like DSPy — poses significant limitations: it is time-intensive, demands substantial expertise, and lacks scalability. To overcome these challenges, we introduce AutoDSPy, the first framework to fully automate DSPy pipeline construction using reinforcement learning (RL). AutoDSPy leverages an RL-tuned policy network to dynamically select optimal reasoning modules — such as Chain-of-Thought for logical tasks or ReAct for tool integration — along with input-output signatures and execution strategies, entirely eliminating the need for manual configuration. Experimental results on the GSM8K and HotPotQA benchmarks demonstrate that AutoDSPy outperforms traditional DSPy baselines, achieving accuracy gains of up to 4.3% while reducing inference time, even with smaller models like GPT-2 (127M). By integrating RL-based automation, AutoDSPy enhances both efficiency and accessibility, simplifying the development of structured, high-performing LLM solutions and enabling scalability across a wide range of tasks.

Introduction

LLMs have showcased extraordinary abilities in generating human-like text and tackling complex reasoning tasks, yet their effectiveness hinges heavily on how prompts are crafted and pipelines configured. Manual prompt engineering is a labor-intensive process, often requiring extensive trial-and-error, deep expertise, and significant time investment — sometimes yielding suboptimal results.

Figure 1: Comparison of pipeline construction methods. (a) Traditional DSPy requires manual, static configuration of modules and signatures — done entirely by the user. (b) LLM-driven refinement iteratively improves prompts using a pretrained LLM (e.g., PromptWizard). (c) AutoDSPy (Ours) uses reinforcement learning to train a finetuned policy network that builds optimized pipelines based on execution feedback — fully automated end-to-end.

DSPy introduced a structured, modular framework for building LLM pipelines using three components: signatures (typed I/O mappings), modules (reasoning strategies: ChainOfThought, ReAct, ProgramOfThought), and teleprompters (demonstration-based optimizers). However, DSPy still requires manual selection of modules, specification of signatures, and choice of optimization strategies — constraining scalability in dynamic or heterogeneous task environments.

AutoDSPy addresses these limitations by automating pipeline synthesis entirely through a policy network (PN) π_θ.

🎯 Key Contributions

• AutoDSPy Framework — The first fully automated extension of DSPy, using RL to eliminate manual pipeline configuration.
• Policy Network Design — An RL-tuned LLM that dynamically selects modules, signatures, and strategies for task-adaptive pipeline synthesis.
• Three RL Strategies — REINFORCE (◇), PPO (♡), and GRPO (♣) — each offering distinct efficiency-accuracy trade-offs.
• Small-Model Competitiveness — GPT-2 (127M) with AutoDSPy outperforms all DSPy baselines on GSM8K, demonstrating accessibility without high-compute policy models.
• Empirical Superiority — Proven gains in accuracy and efficiency on GSM8K and HotPotQA, surpassing manual DSPy while preserving modularity and flexibility.

Methodology

Problem Formulation

A pipeline P of length L is defined as a sequence of module-signature pairs optimized under a selected teleprompter:

where each M_k ∈ M (reasoning module), s_k ∈ S (signature), and t ∈ T (teleprompter). The pipeline executes as:

The goal is to learn a Policy Network π_θ that maps input x to an optimal pipeline P, maximizing:

Binary reward R = 1{o_{L-1} = y} with semantic fallback via LLM assessment for partial correctness.

Policy Network

The Policy Network (PN) π_θ is a finetuned LLM that translates input prompts into pipelines. Seven transformer-based LLMs tested as policy networks: GPT-2-127M, LLaMA-3.2-1B, Gemma-3-1B, DeepSeek-R1-1.5B, Qwen-2.5-1.5B, Mistral-v0.1-7B, Phi-4-14B.

RL Algorithms

1. REINFORCE (◇) — Stochastic gradient ascent with EMA baseline: \(\mathcal{L} = -\sum_t \log \pi_\theta(a_t | s_t) \cdot (r - b); \quad b \leftarrow 0.9b + 0.1r\)

2. PPO (♡) — Stabilizes training via clipped objectives and entropy regularization with advantage estimates A_t.

3. GRPO (♣) — Reduces variance by normalizing rewards within prompt-specific groups: \(\bar{r}_i = \frac{r_i - \text{mean}(r)}{\text{std}(r)}, \quad A_i = \bar{r}_i - \bar{r}\)

AutoDSPy Detailed Workflow

Figure 2: AutoDSPy detailed workflow. Training (outside dashed region): randomly selects (x,y) from D; PN π_θ generates several sample pipelines P from DSPy’s pool of modules M, signatures S, and teleprompters T; reward R(o_{L-1}, y) = 1{o_{L-1}=y} computed on final output; loss L computed via one of three RL algorithms (REINFORCE/GRPO/PPO); backpropagation updates θ ← θ − η∇θL. Repeat for N=200 episodes. Inference (dashed region): user prompt → PN generates one pipeline → LLM executes it.

Hyperparameters (Table 3)

Experimental Setup

• Datasets: GSM8K (1,300 test samples, mathematical reasoning); HotPotQA (1,000 test samples, multi-hop QA)
• Training: 200 training samples, 200 episodes
• Inference LMs: LLaMA-3.1-8B and Qwen-2.5-14B
• DSPy baselines: Predict, CoT, MIPROv2*, BetterTogether* (* used with DSPy-CoT pipeline)
• Hardware: NVIDIA T4 GPU, RTX 4090, TPU VM v3-8

📊 Results

Table 1: Main Performance — GSM8K & HotPotQA (Accuracy % / Inference Time s)

Bold Red = best; Blue = second best. No teleprompters used for AutoDSPy.

Key Findings

• GRPO (♣) best overall: GPT-2-127M GRPO achieves 82.4% GSM8K (vs. DSPy-CoT 79.9%), reducing inference from 11.1s → 8.1s. A 127M policy model beats manual expert design.
• GRPO (♣) best HotPotQA: DeepSeek-R1-1.5B GRPO achieves 77.2% HotPotQA (vs. DSPy-CoT 73.5%) — +3.7% multi-hop gain.
• REINFORCE (◇) fastest: 5.1s on GSM8K while maintaining competitive accuracy — ideal for latency-critical deployment.
• PPO (♡) balanced: 57.7% → 76.6% on GSM8K with 300 episodes, showing strong policy optimization under more training.
• MIPROv2 collapses on HotPotQA: Only 29.6% — multi-hop reasoning degrades with this teleprompter despite 76.1% on GSM8K.
• BetterTogether failure on HotPotQA: 31.8% vs. 80.4% on GSM8K — teleprompters tuned for arithmetic fail to transfer to multi-hop QA.

Table 2: Generalizability — MGSM, MBPP, LegalBench, PubMedQA (Accuracy % / Time s)

No teleprompters used for AutoDSPy. MBPP metric: pass@1. Best in bold red, second in blue.

AutoDSPy consistently outperforms DSPy baselines across all four additional benchmarks: +2.2% on MGSM (multilingual math), +3.4% on MBPP (code synthesis), +3.0% on LegalBench (legal reasoning), +2.4% on PubMedQA (biomedical QA). GRPO (♣) achieves top accuracy; REINFORCE (◇) prioritizes efficiency.

📊 Ablation Study

Figure 3: Learning Curves & RL Ablation

Figure 3 (a): Accuracy vs training episodes/samples on GPT-2-127M with GRPO (LLaMA-3.1-8B inference). Accuracy increases steadily to 200 episodes/samples and mostly plateaus at 300. However, 300 episodes causes a significant drop on GSM8K (82.4% → 58.8%), indicating overfitting. (b): Impact of varying RL training on GPT-2 across REINFORCE (◇), PPO (♡), GRPO (♣). PPO (♡) improves substantially with more episodes (57.7% → 76.6% on GSM8K). Larger training samples often reduce accuracy due to insufficient episodes to process additional data.

Effect of Training Episodes (200 vs. 300)

200 episodes is optimal for most configurations. 300 episodes causes severe accuracy drop for GRPO on GSM8K (82.4% → 58.8%) due to overfitting, while PPO benefits substantially (+19.0%), underscoring the need for precise episode tuning.

Figure 4: Teleprompter Impact & GRPO k Sensitivity

Figure 4: Teleprompters (LabeledFewShot, BootstrapFewShot, BootstrapFewShotWithRandomSearch, KNNFewShot, COPRO, BootstrapFinetune, Ensemble) evaluated across all three RL algorithms on GSM8K (G) and HotPotQA (H). Teleprompters increase latency across all algorithms (REINFORCE: 5.1s → 24.5s; GRPO: 8.1s → 30.8s; PPO: 7.9s → 28.1s on GSM8K) with inconsistent accuracy gains (GRPO drops 82.4% → 74.6%). Right: GRPO k sensitivity — k=5 optimal for GSM8K precision (82.4%), k=15 benefits HotPotQA (73.9% → 77.1%).

Teleprompter Analysis

Seven teleprompters tested; all excluded from final AutoDSPy design:

Teleprompters excluded from final framework: the latency overhead (up to 5×) is impractical for industrial deployment and accuracy gains are inconsistent — GRPO accuracy drops 7.8% on GSM8K and 4.6% on HotPotQA when teleprompters are added.

📊 Statistical Analysis

Figure 5: Exact-Match Rates Across 6 Benchmarks

Figure 5: Exact-match rates (%) on 100 representative samples per dataset comparing DSPy methods, baseline policies (Random, Hardcoded), and AutoDSPy RL approaches across GSM8K (GSM), HotPotQA (HP), MGSM (MG), MBPP (MB), LegalBench (LB), PubMedQA (PM). GRPO (♣) achieves the best performance in the 81–89% range across all six benchmarks. Performance hierarchy: GRPO > PPO > REINFORCE holds consistently across all datasets (p < 0.01 in all paired t-tests vs. CoT baseline).

Relative Improvements of GRPO over DSPy-CoT

Table 4: Performance Consistency — 5 Independent Runs (GPT-2-127M, GRPO, GSM8K)

Coefficient of variation < 1.2% — GRPO consistently discovers high-quality pipeline configurations regardless of random initialization. Semantic fallback (cosine similarity ≥ 0.92) contributes an additional 2–3% accuracy improvement over exact-match evaluation across all datasets.

📚 Citation

@inproceedings{azim2025autodspy,
  title={AutoDSPy: Automating Modular Prompt Design with Reinforcement Learning for Small and Large Language Models},
  author={Azim, Nafew and Alam, Abrar Ur and Omar, Hasan Bin and Jami, Abdullah Mohammad Muntasir Adnan and Ahad, Jawad Ibn and Kabir, Muhammad Rafsan and Hossain, Md. Ismail and Rahman, Fuad and Amin, Mohammad Ruhul and Rahman, Shafin and Mohammed, Nabeel},
  booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track},
  pages={2881--2896},
  year={2025},
  organization={Association for Computational Linguistics}
}