LAET: A Layer-Wise Adaptive Ensemble Tuning Framework for Pretrained Language Models

✅ Accepted at IEEE International Conference on Big Data 2025

📄 Paper: arXiv:2511.11315 · IEEE Xplore

Abstract

We propose LAET (Layer-wise Adaptive Ensemble Tuning), a novel fine-tuning strategy for pretrained LLMs that identifies and fine-tunes only the most task-relevant layers while freezing less critical ones. By probing hidden state representations per layer through a lightweight classifier, LAET reduces trainable parameters by up to 60% while improving task-specific performance. Ensemble predictions across selected layers further improve robustness. Across 23 financial NLP benchmarks spanning textual analysis, risk management, and forecasting, LAET (on ~3B models) consistently outperforms GPT-4, GPT-4o, DoRA, LoRA, and specialized financial LLMs.

Introduction

Fine-tuning all parameters of large pretrained LLMs is computationally prohibitive and often unnecessary — evidence shows that only a fraction of layers contribute meaningfully to any downstream task. Existing PEFT methods (LoRA, DoRA, AdaLoRA) apply uniform or heuristically distributed parameter budgets, ignoring per-layer utility. LAET addresses this by probing each layer independently to score task relevance, then selectively fine-tuning only high-value layers.

Models used (ℳ₁, ℳ₂, ℳ₃):

• ℳ₁: Gemma-2-2B (2B parameters)
• ℳ₂: Llama-3.2-3B (3.2B parameters)
• ℳ₃: Phi-3.5-mini (3.8B parameters)

🎯 Key Contributions

• Layer-wise Probing — A shared lightweight classifier trained on frozen per-layer hidden states scores each layer’s task relevance via accuracy (m₁) and F1 (m₂).
• Adaptive Layer Selection — Dynamic thresholds δ = α·σ based on score standard deviations automatically select the best-performing layers.
• Ensemble Voting — Majority vote over predictions from all selected layers reduces sensitivity to individual layer noise.
• 60% Parameter Savings — Frozen layers do not receive gradient updates; only selected layers are fine-tuned.
• Outperforms GPT-4 — ~3B parameter models with LAET beat GPT-4 and GPT-4o on financial NLP benchmarks.

Introduction

Figure 1: Comparison of full fine-tuning, parameter-efficient fine-tuning (PEFT), and LAET. Full fine-tuning updates all layers; PEFT applies uniform low-rank adapters. LAET identifies and fine-tunes only the most task-relevant layers, leaving the rest frozen — reducing trainable parameters by up to 60% while improving task performance.

Methodology

Figure 2: LAET methodology. (a) For each layer l in pretrained model ℳ, hidden state representations r_i^(l) are extracted from frozen layers; a shared lightweight classifier ℱ_φ computes logits z_i^l and cross-entropy loss ℒ_l independently per layer. (b) The best-performing layers are selected based on evaluation metrics (accuracy m₁, F1 m₂) and their deviations from maximum values using adaptive threshold δ = α·σ. Only selected layers ℬ are unfrozen for fine-tuning; inference uses majority voting across all selected layers.

Problem Formulation

Let ℳ be a pretrained LLM with L layers. For input tokens x, each layer l produces hidden states H_l = {h₁⁽ˡ⁾, …, hₙ⁽ˡ⁾}.

The last-token representation per layer: r_l = h_n^(l)[:, r] ∈ ℝᵈ

A lightweight classifier ℱ_φ with shared weights is trained independently on each layer’s frozen representations.

Layer Probing

Figure 3: Layer-wise probing evaluation across three hidden state representation strategies: Last Token (LT), Sum of All Tokens (SaT), and Average of All Tokens (AvT). Results show which strategy produces the most task-discriminative representations per layer for adaptive selection.

Per-layer cross-entropy loss:

ℒₗ = -(1/N) Σᵢ Σⱼ 𝒫[yᵢ=j] · log(pᵢˡ[j])

where pᵢˡ = softmax(ℱ_φ(rᵢ⁽ˡ⁾)) ∈ ℝᵏ.

Adaptive Layer Selection

For performance metrics m₁ (accuracy), m₂ (F1) per layer:

σₘ₁ = std({m₁¹,...,m₁ᴸ}),   σₘ₂ = std({m₂¹,...,m₂ᴸ})
δₘ₁ = α·σₘ₁,                  δₘ₂ = β·σₘ₂

Layer l is selected into set ℬ if no other layer l’ satisfies: m₁ˡ' ≥ m₁ˡ + δₘ₁ AND m₂ˡ' ≥ m₂ˡ + δₘ₂

Fine-tuning and Ensemble

Only layers l ∈ ℬ are unfrozen. Training objective:

ℒ_ℬ = (1/|ℬ|) Σₗ∈ℬ ℒₗ(θₗ, φ)

Inference — majority voting:

ŷ = argmax_{c∈𝒞} Σₗ∈ℬ 𝕀(ŷₗ = c)

Ensemble error bound (conditional independence, average error ε̄):

𝒫(ŷ ≠ y) ≤ exp(−2|ℬ|(0.5 − ε̄)²)

Figure 4: Ablation study comparing LAET layer selection methods across four datasets. Results validate that the adaptive threshold selection (δ = α·σ) outperforms static or random layer selection, confirming that task-relevant layer identification is the key driver of LAET’s performance gains.

📊 Table I — Dataset Statistics (23 Financial NLP Benchmarks)

📊 Table II — Textual Analysis (TA) Results

📊 Table III — Risk Management (RM) Results

📊 Table IV — Forecasting (FO) Results

📚 Citation

@inproceedings{ahad2025laet,
  title={LAET: A Layer-Wise Adaptive Ensemble Tuning Framework for Pretrained Language Models},
  author={Ahad, Jawad Ibn and Kabir, Muhammad Rafsan and Krambroeckers, Robin and Momen, Sifat and Mohammed, Nabeel and Rahman, Shafin},
  booktitle={IEEE International Conference on Big Data (BigData)},
  year={2025}
}