Dynamic Temperature Scheduler for Knowledge Distillation

✅ Accepted at IEEE International Conference on Big Data 2025

📄 Paper: arXiv:2511.13767 · IEEE Xplore

Abstract

We introduce the Dynamic Temperature Scheduler (DTS) for knowledge distillation (KD), a method that adaptively adjusts the softening temperature during training based on the cross-entropy loss gap between teacher and student models. Fixed-temperature KD is suboptimal: early training benefits from high T (softer distributions for broad learning) while later training requires low T (sharper distributions for precise mimicry). DTS eliminates manual temperature tuning and integrates seamlessly as a drop-in replacement for any existing KD method. Validated across vision (CIFAR-100, Tiny-ImageNet) and NLP tasks (GLUE, Dolly, SelfInst), DTS consistently improves performance over AKD and CTKD baselines and provides additive gains on top of DKD, MLKD, CRD, and other state-of-the-art KD methods.

Introduction

Knowledge distillation trains a student model to match the teacher’s soft probability distributions. The temperature T controls distribution softness:

• High T → soft distributions → student learns broad inter-class relationships early in training
• Low T → sharp distributions → student learns precise discrimination in later stages

Problem: No single fixed T is optimal across all training steps. Grid search for T is expensive, and heuristic schedules (AKD, CTKD) are not responsive to actual learning dynamics.

DTS Solution: Monitor the teacher-student loss divergence at each batch and adapt T accordingly — no hyperparameter search required.

🎯 Key Contributions

• Loss-Adaptive Temperature — T updated each batch based on divergence d_loss = L_t − L_s between teacher and student cross-entropy losses.
• Cosine + Momentum Schedule — Cosine decay provides smooth base annealing; momentum averaging prevents abrupt changes.
• Plug-and-Play — Works with any existing KD method (KD, DKD, MLKD, CRD, FitNet, AT) without architectural changes.
• No Grid Search — Removes costly T hyperparameter sweep entirely.
• Cross-Domain Validation — Vision (CNN, ViT) + NLP (GPT-2, OPT, BERT) tasks validated.

Methodology

Figure: DTS update mechanism. At each training step: (1) compute loss divergence between teacher and student; (2) scale via adaptive α; (3) apply cosine decay; (4) clamp to [T_min, T_max]; (5) apply momentum smoothing for stable temperature update.

DTS Formula

Step 1 — Cosine Scheduling:

p = e_c / e_t                        (progress ∈ [0,1])
S(p) = 0.5 · (1 + cos(π · p))       (ranges 1.0 → 0.0)

Step 2 — Loss Divergence Adaptive Scaling:

d_loss = L_teacher − L_student

If L_s > L_t (student worse than teacher):
    α = d_loss / (d_loss + 1 + β)       (β prevents ÷0)

If L_s < L_t (student better):
    α = −d_loss / (−d_loss + 1 + β)    (still > 0)

Step 3 — Clamped Temperature:

x = T_init · S(p) · α
T_clamped = clamp(x, T_min, T_max)

Step 4 — Momentum Smoothing:

T_current = μ · T_prev + (1 − μ) · T_clamped      (μ = 0.9)

📊 Table II — CIFAR-100 Cross-Architecture (Acc %)

📊 Table III — CIFAR-100 Same-Architecture, Multiple KD Methods (Acc %)

📊 Table IV — CIFAR-100 Cross-Architecture, DTS on Top of KD Methods (Acc %)

📊 Table V — Tiny-ImageNet Results (Acc %)

(a) CNN-based Student Models:

(b) ViT-based Student Models (Teacher: ResNet50):

📊 Table VI — NLP Tasks: ROUGE-L Scores

📊 Table VII — GLUE Tasks: TinyBERT-4L-312D from BERT_base_uncased

📚 Citation

@inproceedings{islam2025dts,
  title={Dynamic Temperature Scheduler for Knowledge Distillation},
  author={Islam, Sibgat Ul and Ahad, Jawad Ibn and Rahman, Fuad and Amin, Mohammad Ruhul and Mohammed, Nabeel and Rahman, Shafin},
  booktitle={IEEE International Conference on Big Data (BigData)},
  year={2025}
}