Beyond Labels: Aligning Large Language Models with Human-like Reasoning

✅ Accepted at ICPR 2025 (27th International Conference on Pattern Recognition)

📄 Paper: Springer · arXiv:2408.11879
💻 Code & Dataset: github.com/apurba-nsu-rnd-lab/DFAR

Abstract

We introduce the Dataset for Aligning Reasons (DFAR), a modified version of the ETHICS dataset, consisting of ethical statements, their classification labels, and human-annotated reasons explaining the ethical judgments. Unlike standard fine-tuning that trains only on labels, our novel approach jointly trains LLMs on both labels and reasons, enabling the model to generate not just a classification but the reasoning behind ethical decisions. This aligns LLM behavior more closely with human moral reasoning, improving interpretability and classification accuracy.

Introduction

Current LLM fine-tuning for ethical classification treats the task as a black-box label prediction problem. Models learn to output “ethical” or “unethical” without grounding their predictions in human-understandable rationales. This creates opacity and limits reliability when the model encounters novel ethical dilemmas.

We argue that training with reasons — alongside labels — provides a richer supervision signal. The DFAR dataset is constructed by collecting human annotations explaining why each ethical statement is classified as ethical (label=0) or unethical (label=1), enabling joint label+reason supervision.

🎯 Key Contributions

• DFAR Dataset — 5,000 ethical statements (from ETHICS dataset) annotated with classification labels and human-written reasons; 4,500 train / 500 test (90/10 split).
• Joint Label+Reason Fine-tuning — A novel training approach where the LLM generates [label; reason] jointly, trained via unified language modeling loss.
• Cross-dataset Generalization — Models fine-tuned only on DFAR tested on ETHOS without any additional training, showing strong out-of-distribution reasoning.
• Improved Interpretability — Models produce human-readable justifications alongside predictions, enabling auditability of ethical decisions.

Methodology

Figure: (a) Standard fine-tuning: LLM trained on label only — outputs ŷᵢ given input xᵢ, loss computed on label alone. (b) Our approach: LLM jointly generates label ŷᵢ and reason r̂ᵢ given xᵢ, with loss computed over the concatenated label+reason sequence. Weights updated to minimize joint label-reason prediction error.

Dataset: DFAR Statistics

Training Setup

• Models: Mistral 7B, Llama-2 7B
• Method: QLoRA (4-bit quantization + LoRA adapters)
• Loss: Cross-entropy over [label; reason] sequence
• Baselines: Label-only fine-tuning, classical ML (SVM, RF, BERT, DistilBERT)

📊 Results

MAR = Misalignment Rate (↓ lower is better). L+R = Labels + Reasons (our approach). Classical ML models not evaluated on ETHOS in the paper.

Joint label+reason training (L+R) dramatically outperforms all baselines: +26.6% accuracy over classical ML best (RF 78.6%) and +26.6% over label-only fine-tuning for Llama-2, with the lowest misalignment rate (9.4%).

📚 Citation

@inproceedings{kabir2025beyond,
  title={Beyond Labels: Aligning Large Language Models with Human-like Reasoning},
  author={Kabir, Muhammad Rafsan and Sultan, Rafeed Mohammad and Asif, Ihsanul Haque and Ahad, Jawad Ibn and Rahman, Fuad and Amin, Mohammad Ruhul and Mohammed, Nabeel and Rahman, Shafin},
  booktitle={International Conference on Pattern Recognition (ICPR)},
  pages={},
  year={2025},
  publisher={Springer}
}