Beyond Real Weights: Hypercomplex Representations for Stable Quantization

✅ Accepted at IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026

📄 Paper: arXiv:2512.08524 · CVF Proceedings
💻 Code: github.com/milab-nsu/PHM

Abstract

Multimodal large language models (MLLMs) incur substantial inference cost from dense feed-forward network (FFN) layers. We propose a progressive hypercomplex reparameterization framework that compresses MLLMs by gradually replacing dense FFN blocks with compact Parameterized Hypercomplex Multiplication (PHM) operators. A residual interpolation schedule, combined with lightweight reconstruction and knowledge distillation losses, ensures PHM modules inherit the functional behavior of their dense counterparts during training. PHM operators achieve a compression factor of 4/B (B ∈ {2,3}) in parameter count and reduce FLOPs, with selective capacity allocation assigning higher B to performance-sensitive upper language layers. Evaluated on image captioning (Flickr, NoCaps, COCO) and multimodal QA tasks, the PHM variants maintain performance comparable to the dense baselines while achieving significant efficiency gains.

Introduction

Dense FFN weight matrices in transformer-based MLLMs dominate parameter counts. Naive quantization methods cause instability due to the distribution mismatch between compressed and original operators. We replace real-valued dense layers with hypercomplex operators that exploit algebraic structure in weight space, achieving structured compression without abrupt capability loss.

Key insight: Parameterized Hypercomplex Multiplication layers express the weight matrix as a sum of Kronecker products of fixed basis matrices and small learnable matrices. This decomposition imposes algebraic structure that reduces parameter count while preserving representational capacity via the hypercomplex product geometry.

🎯 Key Contributions

• PHM Reparameterization — Dense weight W replaced by Σ Hb ⊗ Ab, compressing parameter count by factor 4/B while maintaining expressiveness.
• Residual Interpolation — Smooth transition W_res(α) = (1-α)W_dense + α·W_PHM over training steps from α=0 to α=1, avoiding distribution shock.
• Dual Stabilization — Knowledge distillation (dense-teacher → PHM-student) + operator-level reconstruction loss ||W_PHM·x − W_dense·x||² act jointly to transfer functional behavior.
• Selective Capacity Assignment — B=3 for upper K language layers (higher risk, richer hypercomplex space), B=2 for lower layers and vision components.
• Unified Training — PHM, LoRA-based PEFT for text attention, and KD integrated in a single two-stage pipeline.

Methodology

Figure: Progressive PHM reparameterization framework. Dense FFN layers are replaced via a residual interpolation schedule. A dense-teacher model provides KD supervision while a reconstruction loss at the operator level ensures the PHM modules learn to mimic their dense counterparts.

PHM Layer Definition

Dense weight W ∈ ℝ^(d_out × d_in), with d_in = 2m, d_out = 2n:

W_PHM = Σ_{b=1}^{B} Hb ⊗ Ab

where Hb ∈ ℝ^{2×2} are fixed hypercomplex bases
      Ab ∈ ℝ^{n×m} are learnable matrices

Parameter count: |W_PHM| = B·nm vs |W_dense| = 4nm → compression factor 4/B

For B=2 (canonical bases I, J):

W_PHM = [[A1, -A2],
          [A2,  A1]]

For B=3 (adds H3 = diag(1,-1)):

W_PHM = [[A1+A3, -A2 ],
          [A2,  A1-A3]]

Dense-to-PHM Initialization

Solve: min_{Ab} ||W - Σ Hb ⊗ Ab||_F²

For B=2: closed-form solution A1 = ½(p+s), A2 = ½(r-q) where p,q,r,s are blocks of W.

Residual Interpolation Schedule

W_res(α) = (1-α)·W_dense + α·W_PHM
α(t) = min(1, t / T_fade)

Training Objective

ℒ(t) = (1 - λ(t))·ℒ_CE + λ(t)·ℒ_KD + μ·ℒ_recon

ℒ_CE  = cross-entropy with label smoothing ε
ℒ_KD  = T² · KL(softmax(z_T/T) || softmax(z_S/T))
ℒ_recon = (1/|M|) Σ_ℓ∈M E_x ||W_ℓ^PHM·x − W_ℓ^dense·x||²

λ(t) = λ_max · min(1, t/T_fade),   μ ≪ 1

Selective Capacity Assignment

For language stack with L_lang layers, threshold t* = L_lang - K:

Bℓ = 3  if ℓ ∈ {t*, t*+1, ..., L_lang-1}  (upper K layers)
Bℓ = 2  otherwise

FLOPs: Dense: ≈ 2N·d_in·d_out, PHM: ≈ B·N·d_in·d_out + O(N·(d_in+d_out))

For B < 4, PHM shows substantial parameter savings with FLOPs no worse than dense.

📊 Table 1 — Image Captioning Performance

(B@4 = BLEU-4, M = METEOR, R = ROUGE-L; PHM variants use B=2/3 selective capacity)

📊 Table 2 — ScienceQA and Financial MME

📊 Table 3 — Computational Efficiency

📊 Table 4 — Ablation: Stabilization Components (Qwen2.5 VL-7B + PHM)

All three stabilization components (KD, Recon Loss, Residual Interpolation) are necessary. Removing any one causes significant CIDEr degradation.

📚 Citation

@inproceedings{ahad2026beyond,
  title={Beyond Real Weights: Hypercomplex Representations for Stable Quantization},
  author={Ahad, Jawad Ibn and Rahman, Maisha and Biswas, Amrijit and Kabir, Muhammad Rafsan and Krambroeckers, Robin and Momen, Sifat and Mohammed, Nabeel and Rahman, Shafin},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2026}
}