Abstract: Albeit great performance of Transformer-based speech self-supervised learning (SSL) models, their large parameter size and computational cost make them unfavorable to utilize. In this study, ...