Article ; Online: Improving Deep Mutual Learning via Knowledge Distillation
Applied Sciences, Vol 12, Iss 15, p
2022 Volume 7916
Abstract: Knowledge transfer has become very popular in recent years, and it is either based on a one-way transfer method used with knowledge distillation or based on a two-way knowledge transfer implemented by deep mutual learning, while both of them adopt a ... ...
Abstract | Knowledge transfer has become very popular in recent years, and it is either based on a one-way transfer method used with knowledge distillation or based on a two-way knowledge transfer implemented by deep mutual learning, while both of them adopt a teacher–student paradigm. A one-way based method is more simple and compact because it only involves an untrained low-capacity student and a high-capacity teacher network in the knowledge transfer process. In contrast, a two-way based method requires more training costs because it involves two or more low-cost network capacities from scratch simultaneously to obtain better accuracy results for each network. In this paper, we propose two new approaches, namely full deep distillation mutual learning (FDDML) and half deep distillation mutual learning (HDDML), and improve convolutional neural network performance. These approaches work with three losses by using variations of existing network architectures, and the experiments have been conducted on three public benchmark datasets. We test our method on some existing KT task methods, showing its performance over related methods. |
---|---|
Keywords | image classification ; knowledge distillation ; mutual learning ; convolutional neural network ; Technology ; T ; Engineering (General). Civil engineering (General) ; TA1-2040 ; Biology (General) ; QH301-705.5 ; Physics ; QC1-999 ; Chemistry ; QD1-999 |
Subject code | 006 |
Language | English |
Publishing date | 2022-08-01T00:00:00Z |
Publisher | MDPI AG |
Document type | Article ; Online |
Database | BASE - Bielefeld Academic Search Engine (life sciences selection) |
Full text online
More links
Kategorien
Inter-library loan at ZB MED
Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.