Article ; Online: Long short-term memory with activation on gradient.
Neural networks : the official journal of the International Neural Network Society
2023 Volume 164, Page(s) 135–145
Abstract: As the number of long short-term memory (LSTM) layers increases, vanishing/exploding gradient problems exacerbate and have a negative impact on the performance of the LSTM. In addition, the ill-conditioned problem occurs in the training process of LSTM ... ...
| Abstract | As the number of long short-term memory (LSTM) layers increases, vanishing/exploding gradient problems exacerbate and have a negative impact on the performance of the LSTM. In addition, the ill-conditioned problem occurs in the training process of LSTM and adversely affects its convergence. In this work, a simple and effective method of the gradient activation is applied to the LSTM, while empirical criteria for choosing gradient activation hyperparameters are found. Activating the gradient refers to modifying the gradient with a specific function named the gradient activation function. Moreover, different activation functions and different gradient operations are compared to prove that the gradient activation is effective on LSTM. Furthermore, comparative experiments are conducted, and their results show that the gradient activation alleviates the above problems and accelerates the convergence of the LSTM. The source code is publicly available at https://github.com/LongJin-lab/ACT-In-NLP. |
|||||
|---|---|---|---|---|---|---|
| MeSH term(s) | Neural Networks, Computer ; Memory, Short-Term ; Memory, Long-Term ; Software | |||||
| Language | English | |||||
| Publishing date | 2023-04-25 | |||||
| Publishing country | United States | |||||
| Document type | Journal Article | |||||
| ZDB-ID | 740542-x | |||||
| ISSN | 1879-2782 ; 0893-6080 | |||||
| ISSN (online) | 1879-2782 | |||||
| ISSN | 0893-6080 | |||||
| DOI | 10.1016/j.neunet.2023.04.026 | |||||
| Shelf mark |
|
|||||
| Database | MEDical Literature Analysis and Retrieval System OnLINE |
Full text online
More links
Kategorien
In stock of ZB MED Cologne/Königswinter
| Zs.A 2365: Show issues | Location: Je nach Verfügbarkeit (siehe Angabe bei Bestand) bis Jg. 1994: Bestellungen von Artikeln über das Online-Bestellformular Jg. 1995 - 2021: Lesesall (2.OG) ab Jg. 2022: Lesesaal (EG) |
Order via subito
This service is chargeable due to the Delivery terms set by subito. Orders including an article and supplementary material will be classified as separate orders. In these cases, fees will be demanded for each order.