Artikel ; Online: An Efficient Fisher Matrix Approximation Method for Large-Scale Neural Network Optimization.
IEEE transactions on pattern analysis and machine intelligence
2023 Band 45, Heft 5, Seite(n) 5391–5403
Abstract: Although the shapes of the parameters are not crucial for designing first-order optimization methods in large scale empirical risk minimization problems, they have important impact on the size of the matrix to be inverted when developing second-order ... ...
Abstract | Although the shapes of the parameters are not crucial for designing first-order optimization methods in large scale empirical risk minimization problems, they have important impact on the size of the matrix to be inverted when developing second-order type methods. In this article, we propose an efficient and novel second-order method based on the parameters in the real matrix space [Formula: see text] and a matrix-product approximate Fisher matrix (MatFisher) by using the products of gradients. The size of the matrix to be inverted is much smaller than that of the Fisher information matrix in the real vector space [Formula: see text]. Moreover, by utilizing the matrix delayed update and the block diagonal approximation techniques, the computational cost can be controlled and is comparable with first-order methods. A global convergence and a superlinear local convergence analysis are established under mild conditions. Numerical results on image classification with ResNet50, quantum chemistry modeling with SchNet, and data-driven partial differential equations solution with PINN illustrate that our method is quite competitive to the state-of-the-art methods. |
---|---|
Sprache | Englisch |
Erscheinungsdatum | 2023-04-03 |
Erscheinungsland | United States |
Dokumenttyp | Journal Article |
ISSN | 1939-3539 |
ISSN (online) | 1939-3539 |
DOI | 10.1109/TPAMI.2022.3213654 |
Datenquelle | MEDical Literature Analysis and Retrieval System OnLINE |
Zusatzmaterialien
Kategorien
Über subito bestellen
Dieser Service ist kostenpflichtig (siehe Lieferbedingungen von subito). Bestellungen, die einen Artikel nebst Supplementary Material umfassen, werden grundsätzlich wie mehrfache Bestellungen bearbeitet. Gebühren fallen in diesen Fällen für jede einzelne Bestellung an.
Fernleihe an ZB MED
Sie können sich den gewünschten Titel als lokale Nutzerin oder lokaler Nutzer von ZB MED direkt an den Standort Köln schicken lassen.