Article ; Online: Achieving Sharp Upper Bounds on the Expressive Power of Neural Networks via Tropical Polynomials.
IEEE transactions on neural networks and learning systems
2024 Volume PP
Abstract: The expressive power of neural networks describes the ability to represent or approximate complex functions. The number of linear regions is the standard and most natural measure of expressive power. However, a major challenge in utilizing the number of ... ...
| Abstract | The expressive power of neural networks describes the ability to represent or approximate complex functions. The number of linear regions is the standard and most natural measure of expressive power. However, a major challenge in utilizing the number of linear regions as a measure of expressive power is the exponential gap between the theoretical upper and lower bounds, which becomes more pronounced as the neural network capacity increases. In this article, we aim to derive a sharp upper bound on piecewise linear neural networks (PLNNs) to bridge this gap. Specifically, we first establish the relationship between tropical polynomials and PLNNs. In the unexpanded tropical polynomials form, we make the proposition that hyperplanes are not all in the general positions, thereby reducing the number of intersecting hyperplanes. We propose a rank-based approach and present the empirical analysis that this approach outperforms previous Zaslavsky's theorem-based methods. In the expanded tropical polynomials form, accounting for limitations in weight initialization and model computational precision, we raise the concept that the values range of each term is bounded. We propose a precision-based approach that transforms the approximate exponential growth of the number of linear regions into polynomial growth with width, which is effective at larger layer widths. Finally, we compare the number of linear regions that can be represented by each hidden layer in both forms and derive a sharp upper bound for PLNNs. Empirical analysis and experimental results provide compelling evidence for the efficacy and feasibility of this sharp upper bound on both simulated experiments and real datasets. |
|---|---|
| Language | English |
| Publishing date | 2024-02-05 |
| Publishing country | United States |
| Document type | Journal Article |
| ISSN | 2162-2388 |
| ISSN (online) | 2162-2388 |
| DOI | 10.1109/TNNLS.2024.3350786 |
| Database | MEDical Literature Analysis and Retrieval System OnLINE |
More links
Kategorien
Order via subito
This service is chargeable due to the Delivery terms set by subito. Orders including an article and supplementary material will be classified as separate orders. In these cases, fees will be demanded for each order.
Inter-library loan at ZB MED
Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.