- Journal Home
- Volume 21 - 2024
- Volume 20 - 2023
- Volume 19 - 2022
- Volume 18 - 2021
- Volume 17 - 2020
- Volume 16 - 2019
- Volume 15 - 2018
- Volume 14 - 2017
- Volume 13 - 2016
- Volume 12 - 2015
- Volume 11 - 2014
- Volume 10 - 2013
- Volume 9 - 2012
- Volume 8 - 2011
- Volume 7 - 2010
- Volume 6 - 2009
- Volume 5 - 2008
- Volume 4 - 2007
- Volume 3 - 2006
- Volume 2 - 2005
- Volume 1 - 2004
Int. J. Numer. Anal. Mod., 21 (2024), pp. 674-696.
Published online: 2024-10
Cited by
- BibTex
- RIS
- TXT
Fully connected deep neural networks are successfully applied to classification and function approximation problems. By minimizing the cost function, i.e., finding the proper weights and biases, models can be built for accurate predictions. The ideal optimization process can achieve global optima. However, do global optima always perform well? If not, how bad can it be? In this work, we aim to: 1) extend the expressive power of shallow neural networks to networks of any depth using a simple trick, 2) construct extremely overfitting deep neural networks that, despite having global optima, still fail to perform well on classification and function approximation problems. Different types of activation functions are considered, including ReLU, Parametric ReLU, and Sigmoid functions. Extensive theoretical analysis has been conducted, ranging from one-dimensional models to models of any dimensionality. Numerical results illustrate our theoretical findings.
}, issn = {2617-8710}, doi = {https://doi.org/10.4208/ijnam2024-1027}, url = {http://global-sci.org/intro/article_detail/ijnam/23448.html} }Fully connected deep neural networks are successfully applied to classification and function approximation problems. By minimizing the cost function, i.e., finding the proper weights and biases, models can be built for accurate predictions. The ideal optimization process can achieve global optima. However, do global optima always perform well? If not, how bad can it be? In this work, we aim to: 1) extend the expressive power of shallow neural networks to networks of any depth using a simple trick, 2) construct extremely overfitting deep neural networks that, despite having global optima, still fail to perform well on classification and function approximation problems. Different types of activation functions are considered, including ReLU, Parametric ReLU, and Sigmoid functions. Extensive theoretical analysis has been conducted, ranging from one-dimensional models to models of any dimensionality. Numerical results illustrate our theoretical findings.