TY - JOUR
T1 - Improving the Expressive Power of Deep Neural Networks Through Integral Activation Transform
AU - Zhang , Zezhong
AU - Bao , Feng
AU - Zhang , Guannan
JO - International Journal of Numerical Analysis and Modeling
VL - 5
SP - 739
EP - 763
PY - 2024
DA - 2024/10
SN - 21
DO - http://doi.org/10.4208/ijnam2024-1030
UR - https://global-sci.org/intro/article_detail/ijnam/23451.html
KW - Integral transform, generalized neural network, continuous ReLU activation pattern, expressive power of neural network.
AB - <p style="text-align: justify;">The impressive expressive power of deep neural networks (DNNs) underlies their
widespread applicability. However, while the theoretical capacity of deep architectures is high,
the practical expressive power achieved through successful training often falls short. Building on
the insights gained from Neural ODEs, which explore the depth of DNNs as a continuous variable,
in this work, we generalize the traditional fully connected DNN through the concept of continuous
width. In the Generalized Deep Neural Network (GDNN), the traditional notion of neurons in
each layer is replaced by a continuous state function. Using the finite rank parameterization of the
weight integral kernel, we establish that GDNN can be obtained by employing the Integral Activation Transform (IAT) as activation layers within the traditional DNN framework. The IAT maps
the input vector to a function space using some basis functions, followed by nonlinear activation in
the function space, and then extracts information through the integration with another collection
of basis functions. A specific variant, IAT-ReLU, featuring the ReLU nonlinearity, serves as a
smooth generalization of the scalar ReLU activation. Notably, IAT-ReLU exhibits a continuous
activation pattern when continuous basis functions are employed, making it smooth and enhancing
the trainability of the DNN. Our numerical experiments demonstrate that IAT-ReLU outperforms
regular ReLU in terms of trainability and better smoothness.</p>