TY - JOUR
T1 - Convergence of Backpropagation with Momentum for Network Architectures with Skip Connections
AU - Agarwal , Chirag
AU - Klobusicky , Joe
AU - Schonfeld , Dan
JO - Journal of Computational Mathematics
VL - 1
SP - 147
EP - 158
PY - 2020
DA - 2020/09
SN - 39
DO - http://doi.org/10.4208/jcm.1912-m2018-0279
UR - https://global-sci.org/intro/article_detail/jcm/18282.html
KW - Backpropagation with momentum, Autoencoders, Directed acyclic graphs.
AB - <p style="text-align: justify;">We study a class of deep neural networks with architectures that form a directed acyclic
graph (DAG). For backpropagation defined by gradient descent with adaptive momentum,
we show weights converge for a large class of nonlinear activation functions. The proof
generalizes the results of Wu et al. (2008) who showed convergence for a feed-forward
network with one hidden layer. For an example of the effectiveness of DAG architectures,
we describe an example of compression through an AutoEncoder, and compare against
sequential feed-forward networks under several metrics.</p>