TY - JOUR
T1 - Enhancing Accuracy in Deep Learning Using Random Matrix Theory
AU - Berlyand , Leonid
AU - Sandier , Etienne
AU - Shmalo , Yitzchak
AU - Zhang , Lei
JO - Journal of Machine Learning
VL - 4
SP - 347
EP - 412
PY - 2024
DA - 2024/11
SN - 3
DO - http://doi.org/10.4208/jml.231220
UR - https://global-sci.org/intro/article_detail/jml/23500.html
KW - Deep learning, Marchenko-Pastur distribution, Random matrix theory, Increasing accuracy, Pruning.
AB - <p style="text-align: justify;">We explore the applications of random matrix theory (RMT) in the training of deep neural networks
(DNNs), focusing on layer pruning that reduces the number of DNN parameters (weights). Our numerical
results show that this pruning leads to a drastic reduction of parameters while not reducing the accuracy of
DNNs and convolutional neural network (CNNs). Moreover, pruning the fully connected DNNs actually increases the accuracy and decreases the variance for random initializations. Our numerics indicate that this
enhancement in accuracy is due to the simplification of the loss landscape. We next provide rigorous mathematical underpinning of these numerical results by proving the RMT-based Pruning Theorem. Our results
offer valuable insights into the practical application of RMT for the creation of more efficient and accurate
deep-learning models.</p>