In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces. The approximation error of the neural network is $\mathcal{O}(1/\sqrt{m})$ where $m$ is the size of networks. In other words, the error of the network is no dependence on the dimensionality respecting to the number of the nodes in neural networks. The key idea of the approximation is to define a Barron space of functionals.

Learning functionals or operators by neural networks is nowadays widely used in computational and applied mathematics. Compared with learning functions by neural networks, an essential difference is that the input spaces of functionals or operators are infinite dimensional space. Some recent works learnt functionals or operators by reducing the input space into a finite dimensional space. However, the curse of dimensionality always exists in this type of methods. That is, in order to maintain the accuracy of an approximation, the number of sample points grows exponentially with the increase of dimension. 

In this paper, we establish a new method for the approximation of functionals by neural networks without curse of dimensionality. Functionals, such as linear functionals and energy functionals, have a wide range of important applications in science and engineering fields. We define Fourier series of functionals and the associated Barron spectral space of functionals, based on which our new neural network approximation method is established. The parameters and the network structure in our method only depend on the functional. The approximation error of the neural network is $O(1/\sqrt{m})$ where $m$ is the size of the network, which does not depend on the dimensionality.

The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.

The modeling of probability distributions is an important branch of machine learning. It became popular in recent years thanks to the success of deep generative models in difficult tasks such as image synthesis and text conversation. Nevertheless, we still lack a theoretical understanding of the good performance of distribution learning models. One mystery is the following paradox: it is generally inevitable that the model suffers from memorization (converges to the empirical distribution of the training samples) and thus becomes useless, and yet in practice the trained model can generate new samples or estimate the probability density over unseen samples. Meanwhile, the existing models are so diverse that it has become overwhelming for practitioners and researchers to get a clear picture of this fast-growing subject. This paper provides a mathematical framework that unifies all the well-known models, so that they can be systemically derived based on simple principles. This framework enables our analysis of the theoretical mysteries of distribution learning, in particular, the paradox between memorization and generalization. It is established that the model during training enjoys implicit regularization, so that it approximates the hidden target distribution before eventually turning towards the empirical distribution. With early stopping, the generalization error escapes from the curse of dimensionality and thus the model generalizes well.