Loss functions with non-isolated minima have emerged in several machine-learning problems, creating a gap between theoretical predictions and observations in practice. In this paper, we formulate a new type of local convexity condition that is suitable to describe the behavior of loss functions near non-isolated minima. We show that such a condition is general enough to encompass many existing conditions. In addition, we study the local convergence of the stochastic gradient descent (SGD) method under this mild condition by adopting the notion of stochastic stability. In the convergence analysis, we establish concentration inequalities for the iterates in SGD, which can be used to interpret the empirical observation from some practical training results.

A huge success in modern machine learning applications is attributed to the feasibility of non-convex optimization via effective optimization techniques such as the stochastic gradient descent (SGD). But at the theoretical level, it is still not clear why the SGD works for the task of non-convex optimization, especially when non-isolated local minima are present. In this paper, we present a local convergence analysis of the SGD in non-convex optimization with non-isolated minima. We propose new non-convex conditions that characterize a large class of non-convex loss functions with non-isolated minima. Under those conditions, we prove the concentration inequalities of the SGD that characterized the convergence behavior. Some numerical results confirm the existence of a deep neural network satisfying the weakest non-convex condition among the proposed ones.