We develop a stable finite difference approximation of the three-dimensional viscoelastic wave equation. The material model is a super-imposition of N standard linear solid mechanisms, which commonly is used in seismology to model a material with constant quality factor Q. The proposed scheme discretizes the governing equations in second order displacement formulation using 3N memory variables, making it significantly more memory efficient than the commonly used first order velocity-stress formulation. The new scheme is a generalization of our energy conserving finite difference scheme for the elastic wave equation in second order formulation [SIAM J. Numer. Anal., 45 (2007), pp. 1902–1936]. Our main result is a proof that the proposed discretization is energy stable, even in the case of variable material properties. The proof relies on the summation-by-parts property of the discretization. The new scheme is implemented with grid refinement with hanging nodes on the interface. Numerical experiments verify the accuracy and stability of the new scheme. Semi-analytical solutions for a half-space problem and the LOH.3 layer over half-space problem are used to demonstrate how the number of viscoelastic mechanisms and the grid resolution influence the accuracy. We find that three standard linear solid mechanisms usually are sufficient to make the modeling error smaller than the discretization error.