In recent years, Fourier spectral methods have emerged as competitive numerical methods for large-scale phase field simulations of microstructures in computational materials sciences. To further improve their effectiveness, we recently developed a new adaptive Fourier-spectral semi-implicit method (AFSIM) for solving the phase field equation by combining an adaptive moving mesh method and the semi-implicit Fourier spectral algorithm. In this paper, we present the application of AFSIM to the Cahn-Hilliard equation with inhomogeneous, anisotropic elasticity. Numerical implementations and test examples in both two and three dimensions are considered with a particular illustration using the well-studied example of mis-fitting particles in a solid as they approach to their equilibrium shapes. It is shown that significant savings in memory and computational time is achieved while accurate solutions are preserved.