We report an efficient phase field formalism to compute the stress distribution in polycrystalline materials with arbitrary elastic inhomogeneity and anisotropy. The dependence of elastic stiffness tensor on grain orientation is taken into account, and the elastic equilibrium equation is solved using a spectral iterative perturbation method. We discuss its applications to computing residual stress distribution in systems containing arbitrarily shaped cavities and cracks (with zero elastic modulus) and to determining the effective elastic properties of polycrystals and multilayered composites.