Boundary integral equations provide a powerful tool for the solution of scattering problems. However, often a singular kernel arises, in which case the standard quadratures will give rise to unavoidable deteriorations in numerical precision, thus special treatment is needed to handle the singular behavior. Especially, for inhomogeneous media, it is difficult if not impossible to find out an analytical expression for Green’s function. In this paper, an efficient fourth-order accurate Cartesian grid-based method is proposed for the two-dimensional Helmholtz scattering and transmission problems with inhomogeneous media. This method provides an alternative approach to indirect integral evaluation by solving equivalent interface problems on Cartesian grid with a modified fourth-order accurate compact finite difference scheme and a fast Fourier transform preconditioned conjugate gradient (FFT-PCG) solver. A remarkable point of this method is that there is no need to know analytical expressions for Green’s function. Numerical experiments are provided to demonstrate the advantage of the current approach, including its simplicity in implementation, its high accuracy and efficiency.