Piecewise constant images, which are sampled from piecewise constant functions, are an important kind of images data. Typical examples include QR codes (Quick Response codes), logos and text images, which are widely used in both general commercial and automotive industry use. In this paper, we consider the problem of removing Gaussian noise from this kind of images. A novel method based on selective averaging and outlier removal is proposed. The selective averaging updates the intensity value at each pixel by averaging pixels in its homogeneous neighborhood. This scheme prevents the diffusion between pixels belonging to different homogeneous regions. Thus, it preserves image edges quite well. The outlier removal is adopted to detect and suppress outliers appearing in the output of selective averaging. The experiments on both gray and color image denoising show that our method is feasible and effective for piecewise constant image restoration, and achieves superior performance among all the compared methods.