Color image segmentation is crucial in image processing and computer vision. Most traditional segmentation methods simply regard an RGB color image as the direct combination of the three monochrome images and ignore the inherent color structures within channels, which contain some key feature information of the image. To better describe the relationship of color channels, we introduce a quaternion-based regularization that can reflect the image characteristics more intuitively. Our model combines the idea of the Mumford-Shah model-based two-stage segmentation method and the Saturation-Value Total Variation regularization for color image segmentation. The new strategy first extracts features from the color image and then subdivides the image in a new color feature space which achieves better performance than methods in RGB color space. Moreover, to accelerate the optimization process, we use a new primal-dual algorithm to solve our novel model. Numerical results demonstrate clearly that the performance of our proposed method is excellent.