Simultaneous multi-class pixel labeling over coherent image sets

Multi-class pixel labeling is an important problem in computer vision that has many diverse applications, including interactive image segmentation, semantic and geometric scene understanding, and stereo reconstruction. Current state-of-the-art approaches learn a model on a set of training images and then apply the learned model to each image in a test set independently. The quality of the results, therefore, depends strongly on the quality of the learned models and the information available within each training image. Importantly, this approach cannot leverage information available in other images at test time which may help to label the image at hand. Instead of labeling each image independently, we propose a semi-supervised approach that exploits the similarity between regions across many images in coherent image subsets. Specifically, our model finds similar regions in related images and constrains the joint labeling of the images to agree on the labels within these regions. By considering the joint labeling, our model gets to leverage contextual information that is not available when considering images in isolation. We test our approach on the popular 21-class MSRC multi-class image segmentation dataset and show improvement in accuracy over a strong baseline model.