Tracking by detection for interactive image augmentation in laparoscopy

We present a system for marking, tracking and visually augmenting a deformable surgical site by the robust automatic detection of natural landmarks (image features) in laparoscopic surgery. In our system, the surgeon first selects a frame containing an organ of interest, and this is used by our system both to detect every instance of the organ in a laparoscopic video feed, and to recover the nonrigid deformations. The system then augments the video with customizable visual information such as the location of hidden or weakly visible structures (cysts, vessels, etc), or planned incision points, acquired from pre-operative or intra-operative data. Frame-rate organ detection is performed via a novel procedure that matches the current frame to the reference frame. Because laparoscopic images are known to be extremely difficult to match, we propose to use Shape-from-Shading and conformal flattening to cancel out much of the variation in appearance due to perspective foreshortening, and we then apply robust matching to the flattened surfaces. Experiments show robust tracking and detection results on a laparoscopic procedure with the uterus as target organ. As our system detects the organ in every frame, it is not impaired by target loss, contrary to most previous methods.