Segment Pool Tracking is the framework we presented for the video segmentation problem. The figure above illustrates the core parts of our approach. First, we generate a pool of segmentation for each frame using the CPMC method. Then, image color features (eg. Color-SIFT) are extracted and appearance models are trained incrementally to track multiple segments in consecutive frames. A main contribution is an efficient least-squares formulation to make simultaneously tracking 1,000 targets almost as efficient as tracking a single target. Since the appearance model for each target is learnt over multiple frames on many segments, it is robust to appearance changes and partial occlusions. Lastly, we use Composite Statistical Inference (CSI) to refine segment tracks by infering on high-order appearance terms while imposing temporal consistency.

During tracking, greedy assignment is applied that serves as non-maximum suppression on segment tracks (see figure below). Tracks that are not consistent in appearance are filtered out automatically. Therefore, although we initialize with more than 1,000 tracks, on average only 60 tracks remain at the end of each sequence, while capturing most of the interesting objects.