Multi-object tracking is one of the key technologies in video surveillance. In this paper, a new PCR-based multi-object tracking algorithm is proposed and principal colors are used to characterize the appearance of each object. Combined with the kalman filter prediction a PCR-based mean shift procedure is applied to estimate the current location of each object. Then an association matrix between the estimate locations and foreground blobs is built to distinguish many different situations that possibly occur in complex real world applications. Lastly the state of object including location, size and color model is updated according to different situations. Experimental results on many image sequences have shown that the proposed algorithm can efficiently deal with complex occlusions. The performance of this new tracking algorithm is robust.