IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 8, AUGUST 2015


Spatiotemporal Saliency Detection for Video
Sequences Based on Random Walk with Restart

Supplemantary Materials


Hansang Kim     Youngbae Kim     Jae-Young Sim     Chang-Su Kim
hansangkim@mcl.korea.ac.kr     youngbaekim@mcl.korea.ac.kr     jysim@unist.ac.kr     cskim@mcl.korea.ac.kr
Korea University     Korea University     UNIST     Korea University


Abstract

A novel saliency detection algorithm for video sequences based on the random walk with restart (RWR) is proposed in this work. We adopt RWR to model the human visual system, which first perceives temporal information and then spatial information in each frame. Specifically, we find a temporal saliency distribution using the features of motion distinctiveness, temporal consistency, and abrupt change. Among them, the motion distinctiveness is derived by comparing the motion profiles of image patches. Then, we employ the temporal saliency distribution as a restarting distribution of the random walker. Also, we design the transition probability matrix for the walker using the spatial features of intensity, color, and compactness. Finally, we estimate the spatiotemporal saliency distribution by finding the steady-state distribution of the walker. The proposed algorithm detects foreground salient objects faithfully, while suppressing cluttered backgrounds effectively, by incorporating the spatial transition matrix and the temporal restarting distribution systematically. Experimental results on various video sequences demonstrate that the proposed algorithm outperforms conventional saliency detection algorithms qualitatively and quantitatively.




Comparison with the Conventional Algorithms

1. MPEG dataset [1]
(3 video sequences) 		  2. NTT dataset [2]
(9 video sequences) 		  3. MCL dataset
(9 video sequences)
Coastguard Hall1 Stefan Bird1 Bird2 Board Flight Fox Horse Rhino Ski Wildcat Ball Car Campus Court Crowd Hall2 Road Square Stair
GT
STC
[3]
GBVS
[4]
SRA
[5]
DVA
[6]
SSR
[7]
FSR
[8]
ISS
[9]
MCE
[10]
HBC
[11]
RBC
[11]
STSD
[12]
CAS
[13]
LSD
[14]
QUAT
[15]
GMR
[16]
HSD
[17]
Proposed



Comparative Video Sequences







References

[1] MPEG database, http://media.xiph.org/video/derf/.
[2] NTT database, http://www.brl.ntt.co.jp/people/akisato/saliency3.html.
[3] Y. Zhai and M. Shah, "Visual attention detection in video sequences using spatiotemporal cues," in Proc. ACM Int. Conf. Multimedia, 2006, pp. 815-824.
[4] J. Harel, C. Koch, and P. Perona, "Graph-based visual saliency," in Proc. Adv. Neural Inf. Process. Syst., 2006, pp. 545-552.
[5] X. Hou and L. Zhang, "Saliency detection: a spectral residual approach," in Proc. IEEE CVPR, Jun. 2007, pp. 1-8.
[6] X. Hou and L. Zhang, "Dynamic visual attention: searching for coding length increments," in Proc. Adv. Neural Inf. Process. Syst., 2008, pp. 681-688.
[7] H. J. Seo and P. Milanfar, "Static and space-time visual saliency detection by self-resemblance," Journal of Vision, vol. 9, no. 12, pp. 1-27, Nov. 2009.
[8] R. Achanta, S. Hemami, F. Estrada, and S. Süsstrunk, "Frequency-tuned salient region detection," in Proc. IEEE CVPR, Jun. 2009, pp. 1597-1604.
[9] Y. Li, Y. Zhou, L. Xu, X. Yang, and J. Yang, "Incremental sparse saliency detection," in Proc. IEEE ICIP, Sep. 2009, pp. 3093-3096.
[10] Y. Li, Y. Zhou, J. Yan, Z. Niu, and J. Yang, "Visual saliency based on conditional entropy," in Proc. Asian Conf. Comput. Vis., vol. 5994, 2009, pp. 246-257.
[11] M.-M. Cheng, G.-X. Zhang, N. J. Mitra, X. Haung, and S.-M. Hu, "Global contrast based salient region detection," in Proc. IEEE CVPR, Jun. 2011, pp. 409-416.
[12] W. Kim, C. Jung, and C. Kim, "Spatiotemporal saliency detection and its applications in static and dynamic scenes," IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 4, pp. 446-456, Apr. 2011.
[13] S. Goferman, L. Zelnik-Manor, and A. Tal, "Context-aware saliency detection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 10, pp. 1915-1926, Oct. 2012.
[14] Y. Xue, X. Guo, and X. Cao, "Motion saliency detection using low-rank and sparse decomposition," in Proc. IEEE ICASSP, Mar. 2012, pp. 1485-1488.
[15] B. Schauerte ad R. Stiefelhagen, "Quaternion-based spectral saliency detection for eye fixation prediction," in Proc. ECCV, 2012, pp. 116-129.
[16] C. Yang, L. Zhang, H. Lu, X. Ruan, and M.-H. Yang, "Saliency detection via graph-based manifold ranking," in Proc. IEEE CVPR, 2013, pp. 3166-3173.
[17] Q. Yan, L. Xu, J. Shi, and J. Jia, "Hierarchical saliency detection," in Proc. IEEE CVPR, 2013, pp. 1155-1162.

 

Downloads

Paper / Source code / MCL dataset