A novel motion-compensated frame interpolation (MCFI) algorithm to increase video temporal resolutions based on multihypothesis motion estimation (MHME) and texture optimization is proposed in this work. First, we form multiple motion hypotheses for each pixel by employing different motion estimation parameters, i.e. different block sizes and different directions. Then, we determine the best motion hypothesis for each pixel by solving a labeling problem and optimizing the parameters. In the labeling problem, the cost function is composed of color, shape, and smoothness terms. Finally, we refine the motion hypothesis field based on the texture optimization technique and blend multiple source pixels to interpolate each pixel in the intermediate frame. Simulation results demonstrate that the proposed algorithm provides significantly better MCFI performance than conventional algorithms.
We compare the performance of the proposed algorithm with those of conventional MCFI algorithms. In the experiment, even frames in the test sequences are skipped and then interpolated. These results confirm that the proposed algorithm outperforms the conventional algorithms. Specifically, whereas the conventional algorithms yield flickering artifacts, the proposed algorithm provides smoother transitions between input frames and intermediate frames. [download video (30.2MB @15fps)]
In addition, we provide the PSNR performances and interpolation results of the proposed algorithm and the conventional algorithms. Please select the test sequence using the select box to show PSNR curves, and then click a point in the curve to see the corresponding interpolation results, which will appear beneath the graph.
PSNR performances on sequence |
|
| PSNR (dB) | |
| Frame no. | |
| |
| [1] | T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, "High accuracy optical flow estimation based on a theory of warping," in Proc. ECCV, May 2004, pp. 25–36. |
| [2] | B.-D. Choi, J.-W. Han, C.-S. Kim, and S.-J. Ko, "Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation," IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 4, pp. 407–416, Apr. 2007. |
| [3] | C. Wang, L. Zhang, Y. He, and Y.-P. Tan, "Frame rate up-conversion using trilateral filtering," IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 886–893, Jun. 2010. |
| [4] | S.-G. Jeong, C. Lee, and C.-S. Kim, "Exemplar-based frame rate up-conversion with congruent segmentation," in Proc. IEEE ICIP, Sep.–Oct. 2012, pp. 845–848. |