MISO-VFI: A Multi-In-Single-Out Network for Video Frame Interpolation without Optical Flow

abstract

In general, deep learning-based video frame interpolation (VFI) methods have predominantly focused on estimating motion vectors between two input frames and warping them to the target time. While this approach has shown impressive performance for linear motion between two input frames, it exhibits limitations when dealing with occlusions and nonlinear movements. Recently, generative models have been applied to VFI to address these issues. However, as VFI is not a task focused on generating plausible images, but rather on predicting accurate intermediate frames between two given frames, performance limitations still persist. In this paper, we propose a multi-in-single-out (MISO) based VFI method that does not rely on motion vector estimation, allowing it to effectively model occlusions and nonlinear motion. Additionally, we introduce a novel motion perceptual loss that enables MISO-VFI to better capture the spatio-temporal correlations within the video frames. Our MISO-VFI method achieves state-of-the-art results on VFI benchmarks Vimeo90K, Middlebury, and UCF101, with a significant performance gap compared to existing approaches.

Citation

@article{lee2023multi,
    title={A Multi-In-Single-Out Network for Video Frame Interpolation without Optical Flow},
    author={Lee, Jaemin and Seo, Minseok and Lee, Sangwoo and Park, Hyobin and Choi, Dong-Geol},
    journal={arXiv preprint arXiv:2311.11602},
    year={2023}
}

Architecture

MISO-VFI’s T-shaped architecture consists of an encoder, predictor, and decoder. The target time t is incorporated via sinusoidal positional embedding and an MLP layer. The model employs 2D perceptual loss, motion perceptual loss, and MSE loss

Qualitative results