When a very fast dynamic event is recorded with a low-framerate camera, the resulting video suffers from severe motion blur (due to exposure time) and motion aliasing (due to low sampling rate in time). True Temporal Super-Resolution (TSR) is more than just Temporal-Interpolation (increasing framerate). It also recovers new high temporal frequencies beyond the temporal nyquist limit of the input video, thus resolving both motion-blur and motion-aliasing. In this paper we propose a "Deep Internal Learning" approach for true TSR. We train a video-specific CNN on examples extracted directly from the low-framerate input video. Our method exploits the strong recurrence of small space-time patches inside a single video sequence, both within and across different spatio-temporal scales of the video. We further observe (for the first time) that small space-time patches recur also across-dimensions of the video sequence - i.e., by swapping the spatial and temporal dimensions. In particular, the higher spatial resolution of video frames provides strong examples as to how to increase the temporal resolution of that video. Such internal video-specific examples give rise to strong self-supervision, requiring no data but the input video itself. This results in Zero-Shot Temporal-SR of complex videos, which removes both motion blur and motion aliasing, outperforming previous supervised methods trained on external video datasets.
00:00 Intro
02:43 Fast Events Captured with a Low Frame Rate
05:43 Small ST video Patches Recur
14:19 New Observation: Patches Also Recur
20:28 How strong is this recurrence?
23:12 Why does this Happen?
25:52 Deep Learning Augmentations
27:26 Generating Training Examples From a Single Video
37:49 Coarse-to-Fine Scheme
38:47 A Simple 3D Fully Convolutional Network
50:34 Creating a Challenging Dataset
56:39 Discussion
[Chapters were auto-generated using our proprietary software - contact us if you are interested in access to the software]
Lecture slides: [ Ссылка ]
Cubic interpolation has a wikipedia page here: [ Ссылка ]
Actual code we used for all our resizing can be found here: [ Ссылка ]
see line 202 for cubic interpolation polynom
The talk is based on the speaker's paper:
Across Scales & Across Dimensions: Temporal Super-Resolution using Deep Internal Learning (ECCV 2020)
arxiv:
[ Ссылка ]
project page:
[ Ссылка ]
open sources:
[ Ссылка ]
Presenter BIO:
Liad Pollak Zuckerman is a Machine Learning Applied Researcher at General Motors. She is an M.Sc graduate from Weizmann Institute. Her research topics include single video and single 3D image super-resolution using deep internal learning.
-------------------------
Find us at:
Newsletter for updates about more events ➜ [ Ссылка ]
Sub-reddit for discussions ➜ [ Ссылка ]
Discord server for, well, discord ➜ [ Ссылка ]
Blog ➜ [ Ссылка ]
AI consultancy Abelians ➜ [ Ссылка ]
