Apparent Resolution Enhancement for Motion Videos, SAP 2012
Floraine Berthouzoz, Raanan Fattal
Abstract
In this work we increase the apparent resolution of videos when viewed on a high-refresh rate display by making use of perceptual properties of the visual system. We achieve this enhancement by exploiting the viewer’s natural tendency to track moving objects in videos which causes the screen pixels to be projected at different sub-pixel offsets onto the retina. We estimate the eye motion using optical flow and use it to compute multiple low-resolution frames
for each input frame. By watching these new frames at a high frame-rate, the viewer’s eyes integrate them over time and merges them into a single perceived frame with a denser pixel layout. In this work we also advance the existing approaches for resolution enhancement in the following ways. We combine current display resolution enhancement with super-resolution methods to enhance input videos that are at the display resolution. We derive a new perceived video model that accounts for actual camera sensor and display pixel shapes in order to achieve optimal enhancement. We analyze the degeneracies that certain motion velocities introduce to super-resolution and resolution enhancement, and offer algorithmic solutions for handling these scenarios as well as other difficulties that arise when dealing with the optical flow of natural videos.
A user study finds that our approach achieves a noticeable increase in the apparent resolution for videos even when viewed on regular hardware (60Hz), and further enhances resolution when viewed on higher refresh rate displays (120Hz).
A comparison between frames a viewer perceives when watching the input video (a), and the ones produced by our method (b).
Research Paper: [PDF] (8MB)
Supplemental Materials
These videos need to be viewed at 60Hz. Please download the videos and close other applications on your machine (don’t view them with your web browser!)
Videos from User Study;
All input videos were downscaled to the display resolution before we processed them. We compare the input videos to our videos and sharpened videos (we apply a sharpening filter to the input video). The sharpened videos only increase the contrast of the frames, but cannot increase the actual resolution of the video. For the user study, we did not display the labels below each video and randomized the order of the 3 types of videos.
Comparison to Templin et al.
Note that Templin et al’s videos play 3 times faster than our videos because they require 90fps input videos, but we only had 30fps input videos available. In terms of resolution enhancement, both approaches produce similar enhancement.
Related Publication
Resolution Enhancement by Vibrating Displays
Floraine Berthouzoz, Raanan Fattal
ACM Transactions on Graphics (Volume 31, Issue 2, April 2012) 15:1-15:14.