ATTENTION: This is a web archive! The IMS Group was split up in 2018 and does not exist anymore. Recent work of former members can be found at the VR/AR Group and the Computer Vision Group.

Interactive Media Systems, TU Wien

Temporal-Consistent Stereo Matting for High-Quality Novel View Synthesis and Visual Effects

Research project in the area of Image and Video Analysis & Synthesis.

Keywords: video processing, stereo analysis, 3D reconstruction, special effects, image matting.

About this Project

This project develops new image processing techniques for the emerging field of 3D television. On the basis of two videos that are recorded by slightly displaced cameras, we aim to extract (1) the opacity values of individual pixels, (2) a depth reconstruction of the scene and (3) the temporal relationship between the images of a sequence. The computed information can serve as the input for an autostereoscopic display that uses these results to provide the user with temporal-smooth depth impression. Moreover, our results may offer the exciting possibility to realize free-viewpoint video where the user has control over a virtual camera and can select the preferred viewpoint. To suit the needs of these applications, our focus lies on generating results of high quality that outperform the current state of the art. Due to addressing the image matting problem, we will also be able to handle fuzzy and hairy objects that are traditionally difficult in image processing.

Funding provided by

WWTF Wiener Wissenschafts-, Forschu und Technologiefonds

Additional Information

The project has been funded by the Vienna Science and Technology Fund (WWTF) through project ICT08-019.

People

Principal Investigator: Margrit Gelautz

Postdoctoral Researchers: Michael Bleyer, Christoph Rhemann

Partner: Carsten Rother (Microsoft Research)

Results / Publications

2014

  • S. Ghuffar, N. Brosch, N. Pfeifer, M. Gelautz: "Motion Estimation and Segmentation in Depth and Intensity Videos"; Integrated Computer-Aided Engineering 2014. Detail information

2013

  • A. Hosni, M. Bleyer, M. Gelautz. “Secrets of Adaptive Support Weight Techniques for Local Stereo Matching”, Journal of Computer Vision and Image Understanding (CVIU) 2013. Detail information
  • A. Hosni, C. Rhemann, M. Bleyer, C. Rother, M. Gelautz: "Fast Cost-Volume Filtering for Visual Correspondence and Beyond", IEEE Transactions on Pattern Analysis and Machine Intelligence 2013. Detail information
  • M. Bleyer, C. Breiteneder: "Stereo Matching - State-of-the-Art and Research Challenges"; in: "Advanced Topics in Computer Vision" 2013. Detail information

2012

  • M. Bleyer, C. Rhemann, C. Rother: "Extracting 3D Scene-Consistent Object Proposals and Depth from Stereo Images", ECCV 2012. Detail information
  • A. Hosni, M. Gelautz, M. Bleyer: "Accuracy-Efficiency Evaluation of Adaptive Support Weight Techniques For Local Stereo Matching", DAGM and OAGM Symposium 2012 (poster presentation). Detail information
  • N. Brosch, A. Hosni, C. Rhemann, M. Gelautz: "Spatio-temporally Coherent Interactive Video Object Segmentation via Efficient Filtering", DAGM and OAGM Symposium 2012 (poster presentation). Detail information
  • N. Brosch, A. Hosni, C. Rhemann, M. Gelautz: "Interactive Video Segmentation via Efficient Filtering", International Computer Vision Summer School 2012 (poster presentation). Detail information
  • P. Kohli, H. Nickisch, C. Rother, C. Rhemann: "User-Centric Learning and Evaluation of Interactive Segmentation Systems", Int. J. Computer Vision 2012. Detail information

2011

  • L. He, M. Bleyer, M. Gelautz: "Object Removal by Depth-Guided Inpainting" , ÖAGM/AAPR 2011. PDF
  • N. Brosch, C. Rhemann, M. Gelautz: "Segmentation-Based Depth Propagation in Videos" , ÖAGM/AAPR 2011. PDF
  • C. Rhemann, A. Hosni, M. Bleyer, C. Rother, M. Gelautz: “Fast Cost-Volume Filtering for Visual Correspondence and Beyond” , CVPR 2011, Colorado Springs. Detail information
  • K. He, C. Rhemann, C. Rother, X. Tang, J. Sun: “A Global Sampling Method for Alpha Matting” , CVPR 2011, Colorado Springs. PDF
  • A. Hosni, C. Rhemann, M. Bleyer, M. Gelautz: "Temporally Consistent Disparity and Optical Flow via Efficient Spatio-temporal Filtering" , PSIVT 2011. Detail information
  • M. Bleyer, C. Rhemann, C. Rother: “PatchMatch Stereo - Stereo Matching with Slanted Support Windows” , BMVC 2011. Detail information
  • M. Bleyer, C. Rother, P. Kohli, D. Scharstein, S. Sinha: “Object Stereo – Joint Stereo Matching and Object Segmentation” , CVPR 2011. PDF
  • A. Hosni, M. Bleyer, C. Rhemann, M. Gelautz, C. Rother: “Real-time Local Stereo Matching Using Guided Image Filtering” , IEEE International Workshop on Hot Topics in 3D 2011. Detail information

2010

  • H. Nickisch, C. Rother, P. Kohli, C. Rhemann: “Learning an Interactive Segmentation System” , ICVGIP 2010, Chennai . PDF; TR: PDF
  • C. Rhemann, C. Rother, P. Kohli, M. Gelautz: “A Spatially Varying PSF-based Prior for Alpha Matting” , CVPR 2010, San Francisco. PDF; Supplementary Material: ZIP (video and additional results); TR: PDF
  • M. Bleyer, C. Rother, P. Kohli: “Surface Stereo with Soft Segmentation” , CVPR, San Francisco 2010. PDF
  • A. Hosni, M. Bleyer, M. Gelautz: ”Near Real-Time Stereo With Adaptive Support Weight Approaches” , 3DPVT 2010, Paris. PDF
  • M. Bleyer, S. Chambon: ”Does Color Really Help in Dense Stereo Matching?” , 3DPVT 2010, Paris. PDF

2009

  • A. Hosni, M. Bleyer, M. Gelautz: “Image Segmentation Via Iterative Geodesic Averaging” , ICIG 2009, Xi’an, China. PDF

Downloads

1291-supp.zip 7.81 MB Zip archive Download
1291-tr-final.pdf 5.51 MB PDF document Download
1291.pdf 2.37 MB PDF document Download
cvpr11matting.pdf 2.85 MB PDF document Download
does color text 1.27 MB PDF document Download
final-iasyslearn-supp.pdf 296 KB PDF document Download
final-iasyslearn.pdf 2.15 MB PDF document Download
image segmentation via iterative geodesic averaging 2.08 MB PDF document Download
near real time stereo with adaptive support weight 620 KB PDF document Download
object stereo joint stereo matching 925 KB PDF document Download
segmentation based depth propagation 4.36 MB PDF document Download
surface stereo with soft segmentation 861 KB PDF document Download
VCIP14_supplementary_material 2.12 MB PDF document Download

Publications

2014

Efficient Depth Propagation in Videos with GPU-acceleration In this paper we propose an optimized semi-automatic approach for efficient 2D-to-3D video conversion. It is based on a conversion algorithm [1] that… M. Ivancsics, N. Brosch, M. Gelautz Details
Efficient quality enhancement of disparity maps based on alpha matting We propose an efficient disparity map enhancement method that improves the alignment of disparity edges and color edges even in the presence of mixed… N. Brosch, M. Nezveda, M. Gelautz, F. Seitner Details
Effiziente Tiefenpropagierung in Videos mit GPU-Unterstützung The commercial availability of stereoscopic displays increases the demand for 3D content. Therefore, fast and robust tools for the generation of such… M. Ivancsics Details
Motion estimation and segmentation in depth and intensity videos This paper investigates motion estimation and segmentation of independently moving objects in video sequences that contain depth and intensity… S. Ghuffar, N. Brosch, N. Pfeifer, M. Gelautz Details

2013

Stereo Matching - State-of-the-Art and Research Challenges M. Bleyer, C. Breiteneder: "Stereo Matching - State-of-the-Art and Research Challenges"; in: "Advanced Topics in Computer Vision", issued by:… M. Bleyer, C. Breiteneder Details
Secrets of Adaptive Support Weight Techniques for Local Stereo Matching In recent years, local stereo matching algorithms have again become very popular in the stereo community. This is mainly due to the introduction of… A. Hosni, M. Bleyer, M. Gelautz Details
Fast Cost-Volume Filtering for Visual Correspondence and Beyond Many computer vision tasks can be formulated as labeling problems. The desired solution is often a spatially smooth labeling where label transitions… A. Hosni, C. Rhemann, M. Bleyer, C. Rother, M. Gelautz Details

2012

Extracting 3D Scene-consistent Object Proposals and Depth from Stereo Images This work combines two active areas of research in computer vision: unsupervised object extraction from a single image, and depth estimation from a… M. Bleyer, C. Rhemann, C. Rother Details
Accuracy-Efficiency Evaluation of Adaptive Support Weight Techniques For Local Stereo Matching Adaptive support weight (ASW) strategies in local stereo matching have recently attracted many researchers due to their compelling results. In this… A. Hosni, M. Gelautz, M. Bleyer Details
Spatio-temporally Coherent Interactive Video Object Segmentation via Efficient Filtering In this paper we propose a fast, interactive object segmentation and matting framework for videos that allows users to extract objects from a video… N. Brosch, A. Hosni, C. Rhemann, M. Gelautz Details
Interactive Video Segmentation via Efficient Filtering In [2], we present a fast, interactive segmentation and matting framework for videos. It is based on recent work [5] that recovers high-quality… N. Brosch, A. Hosni, C. Rhemann, M. Gelautz Details
User-Centric Learning and Evaluation of Interactive Segmentation Systems Many successful applications of computer vision to image or video manipulation are interactive by nature. However, parameters of such systems are… P. Kohli, H. Nickisch, C. Rother, C. Rhemann Details

2011

Temporally Consistent Disparity and Optical Flow via Efficient Spatio-temporal Filtering This paper presents a new efficient algorithm for computing temporally consistent disparity maps from video footage. Our method is motivated by… A. Hosni, C. Rhemann, M. Bleyer, M. Gelautz Details
PatchMatch Stereo - Stereo Matching with Slanted Support Windows Common local stereo methods match support windows at integer-valued disparities. The implicit assumption that pixels within the support region have… M. Bleyer, C. Rhemann, C. Rother Details
Real-time Local Stereo Matching Using Guided Image Filtering Adaptive support weight algorithms represent the state-of-the-art in local stereo matching. Their limitation is a high computational demand, which… A. Hosni, M. Bleyer, C. Rhemann, M. Gelautz, C. Rother Details
A Global Sampling Method for Alpha Matting Alpha matting refers to the problem of softly extracting the foreground from an image. Given a trimap (specifying known foreground/background and… K. He, C. Rhemann, C. Rother, X. Tang, J. Sun Details
Fast Cost-Volume Filtering for Visual Correspondence and Beyond Many computer vision tasks can be formulated as labeling problems. The desired solution is often a spatially smooth labeling where label transitions… C. Rhemann, A. Hosni, M. Bleyer, C. Rother, M. Gelautz Details
Object Stereo - Joint Stereo Matching and Object Segmentation This paper presents a method for joint stereo matching and object segmentation. In our approach a 3D scene is represented as a collection of visually… M. Bleyer, C. Rother, P. Kohli, D. Scharstein, S. Sinha Details
Object Removal by Depth-guided Inpainting Object removal by image inpainting aims at the visual uniformity of the inpainted blanks among their surroundings. Most inpainting algorithms pursue… L. He, M. Bleyer, M. Gelautz Details
Segmentation-Based Depth Propagation in Videos In this paper we propose a simple yet effective approach to convert existing 2D video content into 3D. We present a new semi-automatic algorithm that… N. Brosch, C. Rhemann, M. Gelautz Details

2010

Learning an Interactive Segmentation System Many successful applications of computer vision to image or video manipulation are interactive by nature. However, parameters of such systems are… H. Nickisch, C. Rother, P. Kohli, C. Rhemann Details
Surface Stereo with Soft Segmentation (Poster presentation - ICVSS 2010) M. Bleyer, C. Rother, P. Kohli: "Surface Stereo with Soft Segmentation (Poster presentation - ICVSS 2010)"; Poster: International Computer Vision… M. Bleyer, C. Rother, P. Kohli Details
Surface Stereo with Soft Segmentation (Talk for best presentation award - ICVSS 2010) M. Bleyer, C. Rother, P. Kohli: "Surface Stereo with Soft Segmentation (Talk for best presentation award - ICVSS 2010)"; Talk: International Computer… M. Bleyer, C. Rother, P. Kohli Details
A Spatially Varying PSF-based Prior for Alpha Matting C. Rhemann, C. Rother, P. Kohli, M. Gelautz: "A Spatially Varying PSF-based Prior for Alpha Matting"; Poster: International Computer Vision Summer… C. Rhemann, C. Rother, P. Kohli, M. Gelautz Details
A Spatially Varying PSF-based Prior for Alpha Matting In this paper we considerably improve on a state-of-the-art alpha matting approach by incorporating a new prior which is based on the image formation… C. Rhemann, C. Rother, P. Kohli, M. Gelautz Details
Near Real-Time Stereo With Adaptive Support Weight Approaches Algorithms based on the adaptive support weight strategy currently represent the state-of-the-art in local stereo matching. Unfortunately, their… A. Hosni, M. Bleyer, M. Gelautz Details
Does Color Really Help in Dense Stereo Matching? This paper investigates the role of color in global stereo matching approaches. In our evaluation study, we build various energy functions by… M. Bleyer, S. Chambon Details

2009

Image Segmentation Via Iterative Geodesic Averaging We present a simple and fast method for performing unsupervised segmentation. Our method works by centering a square window on each pixel of the… A. Hosni, M. Bleyer, M. Gelautz Details

Press Coverage

October 2011

Der Standard (Wissenschaft) Algorithmen zum Angreifen Oct 11, 2011 Screenshot

May 2011

Futurezone.at TU Wien: Software für 3-D ohne Brille May 23, 2011 Screenshot

May 2010

Sg.hu (Hungarian) 3D-s tévézés különleges szemüveg nélkül May 28, 2010 Screenshot
Futurezone.at 3-D-Fernsehen ohne Brille May 26, 2010 Screenshot

January 2010

ORF TV show "Newton" 3D ohne Brille - An der Zukunft wird schon geforscht Jan 09, 2010 Screenshot

April 2009

Forschen und Entdecken Frau der Stunde: Leidenschaft für Telematik (siehe S. 5) Apr 01, 2009 –