A MAP Estimation Framework
for HDR Video Synthesis


ABSTRACT

High dynamic range (HDR) image synthesis from multiple low dynamic range (LDR) exposures continues to be a topic of great interest. Its extension to HDR video is also a topic of significant research interest due to its increasing demand and economic costs. However, limited work has been done in this area because of its major challenge in accurate correspondence estimation; in particular, loss of data resulting from poor exposures and varying intensity make conventional optical flow methods highly inaccurate. We avoid exact correspondence estimation by proposing a statistical approach via maximum a posterior (MAP) estimation, and under appropriate statistical assumptions and choice of priors and models we reduce it to an optimization problem of solving for the foreground and background of the target frame. We obtain the background through rank minimization, and estimate the foreground via a novel multiscale adaptive kernel regression technique, which implicitly captures local structure and temporal motion by solving an unconstrained optimization problem. Extensive experimental results on both real and synthesized datasets demonstrate that our algorithm is more capable of delivering high-quality HDR videos than current state-of-the-art methods, under either subjective or objective assessment. Furthermore, a thorough complexity analysis reveals that our algorithm achieves better complexity-performance trade-off than the others.

Data Sets

  • Real world dataset (Groundtruth HDR unavailable):

    • The Fire dataset could be found here[1].

    • The LDR version can be downloaded here.

  • Synthesized dataset (Groundtruth HDR available):

    • All the original HDR files (in .exr format) could be found here[2].

    • The Hallway2 dataset could be downloaded here.

    • The Bridge2 dataset could be downloaded here.

    • The ParkingLot dataset could be downloaded here[3].

    • The Students dataset could be downloaded here.

Source Code

Click here to download the source code for our prosed HDR video synthesis algorithm (MAP_HDR).

Reconstructed Videos

Click here to download all the reconstructed videos.

Frame Results

Fire dataset (Click to enlarge)[1]

Fig. HDR video synthesis results. The 4--8th frames of the input Fire sequence[1] in the first row are synthesized by Kang et al.'s algorithm[4] in the second row, Mangiat and Gibson's algorithm[5] in the third row, Kalantari et al.'s algorithm[1] in the fourth row, and the MAP-HDR algorithm in the fifth row.

Bridge2 dataset (Click to enlarge)[2]

HDR video synthesis results. The 3--7th frames of the input Bridge2 sequence[2] in the first row are synthesized by Kang et al.'s algorithm[4] in the second row, Mangiat and Gibson's algorithm[5] in the third row, Kalantari et al.'s algorithm[1] in the fourth row, and the MAP-HDR algorithm in the fifth row.

Video Results

Bridge2 dataset[2]

Bridge2 dataset DRIVQM assessment[2][6]

Fire dataset[1]

Hallway2 dataset[2]

Hallway2 dataset DRIVQM assessment[2][6]

ParkingLot dataset[3]

ParkingLot dataset DRIVQM assessment[3][6]

Students dataset[2]

Students dataset DRIVQM assessment[2][6]

Failure case: huge object motions

Skatingboarder dataset[1]

Related Publications

  1. Y. Li, C. Lee. and V. Monga, "A MAP Estimation Framework for HDR Video Synthesis", IEEE International Conference on Image Processing, Quebec City, Canada, Sep 27th-30th, 2015.[IEEE Xplore]

  2. Y. Li, C. Lee and V. Monga, "A Maximum A Posteriori Estimation Framework for Robust High Dynamic Range Video Synthesis", IEEE Transactions on Image Processing, volume 26, issue 3, pages 1143-1157, March 2017. [Local PDF][IEEE Xplore]

Selected References

  1. P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Goldman, and E. Shechtman, Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graphics, vol. 31, no. 6, pp. 203:1–11, Nov. 2012.

  2. J. Kronander, S. Gustavson, G. Bonnet, A. Ynnerman, and J. Unger, A unified framework for multi-sensor HDR video reconstruction, Signal Process.: Image Commun., vol. 29, no. 2, pp. 203–215, Feb. 2014.

  3. C. Lee and C.-S. Kim, Rate-distortion optimized layered coding of high dynamic range videos, J. Vis. Commun. Image R., vol. 23, no. 6, pp. 908–923, Aug. 2012.

  4. S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, “High dynamic range video,” ACM Trans. Graphics, vol. 22, no. 3, pp. 319–325, Jul. 2003.

  5. S. Mangiat and J. Gibson, “Spatially adaptive filtering for registration artifact removal in HDR video,” in IEEE Int’l Conf. Image Process., Sep. 2011, pp. 1317–1320.

  6. T. O. Aydin, M. Cad´k, K. Myszkowski, and H.-P. Seidel, Video quality assessment for computer graphics applications, ACM Trans. Graphics, vol. 29, no. 6, pp. 161:1–12, Dec. 2010.

Email
ipal.psu@gmail.com

Address
104 Electrical Engineering East,
University Park, PA 16802, USA

Lab Phone:
814-863-7810
814-867-4564