3D City Modeling Cognitive Loops
Authors: Nico Cornelis, , Kurt Cornelis,
In this video [1] we show the combined results from two recent publications [2], [3]. In [2], we introduce a real-time 3D City Modeling algorithm which is able to build compact 3D representations of cities using the assumption that building facades and roads can be modeled by simple ruled surfaces. The main advantage of this algorithm is its exceptional speed. It can process the full Structure-from-Motion and dense reconstruction pipeline at 25-30fps -- thus, the reconstructed model can directly be created online, while the survey vehicle is driving through the streets. However, due to the simple geometry assumptions, this original algorithm is unable to model cars which are everpresent in cities and obviously visually degrade our resulting 3D city model.
In [3], we therefore propose to combine the 3D reconstruction with an object detection algorithm based on Implicit Shape Models. The two components are integrated in a cognitive feedback loop. The 3D reconstruction modules inform object detection about the scene geometry, which greatly helps to improve detection precision. Using the knowledge of camera parameters and scene geometry from [2], the 2D car detections are temporally integrated in a world coordinate frame, which allows to obtain precise 3D location and orientation estimates. Those can then be used to instantiate the virtual 3D car models which improve the visual realism of our final 3D city model.
Our final system is able to create an automatic 3D city model from the input video streams of a survey vehicle, identify the locations of cars in the recorded real-world scene, and replace them by virtual 3D models in the reconstruction. Besides improving the visual realism of the final 3D model, this has as the additional benefit that it also solves privacy issues by removing personalized information from the resulting final city model. Therefore, object recognition can aid 3D reconstruction in achieving more realistic results. On the other hand, the object recognition algorithm itself can benefit from the higher-level scene knowledge which is available through 3D reconstruction. It is exactly this bidirectional nature of interactions between both the reconstruction and recognition algorithm which earns it the name of cognitive loop.
References:
[1] N. Cornelis, B. Leibe, K. Cornelis, L. Van Gool,
"external page 3D City Modeling Using Cognitive Loops",
3rd International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06), Chapel Hill, USA, June 2006.
and
Video Proceedings for CVPR 2006 (VPCVPR'06), New York, June 2006.
[2] N. Cornelis, K. Cornelis, L. Van Gool,
"external page Fast Compact City Modeling for Navigation Pre-Visualization",
In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR'06), New York, 2006.
[3] B. Leibe, N. Cornelis, K. Cornelis, L. Van Gool,
"external page Integrating Recognition and Reconstruction for Cognitive Traffic Scene Analysis from a Moving Vehicle",
In DAGM Annual Pattern Recognition Symposium, Berlin, Germany,
LNCS Vol. 4174, pp. 192-201, Springer, September 2006.
CVPR'06 Video Proceedings Best Video Award