We combine high resolution HDR-photography and precise 3D laser scanning to create photo-realistic and highly detailed digital models of real locations that allow a virtual observer to move around interactively and explore the place from every possible perspective. On-site we scan the scene from multiple positions with a rate of 100.000 samples per second. We apply a pipeline of sophisticated algorithms to align and merge the individual scans into a dense point cloud with millions of samples which provides a highly detailed 3D model with a precision in the range of a few millimeters. During scanning, an HDR-camera automatically takes high-resolution photos which are then registered to the 3D model, color calibrated, and eventually used as a seamless texture map on the 3D model.
To interactively visualize the massive amount of point and image data, we developed an innovative and highly efficient rendering engine for textured 3D point clouds. By expanding 3D points into small surface elements, the observer gets the visual impression of a continuous surface even at close-up views. Using high-resolution HDR images as textures drastically increases the visual detail and provides a truly photo-realistic appearance of the scanned object or scene.
View dependent reflections and soft shadows are key ingredients for a captivating and immersive visual experience and make the renderings of our virtual 3D models quite compelling. By using advanced rendering techniques often applied in the game or movie industry, our software is able to handle hundreds of light sources, each with its own brightness and color temperature.
Similar to the process of stitching multiple photos to one single panoramic image, the first step of our processing pipeline is to merge the individual scans into one large point cloud representing the whole object or scene. Mathematically this can be modeled as a non-linear global optimization problem to find the unknown relative scanner poses and we developed an automatic method to solve this optimization efficiently and robustly. Intuitively, the method moves and rotates all scanner poses such that the overlapping scans fit together with a precision of a few millimeters.
Registration 2: Images
When equipped with a precisely manufactured lens, modern SLR cameras like the NIKON D800 are able to produce incredibly sharp, high-resolution and high-dynamic-range images. Common techniques use these or even lower quality images to add color information to a point cloud. Since point clouds usually have a coarser resolution than photos, this process discards valuable visual information such that the resulting renderings often appear blurry. In contrast, our approach projects the photos directly onto the scanned surface and thereby rather adds depth information to high-quality photos. This requires precisely calibrated and aligned photos. During projection, even the smallest deviations from the true camera position (fractions of a millimeter) can lead to ghosting artifacts on surfaces. We developed automatic methods which simultaneously correct the projections of hundreds of photos taken during the scan such that all images project up to pixel accuracy onto the surfaces. Moreover since some 3D points might be visible in multiple images, we automatically select the „best“ image (fragment).
Selecting "good" images
In most physical setups, in particular when high-quality SLR cameras are used, the HDR images cannot be taken from the exact same positions as the 3D scans (e.g. because the camera is mounted on top of the scanner). This is problematic since even if the pose registration of the images with respect to the 3D data is extremely precise, there are still parallax issues because parts of the scene might be visibile from the scanner or the camera but not from both. In our 3D processing pipeline, however, we can exploit the combined 3D information from all scans in order to reliably determine those images which actually „see“ a given surface patch. From these candidates we can select the best image texture.
Intensity and Color Adaption
Even with highly efficient reconstruction algorithms available, extensive on-site scanning jobs may take hours. During this time the illumination conditions may have changed significantly since, e.g., the sun and thereby the shadows move. This results in photos with different brightness levels and color temperatures. When projecting these images to neighboring surface patches the differences manifest in hard, visible and unnatural seems. In the interior of a building like the Aachen Cathedral this can be quite dramatic: At nightfall the natural illumination through the windows switches to artificial illumination. However, scanning over night was not an option, because the colored glas windows should be visible in the final 3D model. Common digital image processing software is only capable to adust a single image or to adjust a set of images with the same settings. We created automatic methods to adapt intensity and color temperatures of all images on a per pixel basis to colorize the scanned surfaces seamlessly.
Removal of specular Highlights
The interior walls and floors of many historic buildings are made of slightly glossy materials like polished stone or marble. These materials cast bright light reflections which depend on the relative location of the viewer and the light sources. Taking a photo from a certain position will also capture specular highlights visible from that particular view. When these images are used as textures for the 3D model, those highlights will look quite unnatural when seen from a different perspective which results in a very implausible look. The last step of our processing pipeline deals with this problem. Image regions with a specular highlight are automatically detected or marked manually by the user. Based on state-of-the-art image completion techniques, we developed an approach that suppresses specular highlights while still preserving the original image details.