Imprint: The content of this blog entry is based on VR experiments in the context of the Biennale 4D project that were carried out in the scope of the authors masters degree studies at FHNW University of Applied Sciences and Arts Northwestern Switzerland under the supervision of Prof. Dr. Doris Agotai.
Biennale 4D is an immersive Virtual Reality experience that allows users to explore the archives of past art exhibitions. In addition to digital documentation of the individual art works in form of images and metadata, the Biennale archives contain additional artefacts such as expressive historic shots taken during the actual exhibitions. These pictures offer unique insights and share more of the spirit and atmosphere of the past events. The quest is to make these artefacts available to the user in a suitable way within an immersive VR experience.
Context
While some of these artefacts have been published previously on the web and in printed products, Virtual Reality offers an unparalleled opportunity to display these pictures in the original context of the exhibition space. The virtual environment allows the user to see an archive photo from exactly the same point of view from where the photographer originally captured the shot. This additional context added to the photograph offers a richer, more immersive experience to the user and lets him dive into that scene. Figuratively speaking, it breaths new life into these old shots and lets them become alive.
2D vs. 3D
The design challenge posed is contriving an appropriate way to display two-dimensional archive photos in a three-dimensional world without loosing the sense of immersion and presence provided by the application. Some of the common practices to do this include inserting the photo directly into the virtual room, either by projecting it onto on an complemented artificial element like some kind of partition wall or a head-up display or displaying it “free-floating” in the virtual space. The first approach works well as long as the user is standing directly in front of the picture, once he is positioned sidewise, the picture disappears and becomes merely a line with the width of the element. A solution that is often applied to fix this issue is displaying the picture on a “camera-facing” billboard, meaning that the billboard always aligns itself to the viewpoint of the user. However, by employing this technique, the photo is no more longer displayed precisely in the original position. And additionally, the notion of being within the scene is destroyed by the constant wiggle movements of the photo always aligning itself to the user’s viewpoint.
Texture Mapping
An impressive example for a completely different approach is offered by the Russian artist Alexey Zakharov in the photo-based animation project “The Old New World” [1], where he animated pictures from the early 20th century. He does this by reconstructing the scene with animated wireframes and mapping the historic photo onto these structures. Inspired by Zakharov’s work the idea of mapping the archive photos as textures onto the walls of the model evolved. This concept has many parallels with projection mapping, a light projection technique applied inter alia in many urban installation works of the famous Spanish artist Pablo Valbuena [2] such as “augmented sculpture” or “quadratura”, where a pattern of shadows and lights is projected onto a surface which then creates the illusion of depth.
For texture mapping as well as projection mapping the image needs to be transformed to match with the shape of the correlative surface and in order to result into an undistorted view. For texture mapping specifically, a transformation of the perspective view from the original photo into Bird’s eye view perspective textures for the individual surfaces is required. For this purpose the photo has to be divided into separate perspective segments and each patch needs to be normalized before it then can be applied as textures to the according faces. Yet again, the full immersion of this effect only works from one specific view point: the spot that corresponds to the original camera position [see figure 1]. The further the users moves his position away from this particular spot, the more of the distortion is revealed [see figure 2] until the intended illusion effect is completely gone. With this in mind we introduced frames [see figures 3 and 4] in our experiment to indicate to the user the correct view point and by this means optimising the user experience.
Screenshots of texture mapping experiment
Figure 1 (left): Texture mapping seen from original viewpoint.
Figure 2 (right): Texture mapping seen from other view point revealing distortions.
Figure 3 (left): The picture frame serves as a hint for the user where to position himself to discover view points with the best user experience.
Figure 4 (right): As the user approaches the frame and lines his point of view up with it, the historic shots get revealed within the original context.
[1] Zakharov, Alexey (2016): “The Old New World”. URL: https://vimeo.com/160024074
[2] Valbuena, Pablo. URL: http://www.pablovalbuena.com/