Exploring History

Embodied Interaction for the Exploration of Image Collections in Mixed Reality (MR) for Museums and other Public Spaces based on the use case of the Ringier Bildarchiv

Although museums and archives dispose of extensive cultural heritage data, suitable interfaces for exploration of these collections are lacking. The «Exploring History» project is an applied research based on an archive of historic photographs. Our main objective is to engage the general audience in the exploration of this data and thus facilitate discovery and insight. Embodied interaction promises much potential for this endeavour and is highly suitable for museums and exhibition spaces when implemented with vision based technology as no contact with hardware is required. In addition, MR output is attractive for surrounding audiences.
To test this assertion, an immersive interactive MR exploration space was developed applying embodied concepts, where the user’s position and movement in the interaction space combined with in-air gestures are used for application control of the super-positioned collection data space. A user study was conducted to measure differences in user engagement, exploration behaviour and discovery between this embodied interaction concept and traditional WIMP interaction. With embodied interaction, a significantly higher degree of exploration of the collection space was measured and users discovered slightly more novelties and connections. Additionally, a higher sense of immersion, fresh perspective on the contents and increased interest in the collection were reported. Hedonic qualities (e.g. stimulation, aesthetic appeal, reward factor) scored significantly higher while pragmatic qualities such as perceived usability were lower.
The outcomes of this research is made available by the means of the «EmbodiedExplorationSpace» Unity package, that allows the application of embodied interaction modalities for exploration of other data sets. Further research questions resulting from this work include the investigation of auxiliary interaction features for increased expressiveness, the connection between spatial movement and gained insight, the impact of emotional references on and the role of curiosity in the exploration task. And it remains to investigate how a shared experience and gamification approaches could further increase the level of involvement and thus engage users even more in exploration of historic artifacts.

Keywords: Embodied Interaction, Natural User Interaction, Human Computer Interaction, Data Exploration, Cultural Heritage, Mixed Reality, User Engagement, User Experience.

Starting Position, Motivation and Objective

In previous works (e.g. Biennale 4D, V-aRt-Explorer), the question was explored of how the untapped potential of cultural heritage data collections can be made accessible to a larger target group, primarily lay users, using interactive, immersive technologies. «Biennale 4D» dealt with the creation of a VR experience to explore the archives of the Swiss pavilion at the Biennale Venezia [1] [2] [3]. «V-aRt-Explorer» – also in VR – dealt with the exploration of cultural heritage data and its geographic reference using the use case of the collection from the Museum Rietberg Zurich [4].
Immersive data visualization offers promising prospects for free data exploration, since in immersive representation, on the one hand, the space is expanded with an additional dimension compared to conventional 2D representations, and spatial perception and spatial memory can be used, thereby increasingly recognizing patterns and special features. Along with data visualization, Idreos et al. stress the importance of suitable interactive exploration interfaces [5] and Cetintemel et al. also emphasize the importance of interactivity in data exploration [6]. One reason why the potential of these cultural heritage databases has not yet been exploited is therefore the lack of suitable, interactive interfaces for exploration.
The aim of the work was to use new interactive, immersive technologies to provide access to these data collections for lay users and to stimulate this target group to explore the available data. The Ringier Bildarchiv (picture archive) offered itself as a suitable use case for this investigation.

Results

In the scope of this project, an overview of the theoretical foundations regarding spatial interaction design in general and embodiment specifically was developed. The state of the art for immersive data exploration and the technological capabilities for gesture recognition were examined. Tangible results include a large number of iteratively developed rapid prototypes for evaluating possible solutions, a prototype validated with a user study and the «EmbodiedExplorationSpace» Unity package, which can be used to explore any image data set using the Embodiment approach. Further results from the work are significant insights into the opportunities and limitations of spatial gesture interaction and the use of interactive, immersive technologies in the context of museums and exhibitions, but also insights into what factors for usability, user experience and user engagement in this application context are important about the user behavior of different demographic groups with regard to dealing with new interactive technologies and knowledge for the implementation of user studies. A final noteworthy result of the work is the publication of a poster with an associated 8-page paper in the proceedings of the HCI International 2020 conference.

Information about the project

  • Masterthesis by Kathrin Koebel
  • Master of Science in Engineering degree programme, profile Computer Science
  • Project duration and effort: April 2019 – April 2020, 27 ETCS
  • Project Advisor: Prof. Dr. Doris Agotai
  • External Expert: Dr. Stefan Arisona Müller
  • Contact Ringier Bildarchiv: Daniela Nowakowski, Stadtmuseum Aarau

Contact

FHNW Institute for Interactive Technologies (IIT), Prof. Dr. Doris Agotai or Kathrin Koebel

[1] Koebel, K., Kaufmann, O. (2016): Biennale 4D – Erschaffung einer Virtual Reality Experience zur Exploration der Archivbestände des Schweizer Pavillons an der Biennale Venezia. Bachelorthesis FS16 FHNW. URL: http://web.fhnw.ch/technik/projekte/i/bachelor16/kaufmann-koebel (last accessed: 13.04.2020).
[2] Koebel, K., Agotai, D., Arisona, S., Oberli, M. (2017): Biennale 4D – Exploring the Archives of the Swiss Pavilion at the «Biennale di Venezia» Art Exhibition. In International Conference on Interactive Digital Storytelling (pp. 327-331). Springer, Cham.
[3] Koebel, K., Agotai, D., Arisona, S., Oberli, M. (2017): Biennale 4D – A Journey in time: Virtual reality experience to explore the archives of the Swiss pavilion at the “Biennale di Venezia” art exhibition. In 2017 23rd International Conference on Virtual System & Multimedia (VSMM) (pp. 1-8). IEEE.
[4] Koebel, K., Agotai, D., Cöltekin, A. (2020): Exploring Cultural Heritage Collections in immersive analytics: challenges, benefits, and a case study using Virtual Reality. ISPRS journal of Photogrammetry and Remote Sensing (publication will be released in summer 2020).
[5] Idreos, S., Papaemmanouil, O., Chaudhuri, S. (2015): Overview of data exploration techniques. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 277-281.
[6] Cetintemel, U., Cherniack, M., DeBrabant, J., Diao, Y., Dimitriadou, K., Kalinin, A., Papaemmanouil, O., Zdonik, S.B. (2013): Query Steering for Interactive Data Exploration. In Proceedings of 6th Biennial Conference on Innovative Data Systems Research (CIDR ’13).