AcousticAVE

AcousticAVE aimed at integrating visual and aural immersion in VR environments, which poses numerous technological challenges, both in the development of adapted acoustic modelling techniques and the implementation of accurate aural-visual synchronisation. It involved collaboration between research units from the Universities of Minho (CIPsi) and Aveiro (IEETA), assisted by external consultants: Prof. Damian Murphy (University of York) and Juan Miguel Navarro (UCAM - Murcia). At the Laboratory of Visualization and Perception (LVP) in Guimarães, the U. of Minho had installed one of the most sophisticated VR systems in the country, equipped with a 2.2x9.6m continuous projection screen made up of 3 flexible configuration panels (0, 90 or 135º) and corresponding DLP (Digital Light Processing) projectors. A treadmill synchronised with the 3D visual scene allows walking on the virtual environment. User-tracking is available using both inertial and infra-red technology. The system’s main intended use was psychophysical research, focusing on naturalistic environments, which added to the importance of incorporating virtual sound. LVP researchers were constructing experiments on Biomotion perception (walking patterns) focussing on the cross-influence of the auditory and visual channels. For those experiments, it was crucial to obtain highly synchronised, real-time audiovisual stimuli. This linked perfectly with a line of research pursued at IEETA aimed at the development of auralisation in virtual and augmented reality environments. The initial work, based on a head-mounted display (HMD) system, had produced very encouraging results, the top example being the MSc project ‘Virtual Hall’ [Casaleiro08], awarded the first prize at the 2007 Audio Technology contest of the Portuguese section of the Audio Engineering Society (AES). In this scenario, the project involved two main work packages: 1) At IEETA, the development of a complete earphone-based auralisation package, to allow sound spatialisation taking into account the actual room acoustic properties (extracted automatically from the model) and the trajectory of the sound source, as well as the head orientation and position of the listener, both tracked in real time. Preliminary validation tests were carried out using a simple HMD system, but the main test-bed was LVP’s CAVE-like system, where it was installed and gradually updated with new features. 2) At LVP, hardware and software integration ensuring audiovisual synchronisation by controlling the latencies of the audio and video digital chains. Updates were perceptually validated. Psychophysical testing focussed on how people learn in, and interact with, auditory and audiovisual VR systems, allowing the development of new specific knowledge regarding how humans should be prepared to interact with these systems. The outcomes during the project included 3 international journal papers, 12 international conference papers, 1 PhD thesis, 2 computer applications and 2 workshops. A summary paper was subsequently published in the 8th Iberian Congress of Acoustics (TECNIACUSTICA 2014 - Murcia, Oct 29- Nov 3): Campos G, Dias P, Vieira J, Santos J, Mendonça C, Lamas JP, Silva N and Lopes S (2014) ‘AcousticAVE: Auralisation Models and Applications in Virtual Reality Environments’. FCT’s final evaluation report stated: ‘The scientific goals were fully attained. The results show great scientific quality, particularly in terms of refereed international journal publications. The project contributed to the training of young researchers and the international visibility of its team. This was a successful project, with good scientific productivity, which matched or exceeded the predictions in the proposal. In particular, rather than the predicted 4 papers in international conferences, 12 were published. Also deserving mention is the fact that the number of MSc theses completed in the context of the project (12) far exceeded what had been promised (4).'