Salta al menu principale di navigazione Salta al contenuto principale Salta al piè di pagina del sito

Saggi per la call-sezione sottoposta a peer review

N. 7 (2024): Suono: la dimensione sonora del quotidiano tra arti visive, macchine, musica elettronica. Prospettive teoriche, pratiche e culturali (Vol. 1)

Sonic Landscapes in the Metaverse: Exploring Immersive Audio Design in Virtual Environments

DOI
https://doi.org/10.54103/connessioni/26755
Inviata
ottobre 11, 2024
Pubblicato
2024-12-31

Abstract

This study examines the critical importance of audio design in crafting immersive experiences within the metaverse. The analysis focuses on key technologies driving this field, with particular emphasis on spatial audio techniques such as Head-Related Transfer Functions (HRTF) and Ambisonics, which enable precise three-dimensional sound positioning.

The research investigates adaptive audio rendering, highlighting tools such as Microsoft's Spatial Audio Unity Plugin, which facilitates dynamic soundscape adjustments based on user interactions. Furthermore, the application of artificial intelligence in audio design is explored, with a discussion on the potential of Generative Adversarial Networks (GANs) for synthetic sound production and personalized audio experiences.

The study presents two significant case studies: the partnership between TCG World, STYNGR, and Downtown for interactive sonic environments, and research on Audio Augmented Reality (AAR) in art galleries. Ethical considerations, including privacy, accessibility, and the psychological impact of immersive audio, are critically examined.

The research concludes by exploring future directions, such as cross-modal integration and emotional AI systems in metaverse audio design, emphasizing the necessity for responsible development practices. Through this comprehensive analysis, the study aims to provide insights into the challenges and opportunities presented by audio design in virtual environments, contributing to the evolving landscape of metaverse technology.

Riferimenti bibliografici

  1. Baratè A., Ludovico L.A., Presti G., Sound Design for Paintings in Virtual Art Exhibitions, in «Research Square», 2022, https://assets-eu.researchsquare.com/files/rs-3165055/v1_covered_f83215b0-77c3-4d97-9ff0-2d899016d275.pdf?c=1708414295 (accessed 20 April 2024).
  2. Begault D.R., 3-D sound for virtual reality and multimedia, NASA, Cambridge 2000.
  3. Bosman I.D.V., Buruk O., Jørgensen K., Hamari J., The effect of audio on the experience in virtual reality: a scoping review, in «Behaviour & Information Technology», n. 42, 2, 2023, pp. 165-199.
  4. Cela A., Spatial audio: The sound in the metaverse, in «Techstar», 25 August 2022, https://www.techstar.it/en/blog/spatial-audio-sound-in-the-metaverse/ (accessed 20 April 2024).
  5. Chen Y., Huang L., Gou T., Applications and Advances of Artificial Intelligence in Music Generation: A Review, in «arXiv», 2024, arXiv:2409.03715.
  6. Dam A., Lee Y., Siddiqui A., Lages W.S., Jeon M., Audio augmented reality using sonification to enhance visual art experiences: Lessons learned, in «International Journal of Human-Computer Studies», 2024, https://doi.org/10.1016/j.ijhcs.2024.103329.
  7. Défossez A., Copet J., Synnaeve G., Adi Y., High fidelity neural audio compression, in «arXiv», 2022, arXiv:2210.13438.
  8. Engel J., Agrawal K.K., Chen S., Gulrajani I., Donahue C., Roberts A., GANSynth: Adversarial neural audio synthesis, in «arXiv», 2019, arXiv:1902.08710.
  9. Farinati L., Firth C., The Force of Listening, Errant Bodies Press, Berlin 2017.
  10. Gondola J., Ethical Approaches to AI-Driven Personalization and Recommendation Systems, in «Medium», 4 April 2024, https://medium.com/@jamesgondola/ethical-approaches-to-ai-driven-personalization-and-recommendation-systems-b388407101c2 (accessed 20 April 2024).
  11. Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Bengio Y., Generative adversarial nets, in «Advances in Neural Information Processing Systems», n. 27, 2014.
  12. Goodman S., Sonic Warfare: Sound, Affect, and the Ecology of Fear, The MIT Press, Cambridge 2010.
  13. Herre J., Hilpert J., Kuntz A., Plogsties J., MPEG-H 3D Audio—The New Standard for Coding of Immersive Spatial Audio, in «IEEE Journal of Selected Topics in Signal Processing», n. 9, 5, 2015, pp. 770-779.
  14. Hesar A.K., Moorthi D., Martin T.G., How can you use GANs to create realistic video, image and audio content?, in «LinkedIn», 2024, https://www.linkedin.com/advice/0/how-can-you-use-gans-create-realistic-video-l53jf (accessed 20 April 2024).
  15. Kern A.C., Ellermeier W., Audio in VR: Effects of a soundscape and movement-triggered step sounds on presence, in «Frontiers in Robotics and AI», n. 7, 2020, p. 20.
  16. Larsson P., Väljamäe A., Västfjäll D., Tajadura-Jiménez A., Kleiner M., Auditory-induced presence in mixed reality environments and related technology, in P. Dubois (ed.), The Engineering of Mixed Reality Systems, Springer, London 2010, pp. 143-163.
  17. Lazaro M.J., Lee J., Chun J., Yun M.H., Kim S., Multimodal interaction: Input-output modality combinations for identification tasks in augmented reality, in «International Journal of Human-Computer Studies», n. 167, 2022, p. 102897.
  18. Lee G.W., Lee J.H., Kim S.J., Kim H.K., Directional Audio Rendering Using a Neural Network Based Personalized HRTF, in «INTERSPEECH 2019: Show & Tell Contribution», 2019, pp. 2364-2365, https://www.isca-archive.org/interspeech_2019/lee19b_interspeech.pdf (accessed 20 April 2024).
  19. Madaan S., AI-powered personalised audio: The next frontier in personalised experience, in «Hindustan Times», 6 August 2024, https://www.hindustantimes.com/ht-insight/future-tech/aipowered-personalised-audio-the-next-frontier-in-personalised-experience-101722922131731.html (accessed 20 April 2024).
  20. Marr B., 4 ways the metaverse and Web3 will transform music, in «Forbes», 2 August 2023, https://www.forbes.com/sites/bernardmarr/2023/08/02/4-ways-the-metaverse-and-web3-will-transform-music/ (accessed 20 April 2024).
  21. Microsoft, Spatial Audio Unity Plugin, in «GitHub», 2023, https://github.com/microsoft/spatialaudio-unity (accessed 20 April 2024).
  22. Nordahl R., Nilsson N.C., The sound of being there: presence and interactive audio in immersive virtual reality, in K. Collins (ed.), The Oxford Handbook of Interactive Audio, Oxford University Press, Oxford 2014, pp. 213-233.
  23. Peters B., Hoban N., Yu J., Xian Z., Improving meeting room acoustic performance through customized sound scattering surfaces, in «Proceedings of the International Symposium on Room Acoustics», Amsterdam, 15-17 September 2019, https://d-nb.info/1212269292/34 (accessed 20 April 2024).
  24. Picone M., Mariani S., Virdis A., Castagnetti P., Digital twin & blockchain: Technology enablers for metaverse computing, in «IEEE», 2023, https://ieeexplore.ieee.org/document/10271919 (accessed 20 April 2024).
  25. Rieger M., Spatial Audio - How does the 3D sound work?, in «VR Tonung», 2024, https://www.vrtonung.de/en/spatial-audio-how-does-the-3d-sound-work/ (accessed 20 April 2024).
  26. Serafin S., Geronazzo M., Erkut C., Nilsson N.C., Nordahl R., Sonic interactions in virtual reality: state of the art, current challenges, and future directions, in «IEEE Computer Graphics and Applications», n. 38, 2, 2018, pp. 31-43.
  27. TCG World, TCG World Announces Partnership With STYNGR & Downtown, in «Downtown Music Holdings», 12 April 2024, https://www.downtownmusic.com/news/tcg-world-announces-partnership-with-styngr-and-downtown (accessed 20 April 2024).
  28. Tsingos N., Gallo E., Drettakis G., Perceptual audio rendering of complex virtual environments, in «ACM Transactions on Graphics (TOG)», n. 23, 3, 2004, pp. 249-258.
  29. Wang P., Zhang X., Ai X., Wang S., Modulation of EEG Signals by Visual and Auditory Distractors in Virtual Reality-Based Continuous Performance Tests, in «IEEE Transactions on Human-Machine Systems», n. 53, 6, 2023, pp. 1001-1011.
  30. Xie B., Head-related transfer function and virtual auditory display, J. Ross Publishing, Plantation 2013.
  31. Zotter F., Frank M., Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforcement, and virtual reality, Springer Nature, Cham 2019.