Music Syncing as Intermedial Translation

In this article I discuss the notion of intermedial translation in the context of audiovisual works. I begin by presenting Lars Elleström’s modality model as it provides the larger framework under which I then specify the scope of intermedial translation. After confronting different interpretation of the concept, I present my main contribution, which links this theoretical notion to the practice of music synchronization. I argue that the resulting approach provides a unique tool for the analysis of audiovisual artefacts. In order to further clarify this analytical perspective and prove its usefulness, I then proceed to apply it to artifacts created through a variety of software, ranging from music visualizers to video games. A look into 2006 Flash game Line Rider and the online community that spawned around it completes the article as my main case study.

https://doi.org/10.54103/sss18678

HTML

PDF

References

Aasen, Solveig. “Crossmodal Aesthetics: How Music and Dance can Match.” The Philosophical Quarterly 71, no. 2 (2021): 223–40.

Bailie, Joanna. “Film Theory and Synchronization.” CR: The New Centennial Review 18, no. 2 (2018): 69–74.

Bruhn, Siglind. Musical Ekphrasis: Composers Responding to Poetry and Painting. Hillsdale: Pendragon Press, 2000.

———. “Penrose, ‘Seeing is Believeing’: Intentionality, Mediation and Comprehension in the Arts.” In Media Borders, Multimodality and Intermediality, edited by Lars Elleström, 137–49. London: Palgrave Macmillan, 2010.

Chesterman, Andrew. “Cross-disciplinary Notes for a Study of Rhythm.” Adaptation 12, no. 3 (2019): 271–83.

Chion, Michel. Audio-Vision: Sound on Screen. Translated by Claudia Gorbman. 2nd ed. New York: Columbia University Press, 2019.

Compton, Katherine. “Casual Creators: Defining a Genre of Autotelic Creativity Support Systems.” PhD diss., University of California Santa Cruz, 2019.

Cook, Nicholas. Analysing Musical Multimedia. Oxford: Clarendon Press, 1998.

Donnelly, Kevin J. Occult Aesthetics: Synchronization in Sound Film. New York: Oxford University Press, 2014.

———. “The triple lock of synchronization,” in The Cambridge Companion to Video Game Music, edited by Melanie Fritsch, Tim Summers, 94-109. Cambridge: Cambridge University Press, 2021.

Eco, Umberto. Experiences in Translation. Translated by Alastair McEwen. Toronto: University of Toronto Press, 2003.

———. Mouse or Rat? Translation as Negotiation. London: Phoenix, 2003.

Elleström, Lars, ed. Beyond Media Borders, Volume 1: Intermedial Relations Among Multimodal Media. London: Palgrave Macmillan, 2021.

Elleström, Lars, ed. Media Borders, Multimodality and Intermediality. London: Palgrave Macmillan, 2010.

Elleström, Lars. “The Modalities of Media: A Model for Understanding Intermedial Relations.” In Elleström, Media Borders, 11–48.

———. “The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations.” In Elleström, Beyond Media Borders, Volume 1, 2–91.

Englund, Axel. “Intermedial Topography and Metaphorical Interaction”. In Elleström, Media Borders, 69–80.

Fahlenbrach, Kathrin. “Aesthetics and Audiovisual Metaphors in Media Perception.” CLCWeb: Comparative Literature and Culture 7, no. 4 (2005). https://doi.org/10.7771/1481-4374.1280.

Filimowicz, Michael, and Jack Stockholm. “Towards a Phenomenology of the Acoustic Image.” Organised Sound 15, no. 1 (2010): 5–12.

Garro, Diego. “From Sonic Art to Visual Music: Divergences, Convergences, Intersections.” Organised Sound 17, no. 2 (2012): 103–113.

Giannakopoulou, Vasso. “Introduction: Intersemiotic Translation as Adaptation.” Adaptation 12, no. 3 (2019): 199–205.

Jakobson, Roman. “On Linguistic Aspects of Translation.” In On Translation, edited by Reuben Arthur Brower, 232–39. Cambridge, MA: Harvard University Press, 1959.

Johnson, Phylis, and Donald Petit. Machinima: The Art and Practice of Virtual Filmmaking. Jefferson: McFarland & Company, 2012.

Kaindl, Klaus. “Multimodality and Translation.” In The Routledge Handbook of Translation Studies, edited by Carmen Millán, Francesca Bartrina, 257–69. London: Routledge, 2013.

———. “A Theoretical Framework for a Multimodal Conception of Translation.” In Translation and Multimodality. Beyond Words, edited by Monica Boria, Ángeles Carreres, María Noriega-Sánchez, Marcus Tomalin, 49–70. London: Routledge, 2019.

Kanellos, Emmanouil. “Visual Trends in Contemporary Visual Music Practice.” Body, Space & Technology 17, vol. 1 (2018): 22–33. http://doi.org/10.16995/bst.294.

Krumhansl, Carol L., and Diana Lynn Schenck. “Can Dance Reflect the Structural and Expressive Qualities of Music? A Perceptual Experiment on Balanchine’s Choreography of Mozart’s Divertimento N. 15.” Musicae Scientiae 1, no. 1 (1997): 63–85.

Marshall, Sandra K., and Annabel J. Cohen. “Effects of Musical Soundtracks on Attitudes Toward Animated Geometric Figures.” Music Perception 6, no. 1 (1988): 95–112.

Miller, Kiri. Playing Along: Digital Games, YouTube, and Virtual Performance. New York: Oxford University Press, 2012.

Milstein, Dana. “Case Study: Anime Music Videos.” In Music, Sound and Multimedia. From the Live to the Virtual, edited by Jamie Sexton, 29–47. Edinburgh: Edinburgh University Press, 2007.

Mitchell, Robert W., and Matthew C. Gallaher. “Embodying Music: Matching Music and Dance in Memory.” Music Perception 19, no. 1 (2001): 65–85

Montesi, Vanessa. “Translating Paintings into Dance: Marie Chouinard’s The Garden of Earthly Delights and the Challenges Posed to a Verbal-Based Concept of Translation.” The Journal of Specialised Translation 35 (2021): 166–85.

Moseley, Roger. Keys to Play: Music as a Ludic Medium from Apollo to Nintendo. Oakland: University of California Press, 2016.

Naumann, Sandra. “The Expanded Image: On the Musicalization of the Visual Arts in the Twentieth Century.” In Audiovisuology, A Reader, Vol. 2: Essays, edited by Dieter Daniels, Sandra Naumann, 504–33. Köln: Walther König, 2015.

Ouzounian, Gascia. Stereophonica: Sound and Space in Science, Technology, and the Arts. Cambridge, MA: The MIT Press, 2021.

Queiroz, João, and Pedro Atã. “Intersemiotic Translation, Cognitive Artefact, and Creativity.” Adaptation 12, no. 3 (2019): 298–314.

Rajewski, Irina. “Border Talks: The Problematic Status of Media Borders in the Current Debate About Intermediality.” In Elleström, Media Borders, 51–69.

Roux-Girard, Guillaume. “Sound and Videoludic Experience.” In The Oxford Handbook of Interactive Audio, edited by Karen Collins, Bill Kapralos, Holly Tessler, 131–44. Oxford: Oxford University Press, 2014.

Schober, Regina. “Translating Sounds: Intermedial Exchanges in Amy Lowell’s ‘Stravinsky’s Three Pieces “Grotesques” for String Quartet’”. In Elleström, Media Borders, Multimodality and Intermediality, 163–74.

Summers, Tim. Understanding Video Game Music. Cambridge: Cambridge University Press, 2016.

Tobias, James. Sync: Stylistics of Hieroglyphic Time. Philadelphia: Temple University Press, 2010.

Tymoczko, Maria. Enlarging Translation, Empowering Translators. Manchester: St. Jerome Publishing, 2007.

Woolman, Matt. Sonic Graphics: Seeing Sound. London: Thames & Hudson, 2000.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.