virtual reality. From controlled hallucination to Resident Evil 7: Biohazard

In this paper, I will work on the relationship between perception and imagination in Virtual Reality, claiming that “hallucination” is the ordinary motor also for online perception and not only a deviant form of it. First, I will deal with the problem of perception from the point of view of cognitive semiotics and I will try to underline the crucial role of imagination, claiming that perception is a form of “controlled hallucination”. Later, I will focus on the relationship between perception, hallucination, and memory in Virtual Reality. On the one hand, I will claim that Virtual Reality expresses the transition from actual perception to imagination, memory or dream through another actual perception. On the other hand, I will claim that it can express it without any problems through the old techniques coming from cinema and other audiovisual languages, since they par-tially share the very same formal apparatus of enunciation. I will demonstrate all of this by analyzing Resident Evil 7: Biohazard . Abstract

Perception, imagination, and the control of the reality First, I will deal with the problem of perception from the point of view of cognitive semiotics. 1 I will try to underline the crucial role of imagination, claiming that perception is a form of "controlled hallucination", 2 where, by "controlled hallucination", I mean the product of the imagination controlled by the world. The main idea is that "hallucination" is the model of perception and not a deviant form of it. With "hallucination", as defined in perception studies and in the neurogeometry of vision, 3 I mean the morphological activity of the production of forms by the imagination, which remains crucial both when it is not controlled by the world -as in the case of hallucination, imagination, or dream -and when it is controlled by the world, as in the case of online perception. In Virtual Reality, the world that controls perception is substituted by a technology, a prosthesis capable of creating a strong effect of reality, a simulacrum with an effect of presence that no other audiovisual has ever been able to build. 4 In Augmented Reality, on the other side, the technology adds elements to the world that controls perception, without a full-blown substitution, as it happens with Virtual Reality.
Maybe, the word "hallucination" can be misleading, since perceptual phenomena under the aegis of hallucination may seem to lose the concreteness that I want to characterize them as having. It is possible that "figuration" would fit better with the ideas I will develop here, since no Sartrean "derealization" is involved. 5 However, since our brains try to guess what is out there, and to the extent that that guess matches the evolving sensory data, we perceive the world, in Surfing Uncertainty, Clark recalls the slogan coined by the vision scientist Ramesh Jain that perception is "controlled hallucination". This is the direction I am going to take. But, precisely because "this view of perception puts us in genuine cognitive contact with the salient aspects of our environment", Clark suggests we consider hallucination as a form of "uncontrolled perception". 6 However, Clark's view -if it is put like that -is the classical one that thinks of perception as grounding both hallucination and imagination, which are supposed to be "deviant" or "uncontrolled" forms of perception. Since I want to claim the opposite, I will continue to use "hallucination", and since there is a well-established tradition regarding this concept in the field of perception studies, I will do so with the caveat that "hallucination" does not imply any kind of "derealization" of perceptual phenomena from a phenomenological point of view. 7 Indeed, with "hallucination", or "figuration", I indicate a process of microgenesis 8 that continuously produces the next thread of perceptual experience while the current one fades, and does so without voluntary control. 9 I claim that meaning guides this microgenesis, or hallucination, and that imagination is the engine of this microgenesis guided by meaning.
For instance, in Kanizsa's triangle (Fig. 1), the triangle is totally a product of our hallucination, since it is the only thing that is not present in the stimuli. It is not by chance that, as Reddy and colleagues show, not only are online perception and imagination closely related in the brain, but perception, as it occurs in creatures like us, is co-emergent with imagination. 10 What we perceive is literally (not metaphorically) the future, not the present, because perception is the anticipation of the next thread of sensory information through previous knowledge. Indeed, in perception we build through imagination the world that we expect.
There is a beautiful experiment by Adams and colleagues that shows that, in some circumstances, we hear the presence of the absence, that is, we hallucinate something that is not there, but we expect to be there. 11 When silence arrives, we literally do not hear it as we are supposed to do (i.e. as nothing playing). In its place, we hear the presence of the absent sound that we were  expecting. Adams and colleagues' experiment runs as follows. They used a simple computer simulation of birdsong. A multi-layer prediction machine processes sequences of simulated bird-chirps. The simulations were then repeated but omitting the last three chirps of the original signal. At the first missing chirp, the network responds with a strong precise moment where the first missing chirp should have occurred, the system generated a brief, transient illusory percept. This hallucinated percept was not strong, but the timing was correct with respect to the missing chirp. Thus, our perceptive system first dimly "perceived" (hallucinated) the missing chirp, before responding with a strong error signal when the actual absence of the anticipated sensory evidence became apparent. Of course, Kanizsa's triangle is a visual correspondent of Adams and colleagues' experiment. This is what I call perception as a "controlled hallucination", which is the general functioning of our rich, world-revealing perception at any level, since it concerns an organism structurally coupled with its environment trying to minimize disorder and surprise. 12 The Goethean account of perception I refer here to what Jan Koenderink, a cognitive scientist and mathematician who works on the connection between theory of singularities and perception, used to call the "Marrian" and the "Goethean" accounts of perception. ■ According to the Marrian account: perception is the result of standard computations on optical data. ■ According to the Goethean account: perception is controlled hallucination, or "controlled figuration".
The mainstream view in cognitive science and neuroscience, which is often also the commonsense view, is that perception is all about a kind of passive imprinting of the world upon the sense organs and the brain. As Egner and colleagues say, "on traditional accounts the visual system was seen as a passive analyzer of bottom-up sensory information". 13 This is a view of the perceiving brain as highly stimulus-driven, taking energetic inputs from the senses and turning them into a coherent percept by a kind of inwards flowing stream.
The Predictive Processing account of perception takes a different direction and includes a top-down predictive aspect in its account. 14 However, Predictive Processing thinks of perception as a kind of new schematism between aesthetics (the sensory data) and the concepts (the priors). According to this view, our brains are proactive: they are constantly buzzing as they try to predict the sensory signals arriving across all modalities. When such proactive brains "match" the incoming sensory signal, we perceive the world, understand it, and are immediately positioned to imagine it so as to act in it too. However, "hallucination" is different from the mainstream notion of "prior". A prior -as used in Bayesian inference -is a generic, usually statistical, property. 15 For example, light comes from above is such a prior (if put in suitable format). It applies, on the average, for terrestrial animals that live in open spaces. Such priors package 'frozen' prior experience as it were. 'Hallucinations' differ by not being frozen, applying to the actual situation. Hallucinations can be regarded as specific, necessarily tentative, instantiations of the observer's present "situational awareness" instead of its average past. This is important, because we do not always update our perceptions according to our past experience and according to the changes in our priors. This is very well seen in the popular Muller-Layer illusion, where even when we learn that the two lines have the same length, we still keep on seeing them in the previous way. In perception, "data" are constructed through an attunement of the organism and the world, where the organism looks for elements that are worth for him to look for. For instance, as in Kanizsa's triangle, we look for edges and we perceive edges even if edges are the only thing that are not present in the stimulus. This is because, given evolutionary pressure and experience driven plasticity, we are wi red to the envi ronment i n order to produce states that track edges when exposed to di sconti nui ti es. The system i s physi cal l y attuned to such thi ngs, ' set up to be set off' by such vi sual di sconti nui ti es. 17 Exploring the world, the organism casts his questions to the environment through imagination and predicts its answers until he encounters resistance. This very action turns optical or sound structure into "data", which are not sent from the world to the organism through senses but are the actual product of the autopoietic structure of the system of perception. Perception occurs when the top down activity of the imagination succeeds at generating the sensory data for itself, building a coherent story that paves the way for efficacious action. When it can generate the future sensory data, the agent perceives the world. When it cannot, encountering a resistance, it tries a new attunement or changes the world through action. Therefore, data are built up because we produce them in looking for what we need for action, in order to minimize our work in the environment.

Moving towards virtual and augmented reality
The difference between the mainstream view, where data are sent by the environment and processed through perception, and the view where they are the product of what we look for in a coupled environment, can be operationalized as follows: perception is sensorily guided potential behavior. 18 Potential is key here. Perception is grounded on imagination, as it concerns the potential behavior connected to a coherent "story" we are building in order to act in the world and minimize disorder. 19 Perception as sensorily guided potential behavior is meant to reveal a world of salient, meaningful, interacting causes selected in the light of human needs and possibilities, which is exactly what pragmatists had in mind when they were telling us that the meaning of something consists in its conceivable practical bearings. This also marks the difference between pragmatism and behaviorism, since perception is neither action nor behavior. This pragmatist idea is consistent with the Affordance Competition Hypothesis originally introduced by Cisek, 20 which is also a key hypothesis for the Predictive Processing by Andy Clark. 21 The brain processes sensory information to specify, in parallel, several potential actions that are currently available. These potential actions compete against each other for further processing, while information is collected to bias this competition until a single response is selected. 22 If we go beyond Cisek's terminology of "processing information", like I think we have to do in order to focus on the main problem of perception as controlled hallucination, we see that, at the moment, we are seeing a good deal of confirmation regarding this "pragmatist" approach from the neurosciences. 23 The classic distinction between perception, cognition and action simply fails to reflect not only the phenomenology of our experience, but also the global functional architecture of the brain, where, for instance, motor systems are active and play a huge part also in perception, decision making, social cognition and problem solving. 24 Increasing and highly suggestive evidence challenges the view of core cognitive capacities (such as planning and deciding) as neurophysiologically distinct from the circuitry of sensorimotor control. For example, decisions concerning eye movements and the execution of eye movements recruit highly overlapping circuits in lateral intraparietal area (LIP), frontal eye fields (FEF), and the superior colliculus [...]. In the same vein, a perceptual decision task (one in which the decision is reported by an arm movement) revealed marked responses within premotor cortex corresponding to the process of deciding upon a response (Romo et al. 2004). Quite generally, wherever a decision is to be reported by (or otherwise invokes) some motor action, there looks to be an entwining of perceptuo-motor processing and decision-making, leading Cisek and Kalaska to suggest that 'decisions, at least those reported through actions, are made within the same sensorimotor circuits that are responsible for planning and executing the associated action' (Cisek and Kalaska 2011: 274). In cortical associative regions such as posterior parietal cortex (PPC), Cisek and Kalaska go on to argue, activity does not seem in any way to respect the traditional divisions between perception, cognition, and action. Instead we find neuronal populations that trade in shifting and context-responsive combinations of perceiving, deciding, and acting, and in which even single cells may participate in many such functions (Andersen and Buneo 2003). 25 If perception is supposed to work as a process that is continuous with action, the Marrian casual chain is inverted. Instead of "data" arriving at the eye, being processed, being further processed and finally resulting in a "representation" of the scene in front of us, the agent explores the world in any conceivable direction, casting his questions and producing data in relation to what he needs for action, until it encounters resistance. This is why imagination is the real engine for online perception. Since imagination is a faculty that allows us to move our consciousness from a proximal to a distal place, which can be in the past (memory), future (prospects) or in an invented reality, this "looking for the future" in online perception, trying to anticipate the next thread of the world, is grounded exactly on imagination.
This places Virtual Reality and Augmented Reality (VAR) in radical continuity with online perception. Indeed, if, in phenomenological terms, hallucination is a "perception in the absence of the external object", and if the objects that appear in virtual reality are artificial simulacra of presence that we perceive without them being anchored in 25 A. Clark, Surfing Uncertainty: 178. our experience of the physical world, we shall understand immersive virtual experience as a form of voluntary and deliberate hallucination. But this does not imply any kind of derealization: it simply implies the substitution of the control of the world with the control of a technology. This is why VAR promises important scientific applications, which, in a few years, will radically change many of our laboratories of psychology, neuroscience and cognitive sciences. Indeed, on the one hand, we want a world in which reality is not so "real" as to modify the experimental results and influence them, and VAR provides us with only a virtual reality, which can be controlled in detail and put into brackets at will. On the other hand, we also want a world that is not so unreal as to be indistinguishable from the normal conditions of a laboratory, which at full capacity lives by cutting off every variable that cannot be controlled and ends up producing data that have the purpose of explaining our experience in a condition that we know to be a radical impoverishment of this very same experience. VARs allows us to create a very strong effect of presence within a world-environment, 26 capable of simulating a reality that remains only virtual and can therefore be controlled in its different parameters. At the same time, VAR allows us to increase the "gradient of reality" inside a laboratory, integrating those experiences and variables that a laboratory usually tends to cut away, in the name of the robustness of its measurements. Thus, the conditions of the laboratory are "augmented" with a reconstructed and simulated reality, which we can see and experience only thanks to the prostheses of VAR. In this respect, VAR is a prosthetic technology that is in its own way unprecedented, capable of generating a controllable world without losing at the same time the phenomenological richness of the world.
Precisely because of its ability to make the world present through perception, without the world being in any way unamendable -since it can be controlled -it is completely normal that people who set up environments in VAR exploit this possibility, preferring online perception to other expressive possibilities and using online perception in order to express dreams, hallucinations or memories. However, this choice does not in any way reside in a technological, semiotic or enunciative limit of VAR and its language. On the contrary, it is only a "stylistic" choice, which can be suspended. As we will see, many of these suspensions, which result in the use of old cinematographic techniques within Virtual Reality, can be seen for example in an extraordinary game for Play Station VR such as Resident Evil 7: Biohazard.
Elsewehere, 27 I tried to show how, from a semiotic point of view, 28 VAR works by holding together a formal apparatus of the prosthetic type of enunciation, typical of audiovisuals, and a formal apparatus of enunciation of the simulacral type, which is instead typical of other semiotic systems, such as verbal language. 29 If this is true, as I believe it is, Virtual Reality and Augmented Reality are not incapable of effectively representing the modifications of perception that cinema has always expressed through its representative strategies, such as dissolve, shift from color to black-and-white, flou or blur, used to indicate, in a point-of-view shot, the transition from actual perception to memory, dream, daydream, fantasy, or hallucination. VAR can use all these techniques without any problems at all, since it shares with cinema and audiovisual languages the very same formal apparatus of enunciation. We can have a dissolve or a shift from color to black-and-white also in VAR. If this is not done, or it is not done so often, it is only because of a choice from the "authors" or the VAR, who clearly opt for a "pan-perceptive way" for stylistic or "aesthetical" reasons.
But this does not mean that VAR cannot express non-perceptive conditions such as memory, dream, daydream, fantasy, or hallucination. On the contrary, it does that through perception, perfectly expressing the transition from actual perception to memory, dream, daydream, fantasy, or hallucination in a "pan-perceptive" way. Indeed, Virtual Reality expresses the transition from the actual perception through the actual perception. And sometimes it incorporates the old audiovisual techniques, embodying them inside this transition from perception through perception.
A conclusion in the form of a case study.

Resident Evil 7: Biohazard
As a case study, I will work here on Resident Evil 7: Biohazard, a survival horror game for PS4 developed by Capcom. The game, which can be played through Play Station VR, is full of shifts from actual perception to memories, dreams, and hallucinations. All these shifts are expressed through pure perception, while incorporating from time to time some techniques originating from cinema or audiovisual languages. Resident Evil 7: Biohazard is set somewhere in the southern United States, after the murder of a three-man TV crew by the infected Baker family members Jack and Mia. Ethan Winters is the protagonist, who is searching for his missing wife, Mia, which leads him to a derelict plantation mansion, home of the Baker family. At the very beginning of the story, the player, in the role of Ethan, finds a videotape containing a short video shot by the television crew and, in an adjacent room, he also finds a VCR and a TV where he can see it. Notice how the videotape is a "technological quotation": it is through an old audiovisual that we come to know what happened in the past, 30 now that we are in VR. The player has some expectations at this point: 31 we know that a TV crew has been there before us, because we have previously explored their abandoned van. However, the narration of this flashback, which also gives the player important information on the topography of the house, takes place by giving the player control of one or three crew members, not surprisingly the cameraman, who must film what happened previously and return it to the eyes of the player, who is controlling him through his joypad. Here the flashback and the memory, in a form that clearly identifies that they are flashbacks and memories, are obtained through the perception of a "machinic" eye that was there, watching for us. And who now identifies with us and our avatar. 32 The very same thing happens when the player controls Mia Winters, in the central part of the plot. Mia hallucinates (we will understand later that she has been infected and her mind is controlled by Eveline) and sees a 10-year-old girl who tells her to watch another videotape, "so they can be a family". The videotape introduces 30 According to Kirkland, in the Resident Evil saga "old media technologies contribute a sense of the real perceived as lacking in digital media, yet central to a generically-significant impression of embodiment". See E. Kirkland, "Resident Evil's typewriter: horror videogames and their media", Games and Culture 4, no. 2 (2009). 31 On the way the Resident Evil saga handles expectations of its players, see C. Reed, "Resident Evil's rhetoric: the communication of corruption in survival horror video games", Games and Culture 11, no. 6 (2016). 32 https://www.youtube.com/watch?v=SBsOqYpx2ng&t=1550s a memory through which we learn that Mia was a scientist, taking care of a bioweapon under the form of a little girl, but something went wrong. Once again, it is through actual perception that memories and hallucinations are performed. It is through perception that Virtual Reality expresses the transition from actual perception. Controlling Mia Winters, playing through her and perceiving through her eyes and ears, we come to know the truth about the story that we are playing as Ethan Winters, to the point that one of our aims is to send Ethan (ourselves) the very same message we receive when Resident Evil 7: Biohazard starts. 33 However, the main moments connected to the relationship between perception, memories and hallucination in Virtual Reality are still to come in Resident Evil 7. At a certain point of the plot, Ethan meets Jack Baker's daughter, Zoe, who also wants to escape Baker's house. Zoe tells Ethan that his wife Mia is still alive, even if Ethan (us) has already killed her, since Mia has been given the very same infection the Bakers have, and this infection gives her body powerful regenerative abilities and extreme mutations. Ethan is told that she and Mia would need to have their infections cured by a special serum first, before leaving Baker's house. Ethan heads out to an old house to find the ingredients for the serum, where he is forced to battle and kill Marguerite Baker. Once he retrieves the ingredient, Ethan begins to have strange visions of a young girl. From this point on, hallucinations, memories and perceptions coexist and alternate in all the Resident Evil 7 gameplay and VAR has no problems at expressing their development throughout the whole story at all, using the very same techniques that audiovisual languages used to employ.
For instance, in the final boss fight, Ethan perceives the world through a grayish film and sees Eveline 33 See a complete walkthrough of this part of the game here: https://www.youtube.com/ watch?v=Gh3CkPI0UpA in her 10-year old girl form, that we know being nothing else but a hallucination, since we are told thanks to the Nexbas document found in the salt mines that almost immediately after the infection, the subject begins to see images of Eveline (though she is not in fact there) and even hear her voice (which is inaudible to anyone else). Auditions with infected subjects throughout the stages of infection reveal that at first, the phantom Eveline appears to be a normal young girl, sometimes desiring companionship or assistance.
However, after being able to approach her and inject the toxin that we have previously synthetized in the neck, we see an explosion of light flood the screen and then dissolve, indicating the transition from actual perception, that is hallucination, to real online perception. Indeed, when dissolve and blur fade, the girl we are holding in our arms reveals herself to be an old monstrous lady whom we have already met (Baker's "grandmother" in the wheelchair), that melts into the ground in a colorful scenario (not greyish as before), telling us "why does everybody hate me? I just wanted a family". 34 In her actual form she attacks the player, giving birth to the final boss fight.
Memories make no exception and are expressed perfectly fine (like hallucinations are) in Virtual Reality. For instance, immediately before, the player is told the true story through the reliving of a painful memory by Mia while he is exploring the house. While we are perceiving the actual scene in the house and after we have found a doll on the ground close to a wheelchair (foreshadowing Eveline's true identity), we hear a buzz sound that indicates a clear discontinuity and Mia shows up in our perceptive field without interacting with us nor seeing that we are there.
What we see is a memory by Mia, while she was taking care of Eveline, the E-Series Biohazard weapon (E-001) built by the company she was working for (a common appearance was selected for the bioweapons; that of a roughly 10-yearold girl, to ensure ease blending in with urban population). It is through these memories perceived through Virtual Reality that we come to know that Eveline infected Mia because she wanted Mia to be her mother. Immediately later, we are inside that memory, we can interact with it and we are part of its compound: indeed, Eveline asks us to be her father ("and if he does not want to be my father, he can die") and we ask ourselves "why am I seeing this?". 35 As we see, VAR has no problems at all at in using all the classic audiovisual techniques in order to express the transition from online perception to memory, dream or hallucination. It simply prefers doing that the majority of the time through its pan-perceptive model. However, not only can VAR utilise the old audiovisual techniques, but it looks like that also the old audiovisual languages used to express hallucinations, memories and dreams through perception: they simply did not give the observer the sensation of "being there". How could cinema express memory, dream or hallucination through dissolve, blur or flou if not through perception? Since both cinema and VAR share the very same prosthetic structure of their formal apparatus of enunciation, they both use a prosthesis (a screen, a mounted display etc.) in order to make us see things that we could not have seen without the text. 36 In this way, from a semiotic point of view, VAR confirms its mixed nature, keeping together a simulacral and a prosthetic structure of its language.