The Handbook of Multimodal-Multisensor Interfaces, Volume 1. Sharon Oviatt

[Cisek 1999]. An important point, then, is that this grounding not only affects our learning and understanding of existing objects by providing expected action-sensory contingencies, it also guides the active production of objects.

      One type of object production that has been shown to facilitate symbol learning, is writing by hand. Because the production of symbols has the defined purpose of communicating through the written word, humans have created writing tools that allow a greater degree of accuracy in the written form and that allow the transfer of permanent marks through ink or similar media. The manipulation of writing implements introduces yet another source of sensory information: haptic and kinesthetic cues that can augment and change motor production and visual perception. Nonetheless, visual-motor guidance of the writing device is still required, and this coupling of multimodal-multisensory information facilitates learning in much the same manner as other forms of active interactions. Handwriting symbols has been repeatedly correlated with increased competencies in symbol understanding (e.g., [Berninger et al. 1998, Richards et al. 2011, Li and James 2016). Although experimental studies are somewhat limited in adults, there is nonetheless a growing amount of research implicating the importance of self-generated action through writing on symbol understanding. In a series of pivotal and still highly influential studies, Freyd and colleagues showed that symbol recognition is significantly affected by our experiences with writing. For example, we are better able to identify static letters of the alphabet whose stroke directions and trajectories conform to how an individual creates those forms through handwriting, compared to similar letters that do not conform to the observer’s own stroke directions and trajectories [Freyd 1983, Babcock and Freyd 1988, Orliaguet et al. 1997].

      The interplay between the motor production of letters and the visual perception of letters has been demonstrated in individuals who have letter perception deficits. For example, individuals with pure alexia have difficulty recognizing letters. However, in some cases, their letter recognition can be facilitated by tracing the letter in the air with their finger or making hand movements that mimic the letter shape [Bartolomeo et al. 2002, Seki et al. 1995]. The interpretation of these case studies rests upon the fact that pure alexia results from damage to a specific brain region. The fact that damage to one location can result in a deficit in letter recognition that can be recovered to some extent by active interaction with the symbol’s form suggests that both visual and motor brain systems are involved in the perception of a symbol. In these cases, the patients’ actions facilitated their visual perceptions, evidence that the neural mechanisms subserving letter perception span both visual and motor brain regions as a result of prior multimodal experience.

      Moreover, the production of symbols may rely upon the same neural mechanisms as the recognition of symbols. When adults are asked to simultaneously identify letters and shapes presented in varying degrees of visual noise, their thresholds for detection were increased (worsened) if they were writing a perceptually similar letter or shape during symbol identification, compared to when writing a perceptually dissimilar shape [James and Gauthier 2006]. The interpretation of this study rests upon theories of neural interference, which suggest that if one behavioral task is significantly affected by another concurrent task, then the two share common mechanisms. The fact that an interference effect was observed, rather than a facilitation effect, suggests that the production of a form and the perception of the form overlapped in terms of their underlying mechanisms. Again, we are provided with evidence that active interactions mold neural systems in such a way that the system will seek expected action-sensory contingencies, either by producing them through manual rotation or, in this case, producing them through the manipulation of writing implements.

2.4 Behavioral Results in Children

      By exploring their surroundings, infants and children discover object properties and uncover possibilities for actions afforded by many objects. Importantly, they also learn about the functional changes that those novel possibilities for action imply for the action capabilities of the limbs [Lockman 2000]. The profound effects that multimodal-multisensory learning has on cognitive development was originally outlined in an extensive psychological theory proposed by Jean Piaget [1952, 1954]. Piaget believed that because information gained by watching others in early life was extremely limited, most of what children learn prior to two years of age is gained through multimodal experience. Importantly, Piaget argued that manual actions served to link information from multiple sensory systems together. For example, vision, somatosensation, and audition could all be linked through action on objects, because an action would result in the simultaneous production of visual, somatosensory, and auditory sensory input.

      Early in life, each sensory modality is still relatively immature. However, some sensory systems are more mature than others. For example, the visual system of infants is very poor (e.g., Banks and Salapatek 1983), but their sense of touch is remarkably mature [Sann and Streri 2008]. This observation highlights one of the major benefits of active interactions in infancy: each action is accompanied by, at least, tactile stimulation. As infants reach and hold objects of interest, they often bring them into sight, and in doing so, produce simultaneous visual, motor, and tactile information. Importantly, the somatosensory modality provides the infant with rich information about the object (e.g., texture, shape, weight, temperature, size) that is simultaneously paired with relatively immature visual percepts (e.g., color, global shape). The tactile information is only gained through actions, and due to the somatosensory system’s relative maturity, has the ability to aid in the development of the visual percept. Therefore, in infancy, although visual and motor systems are immature, actions still provide a wealth of information to the developing brain, because they are inherently multimodal.

      In what follows, we will discuss a sample of empirical work that underlines the importance of multimodal-multisensory learning during development.

       2.4.1 Surface Perception

      Visual perception is affected by early locomotion abilities in very young children. Locomotion is an active interaction that allows children to explore their surroundings and that presents a variety of new multimodal experiences. For infants, the ability to move themselves represents unprecedented opportunity for self-generated actions on a variety of objects that were previously unreachable, but first, on surfaces. Visual competencies can develop from merely experiencing different surfaces. One well-known demonstration of visual development as a result of multimodal experience is the visual cliff paradigm [Gibson and Walk 1960]. These experiments require an apparatus constructed of two surface levels, the lower surface being a large drop from the higher surface. However, glass covers the lower surface at the same height as the higher surface such that one could locomote from the high surface over the low surface by moving across the glass (Figure 2.3). The point of the apparatus is to provide conflicting visual and somatosensory information to an infant. If one relies on vision, one will perceive a large drop off (danger). If one relies on somatosensation, the feel of the glass would reassure the actor that the surface was safe to cross. Most infants that can crawl will not cross the visual cliff to get to their caregiver—relying on visual cues rather than haptic ones [Gibson and Walk 1960, Bertenthal et al. 1984]. These original studies documented this reliance on visual depth cues in numerous diurnal species [Gibson and Walk 1960].

      Figure 2.3 Visual cliff apparatus. From The Richard D. Walk papers, courtesy Drs. Nicholas and Dorothy Cummings Center for the History of Psychology, The University of Akron.

      However, visual development is prolonged compared to somatosensory development, which suggests that there should exist a time point in development when infants should not be so affected by visual cues. As a multimodal experience that binds visual and somatosensation, one would expect that experience with locomotion is important for infants to learn that “clear” (i.e., non-existent) surfaces do not afford locomotion. The early studies did not compare crawlers to non-crawlers to test experimentally whether experience with locomotion was necessary for this behavior to develop. Campos et al. [1992] showed, however, that if one lowers a non-crawling infant over the low side of the visual cliff, they will not demonstrate