The Handbook of Multimodal-Multisensor Interfaces, Volume 1. Sharon Oviatt

3.3.3, we discuss vocabulary development for more informative discrete communicative elements.

      Haptics can provide spatial marking. Highly relevant to guiding interactions, the addition of spatially informative sensations to touched surfaces screens is becoming possible through several emerging technologies, whether the surface is co-located with a graphic display (touchscreen) or mapped to it (as with a trackpad accessed through fingertip or stylus, or a haptically enabled mouse). Most basically, a vibrotactile actuator can jolt an entire touched surface when a finger crosses a boundary; our brain attributes the “bump” to the touched point rather than the entire screen. Variable friction can render textures that mark regions of a surface [Levesque et al. 2011], but because the whole surface has the same coefficient of friction at a given instant, state changes are salient but not felt as edges under the finger.

      Marking traceable edges requires the capacity to independently display different haptic states to skin that touches a surface at different points, through multiple fingers, different parts of the hand, or adjacent points on one finger. Present efforts have not yet simultaneously achieved high resolution, high refresh rate, and optical transparency, nor low cost. Recent advances in shape display use technologies ranging from shape memory polymers (http://www.blindpad.eu) or mechanical structures [Jang et al. 2016]) are promising.

       Improving Specific Performance and General Quality

      Quantifiable performance improvements are always easier to value than more qualitative ones, whether they benefit safety, efficiency or some other monetizable parameter. As for many interface innovations, however, performance improvement often manifests as a fluidity or reduction in effort that lessens fatigue over a period of time where the user is doing many different things, and can be difficult to isolate in causality or to measure precisely.

      Exceptions may be when haptic feedback is applied to error suppression in situations where users are known to be particularly error-prone. For example, drivers often have difficulty with verbal left/right direction commands, whereas spatially delivered haptic cues are likely to improve performance without diverting visual or auditory attention from a driving task. Haptic feedback can also increase dexterity in surgical simulations, teloperated environments, and facilitate simple pointing tasks on a GUI or touchscreens [Poupyrev and Maruyama 2003, Levesque et al. 2011]. These are all changes that can be measured, at least in controlled laboratory settings, with some transfer to real environments inferred.

      More broadly, haptic feedback is often found to contribute to the user’s sense of immersion through addition of a sensory modality, for gaming environments, virtual reality, and teleoperated or minimally invasive surgery. Immersion is generally accepted as beneficial, enabling secondary performance improvements by dint of focus and clarity, or greater engagement and enjoyment and thus product success.

       Affect or Emotion Display

      Haptic elements, both input and output, can be used for affective coloring of an interactive experience, as an overt user expression (overtly, as in “conviction widgets” [Chu et al. 2009]), or deliberate conveyance of emotion to another person [Smith and MacLean 2007]. Incoming to the user, attention to affective haptic design can influence how signals are interpreted [Swindells et al. 2007], make them more understandable and memorable [Klatzky and Peck 2012, Seifi and MacLean 2013], and contribute to a sense of delight in the interaction [Levesque et al. 2011].

      Sometimes the primary purpose of a person-to-person communication is affective in nature. Haptics can contribute to such enrichment. Therapeutically, touch-centric mediums such as haptic social robots can act both socially and physiologically on a human to change emotional state [Inoue et al. 2012, Sefidgar et al. 2015].

3.3 Physical Design Space of Haptic Media

      Designers of effective haptic sensations within a multimodal interaction must understand what properties of haptic signals are manipulable, how they are perceived, and schemas for encoding meaning to them.

       3.3.1 The Sensation

      Delivered through a heterogeneous set of technologies, haptic sensations target different human mechanorceptors, and further vary in energetic state and expressive properties.

      A sensation can be kinesthetic or tactile. The most common type of the proprioceptively targeted haptic display is force feedback, in which the device exerts a force on the user’s body (often a hand) while the user moves the device through space (e.g., handshaking with a robot or teleoperated surgery). Vibrotactile actuators, alone or in arrays, produce the most well known of tactile sensations. Others include programmable friction [Winfield et al. 2007, Levesque et al. 2011], ultrasonic sensations [Carter et al. 2013], and thermal feedback [Ho and Jones 2007].

      A sensation’s salience can vary, from intrusive to ambient. Haptic sensations can be designed to instantly capture the user’s attention (e.g., vibrotactile (VT) notifications) or be present at their attentional background, and referred by the users when needed (Section 3.2.3, [MacLean 2009]). The latter presents information in an ambient manner while the former can interrupt the user’s current state or action to convey the information. Interesting designs are possible by moving between these two ends. For example, a posture correcting chair provides awareness of the user’s posture with ambient pressure sensations at the back of their seat which can gradually move into the user’s attentional foreground when necessary [Zheng and Morrell 2012].

      A designer can engineer the properties of an individual stimulus to create different sensations. In addition to signal amplitude, haptic signals commonly use temporal and/or spatial parameters. For example, vibrotactile signals have several temporal parameters including frequency, rhythm, and pulse envelope (specified by attack, decay, sustain, and release parameters [MacLean 2008b, Ternes and MacLean 2008, Choi and Kuchenbecker 2013] as well as spatial parameters such as location (x,y) and direction when several actuators are combined over a surface (e.g., a haptic seatpad) [Schneider and MacLean 2014, Schneider et al. 2015b]. Variable friction and force feedback devices can provide different signals over space and time depending on the user’s interactions [Levesque et al. 2011, Levesque et al. 2012].

       3.3.2 The Sensation-Human Connection

      In devising effective interactions, designers must consider a device’s connection to the user’s body and the range of haptic sensations perceptible in a given context.

       Physical Connection

      A haptic device’s connection to its user’s body varies with technology and use case, and impacts perception.

      Contact mode can vary. Location, surface area, and tightness are part of the body-device connection; prototypes for wrist, belts, jackets, shoe insoles, or handheld devices vary these parameters. The contact can be persistent (e.g., a wristband) or occasional and on-demand (e.g., a haptic keypad on an ATM or a haptic door knob) [Karuei et al. 2011, MacLean and Roderick 1999].

      Bodily distance can vary. Haptic signals can be felt through an internal mechanism (such as vibrating tattoos [Radivojevic et al. 2014]), an external but contacting device (smartwatches, game controllers), or an external, noncontacting device (ultrahaptic devices [Carter et al. 2013]). The current norm is to feel the sensations through an external and contacting device.

      On the human side, contact can be active or passive. Human-active touching is generally done for a reason. Active or passiveness of user touch is influenced (afforded) by device and interaction design. For example, sensations rendered by today’s variable friction technology can only be felt with active (sliding) finger movement. Conversely, users commonly receive vibrotactile sensations passively as event-based notifications; finger movement yields no additional information.

       Effective Size of the Sensation Space (Signal Set Size)

      The number of sensations that humans can perceptually distinguish is a function of hardware, bodily connection, perceptual capability, and context of use.

      Hardware