The Handbook of Multimodal-Multisensor Interfaces, Volume 1. Sharon Oviatt

Chapter 4 A Background Perspective on Touch as a Multimodal (and Multisensor) Construct

       Ken Hinckley

       4.1 Introduction

       4.2 The Duality of Sensors and Modalities

       4.3 A Model of Foreground and Background Interaction

       4.4 Seven Views of Touch Interaction

       4.5 Summary and Discussion

       Focus Questions

       References

       Chapter 5 Understanding and Supporting Modality Choices

       Anthony Jameson, Per Ola Kristensson

       5.1 Introduction

       5.2 Synthesis of Research on Modality Choices

       5.3 Brief Introduction to the ASPECT and ARCADE Models

       5.4 Consequence-Based Choice

       5.5 Trial-and-Error-Based Choice

       5.6 Policy-Based Choice

       5.7 Experience-Based Choice

       5.8 Other Choice Patterns

       5.9 Recapitulation and Ideas for Future Research

       Focus Questions

       References

       Chapter 6 Using Cognitive Models to Understand Multimodal Processes: The Case for Speech and Gesture Production

       Stefan Kopp, Kirsten Bergmann

       6.1 Introduction

       6.2 Multimodal Communication with Speech and Gesture

       6.3 Models of Speech and Gesture Production

       6.4 A Computational Cognitive Model of Speech and Gesture Production

       6.5 Simulation-based Testing

       6.6 Summary

       Focus Questions

       References

       Chapter 7 Multimodal Feedback in HCI: Haptics, Non-Speech Audio, and Their Applications

       Euan Freeman, Graham Wilson, Dong-Bach Vo, Alex Ng, Ioannis Politis, Stephen Brewster

       7.1 Overview of Non-Visual Feedback Modalities

       7.2 Applications of Multimodal Feedback: Accessibility and Mobility

       7.3 Conclusions and Future Directions

       Focus Questions

       References

       Chapter 8 Multimodal Technologies for Seniors: Challenges and Opportunities

       Cosmin Munteanu, Albert Ali Salah

       8.1 Introduction

       8.2 Senior Users and Challenges

       8.3 Specific Application Areas

       8.4 Available Multimodal-Multisensor Technologies

       8.5 Multimodal Interaction for Older Adults—Usability, Design, and Adoption Challenges

       8.6 Conclusions

       Focus Questions

       References

       PART III COMMON MODALITY COMBINATIONS

       Chapter 9 Gaze-Informed Multimodal Interaction

       Pernilla Qvarfordt

       9.1 Introduction

       9.2 Eye Movements and Eye Tracking Data Analysis

       9.3 Eye Movements in Relation to Other Modalities

       9.4 Gaze in Multimodal Interaction and Systems

       9.5 Conclusion and Outlook

       Focus Questions

       References

       Chapter 10 Multimodal Speech and Pen Interfaces

       Philip R. Cohen, Sharon Oviatt

       10.1 Introduction

       10.2 Empirical Research on Multimodal Speech and Pen Interaction

       10.3 Design Prototyping and Data Collection

       10.4 Flow of Signal and Information Processing

       10.5 Distributed Architectural Components

       10.6 Multimodal Fusion and Semantic Integration Architectures

       10.7 Multimodal Speech and Pen Systems

       10.8 Conclusion and Future Directions

       Focus Questions

       References

       Chapter 11 Multimodal Gesture Recognition

       Athanasios Katsamanis, Vassilis Pitsikalis, Stavros Theodorakis, Petros Maragos

       11.1 Introduction

       11.2 Multimodal Communication and Gestures

       11.3 Recognizing Speech and Gestures

       11.4 A System in Detail

       11.5 Conclusions and Outlook

       Focus Questions

       References

       Chapter 12 Audio and Visual Modality Combination in Speech Processing Applications

       Gerasimos Potamianos, Etienne Marcheret, Youssef Mroueh, Vaibhava Goel, Alexandros Koumbaroulis, Argyrios Vartholomaios, Spyridon Thermos

       12.1 Introduction

       12.2 Bimodality in Perception and Production of Human Speech

       12.3 AVASR Applications and Resources

       12.4 The Visual Front-End

       12.5 Audio-Visual