Spoken Multimodal Human-Computer Dialogue in Mobile Environments

Wolfgang Minker, Dirk Bühler, Laila Dybkjær

Springer Science & Business Media, Feb 18, 2005 - Computers - 406 pages

This book is based on publications from the ISCA Tutorial and Research Workshop on Multi-Modal Dialogue in Mobile Environments held at Kloster Irsee, Germany, in 2002. The workshop covered various aspects of devel- ment and evaluation of spoken multimodal dialogue systems and components with particular emphasis on mobile environments, and discussed the state-- the-art within this area. On the development side the major aspects addressed include speech recognition, dialogue management, multimodal output gene- tion, system architectures, full applications, and user interface issues. On the evaluation side primarily usability evaluation was addressed. A number of high quality papers from the workshop were selected to form the basis of this book. The volume is divided into three major parts which group together the ov- all aspects covered by the workshop. The selected papers have all been - tended, reviewed and improved after the workshop to form the backbone of the book. In addition, we have supplemented each of the three parts by an invited contribution intended to serve as an overview chapter.

Contents

• Multimodal Dialogue Systems: 3
• 2 Varieties of Multimodal Dialogue: 4
• 3 Detecting Intentional User Inputs: 6
• 4 Modes and Modalities: 7
• 6 Domain Reasoning: 8
• 7 Output Planning: 9
• 9 Conclusion: 10
• References: 11
• Speech Recognition Technology in MultimodalUbiquitous Computing Environments: 13
• 2 StateoftheArt Speech Recognition Technology: 14
• 3 Ubiquitous Speech Recognition: 16
• 4 Robust Speech Recognition: 18
• 5 Conversational Systems for Information Access: 21
• 6 Systems for Transcribing Understanding and Summarising Ubiquitous Speech Documents: 24
• 7 Conclusion: 32
• References: 33
• A Robust Multimodal Speech Recognition Method using Optical Flow Analysis: 37
• 1 Introduction: 38
• 2 Optical Flow Analysis: 39
• 3 A Multimodal Speech Recognition System: 40
• 4 Experiments for NoiseAdded Data: 43
• 5 Experiments for RealWorld Data: 48
• 6 Conclusion and Future Work: 49
• References: 52
• Feature Functions for TreeBased Dialogue Course Management: 55
• 2 Basic Dialogue Framework: 56
• 3 Feature Functions: 59
• 4 Computing Dialogue Costs: 63
• 5 Selection of Dialogue StateAction Pairs: 64
• 6 XMLbased Data Structures: 65
• 7 Usability in Mobile Environments: 68
• 8 Results: 69
• 9 Summary and Outlook: 74
• A Reasoning Component for InformationSeeking and Planning Dialogues: 77
• 2 StateoftheArt in Problem Solving Dialogues: 80
• 3 Reasoning Architecture: 81
• 4 Application to Calendar Planning: 85
• 5 Conclusion: 88
• References: 90
• A Model for Multimodal Dialogue System Output Applied to an Animated Talking Head: 93
• 2 Specification: 97
• 3 Interpretation: 103
• 4 Realisation in an Animated Talking Head: 105
• 5 Discussion and Future Work: 109
• References: 111
• System Architecture and Example Implementation: 115
• Overview of System Architecture: 117
• 2 Towards Personal Multimodal Conversational User Interface: 118
• 3 System Architectures for Multimodal Dialogue Systems: 122
• 4 Standardisation of Application Representation: 126
• 5 Conclusion: 129
• References: 130
• XISL A ModalityIndependent MMI Description Language: 133
• 2 XISL Execution System: 134
• 3 Extensible Interaction Scenario Language: 136
• 4 Three Types of FrontEnds and XISL Descriptions: 140
• 5 XISL and Other Languages: 146
• 6 Discussion: 147
• References: 148
• A Path to Multimodal Data Services for Telecommunications: 149
• 2 Application Considerations Technologies and Mobile Terminals: 150
• 3 Projects and Commercial Developments: 154
• 4 Three Multimodal Demonstrators: 156
• 5 Roadmap for Successful Versatile Interfaces in Telecommunications: 161
• 6 Conclusion: 163
• References: 164
• Multimodal Spoken Dialogue with Wireless Devices: 169
• 2 Why Multimodal Wireless?: 171
• 3 Walking Direction Application: 172
• 4 Speech Technology for Multimodal Wireless: 173
• 5 User Interface Issues: 174
• 6 Multimodal Architecture Issues: 179
• 7 Conclusion: 182
• References: 184
• The SmartKom Mobile Car Prototype System for Flexible HumanMachine Communication: 185
• 2 Related Work: 186
• 3 SmartKom Intuitive HumanMachine Interaction: 189
• 4 Scenarios for Mobile Use: 191
• 5 Demonstrator Architecture: 193
• 6 Dialogue Design: 194
• 7 Outlook Towards Flexible Modality Control: 197
• 8 Conclusion: 199
• References: 200
• LARRI A LanguageBased Maintenance and Repair Assistant: 203
• 2 LARRI System Description: 204
• 3 LARRI Hardware and Software Architecture: 208
• 4 Experiments and Results: 213
• 5 Conclusion: 215
• References: 217
• Evaluation and Usability: 219
• Overview of Evaluation and Usability: 221
• 2 StateoftheArt: 223
• 3 Empirical Generalisations: 227
• 4 Frameworks: 234
• and Theory: 236
• 6 Discussion and Outlook: 238
• References: 241
• Evaluating Dialogue Strategies in Multimodal Dialogue Systems: 247
• 2 WizardofOz Experiment: 251
• 3 Overhearer Experiment: 262
• 4 Discussion: 266
• References: 267
• Enhancing the Usability of Multimodal Virtual Codrivers: 269
• 2 The VICO System: 271
• 3 VICO Haptics How and When to Make VICO Listen?: 272
• 4 VICO Graphics When might the Driver Look?: 274
• 5 Who is Driving this Time?: 278
• 6 Modelling the Driver: 280
• 7 Conclusion and Future Work: 284
• References: 285
• Design Implementation and Evaluation of the SENECA Spoken Language Dialogue System: 287
• 1 Introduction: 288
• 2 The SENECA SLDS: 290
• 3 Evaluation of the SENECA SLDS Demonstrator: 301
• 4 Conclusion: 308
• References: 309
• Segmenting Route Descriptions for Mobile Devices: 311
• 2 Structured Information Delivery: 315
• 4 Evaluation: 322
• 5 Conclusion: 326
• References: 327
• Effects of Prolonged Use on the Usability of a Multimodal FormFilling Interface: 329
• 2 The MATIS System: 332
• 3 Methods: 335
• 4 Results and Discussion: 337
• 5 Conclusion: 345
• References: 346
• User Multitasking with Mobile Multimodal Systems: 349
• 1 The Challenge of Multitasking: 350
• 2 Example System: 354
• 4 Analyses of Task Combinations: 359
• 5 Studies with Users: 364
• 6 The Central Issues Revisited: 371
• References: 375
• Speech Convergence with Animated Personas: 379
• 2 Research Goals: 382
• 3 Method: 383
• 4 Results: 387
• 5 Discussion: 391
• 6 Conclusion: 393
• References: 394
• Index: 399

Bibliographic information

• Title: Spoken Multimodal Human-Computer Dialogue in Mobile Environments; Volume 28 of Text, Speech and Language Technology
• Editors: Wolfgang Minker, Dirk Bühler, Laila Dybkjær
• Edition: illustrated
• Publisher: Springer Science & Business Media, 2005
• ISBN: 1402030746, 9781402030741
• Length: 406 pages
• Subjects: Computers / Desktop Applications / Design & Graphics; Computers / Interactive & Multimedia; Computers / Natural Language Processing; Computers / Operating Systems / General; Computers / User Interfaces; Language Arts & Disciplines / Linguistics / General