Estimating Spoken Dialog System Quality with User Models

Front Cover
Springer Science & Business Media, Aug 4, 2012 - Technology & Engineering - 130 pages

Spoken dialog systems have the potential to offer highly intuitive user interfaces, as they allow systems to be controlled using natural language. However, the complexity inherent in natural language dialogs means that careful testing of the system must be carried out from the very beginning of the design process. This book examines how user models can be used to support such early evaluations in two ways: by running simulations of dialogs, and by estimating the quality judgments of users. First, a design environment supporting the creation of dialog flows, the simulation of dialogs, and the analysis of the simulated data is proposed. How the quality of user simulations may be quantified with respect to their suitability for both formative and summative evaluation is then discussed. The remainder of the book is dedicated to the problem of predicting quality judgments of users based on interaction data. New modeling approaches are presented, which process the dialogs as sequences, and which allow knowledge about the judgment behavior of users to be incorporated into predictions. All proposed methods are validated with example evaluation studies.

 

What people are saying - Write a review

We haven't found any reviews in the usual places.

Contents

1 Introduction
1
2 MeMo Usability Workbench
9
3 Evaluation of the MeMo User SimulationUse Case Inspire Smart Home System
21
4 Detection of Usability Problems Using an AdHoc User Simulation
41
5 Prediction of User Judgments
61
6 Application of Prediction Models in a Realistic Usage Scenario
86
7 Conclusions and Future Work
101
Appendix A List of Usability Problems Foundin the BoRIS Database Chapter 4
109
Appendix B List of Usability Issues Covered in the DialogScripts of the Experiment in Chapter 5
117
Appendix C Interaction Parameters Used for JudgmentPrediction Models
119
References
122
Copyright

Other editions - View all

Common terms and phrases

Bibliographic information