Evaluating
Multi-party Multi-modal Systems
October 2000
Laurie E. Damianos, The MITRE Corporation
Jill Drury, The MITRE Corporation
Lynette Hirschman, The MITRE Corporation
Jeff Kurtz, The MITRE Corporation
Beatrice Oshika, The MITRE Corporation
Tari Fanderclai
ABSTRACT
The MITRE Corporation's Evaluation Working Group has
developed a methodology for evaluating multi-modal groupware
systems and capturing data on human-human interactions. The
methodology consists of a framework for describing collaborative
systems, a scenario-based evaluation approach, and evaluation
metrics for the various components of collaborative systems. We
designed and ran two sets of experiments to validate the
methodology by evaluating collaborative systems. In one
experiment, we compared two configurations of a multi-modal
collaborative application using a map navigation scenario
requiring information sharing and decision making. In the second
experiment, we applied the evaluation methodology to a loosely
integrated set of collaborative tools, again using a
scenario-based approach. In both experiments, multi-modal,
multi-user data were collected, visualized, annotated, and
analyzed.
Additional Search Keywords
Collaborative systems, evaluation methodology, multi-modal, human-human
interactions, human-computer interactions, data collection, Map Navigation
Experiment, scenario-based evaluation.
|
