Despite that the need for a common evaluation framework for multimedia and multimodal documents for various use cases, including non-topical use, is widely acknowledged, such a framework is still not in place. Retrieval system evaluation results are not regularly validated in laboratory or field studies; the infrastructure for generalizing results over tasks, users and collections is still missing. This chapter presents a use case-based framework for experimental design in the field of interactive information access. The framework is highlighted by examples that sketch out how the framework can be productively used in experimental design and reporting with a minimal threshold for adoption.
In "Professional Search in the Modern World", Paltoglou, Georgios, Loizides, Fernando, Hansen, Preben (Eds.). Springer. 2014.
