This paper is a report from a workshop on Evaluation of Information Systems in Commercial Settings, inspired by the industrial day at SIGIR 2016. Small and medium size enterprises often lack the resources needed to develop proper evaluation infrastructures, but also to follow the research development in the field of evaluation. Similarly,academics lag behind in (a) understanding real practical issues raised when it comes to the evaluation of real systems – e.g. even depth-k pooling is often infeasible when an SME has a single ranking algorithm developed, and (b) sensing the breadth of applications and tasks on which systems require evaluation and the challenges of them. Large enterprises with the necessary resources and the data sets and flows to work with are hesitant to make their tests public, for both commercial and legal reasons.This workshop brought together representatives from technology companies, large and small, media houses, industrial consultants and academic research in information access for a discussion on practical issues and solutions to these issues.
Kanoulas, Evangelos and Karlgren, Jussi
