Evaluation of visual video summaries: user-supplied constructs and descriptions

Аннотация

Evaluation of video summarization approaches requires more information on the user-perceived qualities of different types of summaries. Also, evaluation measures need to be further developed in a user-led manner. This article reports on a user-centered evaluation of visual video summaries. Four types of summaries (fastforward, user-controlled fastforward, scene clips, and storyboard) were evaluated with a set of existing performance and satisfaction measures. A repertory grid elicitation was conducted with our participants gathering evaluation constructs related to both video summary content and controls. Results showed a lack of correlation between performance and satisfaction measures. User-supplied evaluation constructs were shown to span both the performance and satisfaction dimensions of the video summary evaluation space. Most constructs achieved moderate to good inter-rater agreement in a consequent survey. Free descriptions of videos and respective summaries showed that while users are able to interpret object- and event-related information from short summaries, thematic inference lacked, leading to worse descriptions than for the full videos.