Input for this challenge is a video capture of a free-hand drawing (see example video) in XGA.The metrics for evaluation are "defined" as follows:
Output for this challenge should be a set of “adapted” videos , with the same content in different (smaller) resolutions – for instance, VGA and QVGA. The adapted videos would ideally be regarded by users as perceptually optimal, meaning they hold the same content as the original.
The following criteria could be used, as well as other evaluation metrics that you may devise:However, there are many possibilities to adapt content and evaluate the result which heavily depends on the user's context. Some people may think that the description of the input/output as well as the evaluation criteria is defined too vague and I tend to agree. Let me explain:
- Subjective comparison between the perceptual quality of the original and the “adapted” content.
- Subjective comparison between the perceptual quality of a scaled-down version of the original (using a 5-tap poly-phase filter) and the “adapted” content.
- Real-time Performance.
- The input is given and for the output it is requested to produce "a set of adapted videos" that is "regarded by users as perceptually optimal". However, it's not clear to which context the input shall be adapted. The text says "different (smaller) resolutions" as an example but I can imaging that users will prefer the original video and regard it as perceptually optimal compared to anything else. Thus, in my view it is necessary to specify the context to which the video shall be adapted. The context may include a lot of things such as terminal device, decoding capabilities, network conditions, user location (stationary, mobile), etc., etc.
- Subjective quality assessment is not an easy task and there are many possibilities and many approaches. Form the description above it is not clear how the subjective quality evaluation will be performed. In particular, I wonder whether "real" subjective tests as suggested by the Video Quality Experts Group (VQEG) will be adopted (e.g., DSIS - Double Stimulus Impairment Scale or ACR - Absolute Category Scale to just name two). In my view and in order to provide a fair evaluation it is absolutely necessary to define the exact procedure on how the subjective evaluation of the submissions will be performed. One possibility, of course, is the adoption of a standardized approach and probably DSIS is the right candidate.