Text-to-video (T2V) technology has made significant advancements in the last two years and garnered increasing attention from the general community. T2V products such as Gen2 and Pika have attracted many users. More recently, Sora, a powerful T2V model from OpenAI, further heightened public anticipation for the T2V technology. Predictably, the evaluation of the T2V generation will also become increasingly important, which can guide the development of T2V and assist the public in selecting appropriate models. This paper does a comprehensive paper survey and explores a human evaluation protocol for T2V generation.

Comprehensive evaluation metrics for T2V models. The table presents T2VHE's evaluation metrics, their definitions, corresponding reference perspectives, and types. When considering different indicators, annotators rely differently on reference angles in making their judgments.

Regarding absolute scoring could still result in noisy annotations and pose challenges in reaching consensus among annotator, we use the less challenging comparative scoring method. Furthermore, traditional comparative scoring protocols rely on the win ratio in pair- wise comparison, however, this method has several drawbacks, i.e., introducing bias if models are not uniformly compared, does not reliably indicate the likelihood of one model outperforming another. To overcome these issues, we adopt the Rao and Kupper model, a probabilistic approach that allows for more efficient handling of the results of pairwise comparisons using less data than full comparisons. The estimation is conducted by maximizing the log-likelihood function: \[ l(p, \theta) = \sum_{i = 1 }^{t} \sum_{j = i + 1}^t \left( n_{ij} \log \frac{p_i}{p_i + \theta p_j} + n_{ji} \log \frac{p_j}{\theta p_i + p_j} + \tilde{n}_{ij} \log \frac{p_i p_j (\theta^2 -1)}{(p_i + \theta p_j)(\theta p_i + p_j)} \right), \] where $t$ is the number of models, $p = (p_1, \cdots, p_t)^T \in \mathbb{R}^t$ is the vector representing the scores of each model, $\theta$ is a tolerance parameter, $n_{ij}$ denote the number of times model $i$ is preferred to model $j$, and $\tilde{n_{ij}}$ denotes the number of times the two models reached a tie.

To ensure the quality of the annotations, for each metric, we design a set of annotator training, including their detailed definitions, corresponding reference perspectives, and for each reference perspective, we further provide the corresponding analysis process and examples to help the annotators to make decisions. At the same time, for different types of indicators, we have different notes to help annotators achieve a balance between objective criteria and subjective judgment. Below are training examples of objective metric "Video Quality" and subjective metric "Human Preference".

Example of training for "Video Quality"

Example of training for "Human Preference"

Evaluating multiple video models via traditional pairwise evaluation protocols becomes increasingly resource-intensive as the number of models expands. To efficiently obtain stable model rankings, we propose a pluggable dynamic evaluation module, the specific process is as follows:

Key Principles

• Video Quality Proximity Rule
  • Leverage the scoring results of automated metrics to allow human annotators to prioritize the annotation of samples that are difficult to distinguish with automated metrics.
  • Ensures that initially evaluated video pairs have similar quality levels.
• Model strength Rule
  • Determines the evaluation priority of subsequent video pairs based on model strength scores.
  • Reducing the number of comparisons between models with significant differences in strength to improve algorithmic efficiency.

Evaluation Process

1. Initial Model Strength Assignment
   • Each model is assigned an initial neutral strength value, indicating no prior bias.
2. Automatic Metrics Computation
   • For each video, the following metrics were evaluated by a pre-trained scorer: subject consistency, temporal flickering, motion smoothness, dynamic fegree, aesthetic quality, imaging quality, and overall consistency.
   • Scores are normalized and summed to produce the feature score for each video.
3. Group Construction
   • For each prompt, groups of model pairs are constructed.
   • The absolute value of the difference between the scores of two videos in a pair is input into an exponential decay model.
   • The output value is the score of each video pair, and the sum of these scores forms the total score for the group.
4. Sorting and Grouping
   • Groups were ranked according to their total scores, with higher scores at the top indicating less variation within the group.
   • This preprocessing step does not increase the cost of our evaluation protocol.
5. Human Evaluation Phase
   • An initial set of video pairs is evaluated by humans, and model strengths are updated using the Rao and Kupper model.
   • Comparisons are then split into batches. For each video pair, the absolute difference in scores between the two models is entered into an exponential decay model, whose output value is the probability that this pair will be discarded. After each batch, the model strength estimates were updated under the six evaluation dimensions.
   • The evaluation ends when the model rankings stabilize, meaning the rankings of models under each dimension remain unchanged for several consecutive batches.

Our protocol, augmented by the dynamic evaluation module, cuts annotation costs to about 53% of the original expense while achieving comparable outcomes. Subsequent experiments further confirmed the module's effectiveness and reliability.