...
- Exact Agreement: Set a threshold percentage for raters to meet by matching the true score for all traits on qualification set responses exactly.
- Trait-Specific Exact Agreement: Set a threshold percentage for raters to meet by matching the true score on a specific trait(s) exactly.
- Exact/Adjacent Agreement: The threshold percentage set will be met by achieving an exact or adjacent match to the overall true score on each response.
- Trait-Specific Exact/Adjacent Agreement: The threshold percentage set will be met by achieving an exact or adjacent match to the specific trait's true score on each response.
- Max Discrepancies: The total number of non-adjacent overall scores given on each response by the rater that will result in failing qualification.
- Max Discrepancies Per Trait: The total number of non-adjacent specific trait scores given on each response by the rater that will result in failing qualification.
- Drop a Set: Remove a set that the rater has seen from the final calculation of all Cross-set Requirements. Choose an option for how the set dropped will be determined. Each option is based on the scores for an individual set.
- Keep Best Trait: Define a number of sets for a specific trait or all traits where only the highest agreement results will be used to satisfy the cross-set trait-specific agreement requirements. The number of traits kept will always be equal to the number of sets configured on Must Pass. For this option to be used, there must already be a trait-specific rule set.
- Traits Must Pass- Define : Define a number of sets for a specific trait or all traits where the rater must meet the trait-specific agreement threshold rule(s). For this option to be used, there must already be a trait-specific rule set. Note, when using this rule, it will be combined with any trait-specific requirements that refer to the same trait to create 1 rule.
...