A low Kappa score indicates a lack of agreement, beyond chance, between reviewers undertaking screening for example. This may result in a higher-than-expected number of conflicts requiring resolution.
There are several reasons why inter-rater reliability may be low:
- Lack of clarity or ambiguity in the criteria: If the criteria used to evaluate the phenomenon are unclear or ambiguous, raters may have different interpretations and produce inconsistent ratings.
- Differences in judgment or perception: Raters may have different judgments or perceptions of the phenomenon being evaluated, which can lead to disagreements and inconsistent ratings.
- Rater bias: Raters may have personal biases or preferences that influence their evaluations, leading to inconsistent ratings.
- Inadequate training or lack of experience: If the raters are not adequately trained or do not have a clear understanding of the protocol or criteria, they may produce inconsistent ratings.
- Complexity of the phenomenon: If the phenomenon being evaluated is complex or difficult to evaluate, raters may have difficulty producing consistent ratings.
- Insufficient sample size: If the sample size is too small, it may be difficult to establish inter-rater reliability due to limited data.
Check out our article on What can I do to improve a low Kappa score? for some tips on improving the score.
Click here to go back to Covidence Academy.
Already have an account? Sign in and start screening!
Explore more resources.
