If you are using a dual-reviewer approach to screening there will inevitably be a disagreement in voting. This is known as a ‘conflict’. Conflicts arise in both title and abstract and full text screening.
Title and abstract
Most systematic reviews have three voting options in title and abstract screening:
- Yes
- Maybe/Unsure
- No
Conflicts arise when a reviewer votes ‘Yes’ and the second reviewer votes ‘No’. A conflict can also occur when a reviewer votes ‘Maybe’ and the second reviewer votes ‘No’.
How to resolve it?
Resolving conflicts is usually done by the two reviewers who voted. It’s possible that one of the reviewers missed an eligibility criteria or accidentally selected the wrong option. These conflicts are simply and easily resolved between the two reviewers. It’s important to maintain blinding until the final decision is made and a vote is agreed.
Bringing in a third reviewer is a solution on the rare occasion when a consensus agreement cannot be reached by the two reviewers.
Full text
In full text screening, the reviewers are usually voting to include or exclude a reference. A reason for exclusion is required if a reference is excluded. This reason is used in the PRISMA figure (flow diagram).
Conflicts can arise when a reviewer votes to include and the second reviewer votes to exclude. Conflicts can also arise when both reviewers vote to exclude a reference but they provide a different reason for exclusion (PRISMA only allows one reason per reference).
How to resolve it?
As with title and abstract screening, resolving conflicts is usually done by the two reviewers who voted. A quick email conversation or online call is quite often all that’s needed to resolve the disagreement. It’s possible that one of the reviewers missed an eligibility criteria or accidentally selected the wrong option. It’s important to maintain blinding until the final decision is made and a vote is agreed.
Bringing in a third reviewer is a solution on the rare occasion when a consensus agreement cannot be reached by the two reviewers. Sometimes a senior team member prefers to resolve conflicts as they may have more content/methodological experience.
Some studies have more than one valid reason for exclusion which accounts for the second type of conflict that is often observed in full text screening. A good solution here is to order your list of reasons for exclusion into a hierarchy. This means that when the team is screening and excluding studies they will always select the reason in the same order and this will certainly minimise the number of conflicts generated. In these situations there is often not a right or wrong reason and the two reviewers or a third reviewer will need to come to a final decision about which reason should be recorded.
Resolving conflicts can add time delays ⌚ to your review so it is always a good idea to pilot both title and abstract and full text screening and make sure that the team understands the eligibility criteria and have a hierarchy of reasons for exclusion.
Click here to go back to Covidence Academy.
Already have an account? Sign in and start screening!
Explore more resources.
