An explanation of the (not so easy) process.
Completing a systematic review is a big task. It might even seem overwhelming at the outset as you start to get a feel for the road that lies ahead. But the challenge is worthwhile. We write systematic reviews because we want to know the answer to an important clinical question. And because high-quality evidence can make a real difference for actual people.
Help is at hand. A systematic review is often written by a team because of the wide range of knowledge, skills, and experience needed to do it well. The other good news is that there are a range of practical tools that have been developed over the years to make conducting a review quicker, slicker, and easier 🚀.
Systematic reviews are highly structured and follow a standard process. The process can be broken down into a series of smaller and more manageable steps. So grab a chair, make yourself comfortable 🍿🥤 and let’s take a look at the main stages.
A systematic approach
The seven stages are:
- Identify the research question 🤔
- Define the inclusion and exclusion criteria ✅
- Search for studies🔍
- Select studies 🎯
- Extract data 📊
- Assess quality ⚖
- Synthesize and present results ✍
Now let’s run through a brief description of each one.
1. Identify the research question
The first step is to identify the research question. All the decisions from this point on will be based on the research question so it’s important to think carefully, consult widely, and define clearly.
The review should fill a knowledge gap. You can get an idea for what systematic reviews are already out there by checking the PROSPERO database. Think about who will use the review. Talk to these people about what information they need and why. This will ensure that the review is relevant.
Consider scope. Deciding on a broad or narrow focus will depend on many factors. A broad scope allows for a comprehensive piece of work but demands significant resources. A narrow scope might be more manageable but risks finding too little information to produce a useful synthesis. If the review is likely to produce a large amount of evidence, consider using a tool such as Covidence to manage the study data, document the decision-making and track progress.
Question frameworks provide helpful focus. The choice of framework will depend on the field of research, the nature of the question and the type of data you expect to find. PICO (patient, intervention, comparison, outcome) is commonly used in clinical medicine, SPIDER (sample, phenomenon of interest, design, evaluation, research type) can be used for reviews of qualitative evidence. And there are many more!
2. Define inclusion and exclusion criteria
The inclusion and exclusion criteria are based on the review question. What are the most important factors in your review? If you have a PICO question, you will now specify which patients, interventions and comparisons you will look at. The reporting of outcomes is not usually used to decide study eligibility. That’s because this would risk excluding studies that did actually measure the outcome but did not report it. This selective reporting on the part of the study authors can introduce bias into the review.
These criteria will also be based on how you plan to group the data for synthesis later in the review process (at stage 7). This is to make sure that the search produces the data that you need to perform that synthesis. Your decisions will be explained in a protocol or research plan, which will also describe your methods for synthesising your data. In Covidence, you can save the criteria and display them alongside the search results for easy reference while screening 🤩.
3. Search for studies
Searching the literature is a specialist discipline. Yes, it’s complicated. But with the help of a librarian or information specialist you can develop an effective search strategy. How will you find all potentially relevant studies? How much searching is enough? What software is available via your institutional license? Your librarian knows the answer to all this and more. Proof (if any were needed) that not all heroes wear capes.
The search must be highly sensitive. That is, it must retrieve as many of the potentially relevant studies as possible. Unfortunately, high sensitivity comes at the expense of overall precision. Your challenge is to find an acceptable point in this tradeoff that captures the information you need, gets rid of the information you don’t, and saves you from the fate of trawling through irrelevant references into eternity 🤯.
Example search strategy conducted in PubMed. Source: https://www.bmj.com/content/330/7482/68
4. Select studies
Curious to see what information the search has brought to light? This is an exciting stage but it can also be time-consuming (yes, of course you will need snacks 🍕🥡🧁). You might be feeling a little apprehensive about the amount of information that you now have to sift. Don’t worry: there is a clear two-step process for this.
- First, you will go through the titles and abstracts of each reference and decide whether or not the study described is potentially eligible for inclusion. If you’re unsure about a particular study because you don’t have enough information, you’ll keep it in the mix for now.
- Second, you will take a closer look at each study by screening the full text. This additional information will help to determine eligibility that was unclear at the first screen.
These steps can be managed easily in Covidence, which documents decisions and enables collaboration across your team. For example, when team members reach conflicting decisions about the inclusion or exclusion of studies, Covidence keeps a record of it and responds with an alert so that the conflict can be investigated and resolved quickly and transparently 👍.
5. Extract data
Before you can work on them, the data need to be extracted from the individual studies in a way that is consistent and reliable. The data extraction form is your friend here 📋. The exact format will depend on factors like the type of data and the nature of the question. The key to success, according to the Cochrane Handbook, is “to construct easy-to-use forms and collect sufficient and unambiguous data that faithfully represent the source in a structured and organized manner.”
It is common to find several published reports of the same study. Multiple reports need to be linked to avoid counting the same data several times. In Covidence you can merge reports into one study quickly and easily as part of the data extraction process 👏.
You will collect the data as specified in the protocol or research plan. The extraction process will also follow the relevant reporting guidelines and the PRISMA statement. In Covidence you can easily create, customise, and populate data extraction tables. Having the data and the tables in the same tool streamlines the workflow and saves time.
Pilot your data extraction form to make sure you are getting the right data in the right format for your analysis!
6. Assess quality
Let’s talk about bias. Certain design elements of randomised trials (inadequate blinding, for example) have been shown to bias their results in some cases. This is a problem because biased results can understate or overstate the true effect of the intervention 🤦. To minimise this and other types of bias, review teams must carefully assess the quality of each included trial and, in particular, the risk of bias in its results.
Whether you decide to use Cochrane’s risk of bias tool, or produce a customised assessment template, Covidence makes this process simple. You can set up a group of bias domains that enable a structured examination of the design, conduct, and reporting of the studies included in the review. Covidence’s in-built flexibility means that you can edit these domains as you go, save work in progress and return to it later, and easily check that members of the review team have completed the tasks assigned to them 😎.
7. Synthesise and present results
We’re almost there. It’s now time to produce a synthesis of the study data. Here you will summarise the characteristics of the included studies and bring study data together to produce something that has greater value as a whole than the sum of its parts.
The way in which you present the results will depend on factors such as the type of data, the type of analysis, and the applicable reporting guidelines. This will have been pre-specified in the protocol or research plan but there will be unforeseen issues that you will need to discuss, explain and justify in the review.
You might perform a meta-analysis, a statistical method that combines the results of several trials. Meta-analysis can add value by improving overall precision. But it can be unreliable if it is not conducted with due attention to, yes, bias. We talked about bias in study results at stage 6. By now, there is the additional risk of introducing bias to the results of the synthesis itself. Systematic reviewer pro tip: 🚨 Take advice from a methodologist and proceed with care 🚨.
Example meta-analysis results presented in a forest plot. Source: https://training.cochrane.org/handbook/current/chapter-10
And... you're done!
A completed systematic review is a great achievement and an opportunity to develop your research skills 🏆. We have written in more depth on each of these stages in a series of blog posts.
Find out how Covidence can support you on your systematic review journey. Register for a free trial today.
