Systematic Review Overview: 4. Apply Inclusion and Exclusion Criteria

At the beginning of large systematic reviews, researchers discuss and develop a series of inclusion and exclusion criteria to fit in with their review question and/or the brief provided by whoever is funding the project.

Systematic reviews often exclude studies if they do not conform to specific study designs, are not written in English or within a certain time frame. As a researcher, you should be cautious of any bias you might introduce into the review by adding certain inclusion or exclusion criteria. For example: limiting to studies in English may miss important studies published in other languages, leading to language bias.

All decisions to include or exclude certain studies or groups of studies should be documented in the methods section of the research proposal/protocol - this way it can be demonstrated that a systematic process has been followed.

In large systematic reviews, the inclusion/exclusion criteria are applied by at least 2 reviewers to all the studies retrieved by the literature search. A strategy to resolve any disagreements between the reviewers should be outlined in the protocol, such as bringing in a third screener.

There are two levels of the screening process. The first level of screening involves scanning the titles and abstracts of the articles; those that are clearly irrelevant can be excluded.

Full text papers are obtained for the remaining articles and the criteria are applied again for the second level of screening on the full text. Those that meet the criteria are included in the review (although sometimes if too many papers are obtained, the question and criteria are refined and the process repeated). At this stage of screening, the reason for exclusion(s) must be recorded. This process is represented by the following flow diagram (See PRISMA Flow Diagram).

• Studies should be selected in an unbiased way, based on selection criteria that flow directly from the review questions, and that have been piloted to check that they can be reliably applied.
• Study selection is a staged process involving sifting through the citations located by the search, retrieving full reports of potentially relevant citations and, from their assessment, identifying those studies that fulfill the inclusion criteria.
• Parallel independent assessments should be conducted to minimize the risk of errors of judgment. If disagreements occur between reviewers, they should be resolved according to a predefined strategy using consensus and arbitration as appropriate.
• The study selection process should be documented, detailing reasons for inclusion and exclusion.

While awaiting search strategy development and final citation results:

• Provide clear and explicit inclusion and exclusion criteria, with definitions and explanations where warranted.
• Conduct thorough training for all involved.
• Provide clear guidelines which should be reviewed by all prior to starting the activity.
• Provide pilot testing or beta testing of screening tools/procedures, using samples/subsets of real data (with test inter-rater reliability calculations to determine preliminary agreement or variability).
• Optional: pilot or beta test screeners in pairs: one screener with previous experience paired with a more novice screener.
• Conduct ongoing, active surveillance/auditing of activities (can see if/when going off course)
• Provide ongoing opportunities for discussion, education, and training.