Literature Screening

Once the search has been completed and the bibliographic data has been deduplicated, cleaned, and enhanced, the next step is to systematically screen the literature by applying the pre-defined inclusion/exclusion criteria to the results. In general, screening can be described as a two-phase process in which titles and abstracts are screened first, followed by screening at the full-text level for any items that meet the inclusion criteria. However, in practice, the screening process is more complex to ensure consistency, transparency, and efficiency. This page outlines the elements of the process as well as considerations regarding workflow and reporting.

Training & Screener Agreement

The consistent application of screening criteria is important to ensure the objectivity and reliability of the overall synthesis and findings. When left unchecked, variations in how individuals interpret criteria can introduce inconsistencies that affect the validity of the review. Variations can be caused by differences in background knowledge, experience, misunderstanding, and even unconscious bias so it's important to ensure a clear understanding of how criteria are being applied at each stage and to test for agreement between screeners. It is also important to note that reporting the outcome of pilot screening and screener agreement data is a required reporting element for many of the established guidelines, so planning this ahead of time can help keep the workflow as efficient as possible. 

Screening Titles and Abstracts

Screening at the title and abstract level requires at least two independent reviewers to minimize bias. At this stage, reviewers analyze the title and abstract of search results and make a decision to either exclude articles or to move them into the next round where the full-text of the article will be reviewed. At this level guidelines do not require a reason to be listed when articles are excluded, however, it can be helpful to set up a waterfall checklist as a reference guide for screeners to ensure consistency. When testing for agreement at this level, a waterfall checklist can also help identify which criteria may need further detail and discussion among screeners. Workflows and decision-making in regard to disagreements and potentially relevant articles can differ, teams should consider how these are handled and document their process in detail to ensure they meet reporting guidelines and maintain transparency.

Screening Full-text

Screening at the full-text level should also be completed by at least two reviewers to minimize bias. At this stage, screeners apply selection criteria to the full text of the documents included at title/abstract screening to determine if they meet inclusion criteria. Decisions to exclude articles at this point should be documented with a rationale for inclusion/exclusion. While screening full-text, teams may also work to code and/or extract relevant data needed for synthesis. Assessing agreement at this stage may not be necessary due to the need for exclusionary rationale, though training is still encouraged.

Screening Gray Literature

Due to the nature of gray literature searching/discovery and the lack of standard/available metadata, it can often be more efficient to screen gray literature as it is found, this type of screening is referred to as "in situ". The process of downloading and creating metadata for gray literature is often unfeasible and unnecessary. A well-documented strategy for searching and screening these materials can help maintain a transparent and repeatable process while ensuring teams are able to complete the most comprehensive search possible. Note that any screening done in situ should also be reflected in the ROSES or PRISMA flow diagram

Documenting Decisions

Maintaining detailed records of all processes and decisions is key across the synthesis process. These processes, when well documented, can also help to support future synthesis projects. It's important to review the conducting and reporting guidelines that your synthesis follows to ensure you are providing the level of detail required.