What is a Protocol?

A protocol is the foundation of a systematic evidence synthesis project. It serves as a comprehensive plan that details the research question, methods, and processes you will follow in your synthesis. The protocol ensures that the project is transparent, rigorous, and objective from start to finish. Protocols can be registered in protocol databases or published as peer-reviewed articles. Registering or publishing your protocol is a key component in ensuring transparency and preventing duplication of efforts across the research community. Protocol registration is a required reporting element and any final synthesis without an associated protocol should be critically reviewed as this can often signal that established guidelines were not consulted.

Key Components

• Ensures a rigorous and well-defined review process: The protocol serves as a roadmap, keeping your synthesis on track and aligned with best practices.
• Provides clarity on question formulation: It explains how and why the research question was developed and ensures that it is answerable through systematic methods.
• Outlines a strategy for retrieving relevant studies: The protocol describes the search strategy, including databases, search terms, and grey literature sources to ensure comprehensive coverage.
• Details eligibility criteria for study selection: It specifies inclusion and exclusion criteria, ensuring that all relevant evidence is considered while maintaining consistency in study selection.
• Describes the screening methods: The protocol lays out the process for title/abstract and full-text screening, including the roles of screeners and methods for resolving disagreements.
• Outlines the data extraction and coding strategy: It provides a standardized approach for extracting and coding relevant data, ensuring consistency and accuracy across the review.
• Details the study validity assessment strategy (for evidence reviews): The protocol explains the methods for critically appraising study quality and assessing the risk of bias.
• Describes the data synthesis strategy: Whether using narrative synthesis, meta-analysis, or another method, the protocol explains how the data will be combined and analyzed to answer the research question.
• Addresses any conflicts of interest: The protocol documents any potential conflicts of interest and outlines how they will be managed to maintain objectivity.

Why is Developing a Protocol Important?

Developing a protocol is a crucial step in the evidence synthesis process because it enhances the credibility, reproducibility, and transparency of your work. A well-developed protocol will:

• Prevent bias: By outlining methods and criteria in advance, the protocol reduces the risk of introducing bias into the review process.
• Ensure consistency: The protocol helps maintain consistency across all phases of the project, from study selection to data synthesis, even if new team members join the project.
• Facilitate peer review and collaboration: Registering or publishing the protocol allows peers and stakeholders to review the planned methods and provide input before the review is conducted, strengthening the overall project.
• Promote accountability: Adhering to a pre-defined protocol ensures that deviations from planned methods are transparent and well-documented, contributing to the trustworthiness of the synthesis.

Conducting and Reporting Guidelines

Both conducting and reporting guidelines are critical for a high-quality systematic review or evidence synthesis. Conducting guidelines ensure that the research is conducted with integrity and rigor, while reporting guidelines ensure that the findings are presented transparently and comprehensibly. Together, they help ensure that the review is both well-executed and useful to the wider community, including researchers, policymakers, and practitioners. For example, a well-conducted review that follows Cochrane guidelines might produce robust findings, but if it is not reported following PRISMA or ROSES standards, those findings might be unclear, incomplete, or difficult to trust. Conversely, a perfectly reported review that was not methodologically sound during its execution may mislead readers and lead to incorrect conclusions.

Key Distinctions

• Timing: Conducting guidelines are used throughout the review process to guide the methodology. Reporting guidelines come into play after the review is complete to help authors accurately report what was done and what was found.
• Focus: Conducting guidelines focus on ensuring the quality and integrity of the review process. Reporting guidelines focus on ensuring the clarity and transparency of the review’s findings.
• Use: Conducting guidelines are internal tools that help researchers structure and execute the evidence synthesis. Reporting guidelines are external tools used to communicate the methods and findings to the broader research community and stakeholders.

Established Guidelines & Organizing Bodies

Organizing bodies that develop and maintain guidelines constantly work to improve guidelines and respond to new methods, technology, etc. As mentioned above, guidelines can be focused on conducting research or reporting research and some guidelines serve in both capacities. This list represents the major organizing bodies that work to advance methods and guidelines.

• The Collaboration of Environmental Evidence (CEE) (Environmental Sciences)
• The Campbell Collaboration (Social Sciences)
• Cochrane (Health Sciences)
• The Joanna Briggs Institute (JBI) (Health Sciences)
• The PRISMA Executive (Health Sciences & General Methods)

Protocol Registration

Authors may choose to either publish or register their protocols, both are acceptable means of registration. Publishing a protocol in a journal may have the benefit of guaranteeing the final synthesis will be published (as long as the methods and time to completion are followed within reason). Depending on the journal you select, you may also be required to register a published protocol within a registration database. Many journals and associated guidelines require the registration of protocol to publish a final synthesis. While some journals lack this requirement, an a priori protocol is core to the evidence synthesis process and ensures transparency. Not all syntheses are created equal. Some journals do not have a requirement for protocol registration when publishing a paper as a "Systematic Review" or "Systematic Map". When relying on syntheses for decision making it's important to ensure these standards have been met; completed synthesis projects that lack a registered protocol have likely not been conducted according to established guidelines or conducted systematically. When considering journals it is recommended that you choose a title that requires authors to follow conducting and reporting guidelines as well as requiring the registration or a protocol. The following are examples of Evidence Synthesis registries.

• PROCEED (Environmental Science)
• Campbell Registry (Social Science)
• PROSPERO (Health Science)
• Open Science Framework (Multidisciplinary)