Preparing for a Systematic Research Project

The decision to conduct a systematic research project should only be made after carefully considering objectives, needs, and feasibility. Evidence Syntheses require more time and resources than traditional reviews and while there are significant benefits, teams should only take on syntheses after determining that there is a clear need for synthesis and that there is ample time and resources to successfully complete the synthesis in a timely manner. Teams should also work to determine how their findings can assist in policy, decisions, or guiding practice. While these steps are not formalized in Evidence Synthesis guidance, teams that proactively work through these planning steps will ensure a smoother process and more impactful results.

Needs Assessment

Evidence synthesis projects can address a variety of objectives and information needs. However, the decision to conduct a synthesis should be guided by clearly identified priorities. When considering the type of literature review method to use (including non-systematic methods), it's important to ensure that the level of need is in line with the method selected. 

An evidence synthesis is most valuable when it:

• Informs policy decisions: Syntheses provide reliable, evidence-based insights to guide policy development and ensure that decisions are backed by the best available science.
• Identifies gaps in knowledge and evidence: By systematically reviewing the existing research, syntheses help to highlight where knowledge is lacking, pinpointing areas for future study and investment.
• Guides the allocation of funding and resources: Syntheses identify where research funding can have the greatest impact by addressing critical gaps or supporting the most effective interventions.
• Supports strategic planning: The results of a synthesis can inform agency-wide strategic initiatives, helping to align research efforts with long-term goals and priorities.
• Establishes recommendations for practitioners: Syntheses offer clear, actionable guidance for practitioners, helping them implement evidence-based practices in their work.
• Ensures a holistic understanding of complex topics: Evidence syntheses integrate findings from multiple studies, offering a comprehensive view of complex environmental and management challenges.
• Integrates cross-disciplinary research: Syntheses draw together evidence from different fields, providing a well-rounded perspective that supports multi-faceted decision-making.
• Instills confidence and trust: Syntheses enhance the credibility of recommendations by systematically reviewing and appraising all available evidence, reducing uncertainty.
• Reduces duplication of research efforts: By collating existing evidence, syntheses help prevent redundant studies and direct research efforts toward new areas of inquiry.

Feasibility Assessment

Before embarking on an evidence synthesis project, it's crucial to conduct a thorough feasibility assessment. Evidence syntheses are resource-intensive and time-consuming endeavors, and not every research question may warrant this level of effort. At NOAA, we encourage teams to carefully evaluate the complexity of the research question, the availability of relevant literature, and the capacity of the team to manage the project. Carefully assessing feasibility will help avoid potential challenges later in the process, ensuring that the synthesis can be conducted successfully and within the necessary timeframe.

Key questions to consider during the feasibility assessment:

• Is the research question clearly defined and suitable for synthesis?
  Ensure that the question is specific enough to be addressed through a systematic review or systematic map.
• Is there sufficient existing research available?
  Conduct preliminary scoping to determine whether there is enough data and literature available to justify a full synthesis.
• Do you have the necessary time, expertise, and resources?
  Consider the personnel, funding, and time required to complete the synthesis effectively. Resources can include both human resources (e.g., subject-matter experts, methodologists) and technological tools (e.g., bibliographic management software).

Selecting a Methodology

Once feasibility has been established, the next step is to determine which type of evidence synthesis is most appropriate for the research question. Choosing the appropriate synthesis type is essential for aligning the scope of the project with the desired outcomes.

• Systematic Reviews rigorously synthesize findings from multiple studies to answer a specific, well-defined research question. They involve comprehensive data collection, critical appraisal, and often a meta-analysis to quantitatively assess the evidence.
• Systematic Maps provide a broader overview of the existing research, helping to identify gaps and trends in the evidence base. These maps do not typically provide detailed recommendations or conclusions but offer a visual representation of the available evidence on a given topic.

Establishing a Review Team

Evidence synthesis projects require a multidisciplinary team with a range of expertise, including subject-matter experts, methodologists, data managers, screeners, and synthesis methods experts. It is recommended that a team includes members with specific skills in project management, information retrieval, data coding and extraction, critical appraisal, and data synthesis. Ensuring that your team has the necessary expertise and independence of judgment will help mitigate bias and improve the credibility of the synthesis. It's also critical to address any potential conflicts of interest within the team and document how these are managed to maintain the integrity of the process.

Here are some key team roles you should consider as you build your team:

• Lead Reviewer/Project Manager: Manages the overall project and ensures adherence to protocols.
• Information Specialist: Develops and conducts the search strategy across multiple databases and can guide the review team in scoping the research question, applying machine learning screening techniques, and managing citations across screening phases.
• Screeners: Review titles, abstracts, and full texts according to predefined eligibility criteria. They ensure consistency in decision-making and minimize bias by working independently or in pairs to review each study.
• Data Analyst: Performs data extraction and synthesis, including meta-analysis if needed. Ensures data is handled with precision and consistency.
• Subject-Matter Experts: Provide insights into the specific topic area and assist with the interpretation of findings. They help contextualize the evidence and assess its relevance to the policy or practice in question.
• Synthesis Methods Experts: Oversee the selection and application of appropriate synthesis techniques. Whether conducting narrative synthesis, meta-analysis, or qualitative synthesis, these experts ensure that the data is synthesized in a way that maximizes the robustness and reliability of the findings.