Developing a Search Strategy
Informed by both the research question and the selection criteria, a search strategy lays the foundation for identifying all relevant studies that will inform the review. A well-crafted search strategy ensures that the evidence base is both comprehensive and unbiased, capturing a broad range of research while minimizing the risk of overlooking key studies. This process involves a careful balance between precision (returning only relevant studies to ensure efficiency) and sensitivity (returning as many potentially relevant studies as possible to ensure comprehensiveness). By systematically planning and structuring the search, teams can efficiently navigate vast quantities of literature, enabling them to focus on the most pertinent evidence that will answer their research question. Thoughtful consideration of keywords, databases, and inclusion/exclusion criteria ensures that the search is both efficient and effective.
Search strategies should consider and be shaped by the following:
Search strategies include:
Searching Platforms and Databases
The scope, functionality, and availability of research platforms will drive the selection of which platforms and databases you choose to search. Understanding a resource scope will help you answer the question of why you choose to include that resource while understanding the functionality will help guide how you search. Due to the substantial differences in functionality across resources, search strategies must compensate for these differences and be adjusted to ensure parity when searching. Adjusting a search may include translating syntax, defining and mapping metadata fields, testing and comparing resulting citations, and analyzing resources to understand their unique return.
Concepts and keywords related to the research question/topic are developed iteratively. This process often goes hand in hand with refining the inclusion and exclusion criteria as it is common for questions of scope to arise as long lists of keywords are formed into concept lists. Keywords are normally developed by a team that includes subject matter experts and research librarians and can include the use of text mining and topic modeling to help examine known information sources or naive searches on individual concepts.
Search strings are also developed iteratively. Strings and sub-strings are often tested as they are developed to examine the impact of individual terms on search results. This process of iterative development and testing provides a research team with the ability to examine the impact of terms to ensure a balance between precision and sensitivity. Documenting the iterative development of strings ensures the review team is highly aware of the search strategy and ensures transparency in documenting decisions related to final strings.
Inclusion and exclusion criteria inform search string development and serve as post-search filters. When clearly defined, research teams can develop more effective search strings that balance comprehensiveness and feasibility. For example, a team may not want to include geographic-focused search terms despite a geographically specific question, instead, the team would apply this criterion when screening literature in order to ensure relevant literature that lacks geographic terminology is not excluded in the initial search.
Searching for Gray Literature
The inclusion of gray literature is highly recommended for synthesis projects in environmental sciences due to the amount of evidence that is not published through traditional publication models (peer-reviewed articles, books, etc.). It's important to remember, however, that searching and screening gray literature is extremely time-consuming. Developing a strategy for gray literature discovery and screening is critical to ensuring a smooth project. Our team has developed a guide on gray literature that details the various types of gray literature and sources across governmental, NGO, and research communities. We have also developed a template for gray literature searching to aid in documenting gray literature search efforts associated with synthesis projects.
Supplemental Search Methods
Beyond databases, repositories, and gray literature information sources there are also various supplementary methods that teams can use to discover potentially relevant evidence. These methods include hand-searching non-indexed sources, citation chasing, calls for evidence to stakeholders, review of relevant published syntheses, and the use of novel search tools that incorporate large language models and generative AI.
Managing and Tracking Citation Data
When gathering search results, a research team may have anywhere between 10k - 100k+ bibliographic records before deduplication. This data may be exported from platforms in the form of .ris, .csv, .bib , or other formats that include bibliographic metadata. Teams may also be handling PDF documents, supplementary documentation, etc. In order to successfully manage this data and maintain accurate information that is required for reporting, it is highly recommended that a team use citation management software and maintain all exported bibliometric data in accordance with a data management plan.
It is important to remember that citation data is not standardized. While information sources may provide the ability to export bibliographic data in .ris or .bib formats, the quality, completeness, accuracy, and even the metadata fields will vary widely. After gathering search results, teams should be prepared to spend time on creating (in the case of gray literature), cleaning, and enhancing metadata to ensure that the data is ready for deduplication and screening at the title/abstract level. Citation management should also be considered in regard to how teams will undertake screening, both at the title/abstract and full-text level.
