• limitations of literature reviews
• systematic reviews
• scoping reviews
• meta-analysis reviews
• locating individual research studies.

Limitations of Literature Reviews

The introduction to an empirical article often includes a literature review conducted by the authors. In addition, literature reviews sometimes appear in the published literature. These may provide important information for practitioners to consider, however they are seldom sufficient by themselves. Bronson and Davis (2012) summarized the problems facing social work practitioners who rely on a literature review to inform their practice:

• a great deal of the research literature lacks sufficient intervention detail to inform practitioners about how to implement the best practices reported in the articles;
• outcome studies often present the practitioner with contradictory results;
• authors’ reviews often lack sufficient background concerning underlying assumptions and biases that may influence the credibility and relevance for practice decision-making, particularly within specific practice settings or with specific types of clients.

A literature review is only as thorough as the search for literature; an extensive review is more informative than a search limited by factors such as:

• the discipline where the literature appears, ignoring literature from other professions and disciplines;
• the time frame covered in the search (e.g., limited to just a few years);
• the nation where the work was conducted (e.g., an “Americentric” search might miss important contributions from other countries);
• only reviewing what is easily accessed from a library.

Generic literature reviews also may be conducted in an unsystematic fashion, leaving a reader uncertain as to the extensiveness of the coverage. This leads to discussing the qualities of more systematically conducted types of reviews: systematic, scoping, and meta-analysis reviews.

Systematic Reviews

The topic of systematic reviews was briefly introduced in our earlier course. In this chapter, you are provided with greater detail about these particularly useful tools for practitioners in many disciplines. Systematic review is a methodical process of summarizing evidence that results in a product: the systematic review report. What the systematic review method provides is:

• “An attempt to include all relevant research, including published and unpublished studies and those with contradictory findings;
• A systematic method of collecting data from the existing studies to allow for meaningful synthesis;
• Explicitly stated inclusion and exclusion criteria for identifying relevant research; and
• Transparent search methods to allow for replication”(Bronson & Davis, 2012, p. 15).

The application of systematic review methodology is how systematic reviews differ from general literature reviews. Systematic review reports are published in various dissemination outlets, including:

• professional journals;
• government agencies and government resources, such as:
  • various institutes of the National Institutes of Health/NIH (National Institute of Mental Health/NIMH, National Institute on Alcohol Abuse and Alcoholism/NIAAA, National Institute on Drug Abuse/NIDA, National Institute of Child Health and Human Development/NICHD, National Institute on Aging/NIA, National Institute of Environmental Health Sciences/NIEHS, and National Institute on Minority Health and Health Disparities/NIMHD),
  • Substance Abuse and Mental Health Services Administration/SAMHSA,
  • Agency for Healthcare Research and Quality/AHRQ,
  • Office of Justice Programs/OJP,
  • Office of Juvenile Justice and Delinquency Prevention/OJJDP,
  • Child Welfare Information Gateway at childwelfare.gov/topics/responding/child-protection/evidence/,
  • National Registry of Evidence-based Practices Resource Center at https://www.samhsa.gov/resource-search/ebp , 
  • U.S. Department of Education’s What Works Clearinghouse/WWC at https://ies.ed.gov/ncee/wwc/and,
  • Centers for Disease Control and Prevention/CDC;
• United States Preventive Services Taskforce (USPSTF) at www.uspreventiveservicestaskforce.org;
• Campbell Collaboration at www.campbellcollaboration.org/;
• Cochrane Collaboration database at http://us.cochrane.org/and the Cochrane Community Database of Abstracts of Reviews of Effects (DARE);
• the patient information resource, Informed Health Online, at www.informedhealth.org/;
• PubMed database of systematic reviews at the U.S. National Library of Medicine in NIH at https://www.ncbi.nlm.nih.gov/pubmed/.

Many of these sources address medical practice questions but may also include behavioral health questions of interest to social work practitioners. The Campbell Collaboration may be of particular interest to social work practice and policy (https://www.campbellcollaboration.org/), including umbrella topics such as:

• crime and justice
• disability
• education
• international development
• social welfare
• knowledge translation and implementation

One way to locate systematic reviews about a specific practice topic or question is to place the phrase “systematic review” in your electronic search terms. Let’s consider a practice example where a social worker needed to find evidence to inform practice decisions concerning the practice question: what is the best intervention for promoting development in young children with autism? The social worker wanted to inform a family about options for early and intensive intervention to improve outcomes for their 4-year old child with autism. So, the social worker entered the search terms: “systematic review parenting education intervention” into Google Scholar. The result appears in Figure 3-1:

Figure 3-1. Result of a systematic review search for autism interventions

Reading the abstract for this review indicated that the objective was a good match for the practice question, and the social worker learned generally how the review was conducted (methods). While this looked promising, the results and the conclusions of the published review were somewhat disappointing because the strength of the evidence the reviewers identified was generally low. Two intervention approaches showed benefits in some aspects of development for some children (the Lovaas-based intervention and the Early Start Denver Model), but which children did and did not benefit was poorly understood (Warren et al, 2011); parent intervention showed some short-term gains, but long-term durability was unclear. So, the social worker did not yet have a strong evidence-based answer to the practice question and turned back to the search for systematic reviews related to autism.

The next article that came up reviewed nutritional and dietary intervention strategies (Figure 3-2).

Figure 3-2. Next result of a systematic review search for autism interventions

In reading the abstract, the social worker was not as certain about the goodness-of-fit with the practice questions regarding promoting development in young children with autism but continued reading through the contents of the abstract. Again, the strength of the evidence was low, the results were inconsistent and were not demonstrated to persist over time (Sathe, Andrews, McPheeters, & Warren, 2017). The social worker still had little evidence to inform practice decisions.

Next the social worker considered the possibility that perhaps the search question was too broad and general. Autism is a complex problem with many variants and theories about etiology and factors that influence its course, thus it has many possible avenues for intervention. The social worker then considered what the family described as what they were most interested in pursuing with this child at this time. They were interested in behavioral approaches rather than medication. The next reference that the social worker located was the Systematic Review of Early Intensive Behavioral Interventions for Children with Autism (Howlin, Magiati, & Charman, 2009). This review applied inclusion criteria that the study’s population had to be children under the age of six years, which made it a good fit for the 4-year old family member. The review results indicated that there exists evidence to support the effectiveness of the Early Intensive Behavioral Intervention (EIBI) approach initially developed at UCLA and offered through various programs elsewhere. The review also specified that the observed effectiveness was not consistent across all children, and observed effects typically appeared within the first 12 months of intervention. The results of the review were similar to those for a similar systematic review conducted by another team, and this confirmation served to enhance the social worker’s confidence in the results. This information gave the social worker a starting place to seek local programs based on this approach in referring the family, and a timeline for evaluating the impact of the intervention (12 months). However, the social worker might also want to extend the literature search to see whether more recent evidence exists about this or other relevant interventions.

The results of the EIBI autism review were based on over 640 published studies, making the single review highly useful to a practitioner who does not have time to locate, read, critically analyze, and draw conclusions from so many pieces. Furthermore, individual practitioners may find it difficult to be systematic in the process of critically reviewing many pieces of literature. It helps to understand what goes into a systematic review to know how to trust the authors’ critical review process in place of one’s own critical analysis.

Process of a Systematic Review

Authors of a research skills workbook delineated 12 stages in the systematic review process they called “meta-synthesis” (Corcoran & Secret, 2013). Here is an overview of what they outlined, integrated with content from other resources about systematic reviews (Bronson & Davis, 2012; Higgins & Green, 2011; Littell, Corcoran, & Pillai, 2008).

• Develop specific, answerable research question. The project begins with clearly formulated objectives and the research question is specific. The COPES or PICO approach you learned in Chapter 2 might be applied at this stage.
• Form a review team. The team should include a diverse skill set, covering a range (substantive on the content area and relevant methodological/technical expertise).
• Determine explicit inclusion/exclusion criteria. These criteria concerning which reports ti include in the review should be specific about the conditions, circumstances, populations, types of intervention, intervention settings, outcome indicators, and study design/approach.
• Develop a protocol. This is a set of written guidelines about procedures and methods the team will use in the review process. This is done prescriptively (before the work begins), it is not meant to be a post hoc description after the work is completed.
• Implement a reproducible strategy for study identification. This strategy isdefined in the protocol and incorporates the inclusion/exclusion criteria that were established in prior stages. The strategy needs to result in identification of all relevant studies, including those that might be unpublished. The review should include outcomes that were adverse, beneficial, or “no observed difference” on the meaningful (primary) outcomes.
• Screen titles and abstracts. The team applies their pre-determined criteria and protocol to determine which of the previously identified studies are potentially relevant to the review.
• Retrieve potentially relevant study reports. The team utilizes strong library and electronic search approaches to gain access to the reports (including unpublished reports) identified as potentially relevant. A strong system of organizing the secured reports is important at this stage, as it can reduce duplicated effort.
• Team members judge each study. As in the case of coding qualitative data, two or more individuals should independently determine whether each report represents an eligible study—based on the predetermined eligibility criteria. Reviewers document their decisions, and discrepancies are resolved by the team.
• Data from eligible studies are extracted. For each study deemed eligible, data about the study are copied into standardized forms (created as part of the protocol). As in the judging phase, this is done by at least two team members, and disagreements are resolved by the team.
• Rate study quality. For each study deemed eligible, at least two raters critically appraise its quality (scientific integrity and design adequacy) and its qualities (sampling, data collection methods, results, and conclusions). As before, this is documented on standardized forms and disagreements are resolved by the team.
• Describe key features of included studies. Tables are used to document descriptions of these elements, created as part of the review protocol. Common categories are: author’s/authors’ purpose(s)/study aims; study design and approach; sample information; measures/data collection; data analysis/results; conclusions.
• Synthesize results. Team members work with the data accumulated in the prior steps to draw conclusions, answering the initial research question.

Strength of Evidence in a Systematic Review

Systematic reviews differ, to some extent, in terms of eligibility criteria concerning strength of evidence (scientific rigor) of the studies included. Study replication, where different investigations lead to the same observed results, increases confidence in reliability of the intervention study findings. Also, interventions found to be effective in different contexts (with different populations, in different settings, and/or by different service providers) are considered especially meritorious in terms of strength of the evidence.

In many instances, the only studies included in a systematic review are randomized control trials (RCTs) since these are believed to represent the strongest evidence possible (internal validity). This is the case for the Cochrane review system, for example. A table describing levels of evidence summarizes a prevailing belief system concerning the strength of evidence different studies present (adapted from Ackley, Swan, Ladwig, & Tucker, 2008, p. 7 and the Winona State University library https://libguides.winona.edu/c.php?g=11614&p=61584). In this framework, the “best” evidence for informing practice is associated with the top of the hierarchy (Level 1), the “weakest” evidence is at the bottom (Level 7), progressing from “green means go” toward “orange means caution.” This table follows the decision rules presented in the Oxford guidelines concerning levels of evidence (Centre for Evidence-Based Medicine, https://www.cebm.net/2009/06/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/).

Scoping Reviews

“The scoping review has become an increasingly popular approach for synthesizing research evidence” (Pham et al., 2014, p. 371). This approach is used when the existing literature in an area is wide ranging, complex, and diverse in terms of study approaches, methods, and levels of evidence available. Scoping reviews apply systematic procedures and review criteria, resulting in a “map” depicting an area of research—particularly an emerging area where the approaches to gathering evidence remain diverse.

• the research question tends to be broader, less well-defined in a scoping review;
• greater “tolerance” allowed in the inclusion/exclusion criteria related to quality or level of evidence—this allows reviewers to include evidence of greater diversity (Arksey & O’Malley, 2005).

Meta-Analysis Reviews

Meta-analysis is an interesting hybrid approach to evidence, combining elements of a systematic review with the statistical analytic approaches used in generating results from data. In reviewing literature, investigators often find that there exist multiple published studies on a topic, but that conclusions cannot be confidently drawn from the results because the individual studies were each conducted with relatively small sample sizes. The statistical analytic approach comes into play when investigators statistically combine the smaller samples from these individual studies into a single, larger sample. This increases the investigators’ power to potentially detect relatively small effects in the data that the smaller samples could not detect with confidence.

The difficulty in conducting meta-analysis lies in gaining access to the data needed for the combined analysis. Meta-analysis investigators do not necessarily need access to original data provided by study participants to the original investigators. However, they do need access to certain descriptive statistics and a clear understanding of how the variables of interest were measured in each study.

An example meta-analysis report comes from Kennedy et al., (2016), addressing the question of whether parent-child interaction therapy (PCIT) when utilized with families engaged in or at risk of engaging in child physical abuse is effective in preventing future episodes of child maltreatment. PCIT literature indicated that it was a best practice with other types of families. The article’s authors described the steps taken in conducting this meta-analysis.

• They first conducted a systematic review of literature concerning the use of parent-child interaction therapy (PCIT) specifically with families who either engaged in or were identified as being at risk of physical child abuse. Limiting this review to studies that applied an experimental or quasi-experimental design resulted in their identifying six relevant studies.
• Next, they identified the outcome variable(s) of interest that these studies had in common. The investigators focused on variables that depicted three parent-level outcomes: physical abuse recurrence, child abuse potential, and parenting stress (not child-level outcomes).
• The investigators determined the effect size of PCIT intervention compared to treatment as usual conditions, to see if the statistically significant differences were meaningful. They found that the size of the effect produced by using PCIT rather than the tested alternatives varied considerably across the different individual studies. When analyzed together, the effect size was medium for reducing physical abuse recurrence, small for reducing child abuse potential and parenting stress across the reported studies. This makes it reasonable to consider as a practice option, at least in terms of reducing physical abuse recurrence.

As you can see, the meta-analytic approach is a potentially powerful tool for synthesizing literature on a specific topic. It helps sort out the variability that often exists in individual study results. It follows a specific systematic methodology and is reproducible when well-described, which meets our criteria for empirical literature.

Locating Individual Reports

Conducting a search of literature that will include individual reports and articles is an important adjunct to existing systematic, scoping, and meta-analysis reviews. The handbook on conducting systematic reviews presented by the Cochrane Collaboration (https://handbook-5-1.cochrane.org/) describes how to set up a search for relevant studies that can form a basis for conducting a general literature search (they call it a narrative review as opposed to a systematic review). An important point made in their handbook is the distinction (and tension) between achieving sensitivity and precision.

• Sensitivity refers to greater degrees of comprehensiveness of a search—casting a wide net, so to speak. Operationally, they define sensitivity as “the number of relevant reports identified divided by the total number of relevant reports in existence,” (section 6.4.4) although the latter is like a population—its dimensions can never really be known. The idea here is that the larger this ratio, the greater the sensitivity, and the more the search is inclusive. However, in casting this wider net, the greater the number of irrelevant reports that are also included in the search results.
  
• Precision, on the other hand, refers to the fewest number of irrelevant results—hitting the center of the target, so to speak. Operationally, they define precision as “the number of relevant reports identified divided by the total number of reports identified” (section 6.4.4). In other words, precision is about reducing how much wasted effort went into identifying reports that are irrelevant. It is about being efficient.
  

In practical terms, the sensitivity and precision of your search are dependent on the search terms you use to guide your search. Sensitivity increases with the use of more synonyms, related terms, and variations in spelling. For example, searching for articles about social work interventions for responding to natural disasters might include at least the following search terms:

• natural disaster
• natural disasters
• disaster distress
• disaster relief
• disaster services

A more sensitive, lower precision list might include:

• traumatic events
• traumatic stress events
• tornado relief
• hurricane relief
• earthquake relief
• earthquake disaster recovery
• tsunami relief
• FEMA responses
• Red Cross responses
• ASPCA responses

Even more sensitive, possibly less precise would be naming specific events, such as 911, Hurricane Sandy, Hurricane Maria, Hurricane Katrina, Hurricane Florence, the Oklahoma tornadoes, the San Francisco earthquake of 1989, Mount St. Helens eruption, and other named events. If you wished to include international disaster relief efforts, you might include the December 26^th tsunami, the 2018 North Sulawesi tsunami, and the 2011 earthquake and tsunami in Japan.

The content learned in Module 2 (Chapter 3) from our prior course concerning how and where to locate empirical evidence applies to the pursuit of evidence about interventions, as well. You learned about:

• Different types of journals. In addition to topic specific journals (e.g., Journal of Social Work Practice in the Addictions, Child and Family Social Work, Journal of Gerontological Social Work, Affilia—Journal of Women and Social Work, Social Work in Health Care, Health and Social Work, Social Work in Public Health, Social Work with Groups, Social Work in Mental Health), the profession has numerous generic journals and journals in allied disciplines to consult—just a sample are listed here.
  • Social Work
  • Journal of the Society for Social Work and Research
  • Social Work Research
  • Research on Social Work Practice
  • Journal of Social Work
  • The British Journal of Social Work
  • Journal of Social Work Practice
  • Clinical Social Work Journal
  • Social Work Today
  • Social Service Review
  • International Social Work
  • Social Policy and Administration
  • Journal of Orthopsychiatry
  • Journal of Marriage and Family
  • Family Relations
  • Trauma, Violence and Abuse
  • American Journal of Community Psychology
  • Journal of Social Policy

• Different types of abstracting and indexing databases. A sample are listed here.
  • MEDLINE/PubMed
  • PsycINFO
  • Campbell Collaboration
  • Cochrane Collaboration.
• Government sites (reproduced from your Module 2 Chapter 3 readings).
  • Agency for Healthcare Research and Quality (AHRQ) at https://www.ahrq.gov/
  • Bureau of Justice Statistics (BJS) at https://www.bjs.gov/
  • Census Bureau at https://www.census.gov
  • Centers for Disease Control and Prevention (CDC) at https://www.cdc.gov
  • Child Welfare Information Gateway at https://www.childwelfare.gov
  • Children’s Bureau/Administration for Children & Families at https://www.acf.hhs.gov
  • Forum on Child and Family Statistics at https://www.childstats.gov
  • National Institutes of Health (NIH) at https://www.nih.gov, including:
    • National Institute on Aging (NIA at https://www.nia.nih.gov
    • National Institute on Alcohol Abuse and Alcoholism (NIAAA) at https://www.niaaa.nih.gov
    • National Institute of Child Health and Human Development (NICHD) at https://www.nichd.nih.gov
    • National Institute on Drug Abuse (NIDA) at https://www.nida.nih.gov
    • National Institute of Environmental Health Sciences at https://www.niehs.nih.gov
    • National Institute of Mental Health (NIMH) at https://www.nimh.nih.gov
    • National Institute on Minority Health and Health Disparities at https://www.nimhd.nih.gov
• National Institute of Justice (NIJ) at https://www.nij.gov
• Substance Abuse and Mental Health Services Administration (SAMHSA) at https://www.samhsa.gov/
• United States Agency for International Development at https://usaid.gov

Finally, a helpful way to build a literature search is to look at citations used in articles that you found to be of interest/relevance. On the flip side, you can locate a citation index that lists articles which cited an article that you found to be of interest/relevance.

Working Example: Addressing High School Dropout

Returning to the example about best practices for preventing high school dropout from chapter 2, the school district team examined literature about the risk factors for dropping out. They found that the reasons for dropping out fit into three categories: individual, family, and school-related reasons (Kelly & Franklin, 2011). The team first turned to online clearinghouses, five of them, as well as recent textbooks, and article databases (e.g., EBSCO, PsycInfo, and Social Work Abstracts). The team specified their keyword search terms used in the article searches (e.g., effective school dropout prevention programs, school dropout and prevention, and others). Their effectiveness review criteria were:

1. The program being studied had an experimental or quasi-experimental design with a control or comparison group and a sample size that allowed for statistical power.
2. The studies had been conducted with students that had similar demographics (low to middle-income socioeconomic status [SES] white, African American, and Hispanic) and could be generalized to the specific FGN and FGS student population.
3. Findings from the programs in the study showed sustained treatment effects after a minimum of a one-year follow-up.
4. The study had materials that could be easily accessed for implementation via Web site or contact person with clear instructions on how to receive training related to the intervention (Kelly & Franklin, 2011, p. 149).

In the end, the team identified 12 interventions that met their criteria.

Chapter Summary

In this chapter you examined four types of literature review that practitioners use to inform their practice decisions based on their practice questions: general, systematic, scoping, and meta-analysis reviews. You also reviewed information from our earlier course about locating research evidence and study reports—a discussion expanded to improve our understanding social work interventions. You learned about one model rating levels of evidence, as well as learning about the tension that exists between sensitivity and precision in search strategies. The material was further developed through an example of the process put into practice. The final steps in the EBP process inform our next chapter: integrating the acquired information, implementing an intervention plan, and evaluating the intervention process and outcomes.