5.1 Reading results

If you recall from section 3.1, empirical journal articles are those that report the results of quantitative or qualitative data analyzed by the author. They follow a set structure—introduction, methods, results, discussion/conclusions. This section is about reading the most challenging section: results.

Read beyond the abstract

At this point, I have read hundreds of literature reviews written by students. One of the challenges I have noted is that students will report the results as summarized in the abstract, rather than the detailed findings laid out in the results section of the article. This poses a problem when you are writing a literature review because you need to provide specific and clear facts that support your reading of the literature. The abstract may say something like: “we found that poverty is associated with academic achievement.” For your literature review, you want the details, not the summary. In the results section of the article, you may find a sentence that states: “children living in households experiencing poverty are three times more likely to have failed at least one course.” This more specific statistical information provides a stronger basis on which to build the arguments in your literature review.

Using the summarized results in an abstract is an understandable mistake to make. The results section often contains figures and tables that may be challenging to understand. Often, without having completed more advanced coursework on statistical or qualitative analysis, some of the terminology, symbols, or diagrams may be difficult to comprehend. This section is all about how to read and interpret the results of an empirical (quantitative or qualitative) journal article. Our discussion here will be basic, and in parts three and four of the textbook, you will learn more about how to interpret results from statistical tests and qualitative data analysis.

Remember, this section only addresses empirical articles. Non-empirical articles (e.g., theoretical articles, literature reviews) don’t have results. They cite the analysis of raw data completed by other authors, not the person writing the journal article who is merely summarizing others’ work.

Quantitative results

Quantitative articles often contain tables, and scanning them is a good way to begin reading the results. A table usually provides a quick, condensed summary of the report’s key findings. Tables are a concise way to report large amounts of data. Some tables present descriptive information about a researcher’s sample (often the first table in a results section). These tables will likely contain frequencies (N) and percentages (%). For example, if gender happened to be an important variable for the researcher’s analysis, a descriptive table would show how many and what percent of all study participants are of a particular gender. Frequencies or “how many” will probably be listed as N, while the percent symbol (%) might be used to indicate percentages.

In a table presenting a causal relationship, two sets of variables are represented. The independent variable, or cause, and the dependent variable, the effect. We’ll go into more detail on variables in Chapter 8. Independent variable attributes are typically presented in the table’s columns, while dependent variable attributes are presented in rows. This allows the reader to scan a table’s rows to see how values on the dependent variable change as the independent variable values change. Tables displaying results of quantitative analysis will also likely include some information about which relationships are significant or not. We will discuss the details of significance and p-values later in this section.

Let’s look at a specific example: Table 5.1. It presents the causal relationship between gender and experiencing harassing behaviors at work. In this example, gender is the independent variable (the cause) and the harassing behaviors listed are the dependent variables (the effects).^[1] Therefore, we place gender in the table’s columns and harassing behaviors in the table’s rows.

Reading across the table’s top row, we see that 2.9% of women in the sample reported experiencing subtle or obvious threats to their safety at work, while 4.7% of men in the sample reported the same. We can read across each of the rows of the table in this way. Reading across the bottom row, we see that 9.4% of women in the sample reported experiencing staring or invasion of their personal space at work while just 2.3% of men in the sample reported having the same experience. We’ll discuss p values later in this section.

While you can certainly scan tables for key results, they are often difficult to understand without reading the text of the article. The article and table were meant to complement each other, and the text should provide information on how the authors interpret their findings. The table is not redundant with the text of the results section. Additionally, the first table in most results sections is a summary of the study’s sample, which provides more background information on the study than information about hypotheses and findings. It is also a good idea to look back at the methods section of the article as the data analysis plan the authors outline should walk you through the steps they took to analyze their data which will inform how they report them in the results section.

Statistical significance

The statistics reported in Table 5.1 represent what the researchers found in their sample. The purpose of statistical analysis is usually to generalize from the small number of people in a study’s sample to a larger population of people. Thus, the researchers intend to make causal arguments about harassing behaviors at workplaces beyond those covered in the sample.

Generalizing is key to understanding statistical significance. According to Cassidy and colleagues, (2019)^[2] 89% of research methods textbooks in psychology define statistical significance incorrectly. This includes an early draft of this textbook which defined statistical significance as “the likelihood that the relationships we observe could be caused by something other than chance.” If you have previously had a research methods class, this might sound familiar to you. It certainly did to me!

But statistical significance is less about “random chance” than more about the null hypothesis. Basically, at the beginning of a study a researcher develops a hypothesis about what they expect to find, usually that there is a statistical relationship between two or more variables. The null hypothesis is the opposite. It is the hypothesis that there is no relationship between the variables in a research study. Researchers then can test that relationship with the expectation that they will ultimately reject the null hypothesis and actually find a relationship between the variables.

For example, in Table 5.1 researchers were examining whether gender impacts harassment. Of course, researchers assumed (hypothesized) that women were more likely to experience harassment than men. The null hypothesis, then, would be that that women were not more likely to experience harassment than men. Once we conduct the study, we predict that our results will lead us to reject the null hypothesis and find that indeed, women were more likely to experience harassment than men. We would then generalize from our study’s sample to the larger population of people in the workplace.

Statistical significance is calculated using a p-value which is obtained by comparing the statistical results with a hypothetical set of results that would occur if the researchers re-ran their study a large number of times. Keeping with our example, imagine we re-ran our study with different men and women from different workplaces hundreds and hundreds of times and we test to see if the null hypothesis is true–that women were not more likely to experience harassment than men. If we find that across a bunch of those repetitions that we can’t reject the null hypothesis, then we have to assume that the evidence does not support our hypothesized outcome.

P-Value

Looking at Table 5.1, we see a five reported “p-values”: 0.6230, .4800, .1840, .4070, .039″. The smaller the p-value, the greater the statistical incompatibility with the null hypothesis” (Wasserstein & Lazar, 2016, p. 131).^[3] Generally, researchers in the social sciences have used 0.05 as the value at which a result is reported as significant (p is less than 0.05) or not significant (p is greater than 0.05). The p-value of 0.05 refers to our willingness to accept a risk that 5% of those hypothetical results from re-running our study would have us reject the null hypothesis when it was in fact true. Researchers, however, may choose a stricter standard such as 0.01 in which only 1% of those hypothetical results are more extreme or a more lenient standard like 0.1 in which as many as 10% of those hypothetical results are more extreme than what was actually found in the study.

Why might a researcher choose a standard of p < .01? In many cases, we may want to be really really really sure we aren’t making a mistake–for example in medical research where lives might rest on an very accurate understanding of the effects of an intervention. Similarly, we might be willing to make the case that great latitude for error be tolerated, for example in exploratory research designed to suggest areas for further research. The key is that we identify the significance level we will use in advance and stick with that choice.

Let’s look back at Table 5.1. Which one of the relationships between gender and harassing behaviors is statistically significant? It’s the last one in the table, “staring or invasion of personal space,” whose p-value is 0.039 (under the p<0.05 standard to establish statistical significance). Again, this indicates that if we re-ran our study over and over again and gender did not impact staring/invasion of space (i.e., the null hypothesis was true), only 3.9% of the time would we find similar or more extreme differences between men and women than what we observed in our study. Thus, we conclude that for staring or invasion of space only, there is a statistically significant relationship.

For contrast, let’s look at “being pushed, hit, or grabbed” and run through the same analysis to see if it is statistically significant. If we re-ran our study over and over again and the null hypothesis was true, 48% of the time (p=.48) we would find similar or more extreme differences between men and women. That is too much possibility of error for us to accept, and therefore, we report those results as not statistically significant.

This discussion should also highlight a point we discussed previously: that it is important to read the full results section, rather than simply relying on the summary in the abstract. If the abstract stated that most tests revealed no statistically significant relationships between gender and harassment, you would have missed the detail on which behaviors were and were not associated with gender. Read the full results section! And don’t be afraid to ask for help from a professor in understanding what you are reading, as results sections are often not written to be easily understood.

Statistical significance and p-values have been critiqued recently for a number of reasons, including that they are misused and misinterpreted (Wasserstein & Lazar, 2016)^[4], that researchers deliberately manipulate their analyses to have significant results (Head et al., 2015)^[5], and factor into the difficulty scientists have today in reproducing many of the results of previous social science studies (Peng, 2015).^[6] For this reason, we share these principles, adapted from those put forth by the American Statistical Association,^[7] for understanding and using p-values in social science:

1. P-values provide evidence against a null hypothesis.
2. P-values do not indicate whether the results were produced by random chance alone or if the researcher’s hypothesis is true, though both are common misconceptions.
3. Statistical significance can be detected in minuscule differences that have very little effect on the real world.
4. Nuance is needed to interpret scientific findings, as a conclusion does not become true or false when the p-value passes from p=0.051 to p=0.049.
5. Real-world decision-making must use more than reported p-values. It’s easy to run analyses of large datasets and only report the significant findings.
6. Greater confidence can be placed in studies that pre-register their hypotheses and share their data and methods openly with the public.
7. “By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. For example, a p-value near 0.05 taken by itself offers only weak evidence against the null hypothesis. Likewise, a relatively large p-value does not imply evidence in favor of the null hypothesis; many other hypotheses may be equally or more consistent with the observed data” (Wasserstein & Lazar, 2016, p. 132).

A final note–one of the biggest problems with significance testing is our desire as researchers to “prove” something works and our reluctance as journal editors to publish articles with non-significant findings. In fact, the Journal of Articles in Support of the Null Hypothesis was established to specifically provide a space for research generating non-significant results. If you’re interested in reading more about the Wild West of journal publishing, check out Future Challenges and Opportunities in Academic Publishing.

Confidence intervals

Because of the limitations of p-values, scientists can use other methods to determine whether their models of the world are accurate. One common approach is to use a confidence interval, or a range of values in which the true value is likely to be found. Confidence intervals are helpful because, as principal #5 above points out, p-values do not measure the size of an effect (Greenland et al., 2016).^[8] Remember, something that has very little impact on the world can be statistically significant, and the values in a confidence interval would be helpful. In our example from Table 5.1, imagine our analysis produced a confidence interval that women are 1.2-3.4x more likely to experience “staring or invasion of personal space” than men. As with p-values, calculation for a confidence interval compares what was found in one study with a hypothetical set of results if we repeated the study over and over again. If we calculated 95% confidence intervals for all of the hypothetical set of hundreds and hundreds of studies, that would be our confidence interval. 

Confidence intervals are pretty intuitive. As of this writing, my wife and are expecting our second child. The doctor told us our due date was December 11th. But the doctor also told us that December 11th was only their best estimate. They were actually 95% sure our baby might be born any time in the 30-day period between November 27th and December 25th. Confidence intervals are often listed with a percentage, like 90% or 95%, and a range of values, such as between November 27th and December 25th. You can read that as: “we are 95% sure your baby will be born between November 27th and December 25th because we’ve studied hundreds of thousands of fetuses and mothers, and as a result, we’re 95% sure your baby will be within these two dates.”

Notice that we’re hedging our bets here by using words like “best estimate.” When testing hypotheses, social scientists generally phrase their findings in a tentative way, talking about what results “indicate” or “support,” rather than making bold statements about what their results “prove.” Social scientists have humility because they understand the limitations of their knowledge. In a literature review, using a single study or fact to “prove” an argument right or wrong is often a signal to the person reading your literature review (usually your professor) that you may not have appreciated the limitations of that study or its place in the broader literature on the topic. Strong arguments in a literature review include multiple facts and ideas that span across multiple studies.

You can learn more about creating tables, reading tables, and tests of statistical significance in a class focused exclusively on statistical analysis. We provide links to many free and openly licensed resources on statistics in Chapter 16. For now, we hope this brief introduction to reading tables will improve your confidence in reading and understanding the results sections in quantitative empirical articles.

Qualitative results

Quantitative articles will contain a lot of numbers and the results of statistical tests demonstrating associations between those numbers. Qualitative articles, on the other hand, will consist mostly of quotations from participants. For most qualitative articles, the authors want to put their results in the words of their participants, as they are the experts. Articles that lack quotations make it difficult to assess whether the researcher interpreted the data in a trustworthy manner. These types of articles may also indicate how often particular themes or ideas came up in the data, potentially reflective of how important they were to participants.

Authors often organize qualitative results by themes and subthemes. For example, see this snippet from the results section in Bonanno and Veselak (2019)^[9] discussion parents’ attitudes towards child mental health information sources.

Data analysis revealed four themes related to participants’ abilities to access mental health help and information for their children, and parents’ levels of trust in these sources. These themes are: others’ firsthand experiences family and friends with professional experience, protecting privacy, and uncertainty about schools as information sources. Trust emerged as an overarching and unifying concept for all of these themes.

Others’ firsthand experiences. Several participants reported seeking information from other parents who had experienced mental health struggles similar to their own children. They often referenced friends or family members who had been or would be good sources of information due to their own personal experiences. The following quote from Adrienne demonstrates the importance of firsthand experience:

[I would only feel comfortable sharing concerns or asking for advice] if I knew that they had been in the same situation. (Adrienne)

Similarly, Michelle said: And I talked to a friend of mine who has kids who have IEPs in the district to see, kind of, how did she go about it. (Michelle)

…

Friends/family with professional experience. Several respondents referred to friends or family members who had professional experience with or knowledge of child mental health and suggested that these individuals would be good sources of information. For example, Hannah said:

Well, what happened with me was I have an uncle who’s a psychiatrist. Sometimes if he’s up in (a city to the north), he’s retired, I can call him sometimes and get information. (Hannah)

Michelle, who was in nursing school, echoed this sentiment: At this point, [if my child’s behavioral difficulties continued], I would probably call one of my [nursing] professors. That’s what I’ve done in the past when I’ve needed help with certain things…I have a professor who I would probably consider a friend who I would probably talk to first. She has a big adolescent practice. (Michelle) (p. 402-403)

The terms in bold above refer to the key themes (i.e., qualitative results) that were present in the data. Researchers will state the process by which they interpret each theme, providing a definition and usually some quotations from research participants. Researchers will also draw connections between themes, note consensus or conflict over themes, and situate the themes within the study context.

Qualitative results are specific to the time, place, and culture in which they arise, so you will have to use your best judgment to determine whether these results are relevant to your own work. For example, students in Matt DeCarlo’s class at Radford University in Southwest Virginia may be studying rural populations. Would a study in a large urban city transfer well to a rural area?

Maybe. But even if you were using data from a qualitative study in another rural area, are all rural areas the same? How is the population and sociocultural context in the article similar or different to the one in your study? Qualitative studies have tremendous depth, but researchers must be intentional about drawing conclusions about one context based on a study in another context.

5.2 Organizing information

This section will introduce you to three tools scholars use to organize and synthesize (i.e., weave together) information from multiple sources. First, we will discuss how to build a summary table containing information from empirical articles that are highly relevant—from literature review, to methods and results—to your entire research proposal. These are articles you will need to know the details of back-to-front because they are so highly related to your proposed study.

Second, we’ll discuss what to do with the other articles you’ve downloaded. As we’ve discussed previously, you’re not going to read most of the sources you download from start-to-finish. Instead, you’ll look at the author’s literature review, key ideas, and skim for any relevant passages for your project. As you do so, you should create a topical outline that organizes all relevant facts you might use in your literature that you’ve collected from the abstract, literature review, and conclusion of the articles you’ve found. Of course, it is important to note the original source of the information you are citing.

Finally, we will revisit concept mapping as a technique for visualizing the concepts in your study. Altogether, these techniques should help you create intermediary products—documents you are not likely to show to anyone or turn in for a grade—but that are vital steps to a final research proposal.

Organizing empirical articles using a summary table

Your research proposal is an empirical project. You will collect raw data and analyze it to answer your question. Over the next few weeks, identify about 10-15 articles that are empirically similar to the study you want to conduct. If you plan on conducting surveys of practitioners, it’s a good idea for you to read in detail other articles that have used similar methods (sampling, measures, data analysis) and asked similar questions to your proposal. A summary table can help you organize these Top 10-15 articles: empirical articles that are highly relevant to your proposal and working question.

Using the annotations in Section 4.2 as a guide, create a spreadsheet or Word table with your annotation categories as columns and each source as new row. For example, I was searching for articles on using a specific educational technique in the literature. I wanted to know whether other researchers found positive results, how big their samples were, and whether they were conducted at a single school or across multiple schools. I looked through each empirical article on the topic and filled in a summary table. At the end, I could do an easy visual analysis and state that most studies revealed no significant results and that there were few multi-site studies. These arguments were then included in my literature review. These tables are similar to those you will find in a systematic review article. Note that it’s OK that I found the results of the educational technique to be non-significant. Knowing something doesn’t work any better than other techniques is an important understanding.

A basic summary table is provided in Figure 5.1. A more detailed example is available from Elaine Gregersen’s blog, and you can download an Excel template from Raul Pacheco-Vega’s blog. Remember, although “the means of summarizing can vary, the key at this point is to make sure you understand what you’ve found and how it relates to your topic and research question” (Bennard et al., 2014, para. 10).^[10] As you revisit and revise your working question over the next few weeks, think about those sources that are so relevant you need to understand every detail about them.

A good summary table will also ensure that when you cite these articles in your literature review, you are able to provide the necessary detail and context for readers to properly understand the results. For example, one of the common errors I see in student literature reviews is using a small, exploratory study to represent the truth about a larger population. You will also notice important differences in how variables are measured or how people are sampled, for instance, and these details are often the source of a good critical review of the literature.

Synthesizing facts using a topical outline

If we’re only reading 10 articles in detail, what do we do with the others? Raul Pacheco-Vega recommends using the AIC approach: read the abstract, introduction, and conclusion (and the discussion section, in empirical articles). For non-empirical articles, it’s a little less clear but the first few pages and last few pages of an article usually contain the author’s reading of the relevant literature and their principal conclusions. You may also want to skim the first and last sentence of each paragraph. Only read paragraphs in which you are likely to find information relevant to your working question. Skimming like this gives you the general point of the article, though you should read in detail the most valuable resource of all—another author’s literature review.

It’s impossible to read all of the literature about your topic. You will read about 10 articles in detail. For a few dozen more (there is no magic number), you will read the abstract, introduction, and conclusion, skim the rest of the article, but ultimately never read everything. Make the most out of the articles you do read by extracting as many facts as possible from each. You are starting your research project without a lot of knowledge of the topic you want to study, and by using the literature reviews provided in academic journal articles, you can gain a lot of knowledge about a topic in a short period of time. This way, by reading only a small number of articles, you are also reading their citations and synthesis of dozens of other articles as well.

As you read an article in detail, we suggest copying any facts you find relevant in a separate word processing document. Another idea is to copy anything you’ve annotated as background information in Section 4.2 into an outline. Copying and pasting from PDF to Word can be difficult because PDFs are image files, not documents. To make that easier, use the HTML version of the article, convert the PDF to Word in Adobe Acrobat or another PDF reader, or use the “paste special” command to paste the content into Word without formatting. If it’s an old PDF, you may have to simply type out the information you need. It can be a messy job, but having all of your facts in one place is very helpful when drafting your literature review.

You should copy and paste any fact or argument you consider important. Some good examples include definitions of concepts, statistics about the size of the social problem, and empirical evidence about the key variables in the research question, among countless others. It’s a good idea to consult with your professor and the course syllabus to understand what they are looking for when reading your literature review. Facts for your literature review are principally found in the introduction, results, and discussion section of an empirical article or at any point in a non-empirical article. Copy and paste into your notes anything you may want to use in your literature review.

Importantly, you must make sure you note the original source of each bit of information you copy. Nothing is worse than needing to track down a source for a fact you read who-knows-where. If you found a statistic that the author used in the introduction, it almost certainly came from another source that the author cited in a footnote or internal citation. You will want to check the original source to make sure the author represented the information correctly. Moreover, you may want to read the original study to learn more about your topic and discover other sources relevant to your inquiry.

Assuming you have pulled all of the facts out of multiple articles, it’s time to start thinking about how these pieces of information relate to each other. Start grouping each fact into categories and subcategories as shown in Table 5.2. For example, a statistic stating that single adults who are homeless are more likely to be male may fit into a category of gender and homelessness. For each topic or subtopic you identify during your critical analysis of each paper, determine what those papers have in common. Likewise, determine which differ. If there are contradictory findings, you may be able to identify methodological or theoretical differences that could account for these contradictions. For example, one study may sample only high-income earners or those living in a rural area. Determine what general conclusions you can report about the topic or subtopic, based on all of the information you’ve found.

Create a separate document containing a topical outline that combines your facts from each source and organizes them by topic or category. As you include more facts and more sources in your topical outline, you will begin to see how each fact fits into a category and how categories are related to one another. Keep in mind that your category names may change over time, as may their definitions. This is a natural reflection of the learning you are doing.

A complete topical outline is a long list of facts arranged by category. As you step back from the outline, you should assess which topic areas for which you have enough research support to allow you to draw strong conclusions. You should also assess which areas you need to do more research in before you can write a robust literature review. The topical outline should serve as a transitional document between the notes you write on each source and the literature review you submit to your professor. It is important to note that they contain plagiarized information that is copied and pasted directly from the primary sources. In this case, it is not problematic because these are just notes and are not meant to be turned in as your own ideas. For your final literature review, you must reference these sources in your own words to avoid plagiarism. More importantly, you should keep your voice and ideas front-and-center in what you write as this is your analysis of the literature. Make strong claims and support them thoroughly using facts you found in the literature. We will pick up the task of writing your literature review in section 5.3.

Putting the pieces together: Building a concept map

Developing a concept map or mind map around your topic can be helpful in figuring out how the facts fit together. We talked about concept mapping briefly in Chapter 2, when we were first thinking about your topic and sketching out what you already know about it. Concept mapping during the literature review stage of a research project builds on this foundation of knowledge and aims to improve the “description of the breadth and depth of literature in a domain of inquiry. It also facilitates identification of the number and nature of studies underpinning mapped relationships among concepts, thus laying the groundwork for systematic research reviews and meta-analyses” (Lesley, Floyd, & Oermann, 2002, p. 229).^[11] Its purpose, like other question refinement methods, is to help you organize, prioritize, and integrate material into a workable research area—one that is interesting, answerable, feasible, objective, scholarly, original, and clear.

Think about the topics you created in your topic outline. How do they relate to one another? Within each topic, how do facts relate to one another? As you write down what you have, think about what you already know. What other related concepts do you not yet have information about? What relationships do you need to investigate further? Building a conceptual map should help you understand what you already know, what you need to learn next, and how you can organize a literature review.

This technique is illustrated in this YouTube video about concept mapping. You may want to indicate which concepts and relationships you’ve already found in your review and which ones you think might be true but haven’t found evidence of yet. Once you get a sense of how your concepts are related and which relationships are important to you, it’s time to revise your working question.

Revising your working question

You should be revisiting your working question throughout the literature review process. As you continue to learn more about your topic, your question will become more specific and clearly worded. This is normal, and there is no way to shorten this process. Keep revising your question in order to ensure it will contribute something new to the literature on your topic, is relevant to your target population, and is feasible for you to conduct as a student project.

For example, perhaps your initial idea or interest is how to prevent student dropout. After an initial search of the relevant literature, you realize the topic of student dropout is too broad to adequately cover in the time you have to do your project. You decide to narrow your focus to causes of dropout. After reading some articles on the causes of dropout, you further narrow your search to the influence of student risk factors on dropout. A potential research question might then be, “What student risk factors are associated with dropout in Canada?” You would then need to return to the literature to find more specific studies related to the variables in this question (e.g. dropout, students, risk). You might even add the counterfactual (resilience) to get a better sense of the issue.

Similarly, after an initial literature search for a broad topic such as school performance or grades, examples of a narrow research question might be:

• “To what extent does parental involvement in children’s education relate to school performance over the course of the early grades?”
• “Do parental involvement levels differ by family social, demographic, and contextual characteristics?”
• “What forms of parent involvement are most highly correlated with children’s outcomes? What factors might influence the extent of parental involvement?” (Early Childhood Longitudinal Program, 2011).^[12]

In either case, your literature search, working question, and understanding of the topic are constantly changing as your knowledge of the topic deepens. A literature review is an iterative process, one that stops, starts, and loops back on itself multiple times before completion. As research is a practice behaviour of educators, you should apply the same type of critical reflection to your inquiry as you would to your practice.

5.3 Writing your literature review

Congratulations! By now, you should have discovered, retrieved, evaluated, synthesized, and organized the information you need for your literature review. It’s now time to turn that stack of articles, papers, and notes into a literature review–it’s time to start writing!

Writing about research is different than other types of writing. Research writing is not like a journal entry or opinion paper. The goal here is not to apply your research question to your life or growth as a practitioner. Research writing is about the provision and interpretation of facts. The tone should be objective and unbiased, and personal experiences and opinions are excluded. Particularly for students who are used to reflective practice, research writing can be a challenge. That’s why its important to normalize getting help! If your professor has not built in peer review, consider setting up a peer review group among your peers. You should also reach out to the library to see what writing services on your campus are available to graduate students. No one should feel bad for needing help with something they haven’t done before, haven’t done in a while, or were never taught how to do. 

If you’ve followed the steps in this chapter, you likely have an outline, summary table, and concept map from which you can begin the writing process. But what do you need to include in your literature review? We’ve mentioned it before, but to summarize, a literature review should:

• Introduce the topic and define its key terms.
• Establish the importance of the topic.
• Provide an overview of the important literature related to the concepts found in the research question.
• Identify gaps or controversies in the literature.
• Point out consistent findings across studies.
• Synthesize that which is known about a topic, rather than just provide a summary of the articles you read.
• Discuss possible implications and directions for future research.

Do you have enough facts and sources to accomplish these tasks? It’s a good time to consult your outlines and notes on each article you plan to include in your literature review. If there is something you are missing, you may want to jump back to section 2.3 where we discussed how to search for literature. While you can always fill in material, there is the danger that you will start writing without really knowing what you are talking about or what you want to say. For example, if you don’t have a solid definition of your key concepts or a sense of how the literature has developed over time, it will be difficult to make coherent scholarly claims about your topic.

There is no magical point at which one is ready to write. As you consider whether you are ready, it may be useful to ask yourself these questions:

• Can I talk with someone about what we know about my topic?
• How will my literature review be organized?
• What section headings will I be using?
• How do the various studies relate to each other?
• What contributions do they make to the field?
• Where are the gaps or limitations in existing research?
• And finally, but most importantly, how does my own research fit into what has already been done?

The problem statement

Scholarly works often begin with a problem statement, which serves two functions. First, it establishes why your topic is a problem worth studying. Second, it pulls your reader into the literature review. Who would want to read about something unimportant?

A problem statement generally answers the following questions, though these are far from exhaustive:

• Why is this an important problem to study?
• How many people are affected by this problem?
• How does this problem impact other issues relevant to education?
• Why is your target population an important one to study?

A strong problem statement, like the rest of your literature review, should be filled with empirical results, theory, and arguments based on the extant literature. A research proposal differs significantly from other more reflective essays you’ve likely completed during your studies. If your topic were violence in schools, I’m sure you could come up with answers to the above questions without looking at a single source. However, the purpose of the literature review is not to test your intuition, personal experience, or empathy. Instead, research methods are about gaining specific and articulable knowledge to inform action. With a problem statement, you can take a “boring” topic like the colour of rooms used in school construction, school bussing in major cities, or recycling in schools, and help others see the topic as you see it—an important part of the social world that impacts education practice.

The structure of a literature review

In general, the problem statement belongs at the beginning of the literature review. We usually advise students to spend no more than a paragraph or two for a problem statement. For the rest of your literature review, there is no set formula by which it needs to be organized. However, a literature review generally follows the format of any other essay—Introduction, Body, and Conclusion.

The introduction to the literature review contains a statement or statements about the overall topic. At a minimum, the introduction should define or identify the general topic, issue, or area of concern. You might consider presenting historical background, mentioning the results of a seminal study, and providing definitions of important terms. The introduction may also point to overall trends in what has been previously published on the topic or on conflicts in theory, methodology, evidence, conclusions, or gaps in research and scholarship. We also suggest putting in a few sentences that walk the reader through the rest of the literature review. Highlight your main arguments from the body of the literature review and preview your conclusion. An introduction should let the reader know what to expect from the rest of your review.

The body of your literature review is where you demonstrate your synthesis and analysis of the literature. Again, do not just summarize the literature. We would also caution against organizing your literature review by source—that is, one paragraph for source A, one paragraph for source B, etc. That structure will likely provide an adequate summary of the literature you’ve found, but it would give you almost no synthesis of the literature. That approach doesn’t tell your reader how to put those facts together, it doesn’t highlight points of agreement or contention, or how each study builds on the work of others. In short, it does not demonstrate critical thinking.

Organize your review by argument

Instead, use your outlines and notes as a guide what you have to say about the important topics you need to cover. Literature reviews are written from the perspective of an expert in that field. After an exhaustive literature review, you should feel as though you are able to make strong claims about what is true—so make them! There is no need to hide behind “I believe” or “I think.” Put your voice out in front, loud and proud! But make sure you have facts and sources that back up your claims.

I’ve used the term “argument” here in a specific way. An argument in writing means more than simply disagreeing with what someone else said, as this classic Monty Python sketch demonstrates. Toulman, Rieke, and Janik (1984) identify six elements of an argument:

1. Claim: the thesis statement—what you are trying to prove
2. Grounds: theoretical or empirical evidence that supports your claim
3. Warrant: your reasoning (rule or principle) connecting the claim and its grounds
4. Backing: further facts used to support or legitimize the warrant
5. Qualifier: acknowledging that the argument may not be true for all cases
6. Rebuttal: considering both sides (as cited in Burnette, 2012)^[13]

Let’s walk through an example. If I were writing a literature review on a class size, I might want to lay out the following:

1. Claim: reductions in class size can improve student learning outcomes.
2. Grounds: data comparing learning outcomes in large and small classes, theory regarding why/how learning changes, data from evaluations of class size reduction programs, etc.
3. Warrant: small classes are superior to large classes because they allow teachers and students to engage in more powerful learning experiences (deeper engagement in content, better assessment of needs, better feedback and development).
4. Backing: data demonstrating the beneficial effects of smaller classes.
5. Qualifier: smaller class sizes do not seem to have much impact in later grades. Class size reduction is expensive because of the demand for more space and teachers.
6. Rebuttal: policy should be about creating the best learning environment for students. Student load and not class size may be a better measure of learning conditions in later grades (where teachers may be instructing 150 or more students across multiple classes).

Like any effective argument, your literature review must have some kind of structure. For example, it might begin by describing a phenomenon in a general way along with several studies that provide some detail, then describing two or more competing theories of the phenomenon, and finally presenting a question to answer or hypothesis to test one or more of the theories. Or, it might describe one phenomenon, then describe another that seems inconsistent with the first, then propose a theory that resolves the inconsistency, and finally present a further question to explore or hypothesis to test that theory. In applied research, it might describe a phenomenon or theory, then describe how that phenomenon or theory applies to some important real-world situation, and finally, may suggest a way to examine that phenomenon or test whether it does, in fact, apply to that situation.

Use signposts

Another important issue is signposting. It may not be a term you are familiar with, but you are likely familiar with the concept. Signposting refers to the words used to identify the organization and structure of your literature review to your reader. The most basic form of signposting is using a topic sentence at the beginning of each paragraph. A topic sentence introduces the argument you plan to make in that paragraph. For example, you might start a paragraph stating, “There is strong disagreement in the literature as to whether psychedelic drugs cause people to develop psychotic disorders, or whether psychotic disorders cause people to use psychedelic drugs.” Within that paragraph, your reader would likely assume you will present evidence for both arguments. The concluding sentence of your paragraph should address the topic sentence, discussing how the facts and arguments from the paragraph you’ve written support a specific conclusion. To continue with our example, I might say, “There is likely a reciprocal effect in which both the use of psychedelic drugs worsens pre-psychotic symptoms and worsening psychosis increases the desire to use psychedelic drugs.”

Signposting also involves using headings and subheadings. Your literature review will use APA formatting, which means you need to follow their rules for bolding, capitalization, italicization, and indentation of headings. Headings help your reader understand the structure of your literature review. They can also help if the reader gets lost and needs to re-orient themselves within the document. We often tell our students to assume we know nothing (they don’t mind) and need to be shown exactly where they are addressing each part of the literature review. It’s like walking a small child around, telling them “First we’ll do this, then we’ll do that, and when we’re done, we’ll know this!”

Another way to use signposting is to open each paragraph with a sentence that links the topic of the paragraph with the one before it. Alternatively, one could end each paragraph with a sentence that links it with the next paragraph. For example, imagine we wanted to link a paragraph about barriers to accessing special education services with one about the relationship between the student’s family and teacher. We could use a transition sentence like this: “Even if families overcome these barriers to accessing special education services, the teacher-family relationship can create new barriers to positive learning outcomes.” A transition sentence like this builds a connection between two distinct topics. Transition sentences are also useful within paragraphs. They tell the reader how to consider one piece of information in light of previous information. Even simple transitional words like ‘however’ and ‘similarly’ can help demonstrate critical thinking and link each building block of your argument together.

Many beginning researchers have difficulty incorporating transitions into their writing. Let’s look at an example. Instead of beginning a sentence or paragraph by launching into a description of a study, such as “Williams (2004) found that…,” it is better to start by indicating something about why you are describing this particular study. Here are some simple examples:

• Another example of this phenomenon comes from the work of Williams (2004)…
• Williams (2004) offers one explanation of this phenomenon…
• An alternative perspective has been provided by Williams (2004)…

Now that we know to use signposts, the natural question is “What goes on the signposts?” First, it is important to start with an outline of the main points that you want to make, organized in the order you want to make them. The basic structure of your argument should then be apparent from the outline itself. Unfortunately, there is no formula we can give you that will work for everyone, but we can provide some general pointers on structuring your literature review.

The literature review tends to move from general to more specific ideas. You can build a review by identifying areas of consensus and areas of disagreement. You may choose to present historical studies—preferably seminal studies that are of significant importance—and close with the most recent research. Another approach is to start with the most distantly related facts and literature and then report on those most closely related to your research question. You could also compare and contrast valid approaches, features, characteristics, theories – that is, one approach, then a second approach, followed by a third approach.

Remember that you are assuming your reader knows nothing about the topic—you have to introduce it to us and then walk us through everything we need to know to understand the issue the way you do. This means scaffolding the information you present to help us build meaning and understanding.

Here are some additional tips for writing the body of your literature review:

• Start broad and then narrow down to more specific information.
• When appropriate, cite two or more sources for a single point, but avoid long strings of references for a single idea.
• Use quotes sparingly. Quotations for definitions are okay, but reserve quotes for when something is said so well you couldn’t possible phrase it differently. Never use quotes for statistics.
• Paraphrase when you need to relay the specific details within an article
• Include only the aspects of the study that are relevant to your literature review. Don’t insert extra facts about a study just to sound smart. They take up space better used for other information.
• Avoid reflective, personal writing. It is traditional to avoid using first-person language in the literature review (I, we, us, etc.).
• Avoid informal language like contractions, idioms, and rhetorical questions.
• Note any sections of your review that lack citations from the literature. Your arguments need to be based in empirical or theoretical facts. Do not approach this like a reflective journal entry.
• Point out consistent findings and emphasize stronger studies over weaker ones.
• Point out important strengths and weaknesses of the research base, as well as contradictions and inconsistent findings.
• Implications and suggestions for further research (where there are gaps in the current literature) should be specific.

The conclusion should summarize your literature review, discuss implications, and create a space for further research needed in this area. Your conclusion, like the rest of your literature review, should make a point. What are the important implications of your literature review? How do they inform the question you are trying to answer?

You should consult with your advisor or instructor about the final structure your literature review should take. Here is an example of one possible structure:

• Problem statement
  • Establish the importance of the topic
  • Number and type of people affected
  • Seriousness of the impact
  • Physical, psychological, economic, social, or spiritual consequences of the problem
• Argument 1
  • Definitions of key terms
  • Supporting evidence
  • Common findings across studies, gaps in the literature
• Argument 2
  • Definitions of key terms
  • Supporting evidence
  • Common findings across studies, gaps in the literature
• Argument 3
  • Definitions of key terms
  • Supporting evidence
  • Common findings across studies, gaps in the literature
• Final paragraph
  • Aim
  • Research question(s) and/or hypothesis(es)

Editing your literature review

Literature reviews are more than a summary of the publications you find on a topic. As you have seen in this brief introduction, literature reviews represent a very specific type of research, analysis, and writing. We will explore these topics further in upcoming chapters. As you begin your literature review, here are some common errors to avoid:

• Accepting a researcher’s finding as valid without evaluating methodology and data
• Ignoring contrary findings and alternative interpretations
• Using findings that are not clearly related to your own study or using findings that are too general
• Dedicating insufficient time to literature searching
• Reporting results from a single study, rather than synthesizing the results of multiple studies to provide a comprehensive view of the literature on a topic
• Relying too heavily on secondary sources
• Overusing quotations
• Not justifying arguments using specific facts or theories from the literature

For your literature review, remember that your goal is to construct an argument for the importance of your research question. As you start editing your literature review, make sure it is balanced. Accurately report common findings, areas where studies contradict each other, new theories or perspectives, and how studies cause us to reaffirm or challenge our understanding of your topic.

It is acceptable to argue that the balance of the research supports the existence of a phenomenon or is consistent with a theory (and that is usually the best that researchers in education can hope for), but it is not acceptable to ignore contradictory evidence. A large part of what makes a research question interesting is uncertainty about its answer (University of Minnesota, 2016).^[14]… after all, why take the time and effort to research a question we already know the answer to?

In addition to subjectivity and bias, writer’s block can obstruct the completion of your literature review. Often times, writer’s block can stem from confusing the creating and editing parts of the writing process. Many writers often start by simply trying to type out what they want to say, regardless of how good it is. Author Anne Lamott (1995)^[15] terms these “shitty first drafts,” and we all write them. They are a natural and important part of the writing process. If you’re having trouble getting started, use the text to speech function and pretend you are telling a friend what you know about the topic, or engage in a four-minute-forced-writing to get words on the paper.

Even if you have a detailed outline from which to work, the words are not going to fall into place perfectly the first time you start writing. You should consider turning off the editing and critiquing part of your brain for a while and allow your thoughts to flow. Don’t worry about putting a correctly formatted internal citation (as long as you know which source you used there) when you first write. In fact, I just put in (insert citation) and come back to it later. The goal is to get the information out. Only after you’ve reached a natural stopping point might you go back and edit your draft for grammar, APA style, organization, flow, and more. Divorcing the writing and editing process can go a long way to addressing writer’s block—as can picking a topic about which you have something to say!

As you are editing, keep in mind these questions adapted from Green (2012):^[16]

• Content: Have I clearly stated the main idea or purpose of the paper? Is the thesis or focus clearly presented and appropriate for the reader?
• Organization: How well is it structured? Is the organization spelled out and easy to follow for the reader ?
• Flow: Is there a logical flow from section to section, paragraph to paragraph, sentence to sentence? Are there transitions between and within paragraphs that link ideas together?
• Development: Have I validated the main idea with supporting material? Are supporting data sufficient? Does the conclusion match the introduction?
• Form: Are there any APA style issues, redundancy, problematic wording and terminology (always know the definition of any word you use!), flawed sentence constructions and selection, spelling, and punctuation?

APA Style

Education as a field generally use the APA style guide to format and structure their literature reviews. Most students know APA style only as it relates to internal and external citations. If you are confused about them, consult this amazing APA style guide from the University of Texas-Arlington library. Your university’s library likely has resources they created to help you with APA style, and you can meet with a librarian or your professor to talk about formatting questions you have. Make sure you budget in a few hours at the end of each project to build a correctly formatted references page and check your internal citations. Finally, I strongly recommend spending the money you saved on this book to purchase the actual APA Style Manual.

Of course, APA style is about much more than knowing there is a period after “et al.” or citing the location a book was published. APA style is also about what the profession considers to be good writing. If you haven’t picked up an APA publication manual because you use citation generators, know that I did the same thing when I was in school. Purchasing the APA manual can help you with a common problem we hear about from students. Every professor (and every website about APA style) seems to have their own peculiar idea of “correct” APA style that you can, if needed, demonstrate is not accurate.