In Chapter 2, you developed a “working question” to guide your inquiry and learned how to use online databases to find sources. By now, you’ve hopefully collected a number of academic journal articles relevant to your topic area. It’s now time to evaluate the information you found. Not only do you want to be sure of the source and the quality of the information, but you also want to determine whether each item is an appropriate fit for your literature review.

This is also the point at which you make sure you have searched for and obtained publications for all areas of your research question and that you go back into the literature for another search, if necessary. You may also want to consult with your professor or the syllabus for your class to see what is expected for your literature review. In my class, I have specific questions I will ask students to address in their literature reviews.

It is likely that most of the resources you locate for your review will be from the scholarly literature of your discipline or your topic area. As we have already seen, peer-reviewed articles are written by and for experts in a field. They generally describe formal research studies or experiments with the purpose of providing insight on a topic. You may have located these articles through the four databases in Chapter 2 or through archival searching. You are now probably wondering how to evaluate the utility of the articles you’ve collected so you can use them for your research.

Generally, when we discuss the evaluation of sources, we are referring to the following aspects: accuracy, relevance, bias, reputation, currency, and credibility factors. These measures apply to all works, including books, ebooks, articles, websites, or blog postings to name a few. Before you include a source in your literature review, you should clearly understand what it is and why you are including it. According to Bennard et al. (2014), “Using inaccurate, irrelevant, or poorly researched sources can affect the quality of your own work” (para. 4). When evaluating a work for inclusion in, or exclusion from, your literature review, ask yourself a series of questions about each source.

1. 1. Is the information outdated? Is the source more than 5-10 years old? If so, it will not provide what we currently know about the topic–just what we used to know. Older sources are helpful for historical information, but unless historical analysis is the focus of your literature review, try to limit your sources to those that are current.
   2. How old are the sources used by the author? If you are reading an article from 10 years ago, they are likely citing material from 15-20 years ago. Again, this does not reflect what we currently know about a topic.
   3. Does the author have the credentials to write on the topic? Search the author’s name in a general web search engine like Google. What are the researcher’s academic credentials? What else has this author written? Search by author in the databases and see how much they have published on any given subject.
   4. Who published the source? Books published under popular press imprints (such as Random House or Macmillan) will not present scholarly research in the same way as Sage, Oxford, Harvard, or the University of Washington Press. For grey literature and websites, check the About Us page to learn more about potential biases and funding of the organization who wrote the report.
   5. Is the source relevant to your topic? How does the article fit into the scope of the literature on this topic? Does the information support your thesis or help you answer your question, or is it a challenge to make some kind of connection? Does the information present an opposite point of view, so you can show that you have addressed all sides of the argument in your paper? Many times, literature searches will include articles that ultimately are not that relevant to your final topic. You don’t need to read everything!
   6. How important is this source in the literature? If you search for the article on Google Scholar (see Figure 3.1 for an example of a search result from Google Scholar), you can see how many other sources cited this information. Generally, the higher the number of citations, the more important the article. This is a way to find seminal articles – “A classic work of research literature that is more than 5 years old and is marked by its uniqueness and contribution to professional knowledge” (Houser, 2018, p. 112).
      

      

      ^[1]

   7. Is the source accurate? Check the facts in the article. Can statistics be verified through other sources? Does this information seem to fit with what you have read in other sources?
   8. Is the source reliable and objective? Is a particular point of view or bias immediately obvious, or does it seem objective at first glance? What point of view does the author represent? Are they clear about their point of view? Is the article an editorial that is trying to argue a position? Is the article in a publication with a particular editorial position?
   9. What is the scope of the article? Is it a general work that provides an overview of the topic or is it specifically focused on only one aspect of your topic?
   10. How strong is the evidence in the article? What are the research methods used in the article? Where does the method fall in the hierarchy of evidence?
       • Meta-analysis and meta-synthesis: a systematic and scientific review that uses quantitative or qualitative methods (respectively) to summarize the results of many studies on a topic.
       • Experiments and quasi-experiments: include a group of patients in an experimental group, as well as a control group. These groups are monitored for the variables/outcomes of interest. Randomized control trials are the gold standard.
       • Longitudinal surveys: follow a group of people to identify how variables of interest change over time.
       • Cross-sectional surveys: observe individuals at one point in time and discover relationships between variables.
       • Qualitative studies: use in-depth interviews and analysis of texts to uncover the meaning of social phenomenon

The last point above comes with some pretty strong caveats, as no study is really better than another. Foremost, your research question should guide which kinds of studies you collect for your literature review. If you are conducting a qualitative study, you should include some qualitative studies in your literature review so you can understand how others have studied the topic before you. Even if you are conducting a quantitative study, qualitative research is important for understanding processes and lived experiences. Any article that demonstrates rigor in both thought and methodology is appropriate to use in your inquiry.

At the beginning of a project, you may not know what kind of research project you will ultimately propose. At this point, consulting a meta-analysis, meta-synthesis, or systematic review might be especially helpful as these articles try to summarize an entire body of literature into one article. Every type of source listed here is reputable, but some have greater explanatory power than others.

Thinking about your project

Two of the initial steps in designing a research project are identifying the overarching goals of your project and conducting a literature review. Forming a working research question, as discussed in section 2.1, is another crucial step. Creating and refining your research question will help you identify the key concepts you will study. Once you have identified those concepts, you’ll need to define them and decide how you will know that you are observing them during your data collection. Defining your concepts, and knowing them when you see them, relates to conceptualization and operationalization. Of course, you also need to know what approach you will take to collect your data. Thus, identifying your research method is another important part of research design.

In addition to identifying your research method, you also need to think about who your research participants will be and the larger group(s) they may represent. Last but certainly not least, you should consider any potential ethical concerns that could arise during the course of your research project. These concerns might come up during your data collection, but they might also arise when you get to the point of analyzing or sharing your research results.

Decisions about the various research components do not necessarily occur in sequential order. In fact, you may have to think about potential ethical concerns before you even zero in on a specific research question. Similarly, the goal of being able to make generalizations about your population of interest could shape the decisions you make about your method of data collection. Putting it all together, the following list shows some of the major components you’ll need to consider as you design your research project. Make sure you have information that will help inform how you think about each component.

• Research question
• Literature review
• Research strategy (idiographic or nomothetic, inductive or deductive)
• Units of analysis and units of observation
• Key concepts (conceptualization and operationalization)
• Method of data collection
• Research participants (sample and population)
• Ethical concerns

Being a responsible consumer of research

Being a responsible consumer of research requires you to take your identity as a social scientist seriously. Now that you are familiar with how to conduct research and how to read the results of others’ research, you have some responsibility to put your knowledge and skills to use. To do so, you must be able to distinguish what you know based on research from what you do not know. It is also a matter of having some awareness about what you can and cannot reasonably know as you encounter research findings.

When assessing social scientific findings, think about what information has been provided to you. In a scholarly journal article, you will presumably be given a great deal of information about the researcher’s method of data collection, her sample, and information about how she identified and recruited research participants. All of these details provide important contextual information that can help you assess the researcher’s claims. On the other hand, a discussion of social scientific research in a popular magazine or newspaper will likely fail to provide the same level of detailed information. In this case, what you do and do not know is more limited than in the case of a scholarly journal article. If the research appears in popular media, search for the author or study title in an academic database.

Also, take into account whatever information is provided about a study’s funding source. Most times, the entities that fund a study require that they are acknowledged in the publication, but more popular press may leave out a funding source. In this Internet age, it can be relatively easy to obtain information about how a study was funded. If this information is not provided in the source from which you learned about a study, it might behoove you to do a quick search on the web to see if you can learn more about a researcher’s funding. Findings that seem to support a particular political agenda, for example, might have more or less weight once you know whether and by whom a study was funded.

There is some information that even the most responsible consumer of research cannot know. Because researchers are ethically bound to protect the identities of their subjects, for example, we will never know exactly who participated in a given study. Researchers may also choose not to reveal any personal stakes they hold in the research they conduct. While researchers may “start where they are,” we cannot know for certain whether or how researchers are personally connected to their work unless they choose to share such details. Neither of these “unknowables” are necessarily problematic, but having some awareness of what you may never know about a study does provide important contextual information from which to assess what one can “take away” from a given report of findings.