12.2 Pre-experimental and quasi-experimental design

As we discussed in the previous section, time, funding, and ethics may limit a researcher’s ability to conduct a true experiment. Researchers that wish to conduct a true experiment in medical science or social work may be required to deny necessary treatment to clients, which is violates professional ethics. Even research projects that do not involve administering necessary medications or treatments may limit the researcher’s ability to conduct a classic experiment. When true experiments are not possible, researchers often use quasi-experimental designs.

Quasi-experimental designs are similar to true experiments, but they lack random assignment to experimental and control groups. The most basic of these quasi-experimental designs is the nonequivalent comparison groups design (Rubin & Babbie, 2017). ^[1] The nonequivalent comparison group design resembles the classic experimental design, but it does not use random assignment. In many cases, the groups may already exist. For example, a researcher might conduct research at two different agency sites, one of which receives the intervention and the other does not. The researcher does not need to assigned participants to treatment or comparison groups because the groupings already existed prior to the study. While this method is more convenient for real-world research, researchers cannot be sure that the groups are comparable. Perhaps the treatment group has a characteristic that is unique, such as higher income or different diagnoses, that make the treatment more effective.

Quasi-experiments are particularly useful in social welfare policy research. Social welfare policy researchers like me often look for natural experiments, or situations in which comparable groups are created by differences that already occur in the real world. For example, Stratmann and Wille (2016) ^[2] were interested in seeing how the state healthcare policy called Certificate of Need effected the quality of hospitals. The researchers clearly could not assign states to adopt one set of policies or another, so they used hospital referral regions (the areas from which hospitals draw their patients) that spanned across state lines. Since the hospitals were in the same referral region, the researchers could reasonably assume that the client characteristics were similar. In this way, they could classify patients in experimental and comparison groups without affecting policy or telling people where to live.

There are many important examples of policy experiments that use random assignment, namely the Oregon Medicaid experiment. Oregon’s Medicaid waitlist was so long that state officials decided to conduct a lottery to see which individuals from the wait list would receive the service (Baicker et al., 2013). ^[3] Researchers used the lottery as a natural experiment that included random assignment: people selected to be a part of Medicaid were the experimental group and those on the wait list were in the control group. There are some practical complications with using people on a wait list as a control group, including the possibility of people from the wait list being accepted into the program while data is still being collected. Natural experiments aren’t a specific kind of experiment like quasi- or pre-experimental designs. Instead, they are like a feature of the social world that allows researchers to use the logic of experimental design to investigate the connection between variables.

Another approach to assign participants to experimental and comparison groups in a quasi-experimental design is through matching. Researchers should think about what variables are important in their study, particularly demographic variables or attributes that might impact their dependent variable. Individual matching involves pairing participants with similar attributes. When this is done at the beginning of an experiment, the matched pair is split—with one participant going to the experimental group and the other to the control group. In contrast, an ex post facto control group is when a researcher matches individuals after the intervention is administered to some participants. Finally, researchers may engage in aggregate matching, in which the comparison group is determined to be similar on important variables.

Though it is beyond the scope of this textbook to describe the plethora of quasi-experimental designs, one more design is worth mentioning. The time series design uses multiple observations before and after an intervention. In some cases, experimental and comparison groups are used. In other cases where that is not feasible, a single experimental group is used. By using multiple observations before and after the intervention, the researcher can better understand the true value of the dependent variable in each participant before the intervention starts. Additionally, conducting multiple observations after the intervention allows the researcher to see whether the intervention had lasting effects on participants. Time series designs are similar to single-subjects designs, which we will discuss in Chapter 15.

When true experiments and quasi-experiments are not possible, researchers may turn to a pre-experimental design (Campbell & Stanley, 1963). ^[4] Pre-experimental designs are called such because they often happen before a true experiment is conducted. Often, researchers want to see if their interventions will have an effect on a small group of people before they seek funding and dedicate time to conduct a true experiment. Pre-experimental designs, thus, are usually conducted as a first step towards establishing the evidence for or against an intervention. However, this type of design comes with some unique disadvantages, which we’ll describe as we review the pre-experimental designs available.

If we wished to measure the impact of a natural disaster like Hurricane Katrina, then we might conduct a pre-experiment by identifying an experimental group from a community that experienced the hurricane and a control group from a similar community that had not been hit by the hurricane. This study design, called a static group comparison, has the advantage of including a comparison group that did not experience the stimulus (in this case, the hurricane). Unfortunately, it is difficult to be sure that the groups are truly comparable because the experimental and control groups were determined by factors other than random assignment. Additionally, the design would only allow for posttests, unless one were lucky enough to be gathering the data already before Katrina. As you might have guessed from our example, static group comparisons are useful in cases where a researcher cannot control or predict whether, when, or how the stimulus is administered, as in the case of natural disasters.

In cases where the administration of the stimulus is quite costly or otherwise not possible, a one-shot case study design might be used. In this instance, no pretest is administered, nor is a control group present. In our example of the study of the impact of Hurricane Katrina, a researcher using this design would test the impact of Katrina only among a community that was hit by the hurricane and would not seek a comparison group from a community that did not experience the hurricane. Researchers using a one-shot case study design must be extremely cautious when making claims about the effect of the stimulus, though the design could be useful for exploratory studies that aim to testing one’s measures or the feasibility of further study.

Finally, if a researcher cannot identify a sample that is large enough to split into control and experimental groups, or if they simply do not have access to a control group, they may use a one-group pre-/posttest design. In this instance, pre- and posttests are both taken, but there is no control group to compare the experimental group to. We might be able to study of the impact of Hurricane Katrina using this design if we’d been collecting data on the impacted communities prior to the hurricane. We could then collect similar data after the hurricane. Applying this design involves a bit of serendipity and chance. Without having collected data from impacted communities prior to the hurricane, we would be unable to employ a one- group pre-/posttest design to study Hurricane Katrina’s impact.

The preceding examples where we considered studying the impact of Hurricane Katrina highlight that experiments do not necessarily need to take place in a controlled lab setting. In fact, many applied researchers rely on experiments to assess the impact and effectiveness of various programs and policies. You might recall our discussion of arresting perpetrators of domestic violence in Chapter 6, which is an excellent example of an applied experiment. Researchers did not subject participants to conditions in a lab setting; instead, they applied their stimulus (in this case, arrest) to some subjects in the field and they also had a control group in the field that did not receive the stimulus (and therefore were not arrested).