8 Chapter 8: Experimental Studies

Chapter 8: Experimental Studies

Objectives

After completing this module, you should be able to:

1. Describe the basic principles of experimental studies.

2. Describe common issues encountered in the design and interpretation of findings from experimental studies.

8.1 Introduction

Experimental studies refer to studies in which the investigators allocate exposure to the study participants. The allocated exposure of interest is often called an “intervention”. In experimental studies, researchers allocate an intervention to some participants while other participants receive the “control treatment” and either undergo standard procedures, receive a placebo (a substance that mimics the intervention but has no therapeutic ingredient), or nothing. The participants who receive the intervention are often called the “intervention group”. The other group of participants is often called the “control group”.

Intervention allocation can be either non-randomized (comparing the outcome before and after the intervention, or allocation based on convenience of the investigators) or randomized (participants are randomly assigned to the intervention group or the controlled group and then given either the intervention or the control treatment according to the assignment). Randomization, when properly done, enables the intervention and control groups to be similar in all aspects, including the distribution of the determinants of the outcome other than the allocated exposure, better known as confounders. When the measured characteristics of the two comparison groups are similar, we can also assume that the unmeasured characteristics are similar. Thus, the observed differences between the two groups can be attributed almost entirely to the allocated exposure (i.e., the intervention).

In epidemiology, experimental studies that involve randomization of the study participants are considered the “gold standard” of research owing to this minimization of potential confounding (Bovbjerg, 2020). Since confounding is minimized, the difference in the risk of the outcome between the intervention and control groups can be considered the efficacy of the intervention. Findings of randomized experimental studies are generally considered a stronger level of empirical evidence. Findings from multiple trials across multiple study sites can provide particularly strong empirical evidence to guide decisions regarding the introduction of new treatments or public health measures, as well as the modification of existing treatment guidelines or public health measures.

8.2 Design of Experimental Studies

8.2.1 Types of Experimental Studies

There are many types of experimental and interventional studies. In a randomized clinical trial, the investigator is conducted in a clinical setting, such as a large hospital or a physician’s office. The intervention usually receives a new drug or medical procedure, whereas the control group usually receives the standard-regimen drug or medical procedure. In a community intervention trial, the investigator may randomly select entire communities and deliver a disease-prevention intervention (such as a smoking cessation intervention or an alcohol control campaign) to all residents in the intervention communities (and an alternative program or nothing in the control communities) and measure the outcome during the follow-up visit.

The author summarizes the type of experimental studies by design (Aggarwal & Ranganathan, 2019), as follows:

Randomized controlled trials (RCT): Investigators randomly allocate the participants to either the intervention group or the control group and provide intervention or control treatment according to the study protocol. The random allocation to the intervention vs. control group is generally done at the individual level.

Nonrandomized controlled trial: Investigators allocate participants to the intervention vs. control groups without randomization, and interventions were allocated based on other factors such as the investigator’s own convenience or the ability to afford the trial treatment (Aggarwal & Ranganathan, 2019).

Interventional studies without concurrent controls: Investigators decide to use data collected in the past from a group of persons with similar characteristics to the participants who receive the intervention (also known as “historical controls”).

Before-after (pre-post studies): Investigators measure the outcome of interest among the participants before the allocation of the intervention, deliver the intervention, and then measure the same outcome of interest after intervention delivery is complete. The outcome at the pre-intervention period is compared with the outcome during the post-intervention period.

Factorial study design: Investigators test more than one intervention at a time, and the number of comparison groups is 2ⁿ (n = the number of interventions being tested). For example, in an experimental study on the use of three different medications for treatment (drug A, drug B, and drug C), there will be eight (2³ = 8) comparison groups:

Group 1: Control participants (received standard treatment, placebo, or nothing)

Group 2: Drug A only

Group 3: Drug B only

Group 4: Drug C only

Group 5: Drugs A and B (two drugs combined)

Group 6: Drugs A and C (two drugs combined)

Group 7: Drugs B and C (two drugs combined)

Group 8: Drugs A, B, and C (all three drugs combined)

Crossover study design: Participants receive the intervention or control treatment according to the initial group allocation; then, investigators stop the initial study and switch the participants, and the intervention participants now receive the control treatment and vice versa.

Cluster randomized trials: The intervention is allocated to all members of entire groups of individuals who meet the study criteria (e.g., entire schools or communities) instead of to individual persons. However, the data are still collected at the individual level and analyzed accordingly.

Experimental studies can also be classified according to the objective of comparing the intervention to the control group (Wang et al., 2017), as follows:

Superiority trial: The objective is to show that the outcome in the intervention group is superior to that of the control group.

Non-inferiority trial: The objective is to show that the outcome in the intervention group is not inferior to that of the control group. The intervention group does not necessarily have to be better than the control group; it simply must not be worse than the control group. This type of trial is conducted when the intervention is not expected to yield a better outcome but may be preferable in other ways, such as lower price, fewer side effects, or less invasive procedures (Wang, 2017).

Equivalence trial: The objective is to assess the extent to which the outcome in the intervention group is equal to that of the control group.

8.2.2 Sources of Bias in Experimental Studies

The sources of bias in experimental studies are similar to those in cohort studies, including the primary source of selection bias: loss to follow-up, with differential loss to follow-up between the intervention and control groups posing a greater threat to validity as the loss could influence the association between the intervention and the outcome either toward the null value or away from the null. Information bias can be introduced by both the participants and the investigators.

In experimental studies, information bias can be introduced by the participants when the participants are aware of their intervention vs. control status and are asked to report subjective outcomes, such as pain level, anxiety, or self-reported symptoms. Study staff who collect data and assess the outcome can similarly introduce observer bias to the study findings if they know participants’ intervention vs. control status and their preconceived notion regarding the effect of the intervention (or lack thereof) induced them to examine the participants in a given group more thoroughly than others. Similarly, data analysts can also introduce observer bias if the outcome status is known during data analysis. To overcome this potential source of bias, investigators can choose to conceal the intervention vs. outcome status from the patients, study staff who perform data collection and outcome assessment, and data analysts and other technical personnel using a technique called “blinding”. Blinding is also called “masking” and is generally classified into three levels (Penić et al., 2020): single-blind, double-blind, and triple-blind studies. The difference between blinding levels is generally based on the number of parties from whom the exposure status is concealed. Although definitions and the terminology used to describe blinding in research manuscripts vary widely, single-blind studies most commonly refer to blinding of either the study participants or the data collectors/outcome assessors; double-blind studies most commonly refer to the blinding of both the study participants and the data collectors/outcome assessors; and triple-blind studies most commonly refer to the blinding of the study participants, the data collectors/outcome assessors, and the data analysts (Penić et al., 2020).

8.2.3 Data Analysis in Experimental and Interventional Studies

As in life, some things will not go according to plan when experimental studies are conducted. Participants may not receive all of the intervention as intended; it may become impossible to contact the participants who are then lost to follow-up; or unintentional cross-over may occur (participants in the control group may unintentionally receive the intervention). Investigators have two choices when conducting data analysis: 1) Categorize and compare the participants according to the original allocation of the intervention regardless of actual events, known as intention-to-treat analysis; or 2) Categorize and compare only the participants who remain in the originally allocated group throughout the study, known as per-protocol analysis (Shah, 2011). Intention-to-treat analysis is the method of choice for superiority trials, whereas both intention-to-treat and per-protocol analyses are appropriate for non-inferiority trials.

8.2.4 Comparison of Experimental and Interventional Studies to Cross-sectional, Case-control, and Cohort Studies

Experimental studies differ from observational studies in several aspects but also share the same sources of bias and other concerns. A summary of these differences can be found in Table 8.2.4.1 below.

Table 8.2.4.1 Comparison between experimental studies and observational studies

Component		Observational Studies		Experimental studies
	Cross-sectional studies	Case-control studies	Cohort studies
Purpose	* To estimate the prevalence of exposures and outcomes of interest	* To assess the extent to which an outcome is associated with an exposure	* To compare the risk of the outcome between exposed and non-exposed participants	* To measure the efficacy of an intervention on an outcome
Design	* Sample participants from members of the population of interest and measure their exposure and outcome status	* Select participants with the outcome of interest (preferably those who just developed the outcome, also known as “incident cases”) * Then, select members of the base population that gave rise to the cases who have not become the cases as the “controls”	* Define a group of participants without the outcome of interest who can develop the outcome (i.e., susceptible population), measure their exposure (“baseline” measurement), and measure the incidence (risk) of the outcome of interest in the future after a period of time has passed (“follow-up” measurement)	* Define a group of susceptible participants without the outcome of interest, allocate the exposure (i.e., the intervention) to some or all of the participants, and conduct follow-up data collection to measure the outcome
Measuring the exposure prevalence	* Can measure the prevalence of multiple exposures simultaneously	* Generally, not appropriate for measuring the prevalence of exposures of interest in the general population	* Can measure the prevalence of the exposure at baseline * Can measure the incidence (risk) of the outcome among the susceptible participants	* Cannot measure the prevalence of the exposure. The exposure is allocated by the investigators and does not occur in nature.
Measuring outcome prevalence	* Can measure the prevalence of multiple outcomes simultaneously	* Cannot measure the prevalence of the outcome	* Can measure the outcome prevalence at follow-up, as well as the incidence (risk) of the outcome	* Can measure the outcome prevalence at follow-up, as well as the incidence (risk) of the outcome, among those who did and did not receive the intervention
Use in the study of rare exposures	* Not appropriate if conducted in the general population	* Not appropriate	* Appropriate	* Not applicable. The investigators allocate the exposure to the participants.
Use in the study of rare outcomes	* Not appropriate	* Appropriate	* Appropriate but may incur high costs	* Appropriate but may incur high costs
Common measure(s) of association	* Odds ratio (the ratio of the odds of the outcome among the exposed vs. non-exposed, or vice versa)	* Odds ratio (the ratio of the odds of the exposure among the cases vs. the controls)	* Risk ratio (the ratio of the incidence of the outcome among participants with and without the exposure of interest)	* Risk ratio (the ratio of the incidence of the outcome among the intervention compared to the control)
Common source(s) of selection bias	* Non-response (refusal to participate)	* Control selection, resulting in the exposure odds among the control not reflecting that of the base population that gave rise to the cases * Non-response	* Loss to follow-up (biased toward the null value if differential, biased either toward or away from the null value if non-differential)	* Loss to follow-up (biased toward the null value if differential, biased either toward or away from the null value if non-differential)
Common source(s) of information bias	* Social desirability bias * Self-serving bias * Response acquiescence bias * Observer bias	* Recall bias * Social desirability bias * Self-serving bias * Response acquiescence bias * Observer bias	* Social desirability bias * Self-serving bias * Response acquiescence bias * Observer bias	* Social desirability bias * Self-serving bias * Response acquiescence bias * Observer bias
Use of the study findings	* Monitoring of trends in a population * Generate hypotheses for further studies	* Identify potential determinants of a rare disease * Generate hypotheses for further studies	* Measure the risk of disease * As evidence in support of policy and practice decisions	* As particularly strong evidence in support of policy and practice decisions

8.3 Example of Experimental Studies

Investigators considered reinforcement theory and behavioral economic models of substance use, which suggest that “substance use develops and is maintained in part by few competing rewarding activities in one’s environment and a lack of reinforcement derived from non-substance use-related alternative behaviors” (Daughters et al., 2018) and developed a behavioral activation treatment that can function as a viable alternative to existing techniques. The primary objective of the study is to test the effect of the developed treatment compared to an existing technique among patients at a residential substance use treatment facility on the probability of remaining abstinent from substances at varying lengths of time after treatment.

Investigators conducted a “single-site two-arm parallel-group trial” at a “136-bed residential substance use treatment center in Northeast Washington, DC” among patients receiving standard treatment at the facility (Daughters et al., 2018). Investigators randomly allocated patients to the treatment group (n = 142) and the control group (n = 121). The treatment group received the Life Enhancement Treatment for Substance Use (LETS ACT) behavioral activation treatment. The control group received Supportive Counseling (SC) group counseling in which the therapist “provides unconditional support, utilizes reflective listening technique, and manages group dynamics by encouraging equal participation across patients” (Daughters et al., 2018); however, the therapists were trained to avoid behavioral activation techniques. At varying time periods after treatment, participants completed follow-up assessments where they self-reported demographic characteristics, treatment history, substance use behaviors, and adverse consequences from substance use. Investigators analyzed data using descriptive statistics and logistic regression.

The findings showed that the intervention had a significant effect on the outcome at varying follow-up periods, a modified version of the results table can be found in Table 8.3.1. The investigators made the following remarks in the manuscript’s Results section:

“Abstinence rates were significantly higher for LETS ACT compared to the control condition at all three time points. Abstinence rates at the 12-month follow-up were equivalent to the percent of participants who maintained continuous abstinence for the entire 12-month follow-up. The models adjusting for covariates had a minimal effect.”

Adapted from the Results section in Daughters et al., 2018

Table 8.3.1 Effect of LETS ACT on the primary outcome (abstinence from substances) at various periods after treatment (column percentage)

Outcome	Intervention group (LETS ACT participants) (n = 142)	Control group (Supportive Counseling) (n = 121)	Unadjusted OR (95% CI)	Unadjusted RR (95% CI)
Outcome: Abstinence at 3 months post-treatment
No	83 (58.5%)	91 (75.2%)	Reference	Reference
Yes	59 (41.5%)	30 (24.8%)	2.2 (1.3, 3.7)	1.4 (1.1, 1.7)
Outcome: Abstinence at 6 months post-treatment
No	104 (73.2%)	106 (87.6%)	Reference	Reference
Yes	38 (26.8%)	15 (12.4%)	2.6 (1.3, 5.0)	1.5 (1.2, 1.8)
Outcome: Abstinence at 12 months post-treatment
No	113 (79.6%)	111 (91.7%)	Reference	Reference
Yes	29 (20.4%)	10 (8.3%)	2.9 (1.3, 6.1)	1.5 (1.2, 1.9)

Bold numbers denote statistical significance at a 95% level of confidence

Adapted from Daughters et al., 2018

The investigators also made the following remarks about the findings in the Discussion section:

“Findings suggest that LETS ACT is useful in maintaining abstinence and reducing the adverse consequences from substance use over time, yet not in reducing the frequency of use among those who have relapsed. LETS ACT focuses on building skills that will minimize the likelihood of relapse in response to negative emotions, with less of a focus on harm reduction or reducing frequency of use, and does not include other relapse prevention strategies (i.e., prolapse; (37)). A closer examination of therapy group content in future research may inform hypotheses for why LETS ACT did not have an effect on substance use frequency

…

Findings are in line with prior behavioral economic interventions that aimed to reduce substance use by increasing substance-free positive reinforcement (10, 40) and with the community reinforcement approach (CRA; (41))…Future research is needed to determine whether changes in substance use are mediated by changes in activity levels and/or reward derived from activities (42)

…

Findings must be interpreted in light of study limitations, including use of self-report measures for substance use frequency, substance use-related consequences, and depressive symptoms, and the potential lack of generalizability to other patient groups and treatment settings (i.e., outpatient). Further, given the high percentage of individuals who were referred to treatment by the criminal justice system (73%), findings may not generalize to populations without similar criminal justice monitoring requirements. We were also unable to test the potential effect of group assignment and group dynamics on study outcomes, which will be important in future studies…”

Adapted from the Discussion section in Daughters et al., 2018

8.4 Practical Exercise

Note: the data set and supporting file(s) are available at this webpage: https://www.kaggle.com/wichaiditwit/datasets

Please note that this exercise is recommended but optional. I wish to invite the reader to read this section carefully, as it contains instructions on how to analyze the exercise data set. The solution to the exercise is available at the very end of this chapter after the references.

Assignment: Read the following analysis plan, adapted from a published quasi-experimental study on hand hygiene promotion at a tertiary health facility during the COVID-19 pandemic (Wichaidit et al., 2020), and complete the table at the end of this section.

Title: Observed hand hygiene behaviors before and after the installation of pedal-operated alcohol gel dispensers with behavioral nudges: A hospital-based quasi-experimental study

Objective: To compare the observed hand hygiene behaviors among outpatient department visitors at a tertiary health facility during the COVID-19 pandemic before and after the installation of pedal-operated alcohol gel dispensers with behavioral nudges.

Methods

The complete methodology of this study (Wichaidit et al., 2020) can be found via this link: https://journals.sagepub.com/doi/full/10.4081/jphr.2020.1863

An abridged description of the study methods provided below.

Study Design and Setting

The investigators conducted a hospital-based quasi-experimental (pre- vs. post-intervention comparison) study at a tertiary hospital in southern Thailand. Data in the study were collected at three different departments. However, for the purpose of this practical exercise, we will not disaggregate the findings by department.

Intervention

The intervention in this study was the installation of foot-pedal-operated alcohol gel dispensers with behavioral nudges in the observation areas within the study hospital. The alcohol gel dispensers were donated by a member of the study hospital’s senior management. The nudges were designed based on a review of the literature and a consultation with a graphic designer who worked at Prince of Songkla University. The investigators pilot-tested the prototype images of the nudges by conducting a survey with convenience sampling among outpatient area visitors at the study hospital and used the feedback to finalize the nudges. The details of the finalized nudges can be found in Figure 1 in the published article (Wichaidit et al., 2020).

Study Population and Participants

The purpose of this study was to compare the proportion of events involving potential pathogen transmission before vs. after the intervention. Thus, the unit of analysis for this study was the potential pathogen transmission events among all visitors at the observation areas. These events included: 1) respiratory fluid contact; 2) touching face with bare hands; 3) touching one’s mask or face cover; 4) eating or drinking; and 5) all other events where lack of hand hygiene was observed despite not being able to identify the pathogen transmission act.

Participant Recruitment and Data Collection

The investigators trained three freelance research assistants in surreptitious observation of behaviors using a phone-based questionnaire application. The behaviors are public, and informing the outpatient visitors beforehand could lead to the Hawthorne effect, a type of social desirability bias that occurs among individuals who realize that they are being observed; thus, the investigators received an exemption from obtaining informed consent.

Study Variables

Variable	Definition	Coding
Intervention: Installation of alcohol gel dispensers with behavioral nudges	We compare the outcome at pre-intervention vs. post-intervention period by using a variable that designates the period of the observation made. Use the variable “intervention”, with the following values: 0 = pre-intervention period 1 = post-intervention period	N/A. Use the variable “intervention”.
Outcome: Observed hand hygiene	Observed hand hygiene behavior at any type of event that met the study criteria. Use the variable “hw_sob10_handwashed” with the following values: 0 = hand hygiene not observed 1 = hand hygiene observed	N/A. Use the variable “hw_sob10_handwashed”.

Study Instruments

The study instrument was a structured questionnaire for surreptitious observation of hand hygiene behavior, programmed on the KoboCollect application for Android phones.

Data Analysis

We will use the intention-to-treat analysis approach. We will use descriptive analysis with cross-tabulation. We will assess for the difference in the probability of hand hygiene at pre-intervention vs. post-intervention periods using either the Chi-square test of independence if all cells in the cross-tabulation have more than five observations or Fisher’s exact test if otherwise.

Ethical Considerations

The investigators stated in the article: “The study was approved by the Human Research Ethics Committee of the Faculty of Medicine, Prince of Songkla University (REC. 63-233-19-2). Hand hygiene behaviors at the study sites were considered to be public behaviors and structured observations did not violate privacy and confidentiality, thus the investigators were allowed an exemption from obtaining informed consent.”

Table 8.4.1 Observed hand hygiene behaviors at any eligible events among outpatient visitors before vs. after the installation of hand hygiene stations and behavioral nudges (row percentage)

Period	Hand hygiene not observed	Hand hygiene observed	P-value
Pre-intervention (before installation) (n = … events)
Post-intervention (after installation) (n = … events)

*With the Chi-square test of independence or Fisher’s exact test

8.5 Conclusion

Experimental studies refer to studies in which the investigators allocate exposure to the study participants. The allocated exposure of interest is often called an “intervention”. Experimental studies allocate an intervention to some participants while other participants receive the “control treatment” and either undergo standard procedures, receive a placebo, or do nothing. In this chapter, we have covered the types of experimental studies by design as well as the objective of assessing the difference in outcomes between the intervention and the control groups. We have also covered issues related to potential bias and data analyses.

References

Aggarwal, R., & Ranganathan, P. (2019). Study designs: Part 4—Interventional studies. Perspectives in Clinical Research, 10(3), 137–139. https://doi.org/10.4103/picr.PICR_91_19

Bovbjerg, M. L. (2020). Foundations of Epidemiology. Oregon State University. https://open.oregonstate.education/epidemiology/

Daughters, S. B., Magidson, J. F., Anand, D., Seitz-Brown, C. J., Chen, Y., & Baker, S. (2018). The effect of a behavioral activation treatment for substance use on post-treatment abstinence: A randomized controlled trial. Addiction (Abingdon, England), 113(3), 535–544. https://doi.org/10.1111/add.14049

Penić, A., Begić, D., Balajić, K., Kowalski, M., Marušić, A., & Puljak, L. (2020). Definitions of blinding in randomised controlled trials of interventions published in high-impact anaesthesiology journals: A methodological study and survey of authors. BMJ Open, 10(4), e035168. https://doi.org/10.1136/bmjopen-2019-035168

Shah, P. B. (2011). Intention-to-treat and per-protocol analysis. CMAJ : Canadian Medical Association Journal = Journal de l’Association Medicale Canadienne, 183(6), 696. https://doi.org/10.1503/cmaj.111-2033

Wang, B., Wang, H., Tu, X. M., & Feng, C. (2017). Comparisons of Superiority, Non-inferiority, and Equivalence Trials. Shanghai Archives of Psychiatry, 29(6), 385–388. https://doi.org/10.11919/j.issn.1002-0829.217163

Wichaidit, W., Naknual, S., Kleangkert, N., & Liabsuetrakul, T. (2020). Installation of pedal-operated alcohol gel dispensers with behavioral nudges and changes in hand hygiene behaviors during the COVID-19 pandemic: A hospital-based quasi-experimental study. Journal of Public Health Research, 9(4), https://doi.org/10.4081/jphr.2020.1863

Solutions: Practical Exercise for Chapter 8 (Experimental Studies)

Open the epicalc package and the data file.

library(epicalc)

setwd(“[redacted]”)

use(“Ch8_exercise.csv”)

Exposure: Borderline personality disorder

Use the variable “intervention”.

tab1(intervention)

Outcome: Observed hand hygiene

Use the variable “hw_sob10_handwashed”.

tab1(hw_sob10_handwashed)

Analysis for Table 8.3.1

Use the intention-to-treat analysis approach (use the “intervention” variable as it appears).

Perform descriptive analysis with cross-tabulation.

tabpct(intervention, hw_sob10_handwashed)

Assess the difference in the probability of hand hygiene at pre-intervention vs. post-intervention periods using either the Chi-square test of independence if all cells in the cross-tabulation have more than five observations or Fisher’s exact test if any cell has fewer than five observations.

fisher.test(intervention, hw_sob10_handwashed)