An Experiment in Public Affairs Research Design Instruction
davidpaulcarter
Unbeknownst to them, the public affairs graduate students enrolled in my Fall 2020 Research Design course embarked on an experiment in research design instruction with unclear (to me) outcomes.[1] Instead of spending the semester developing proposals of hypothetical research studies, as their peers had done in prior sections, students of the Fall 2020 semester were to execute research projects from start to finish. The projects were to be “microstudies,” understood as small-scale projects not intended to generate widely generalizable knowledge. Nonetheless, the mandate was considerable: conceive, plan, and execute a research project in its entirety, then write up the results…all in the span of 14 weeks.
The challenge was undertaken in what can be only (and insufficiently) described as an uncertain and stressful time. The semester’s start marked the end of a summer characterized by the brutal murder of George Floyd (among other Black Americans) at the hands of police, and ensuing racial justice protests that spanned a record 2,000 cities and towns across the U.S. (Burch et al, 2020), and at least 40 counties worldwide (Smith, Wu, & Murphy, 2020). As the semester progressed, the turmoil of the summer’s protests gave way to tension over the 2020 presidential election. The turbulent and divisive election process yielded yet further anxiety as President Trump refused to recognize defeat in both popular and electoral college votes (AP, 2020), instead embarking on a campaign to undermine confidence in the electoral process.
And, of course, the semester started months into the novel coronavirus (SARS-CoV-2) pandemic, by which point the term “unprecedented” had become commonplace in relation to virus-related matters. Those fortunate enough to work from home did—often while home-schooling kids, and others headed to work in masks to curtail virus spread. Unemployment had risen to 15%—a level not seen since the Great Depression—largely impacting the most vulnerable tiers of the economic ladder (Soucheray, 2020). Finally, as of the time of this writing 1.62 million deaths are attributed to the disease worldwide, almost 299,000 of which represent U.S. residents (CTP, 2020). Communities of color have borne the brunt of the disease; as of late November, COVID-19 has killed both Black and Indigenous Americans at a rate greater than one in 1,000 (APM, 2020).[2] It is not an exaggeration to say that the fall 2020 semester was a time of both anxiety and collective trauma.
Given all of the above, how did the experiment turn out? My assessment is that the following chapters yield individually-impressive microstudies that sum to a resounding collective success. I set the bar high and these public affairs scholars-in-training cleared it—turbulent context and collective trauma notwithstanding. In the remainder of this introductory chapter, I seek to set the stage upon which this book’s microstudies can be considered and assessed. First, I describe the challenge I lay down for the public affairs students enrolled in my Fall 2020 section of Research Design. I then outline the research through which students responded to the challenge, revealing a set of microstudies that are wonderfully diverse in objectives, epistemological underpinnings, and research designs, data, and methods. I end with a final note on the importance of fostering a safe instructional context for the success of perceived high-risk assignments such as this.
The ASSIGNMENT: COMPLETE A Research Project in 14 Weeks
Students were required to complete their microstudies in groups of no fewer than two and no more than four. The semester was 14 weeks long, with an additional few days for course wrap-up. I set aside the first course week for students to orient themselves to the syllabus and course expectations. Another two were set aside for peer review, as the microstudy assignment was designed to provide students experience giving and receiving critical feedback (I also commented on projects ideas, research, and writing). This effectively left 11 weeks for students to conceive of, plan, and execute the microstudies, as well as write up the results. The assignment timeline is listed in Table 1, below.
Task | Due date |
Microstudy group and research “pitch” | End of week 4 |
Microstudy proposal (research rationale and rough design) | End of week 5 |
Proposal peer review | End of week 6 |
Draft microstudy chapter | End of week 12 |
Chapter peer review | End of week 13 |
Final chapter submission | End of week 14 |
The “Public” in Inquiry of a Public Sort
The microstudies were meant to be “public” in a couple meanings of the word. First, students were directed to choose a research question within the realm of “public affairs,” including matters of “public policy, public administration and management, nonprofit organizations and management, social enterprises, etc.” The requirement was intentionally vague, as a great many topics overlap with the students’ areas of study in some manner or another, and the focus of the project was on the doing of the research projects rather than their results, as discussed further below.
Second, the microstudies were meant to be “public” in the sense that they would be published in this book.[3] Thus, students knew from the get-go that their experience would be shared beyond our classroom’s virtual “walls” – including the successes and challenges that they experienced along the way. The decision to (self-)publish the microstudies was part of a larger pedagogical approach (returned to below), which I hoped would help generate genuine investment in the research and animate the projects. Plus, it seemed a shame for students to work all semester on research projects without sharing them outside of our class.
Considering the Ethical Implications of Social Research
In both developing microstudy ideas and executing the resulting projects, students were directed to carefully consider and articulate the ethical implications of their research—and, in doing so, to ensure that their research followed an ethically conservative and justified course. They were guided by Lee Ann Fujii’s (2012) short article “Research Ethics 101: Dilemmas and Responsibilities.” I chose the essay for several reasons: (i) Fujii extends her discussion beyond the standard treatments of consent and privacy, to include the potential social and political harms that research participation can invite on participants, (ii) Fujii clearly articulates the power imbalance that often, if implicitly, exists between researchers and research participants, and (iii) Fujii specifically addresses the “dilemmas of publication,” which would directly apply to the microstudies’ publication in this book.
Although I conferred with the University of Utah’s Institutional Review Board (IRB) prior to the start of the semester to ensure that none of the microstudies would fall under the U.S. Department of Health and Human Services’ definition of “human subjects research” (thus exempting them from IRB approval and supervision requirements), I am confident that students’ ethical deliberations under Fujii’s guidance was more comprehensive and meaningful than IRB oversight. In the words of Fujii (2012, pg. 722):
Ethical research begins not with an IRB-approved protocol but researchers’ commitment to engage with difficult issues over time…we must remind ourselves that to enter another’s world as a researcher is a privilege, not a right. Wrestling with ethical dilemmas is the price we pay for the privileges we enjoy. It is a responsibility, not a choice, and, when taken seriously, it may be one of the most important benefits we have to offer those who make our work possible.
A Focus on Research Design over Research Results
Finally, we emphasized that the microstudy assignment was about understanding the process of social research design through practice, rather than deriving novel or “important” findings. These were to be microstudies – small in scale and scope by design. And, the challenges that often plague social research (such as low survey response rates) were particularly likely in the case of studies conceived, designed, and executed in a matter of weeks. In the context of anxiety and collective trauma discussed above, “good enough” was a helpful mantra.
In these efforts, we were guided by Norma M. Riccucci’s (2010) observation that the field if public administration (and public affairs, generally) supports and promotes a variety of research traditions. With a range of possible epistemological approaches, data, and data collection and analysis methods at our disposal, the alignment of research goals, epistemic underpinnings, and research designs was crucial. As evidenced by the seven microstudies introduced in the following section, I believe the students of my Fall 2020 Research Design course convincingly demonstrated the strengths of epistemic and methodological diversity in carefully designed public affairs research.
The Response: Seven Microstudies in Public Affairs
This book presents the seven microstudies conceptualized, designed, and executed by students in my Fall 2020 Research Design course. The studies are summarized in Table 2, which identifies the individuals in each microstudy team, alongside the associated research question(s), epistemological approach, and research design and methods. The first three studies (in both the table and book) pursue questions specific to the 2020 novel coronavirus pandemic; otherwise, they are presented in no particular order.
In the first study, titled “COVID-19 and Public Transportation in Utah,” Michael Dillman and Christine Posvitak draw on secondary data from the Utah Transit Authority (UTA) to examine public transit ridership trends during the coronavirus pandemic. By comparing 2020 ridership data to that of prior years, they document the dramatic decline in UTA users which corresponds with the pandemic’s onset. They then draw on Utah Department of Health virus incident data and the timing of local and state pandemic policies to show that neither seem to explain subsequent changes in UTA ridership. The researchers conclude “other factors—such as school operations, economic factors, and factors associated with the passage of time (e.g. “lockdown fatigue”)—likely hold greater sway over [UTA] ridership” (pg. 23).
In the second study, “Impressions of Teleworking among Public Sector Employees During the COVID-19 Pandemic,” Anna Cushing, Alison Cañar, and Russell Facer conducted six in-depth interviews for an interpretive study of public sector employee perceptions of telework. Through an inductive thematic analysis of the resulting qualitative data, they identify a number of telework benefits, such as more flexible schedules and time-saving efficiencies, as well as drawbacks, such as a loss of morale and productivity. They highlight the importance of understanding employee telework perceptions for public sector organizations, suggesting that not doing so could leave government employers at a disadvantage as “the private sector may begin attracting qualified candidates through a commitment to maintaining a work-life balance through [post-pandemic] teleworking” (pg. 46).
Research question(s) | Epistemology; Research design & methods | Researchers |
How has the novel coronavirus changed how people use public transportation along Utah’s Wasatch Front? | Empiricism; descriptive and associational analysis of secondary data | Michael Dillman & Christine Posvistak |
How do public sector employees perceive teleworking during the COVID-19 pandemic? | Interpretivism; small-n qualitative research through semi-structured interviews | Anna Cushing, Alison Canar, & Rusty Facer |
What are the educational modes offered in Utah during the COVID-19 pandemic and why do parents prefer one mode over another? | Empiricism; descriptive analysis of primary survey data | Andre French Lastre, Mishael Garz, Santiago Garcia, & Taylor Greenwell |
How did the Affordable Care Act’s contraceptive mandate impact access to reproductive healthcare? | Interpretivism and critical feminism; small-n qualitative research through semi-structured interviews | Shykell Ledford, Kristen Vanleeuwen, & Logan Waechtler |
To what extent do Salt Lake City’s Climate Positive 2040 goals reflect Intergovernmental Panel on Climate Change recommendations? | Empiricism; synthetic policy document comparative analysis | Luiz Aurelio De Santana Luz, Mauricio Laguan, & Raissa Umba |
To what extent are hazardous waste sited in Salt Lake County more likely to be located in communities of color and/or those that are socioeconomically disadvantaged. | Critical empiricism; descriptive analysis of secondary geographic and quantitative data | Sharah Meservy & Alex Bond |
What barriers and other experiences do college graduates with invisible disabilities face in seeking employment and workplace accommodations? How do these experiences impact other aspects of their lives? | Critical interpretivism; small-n qualitative research through semi-structured interviews | Christine Behle & Sam F. R. Roberts |
In the third study, “Educational Formats in Utah K-12 Schooling During the COVID-19 Pandemic,” Andre French Lastre, Mishael Garz, Santiago Garcia, and Taylor Greenwell conduct a survey of University of Utah College of Social and Behavioral Science students to explore why parents selected different educational modes (online, in-person, or hybrid) during the first months of the pandemic. Although the study’s convenience sample approach precludes generalizations, the findings of their small-n response are consistent with anecdotal accounts of the time. For instance, they find that parents more concerned with exposure to COVID-19 were more likely to chose online education for their kids, while those more concerned with children’s mental health seemed to lean towards in-person or hybrid instruction.
In the fourth study, “A Critical Look at How the ACA’s Contraceptive Mandate Impacted Access to Women’s Reproductive Health Services,” Shykell Ledford, Kristen VanLeeuwen, and Logan Waechtler embark on an interpretive study of women’s health professionals’ perceptions of the Affordable Care Act’s (ACA) Contraceptive Mandate. Their critical qualitative analysis identifies ways in which interviewees believe the Mandate promoted reproductive health services, as well as where it fell short. They further discuss the uncertainty of the Mandate’s future, arguing that “inconsistent implementation of the Mandate and the legal ramifications that encompass it yield an unpredictable fate that has yet to be determined” (pg. 69).
In the fifth study, “Analyzing Salt Lake City’s Climate Positive 2040 Plan through IPCC Policy Goals,” Luiz Aurelio de Santana Luz, Mauricio Laguan, and Raissa Umba design and execute a synthetic policy-comparison approach to assessing Salt Lake City’s plan to tackle climate change and carbon pollution, referred to as “Climate Positive 2040.” By comparing the plan’s goals with recommendations drafted by the Intergovernmental Panel on Climate Change (IPCC), the researchers identify potential limitations and gaps in the Climate Positive 2040 plan. Their findings indicate that while the plan outlines significant policy changes, it does not adopt more than half of the IPCC Assessment Report Mitigation of Climate Change’s recommendations.
In the sixth study, “Critically Examining the Distribution of Hazardous Waste Facilities in Salt Lake County,” Sharah Meservy and Alex Bond use secondary data published by the Environmental Protection Agency (EPA) to examine how environmental racism – a widely accepted and documented phenomenon throughout the U.S. – plays out in the Salt Lake region. Specifically, the researchers examine whether large quantity generators and hazardous waste facilities in Salt Lake County are more likely to be located in communities of color and/or socioeconomically disadvantaged communities. Their review of the data returns seemingly clear evidence of environmental injustice in Salt Lake County and they conclude that until all residents have access to clean and equitable living conditions, regardless of their skin color or financial condition, “environmental justice will not exist in Salt Lake County” (pg. 135).
Finally, in the seventh study, “Invisible Disabilities and Workforce Experiences After Post-Secondary Education,” Christine Behle and Sam F. R. Roberts engage four in-depth interviews in a critical interpretive examination of the lived experiences of individuals with invisible disabilities. They pursued two research questions: What barriers and other experiences do college graduates with invisible disabilities face in seeking employment and workplace accommodations? How do these experiences impact other aspects of their lives? Among the findings they report are reluctance to disclose disabilities for fear of stigma, unstable employment, and long periods of unemployment. The researchers further offer suggestions for future policy action, noting that by addressing the barriers faced by those with invisible disabilities, “we may unlock a new wealth of potential in those who have been previously denied access to full workforce participation” (pg. 168).
A Final Reflection
As indicated by the preceding chapter summaries, although the studies were “micro” in scope and objectives, they nonetheless generated valuable insights that arguably contribute to the localized knowledge within each study’s empirical sphere. In other words, the microstudies were successful not only as class assignments, but as research endeavors in their own rights. I end this introductory chapter with a final note on an ingredient I suspect was foundational in this success—the latitude to undertake calculated risks afforded by an academic context of curricular and instructional safety. I attempted to foster such a context by engaging with students’ work as an advisor and champion, rather than an evaluator. For example, instead of assigning grades, I focused on feedback, dialogue, and development.[4] Although the success exhibited in the chapters that follow is most certainly the a direct result of their authors’ commitment and hard work, I am convinced that a pedagogical approach focused on fostering safety, understanding, and support also played a key role.
References
American Public Media (APM). (2020). The color of coronavirus: COVID-19 deaths by race and ethnicity in the U.S. APM Research Lab. Retrieved November 29, 2020 from https://www.apmresearchlab.org/covid/deaths-by-race
Associated Press (AP). (2020). Retrieved November 29, 2020 from https://www.ap.org/media-center/understanding-the-election
Behle, C. & Roberts, S.F.R. (2020). Invisible disabilities and workforce experiences after post-secondary education (pp 138-170). In D.P. Carter (Ed.). Inquiry of the Public Sort. Salt Lake City, UT: Public Sort Press.
COVID Tracking Project, The (CTP). (2020). U.S. historical data. Retrieved November 29, 2020 from https://covidtracking.com/data/national
Cushing, A., Cañar, A., & Facer, R. (2020). Impressions of teleworking among public sector employees during the COVID-19 pandemic (pp 28-48). In D.P. Carter (Ed.). Inquiry of the Public Sort. Salt Lake City, UT: Public Sort Press.
Burch, A.D.S., Cai, W., Gianordoli, G., McCarthy, M., & Patel, J.K. (June 13, 2020). How Black Lives Matter reached every corner of America. The New York Times. ISSN 0362-4331. Retrieved June 14, 2020.
de Santana Luz, L.A., Laguan, M., & Umba, R. (2020). Analyzing Salt Lake City’s Climate Positive 2040 plan through IPCC policy goals (pp 106-123). In D.P. Carter (Ed.). Inquiry of the Public Sort. Salt Lake City, UT: Public Sort Press.
Dillman, M. & Posvitak, C. (2020). COVID-19 and public transportation in Utah (pp 14-26). In D.P. Carter (Ed.). Inquiry of the Public Sort. Salt Lake City, UT: Public Sort Press.
Fujii, L. A. (2012). Research ethics 101: Dilemmas and responsibilities. PS: Political Science & Politics, 45(4), 717-723.
Lastre, A.F., Garz, M., Garcia, S., & Greenwell, T. (2020). Educational formats in Utah K-12 schooling during the COVID-19 pandemic (pp 49-66). In D.P. Carter (Ed.). Inquiry of the Public Sort. Salt Lake City, UT: Public Sort Press.
Ledford, S., VanLeeuwen, K., & Waechtler. L. (2020). A critical look at how the ACA’s contraceptive mandate impacted access to women’s reproductive health services (pp 67-105). In D.P. Carter (Ed.). Inquiry of the Public Sort. Salt Lake City, UT: Public Sort Press.
Meservy, S. & Bond, A. (2020). Critically examining the distribution of hazardous waste facilities in Salt Lake County (pp 124-137). In D.P. Carter (Ed.). Inquiry of the Public Sort. Salt Lake City, UT: Public Sort Press.
Riccucci, N.M. (2010). Envisioning public administration as a scholarly field in 2020: Rethinking epistemic traditions. Public Administration Review, 70, S304-S306.
Smith, S., Wu, J., & Murphy, J. (June 9, 2020). Map: George Floyd protests around the world. NBC News. Retrieved November 29, 2020 from https://www.nbcnews.com/news/world/map-george-floyd-protests-countries-worldwide-n1228391
- The students enrolled in this class came from across the Master of Public Administration (MPA), Master of Public Policy (MPP), and Master of Science in International Affairs and Global Enterprise (MIAGE) programs. Given the subject of the course in question (research design), I find it necessary to note that the term “experiment” is used here in the popular sense, as an adventure or trial, rather than the more technical use of the term in reference to experimental research. ↵
- COVID-19 related death rates in the U.S. include: 1 in 875 Black Americans, 1 in 925 Indigenous Americans, 1 in 1,275 Latino Americans, 1 in 1,325 Pacific Islander Americans, 1 in 1,625 White Americans, and 1 in 2,100 Asian Americans (APM, 2020) ↵
- Although students were encouraged to include their microstudy in this book, they were not required to do so; groups could omit their study entirely or individual authors could omit their names/use a pseudonym. ↵
- I "ungraded" my courses to the extent allowed by the University of Utah in Fall 2020, inspired by the more humane, inclusive, equitable, and (I suspected) effective approach of focusing on qualitative engagement and feedback. For those interested in the approach, I suggest starting with Jesse Stommel's (un)grading blog posts at jessestommel.com. I should further note that Jesse Stommel and Sean Michael Morris' (seanmichaelmorris.com) joint work also helped inspire this book. ↵