9.1 Empirical vs. ethical questions

Writing a good research question is an art and a science. It is a science because you have to make sure it is clear, concise, and well-developed. It is an art because often your language needs “wordsmithing” to perfect and clarify the meaning. This is an exciting part of the research process; however, it can also be one of the most stressful.

Creating a good research question begins by identifying a topic you are interested in studying. At this point, you already have a working question. You’ve been applying it to the exercises in each chapter, and after reading more about your topic in the scholarly literature, you’ve probably gone back and revised your working question a few times. We’re going to continue that process in more detail in this chapter. Keep in mind that writing research questions is an iterative process, with revisions happening week after week until you are ready to start your project.

Empirical vs. ethical questions

When it comes to research questions, social science is best equipped to answer empirical questions—those that can be answered by real experience in the real world—as opposed to ethical questions—questions where people have moral opinions and which may not be answerable in reference to the real world. While educators have explicit ethical obligations (regarding teaching and students), research projects ask empirical questions to help actualize and support the work of upholding those ethical principles.

In order to help you better understand the difference between ethical and empirical questions, let’s consider a topic about which people have moral opinions. How about SpongeBob SquarePants?^[1] In early 2005, members of the conservative Christian group Focus on the Family (2005)^[2] denounced this seemingly innocuous cartoon character as “morally offensive” because they perceived his character to be one that promotes a “pro-gay agenda.” Focus on the Family supported their claim that SpongeBob is immoral by citing his appearance in a children’s video designed to promote tolerance of all family forms (BBC News, 2005).^[3] They also cited SpongeBob’s regular hand-holding with his male sidekick Patrick as further evidence of his immorality.

So, can we now conclude that SpongeBob SquarePants is immoral? Not so fast. While your mother or a newspaper or television reporter may provide an answer, a social science researcher cannot. Questions of morality are ethical, not empirical. Of course, this doesn’t mean that social science researchers cannot study opinions about or social meanings surrounding SpongeBob SquarePants (Carter, 2010).^[4] We study humans after all, and as you will discover in the following chapters of this textbook, we are trained to utilize a variety of scientific data-collection techniques to understand patterns of human beliefs and behaviors. Using these techniques, we could find out how many people in the United States find SpongeBob morally reprehensible, but we could never learn, empirically, whether SpongeBob is in fact morally reprehensible.

Let’s consider an example. Say a student group wanted to research funding for on reserve indigenous schools and their original research question was: “How can funding for on-reserve indigenous schools be so much lower than the funding for provincial schools?” Outside of the research context, that is a darn good question! It speaks to how colonial systems have disadvantaged indigenous students over time.

Unfortunately, it is an ethical question, not an empirical one. To answer that question, you would have to draw on philosophy and morality, answering what it is about human nature and society that allows such unjust outcomes. However, you could not answer that question by gathering data about people in the real world. If I asked people that question, they would likely give me their opinions about school spending, rural and urban education, affordability, colonialism, and student educational needs. But I wouldn’t get the real answer about why our society tolerates such an imbalance in support for education.

Instead, as you worked on the project the research question would hopefully become more empirical as you read more empirical articles about the topic. One option that might be to ask, “How does provincial funding in schools near indigenous reserves compare to the funding provided to the on reserve school?” Another option might be to ask, “What is the cost of educating students on and off reserve, and how do funding allocations align (or nor) with those costs?” These projects would address the ethical question of funding inequity but do so in a way that gathered and analyzed empirical real-world data. Our job as researchers is to gather social facts about issues, not to judge or determine morality.

9.2 Characteristics of a good research question

Now that you’ve made sure your working question is empirical, you need to revise that working question into a formal research question. So, what makes a good research question? First, it is generally written in the form of a question. To say that your research question is “indigenous funding” or “school violence” or “student assessment” would not be correct. You need to frame your topic as a question, not a statement. A good research question is also one that is well-focused. A well-focused question helps you tune out irrelevant information and not try to answer everything about the world all at once. You could be the most eloquent writer in your class, or even in the world, but if the research question about which you are writing is unclear, your work will ultimately lack direction. In particular, be wary of conjunctions that may conflate variables and embed assumptions in your question.

In addition to being written in the form of a question and being well-focused, a good research question is one that cannot be answered with a simple yes or no. For example, if your interest is in gender norms, you could ask, “Does gender affect a person’s performance in school?” but you will have nothing left to say once you discover your yes or no answer. Instead, why not ask, about the relationship between gender and school performance. Alternatively, maybe we are interested in how or to what extent gender affects a person’s contributions in small group assignments? By tweaking your question in this small way, you suddenly have a much more fascinating question and more to say as you attempt to answer it.

A good research question should also have more than one plausible answer. For example, achievement may be impacted by student stereotyping; by teacher stereotyping; or by assessment type (among other issues). Thinking through the possible relationships between gender and achievement (and even the appropriate definition of the terms gender and achievement) would help a researcher realize that there were many plausible answers to questions about how gender affects a person’s academic achievement. Because gender doesn’t exist in a vacuum, researchers need to consider other characteristics that work together, in this case with gender, to shape people’s behaviours, likes, and dislikes. By doing this, the researcher considers the third feature of a good research question–relationships between concepts. In this case we began with an interest in a single concept—achievement—by asking ourselves what other concepts (such as gender, stereotype, or assessment) might be related to our original interest, we were able to form questions that considered the relationships among those concepts.

We have one final component to consider. Education research questions must contain a target population. Our study would be very different if we were to conduct it on older adult learners or immigrants who just arrived in a new country. The target population is the group of people whose needs your study addresses. Maybe the we noticed issues with achievement as part of our teaching practice with first-generation immigrants, and so we made it our target population. Maybe we wants to address the needs of another community (like fifth grade students in our school). Whatever the case, the target population should be chosen intentionally.

In sum, a good research question generally has the following features:

• It is written in the form of a question
• It is clearly written
• It cannot be answered with “yes” or “no”
• It has more than one plausible answer
• It considers relationships among multiple variables
• It is specific and clear about the concepts it addresses
• It includes a target population

9.3 Quantitative research questions

Quantitative descriptive questions

The type of research you are conducting will impact the research question that you ask. Probably the easiest questions to think of are quantitative descriptive questions. For example, “What is the average student debt load of education students?” is a descriptive question—and an important one. We aren’t trying to build a causal relationship here. We’re simply trying to describe how much debt education students carry. Quantitative descriptive questions like this one are helpful in education practice as part of community scans, in which researchers survey the various needs of the community they serve. If the scan reveals that the community requires more services related to education, governing bodies like the province or school district can use the community scan to create new programs that meet a defined community need (for example, adult education; pre-school services; after-school services).

Quantitative descriptive questions will often ask for percentage, count the number of instances of a phenomenon, or determine an average. Descriptive questions may only include one variable, such as ours about student debt load, or they may include multiple variables. Because these are descriptive questions, our purpose is not to investigate causal relationships between variables. To do that, we need to use a quantitative explanatory question.

Quantitative explanatory questions

Most studies you read in the academic literature will be quantitative and explanatory. Why is that? If you recall from Chapter 2, explanatory research tries to build nomothetic causal relationships. They are generalizable across space and time, so they are applicable to a wide audience. The editorial board of a journal wants to make sure their content will be useful to as many people as possible, so it’s not surprising that quantitative research dominates the academic literature.

Structurally, quantitative explanatory questions must contain an independent variable and dependent variable. Questions should ask about the relationship between these variables. The standard format I was taught in graduate school for an explanatory quantitative research question is: “What is the relationship between [independent variable] and [dependent variable] for [target population]?” You should play with the wording for your research question, revising that standard format to match what you really want to know about your topic.

Let’s take a look at a few more examples of possible research questions and consider the relative strengths and weaknesses of each. Table 9.1 does just that. While reading the table, keep in mind that I have only noted what I view to be the most relevant strengths and weaknesses of each question. Certainly each question may have additional strengths and weaknesses not noted in the table. Each of these questions is drawn from student projects in research methods classes and reflects the work of many students on their research question over many weeks.

Making it more specific

A good research question should also be specific and clear about the concepts it addresses. A student investigating gender and achievement knows what they mean by “achievement.” You likely also have an impression of what “achievement” means. But are your definition and the student’s definition the same? A participant in their study may think that achievement is best represented by GPA, but the researcher may be interested in scores on provincial exams. The only way to ensure your study stays focused and clear is to be specific about what you mean by a concept. The student in our example could pick a achievement item that was interesting to them or that the literature indicated was important—for example, literacy or numeracy. Or, the student could have a broader view of achievement, one that encompasses grades across courses and time (like GPA). Any option is probably okay, as long as the researcher is clear on what they mean by “achievement.” Clarifying these distinctions is important as we look ahead to specifying how your variables will be measured in Chapter 11.

Table 9.2 contains some “watch words” that indicate you may need to be more specific about the concepts in your research question.

It can be challenging to be this specific in education research, particularly when you are just starting out your project and still reading the literature. If you’ve only read one or two articles on your topic, it can be hard to know what you are interested in studying. Broad questions like “What are the causes of chronic homelessness, and what can be done to prevent it?” are common at the beginning stages of a research project as working questions. However, moving from working questions to research questions in your research proposal requires that you examine the literature on the topic and refine your question over time to be more specific and clear. Perhaps you want to study the effect of a specific reading program that you found in the literature. Maybe there is a particular model used to fight antisocial behaviours in school, and create a schoolwide feeling of community, like Caring School Community, that you want to investigate further. You may want to focus on a potential cause of homelessness such as LGBTQ+ discrimination that you find interesting or relevant to your school. As you can see, the possibilities for making your question more specific are almost infinite.

Quantitative exploratory questions

In exploratory research, the researcher doesn’t quite know the lay of the land yet. If someone is proposing to conduct an exploratory quantitative project, the watch words highlighted in Table 9.2 are not problematic at all. In fact, questions such as “What factors influence the academic achievement?” are good because they will explore a variety of factors or causes. In this question, the independent variable is less clearly written, but the dependent variable, academic achievement, is quite clearly written. The inverse can also be true. If we were to ask, “What outcomes are associated with small classes in elementary schools?”, we would have a clear independent variable, small classes, but an unclear dependent variable, outcomes. Because we are only conducting exploratory research on a topic, we may not have an idea of what concepts may comprise our “outcomes” or “factors.” Only after interacting with our participants will we be able to understand which concepts are important.

Remember that exploratory research is appropriate only when the researcher does not know much about topic because there is very little scholarly research. In our examples above, there is extensive literature on the outcomes in small classes and the factors that influence student achievement. Make sure you’ve done a thorough literature review to ensure there is little relevant research to guide you towards a more explanatory question if that is where your interests lie.

9.4 Qualitative research questions

Qualitative research questions differ from quantitative research questions. Because qualitative research questions seek to explore or describe phenomena, not provide a neat nomothetic explanation, they are often more general and openly worded. They may include only one concept, though many include more than one. Instead of asking how one variable causes changes in another, we are instead trying to understand the experiences, understandings, and meanings that people have about the concepts in our research question. These keywords often make an appearance in qualitative research questions.

Let’s work through an example from our last section. In Table 9.1, a student asked, “What is the relationship between school enrolment and adolescents in foster care?” In this question, it is pretty clear that the student believes that adolescents in foster are at a greater risk of school enrolment challenges than their peers not in foster care. This is a nomothetic causal relationship—foster care status causes changes in enrolment.

However, what if the student were less interested in predicting enrolment based on foster care status and more interested in understanding the stories of foster care youth who may be at risk for dropout? In that case, the researcher would be building an idiographic causal explanation. The youths whom the researcher interviews may share stories of how their foster families, peers, teachers, and others treated them. They may share stories about how they thought of their own academics ability and how it changed over time. They may have different ideas about what it means to transition out of foster care.

Because qualitative questions usually center on idiographic causal relationships, they look different than quantitative questions. Table 9.3 below takes the final research questions from Table 9.1 and adapts them for qualitative research. The guidelines for research questions previously described in this chapter still apply, but there are some new elements to qualitative research questions that are not present in quantitative questions.

1. Qualitative research questions often ask about lived experience, personal experience, understanding, meaning, and stories.
2. Qualitative research questions may be more general and less specific.
3. Qualitative research questions may also contain only one variable, rather than asking about relationships between multiple variables.

Qualitative research questions have one final feature that distinguishes them from quantitative research questions: they can change over the course of a study. Qualitative research is a reflexive process, one in which the researcher adapts their approach based on what participants say and do. The researcher must constantly evaluate whether their question is important and relevant to the participants. As the researcher gains information from participants, it is normal for the focus of the inquiry to shift.

For example, a qualitative researcher may want to study how a new truancy rule impacts youth at risk of expulsion. However, after interviewing some of the youth in their community, a researcher might find that the rule is actually irrelevant to their behavior and thoughts. Instead, their participants will direct the discussion to their frustration with the school administrators or the lack of job opportunities in the area. This is a natural part of qualitative research, and it is normal for research questions and hypothesis to evolve based on information gleaned from participants.

However, this reflexivity and openness is unacceptable in quantitative research for good reasons. Researchers using quantitative methods are testing a hypothesis, and if they could revise that hypothesis to match what they found, they could never be wrong! Indeed, an important component of open science and reproducability is the preregistration of a researcher’s hypotheses and data analysis plan in a central repository that can be verified and replicated by reviewers and other researchers. This interactive graphic from 538 shows how an unscrupulous research could come up with a hypothesis and theoretical explanation after collecting data by hunting for a combination of factors that results in a statistically significant relationship. This is an excellent example of how the positivist assumptions behind quantitative research and intepretivist assumptions behind qualitative research result in different approaches to social science.

9.5 Evaluating and updating your research questions

Feasibility and importance

As you are getting ready to finalize your research question and move into designing your research study, it is important to check whether your research question is feasible for you to answer and what importance your results will have in the community, among your participants, and in the scientific literature

Key questions to consider when evaluating your question’s feasibility include:

• Do you have access to the data you need?
• Will you be able to get consent from stakeholders, gatekeepers, and others?
• Does your project pose risk to individuals through direct harm, dual relationships, or breaches in confidentiality? (see Chapter 6 for more ethical considerations)
• Are you competent enough to complete the study?
• Do you have the resources and time needed to carry out the project?

Key questions to consider when evaluating the importance of your question include:

• Can we answer your research question simply by looking at the literature on your topic?
• How does your question add something new to the scholarly or professional literature? (raises a new issue, addresses a controversy, studies a new population, etc.)
• How will your target population benefit, once you answer your research question?
• How will the community, teaching practice, and/or the broader social world benefit, once you answer your research question?

Matching your research question and study design

This chapter described how to create a good quantitative and qualitative research question. In Parts 3 and 4 of this textbook, we will detail some of the basic designs like surveys and interviews that social scientists use to answer their research questions. But which design should you choose?

As with most things, it all depends on your research question. If your research question involves, for example, testing a new intervention, you will likely want to use an experimental design. On the other hand, if you want to know the lived experience of people in a public school, you probably want to use an interview or focus group design.

We will learn more about each one of these designs in the remainder of this textbook. We will also learn about using data that already exists, studying individual students in your class, and evaluating programs, which are other examples of designs. Below is a list of designs we will cover in this textbook:

• Surveys: online, phone, mail, in-person
• Experiments: classic, pre-experiments, quasi-experiments
• Interviews: in-person or via phone or videoconference
• Focus groups: in-person or via videoconference
• Content analysis of existing data
• Secondary data analysis of another researcher’s data
• Program evaluation

The design of your research study determines what you and your participants will do. In an experiment, for example, the researcher will introduce a stimulus or treatment to participants and measure their responses. In contrast, a content analysis may not have participants at all, and the researcher may simply read the marketing materials for a school or look at a politician’s speeches to conduct the data analysis for the study.

I imagine that a content analysis probably seems easier to accomplish than an experiment. However, as a researcher, you have to choose a research design that makes sense for your question and that is feasible to complete with the resources you have. All research projects require some resources to accomplish. Make sure your design is one you can carry out with the resources (time, money, staff, etc.) that you have.

There are so many different designs that exist in the social science literature that it would be impossible to include them all in this textbook. The purpose of the subsequent chapters is to help you understand the basic designs upon which these more advanced designs are built. As you learn more about research design, you will likely find yourself revising your research question to make sure it fits with the design. At the same time, your research question as it exists now should influence the design you end up choosing. There is no set order in which these should happen. Instead, your research project should be guided by whether you can feasibly carry it out and contribute new and important knowledge to the world.