15.1 Evaluation research

As you may recall from the definition provided in Chapter 1, evaluation research is research conducted to assess the effects of specific programs or policies. Evaluation research is often used when some form of policy intervention is planned, such as welfare reform or school curriculum change. The focus on interventions and social problems makes it natural fit for social work researchers. Evaluation research might be used to assess the extent to which intervention is necessary by attempting to define and diagnose social problems in a social worker’s service area. It might also be used to understand whether their agency’s interventions have produced their intended outcomes.

I often remind my students they will eventually have bright ideas about what programs or interventions their agency should try. Moreover, they will eventually be so good at their job they will take on additional administrative and supervisory responsibilities. As a result, you will need to prove to your agency and the people who fund your agency that your interventions are successful. Government and private grants normally require outcomes to be measured and reported to maintain funding.

An outcomes assessment is an evaluation designed to discover if a program achieved its intended outcomes. Much like all of research, it comes with its own peculiar terminology that resembles an assembly line at a factory (Engel & Schutt, 2016). ^[1] Inputs are the resources needed for the program to operate. These include physical location, any equipment needed, staff (and experience/knowledge of those staff), monetary funding, and most importantly, the clients. Program administrators pull together the necessary resources to run an intervention or program. The program is the intervention your clients receive—perhaps giving them access to housing vouchers or enrolling them in a smoking cessation class.

The outputs of programs are tangible results of the program process—i.e., the boring things that come out of your program. Outputs in a program might include the number of clients served, staff members trained to implement the intervention, mobility assistance devices distributed, nicotine patches distributed, etc. By contrast, outcomes speak to the purpose of the program itself.

Outcomes are the observed changes, whether intended or unintended, that occurred due to the program or intervention. By looking at each of these domains, evaluation researchers can obtain a comprehensive view of the program.

Let’s run through an example from my wife’s social work practice. She runs an after-school bicycling club called Pedal Up for children with mental health issues. She has a lot of inputs in her program. First, there are the children who enroll, the volunteer and paid staff members who supervise the kids (and their knowledge about bicycles and children’s mental health), the bicycles and equipment that all clients and staff use, the community center room they use as a home base, the paths of our city where they ride their bikes, and the public and private grants they use to fund the program. Next, the program itself is a twice weekly after-school program in which children learn about bicycle maintenance and bicycle safety for about 30 minutes each day and then spend at least an hour riding around the city on bicycle trails.

There are many ways to measure the outputs of this program. She would probably include the number of children and staff participating in the program or the number of bike rides or lessons given. Other outputs might include the number of miles logged by the children over the school year, the number of bicycle helmets or spare tires distributed, etc. Finally, the outcomes of the program might include providing surveys to family members or teachers assess whether each child’s mental health symptoms have improved. Outcomes may also be measured by counting the number of behavioral issues at school or by conducting a child-friendly survey with the children themselves.

Outcomes assessments are performed at the end of a program or at specific points during the grant reporting process. What if a social worker wants to see if their program is on target to achieve its outcomes during the early stages of implementation? In this case, a process assessment is recommended, which evaluates a program in its earlier stages. Faulkner and Faulkner (2016) ^[2] describe three main goals for conducting a process evaluation.

The first is program description, in which the researcher simply tries to understand what the program looks like in everyday life for clients and staff members. To assess program description in our Pedal Up example, we might measure the number of hours children spent riding their bikes or the number of children and staff in attendance during the first few weeks of the program’s implementation. This data will allow the individuals who are in charge of the program to see how their ideas have translated from the grant proposal to the real world. If child attendance is low or if they are only able to ride their bikes for ten minutes each day, then the assessment may indicate that something is wrong.

Another important goal of process assessment is program monitoring. If you have some social work practice experience already, it’s likely you’ve encountered program monitoring. Agency administrators may look at sign-in sheets for groups, hours billed by clinicians, or other metrics to track how services are utilized over time. They may also assess whether clinicians are following the program correctly or if they are deviating from how the program was designed. This can be an issue when conducting program evaluations of specific treatment models, as any differences between what the administrators conceptualized and what the clinicians implemented jeopardize the internal validity of the evaluation. In our Pedal Up example, we could utilize program monitoring to catch staff members who do not review bike safety each week or staff who do not consistently enforce helmet laws.

The final goal of process assessments is quality assurance. At its most simple level, quality assurance may involve sending out satisfaction questionnaires to clients and staff members. If there are serious issues, it’s better to know them early on so the program can be adapted to meet the needs of clients and staff. Not only is it important to solicit consumer feedback, but it is also imperative to ask for staff feedback. The staff have valuable insight into how the program works in practice and they can identify areas in which the may be falling short. In our example, we could spend some time talking with parents when they pick their children up from the program or hold a staff meeting to provide opportunities for those most involved in the program to provide feedback.

Evaluation research is a part of all social workers’ toolkits. It ensures that social work interventions achieve their intended effects. This protects our clients and ensures that money and other resources are not wasted on programs that do not work. Evaluation research uses the skills of quantitative and qualitative research to ensure clients receive interventions that have been shown to be successful.