2

Once you have established the goals and objectives with your client or client system, you can begin to design your practice evaluation approach. This process involves choosing what are known as process and outcome measures. These days, social workers and the organizations they work for are mostly focused on client outcomes – for which we choose outcome measures, or ways to measure the results of the intervention. In other words, we want to know how the client is doing at the end of the intervention phase or after. Outcomes are sometimes split into outcomes (short term result) and impacts (long term result). For example, an ideal outcome of mental health treatment might be reductions in psychological symptoms, but an impact might be longer-term employability or happiness.

Outcome information is vitally important, but it misses out on something vital, what goes on during the process of the intervention being delivered by the social worker. Sometimes our clients do well. Sometimes our clients don’t do well. And sometimes our clients do well for a bit, and then things drop off. If we only focus on studying client outcomes, we are not likely to have a good sense of what it might have been about our intervention that might have contributed to the outcome. Of course, other factors can come into play regarding how outcomes evolve, but in order to be ethical practitioners, we need to take a look at our own work as well – and we can do this by examining treatment fidelity, as discussed in the previous chapter. Studying treatment fidelity also “ticks the box” with respect to the goal of being reflective and reflexive practitioners.

Given all this, you need to know that practice evaluation includes both process measures (about what happens during intervention) and outcome measures (about what happens at or after the intervention is finished). Let’s consider some examples to make this feel a bit more real. Process and outcome measures need to be valid and reliable (remember, these are two words that do not mean the same thing). See Figure 2.1.

Figure 2.1 Depiction of how the process and the outcome relate to one another

Let’s go back to the mental health social worker we spoke about before. We talked about how a common client goal might be “improvement of mental health symptoms” for a social worker in a mental health setting. That’s the broad wish area, so to speak. Breaking down the goal into measurable objectives might include items such as level of anxiety or frequency of irritability. Both of these objectives could be measured during the process of the therapeutic intervention in order to track the client’s progress. These objectives could also be considered at or after the client completed treatment, this time as outcome measures. The purpose of collecting process information is different than the purpose of collecting outcome information.

The collection of process data often helps us to tweak our intervention by telling us about whether our efforts to date are beginning to help the client make changes or not. If clients are exhibiting symptoms similar to what was measured at the start of treatment, this could indicate that the intervention may not be kicking in yet, may not be going according to plan or that another factor may be influencing the situation.

The collection of outcome data is most often used as a form of accountability. Funders often require information on client outcomes in order to determine whether they should continue to fund a particular service or organization. While this is often the first thing people think about when talking about outcome measures, social workers can use outcome measures to reflect on their body of work as well.

Let’s also go through our macro case example in thinking about process and outcome measures. As you will recall, your home city wants to improve citizens’ feelings of community pride and has hired a social work community organizer to achieve this goal over a two-year period. The goal of improving citizens’ feelings of community pride is the desired outcome. The outcome could be broken down into several measures, such as whether people intended to continue their residence in the community or whether they felt pride in the local park. There are likely many other ways to measure community pride of course, these are just a few.

Evaluating the process of this macro social work intervention might involve a range of things. A social worker engaged in regular contact with the community about the community pride initiative could look at recording the number of people who attended regular community planning meetings related to this effort. This would tell us about community engagement, which could be related to community pride. Another process measure might focus on how community members respond to a set of new murals painted on abandoned buildings, for example, or to new public art placed in several parks. In this scenario, process data tells you if you are on track towards your goal or not and helps you to correct your course if necessary.

So, we’ve walked through a micro and a macro example of practice evaluation involving process and outcome measures. Now, let’s revisit our child welfare social worker who is supporting a family’s efforts to better parent their children. As you will recall, the social worker is using the clinical technique known as motivational interviewing and she needs to know whether she is using the intervention’s parts according to the original evidence-supported model.

In this case, the social worker is mostly interested in process evaluation as her focus is on her own behavior during the course of the intervention. In lay terms, she is essentially asking “am I doing this motivational interviewing thing right?” Process measures can be used to inform practice – mainly to identify areas in need of further training.

There are four processes used in the motivational interviewing technique. They include engaging, focusing, evoking and planning. These processes are used by therapist to assist in conversations about change with clients. How well someone is engaging in any given process, or how often, could be useful information in an ongoing learning/training situation.

Now, let’s shift gears and walk through an example of a fully designed practice evaluation in order to understand the relationship between goals, objectives, process measures and outcome measures. In this scenario, we are focusing on a program treating adults with mild to moderate intellectual disabilities (IDD) living in community-based settings who have substance use problems. A little bit of biopsychosocial context will help with our example.

Use of alcohol and/or drugs by members of this population may be riskier than such use would be for people without IDD. Risk may be inflated for three reasons. First, the prevalence of prescribed psychotropic medications in this population is over 60 percent and the dangers of interactions between alcohol and/or drugs and psychotropic medications are well-known. Second, adults with IDD are at increased risk for victimization of a variety of types, a likelihood that might be exponentiated by alcohol or drug use. Third, adults with IDD are less likely to be able to identify when they have an alcohol and/or drug problem, or to be able to determine where to seek help for such a problem.

Social service providers report that while the prevalence of alcohol and/or drug problems in this population is low, when this problem manifests, the health and social service systems are often stymied with respect to how to address this need (Slayter, 2006). Based on these risk factors alone, evaluating a substance use disorder treatment program for adults with IDD is vital as there are only three studies that explore any kind of intervention for this population

The organization known as ‘A New Leaf’ is an outpatient behavioral health clinic specializing in work with adults with IDD. In response to clinical need, the clinical team has devised an intervention for adults with IDD who have substance use disorders.

This practice evaluation targets all 30 adult clients with IDD with suspected alcohol and/or drug use problems receiving services at A New Leaf between October 2019-April 2020. This evaluation’s stakeholders include adult clients with IDD and suspected alcohol and/or drug use problems, parents and caregivers of this population, members of the clinical and administrative teams at A New Leaf and any clinician in need of guidance for working with this specific population. Adult clients with IDD will be the study’s participants about whom and from whom we will collect data.

The intervention provided by A New Leaf consists of individual therapy and group psychoeducation for clients and a support group for parents and other caretakers (during which data collection will also occur).

The individual therapy is informed by a modified version of cognitive behavioral therapy (CBT) and motivational interviewing (MI). Clients receive weekly individual therapy, and group psychoeducation three times per week for a total of six months. This program has two goals, with several objectives “under” each goal, as follows:

Goal 1: To lessen substance use in the study population

Objective 1a: To reduce clients’ substance use-related problems score.

Objective 1b: To reduce total days of binge drinking each month

Objective 1c: To reduce the number of alcohol or drug-related incident reports in each client’s group home

Objective 1d: To reduce the number of times a client vomits due to alcohol and/or drug use each month

Objective 1e: To reduce the number of times a client passes out due to alcohol and/or drug use each month

Goal 2: To foster the ability for clients to use refusal skills in situations in which they are presented with opportunities to drink and/or use drugs.

Objective 2a: To increase clients’ capacity to link current actions with future consequences.

Objective 2b: To help clients identify alternative strategies to use when offered opportunities to drink and/or use drugs.

Objective 2c: To increase clients’ capacity to role play a refusal scenario to automaticity

Objective 2d: To increase clients’ self-efficacy

Now that we have identified our clinical goals and objectives, we need to think about how to collect information about the process and outcome of the treatment process at A New Leaf.

Parents and caregivers will be providing observational data about behaviors throughout the study. The clinical team will be implementing the intervention as well as collecting data through the use of standardized instruments that are valid and reliable. The organization’s administrative team will also be conducting the data collection, data analysis and data reporting. This evaluation will be structured to track data over time (i.e. ‘longitudinally’), with a baseline measure, weekly process measures, and outcome measures in the short and long-terms.

Let’s start with the process measures. These will be measured weekly and/or monthly in order to see how the process of the intervention is going:

Weekly individual treatment fidelity checklists to be completed by the clinical team in group supervision

Thrice-weekly group checklists to be completed by observer of group psychoeducation session to see if curriculum is being followed (i.e. treatment fidelity)

Days of binge drinking each month (parent or caregiver report)

Instances of vomiting due to alcohol and/or drugs each month (parent or caregiver report)

Instances of passing out due to alcohol and/or drugs each month (parent or caregiver report)

Numbers of alcohol and/or drug-related incident reports each month (parent or caregiver report)

Aside from the first treatment fidelity measure that uses a standardized instrument, all other process measures are unstandardized behavioral observations.

Now, let’s consider the outcome measures. These will be measured at baseline and end of treatment for comparison purposes, as well as in the short term and long term after discharge from the program in order to track progress over time.

You will notice that the clinical objectives have become the outcome measures, but with the addition of specific information about how the concept will be captured or measured. The first two outcome measures use standardized instruments and the rest use unstandardized behavioral observations.

Score on AUDIT scale (standardized measurement of substance use-related problems) (Smith, 1987)

Score on a self-efficacy scale (standardized measurement of self-efficacy) (Bandura, 1999)

Successful connection of current actions to future consequences in role play (clinician report)

Ability to name three alternative strategies/refusal skills in role play done in group home (parent or caregiver report)

Ability to name three alternative strategies/refusal skills in role play done in the clinic (clinician report)

Ability to role play a refusal scenario to automaticity in the clinic (clinician observer coding sheet)

Ability to role play a refusal scenario to automaticity in the group home (parent or caregiver observer coding sheet)

Days of binge drinking each month (parent or caregiver report)

Instances of vomiting due to alcohol and/or drugs each month (parent or caregiver report)

Instances of passing out due to alcohol and/or drugs each month (parent or caregiver report)

Numbers of alcohol and/or drug-related incident reports each month (parent or caregiver report)

At the conclusion of the practice evaluation, we will have helpful information that will tell us whether we achieved our goals and objectives fully or partly. We will also have treatment fidelity information. If we have only partly achieved our objectives, we will be able to have a sense of what specific part of the intervention we need to ‘tweak’ for improvement. Practice evaluation findings will also be used to inform clients, parents, funders and other stakeholders about the program’s efficacy.

In summary, the conduct of practice evaluation can include the collection of process data and/or outcome data. Clinical goals and objectives are used to develop process and outcome measurements, sometimes referred to as measures. Each type of data tells the social worker different types of information.

Outcome evaluation is used more than process evaluation, but the latter can be very helpful in assisting social workers in understanding how the process of intervention is going especially regarding treatment fidelity. Learning how to identify discrete process and outcome measures that are related to intervention goals and objectives is a key skill for social workers to master.

Discussion questions for chapter 2:

• Explain the difference between process and outcome measures as they are used in social work practice evaluation.
• How are process measures most often used in social work?
• Think about your internship placement. If you were to engage in practice evaluation, what might some process and outcome measures be?
• How do process and outcome measures relate to the process of evidence-based practice as we have defined it in this primer?