35 Planning the Outcome Evaluation

Putting together a thoughtful outcome evaluation can provide such rich data for you, the participants in your intervention, your staff and volunteers, the advisory board, and your funders. It is an opportunity to answer questions such as:

• What evidence do we have that our strategies are appropriate?
• What evidence do we have that our strategies are working?
• What evidence do we have that our strategies need to be changed?

Two important concepts are necessary to understand as you plan your evaluation:

• Outcomes
• Outcome indicators

Outcomes

Outcomes of community change work are likely to be:

• Health outcomes (i.e. reduced levels of anxiety)
• Behavior (i.e. social connection)
• Attitude (i.e. happiness)
• Condition (i.e. violence reduction)
• Knowledge (i.e. awareness of child development needs)
• Status (i.e. improved economic well-being)

First, you need to be clear on what your anticipated outcomes are for your community change project or program. What difference are you trying to make? What is the benefit or change for the participants in your program? What may they know at the end of your workshop that they didn’t know before? How might they behave differently? How might a population at risk have improved safety after the implementation of your strategy?

Sample outcomes:

• Students developed new friendships
• People feel an increased sense of connection to the outdoors
• Families feel like they belong in their neighborhood
• Participants increased their economic vitality

Outcome Indicators

Outcome indicators are the observable, measurable characteristics or changes that represent the achievement of an outcome. In other words, how are you going to know if the outcomes were achieved? What evidence is most closely related to your intervention? What are you going to listen for? What are you going to look for?  What are you going to ask? Who are you going to ask?

Sample outcome indicators:

• When asked, students were able to name one new friend
• Participants identified examples of how they have increased their connection to the outdoors
• When surveyed, families identified a greater sense of belonging in their neighborhood
• A survey conducted before and after a program demonstrates an increase in economic vitality

Considerations for Evaluation Planning

Set up the evaluation systems before you begin the program

• The evaluation plan and systems of data collection need to be set up before beginning the program. This ensures that the data collection method can be used consistently and results in data that can be accessed at the time of data collection.
• This is especially important for programs that intend to measure goals and goal attainment, common with housing, health, independent living, recovery programs, or housing support programs. Be sure to set up:
  • a system of initial assessment
  • initial goal identification
  • ongoing documentation of needs, challenges, goal achievement, goal adjustment

Measuring change

• If your outcome indicators include any behavior change, affect, or condition, it is optimal to include measures pre and post-intervention. Often, this is done by including a pre and post-questionnaire that is distributed to participants at the beginning and end of a defined period of time (like one year).
• If you include any initial sign-up for your programming, the pre-program survey could be included.

Use existing assessment interactions to collect data

• For surveys and questionnaires, it is ideal if you can modify data collection tools, such as existing satisfaction questionnaires or initial assessment tools.
• For interviews, it is ideal if you can use times when you already interact with participants (like when they attend meetings or events).

Ensure access to data

• Avoid relying on other people for the data you need to collect so that you can control the evaluation schedule and process.
• Ensure that the data you identify as critical for your evaluation is data you have access to. For example, student grades or attendance are typically unavailable unless they are provided by a parent.
• Ensure that you have permission to gather the data you intend to collect. This is first and foremost considering the people represented by the data.

Ensure the indicator you are planning to measure is directly related to your intervention

• Programs often measure what they think they should rather than the indicator that is most closely correlated to their strategy.
• This is particularly the case with programs focused on children. Programs often want to measure a change in grade, school attendance, etc. However, you should only measure those if they are directly related to your work. If not, you cannot say that the change was a result of your work.

Schedule and communicate the evaluation plan

• Everyone who will be involved in the evaluation of your program needs to be informed about the timing and their role.
• This is particularly important for staff and program participants.

Sample Evaluation Format

As you plan your evaluation, you want the format to be clear and easy to follow, since you and your coworkers are likely to be responsible for implementing the evaluation. Below is an example directed at evaluating a community education program focused on reducing social isolation in older adults.