18

It’s helpful to understand the tools and resources available to evaluate your online course below before you create your course evaluation plan. In order to practice continual course improvement, best practice suggests that you consider a variety of data regarding the effectiveness, efficiency, and appeal of your course. Read through the information provided on this page, then proceed to the Course Evaluation design task for this module.

Course Evaluation

Think back to the first time you finished teaching. Were you and your students completely satisfied with every aspect of the course? Probably not, and the next time, you tweaked this or that to make the course better.

Even though you have poured considerable time and effort into the design and development of your online course thus far, the instructional design process doesn’t stop when you make the course available to students. Best practice involves continuously evaluating the effectiveness of your course design and delivery in order to determine how the course can be improved. Continuous course improvement is built into the eCampus Quality Improvement Program through the use of Quality Matters Peer Reviews and thoughtful course evaluation plans.

In online education, some people tend to use the terms evaluation and assessment interchangeably. These terms have certain meanings depending on the context. We have assessment and evaluation of student learning, assessment, and evaluation, of course, design/delivery, assessment, and evaluation of a program, to name a few.

Generally, evaluation is distinct from assessment in that evaluation makes a value judgment of some kind. Conceicao (2006) offers these definitions:

• Assessment is used to measure learning gains with the purpose of influencing the learning environment or formulating related policies.
• Evaluation focuses on gathering and analyzing data for decision-making and accountability.

This definition of evaluation suggests that the purposes of evaluations are to make high-level decisions about the course or to report success in achieving accountability goals. However, Willis (1993) suggests that evaluations can serve other purposes, as well. Willis states that evaluations can help us understand many things about a course, including the following:

• Student comfort with the method used to deliver the distant instruction
• Appropriateness of assignments
• Clarity of course content
• If class time is well spent
• Teaching effectiveness
• How a course can be improved

We can use these measures to improve a course in three general areas: course effectiveness, course efficiency, and course appeal. In this learning activity, you will have the opportunity to think about the many purposes you have for evaluating your own course. You will also be able to document some design decisions that will allow you to build the course in a way that makes the evaluations well-integrated into the teaching and learning activities that happen throughout the semester.

Formative and Summative Evaluations

The Student Assessment Plan you developed in module 02 focuses on measures of student performance–how student grades may be earned. The Course Evaluation Plan you will develop in this module focuses on measures of course effectiveness, efficiency, and appeal–how well the course helps students achieve the learning objectives within the time and resource constraints, and how much the students enjoy the course. Your Course Evaluation Plan is how you will determine what “grade” you would give the course. Your Course Evaluation Plan may include feedback about course quality from various stakeholders, including students, teaching assistants, faculty in your department, the department chair, and other faculty on campus. Consequently, you may want to share some evaluation results with your students and explain how their input helped improve the course. Likewise, your fellow faculty and your department chair can benefit from hearing about how their feedback resulted in improvements to the course. The end result of collecting and sharing evaluation data may include higher achieving students and increased collegiality with your peers.

Formative Evaluations

Formative evaluations provide data that help you diagnose and solve instructional problems while the course is being taught. According to Hattie’s (2009) research on over 800 meta-analyses covering over 50,000 studies on variables related to student achievement, one of the most effective things we can do to increase student achievement is to help instructors use formative feedback about student achievement to improve instruction. The effect size for providing formative evaluation feedback to instructors is 0.90, which is a 31% gain in student scores when instructors use this strategy. It’s incredibly valuable to identify ways that you will evaluate the effectiveness, efficiency, and appeal of the instructional activities in your course in a formative manner.

Summative Evaluations

Summative evaluations typically occur at the mid-term and end of the course, and the data is used to determine how well the course met the goals set for it by you and your department. The following table provides a more detailed illustration of the differences between formative and summative evaluations.

INSERT TABLE

Boise State Online Course Evaluations

You may have noticed that one example of a summative assessment is the university’s online course evaluation. Since all online courses are evaluated at least once a year, it is important for you to understand the procedure followed for the administration of that particular evaluation. Please select the link below to read more about the online course evaluations.

eCampus Faculty Guidelines: Course Evaluations

If you would like more information about the online course evaluations, including instructions on how to add your own questions to your course evaluation, select the link below.

Boise State: Online Course Evaluations

The online course evaluation is a survey that is made available to students at the end of a course. Most courses are evaluated every semester, although some departments have opted out of this process and have their own alternative evaluations. If your department has its own evaluation, you should review the questions on it so that you are familiar with the information gathered through that survey. If you do participate in the regular online course evaluation process, you will want to review the following questions. These questions will serve as an important source of information for your Course Evaluation Plan. The first set of questions are the standard questions that the university asks on all evaluations. The second set of questions are only asked for online courses.

University Questions for The Instructor

Rate on a scale of 1 to 5 with 1 being Strongly Disagree and 5 being Strongly Agree.

(Strongly Disagree – Disagree – Neutral – Agree – Strongly Agree)

1. 1. The instructor seemed prepared for class.
   2. The instructor fostered learning in this course.
   3. The assessment methods were clearly explained.
   4. The objectives of the course were clearly explained.

Tell us about this course

5. 1. Which aspects of this course were most valuable to your overall learning experience?
   2. What barriers to learning, if any, did you experience in this course?

Bronco Course Evaluation Questions for Online Courses

The eCampus Center approved the following 10 questions that are added to the online course evaluation surveys:

1. 1. While enrolled in THIS online course, I lived: (Please choose one)
   1. 1. on the Boise State campus
      2. in the Idaho Treasure Valley area
      3. in Idaho
      4. in WA, OR, NV, UT, WY, or MT
      5. in the greater United States
      6. in another country outside of the U.S.

Rate on a scale of 1 to 5 with 1 being Strongly Disagree and 5 being Strongly Agree.

Likert Scale (Strongly Disagree – Disagree – Neutral – Agree – Strongly Agree)

2. 1. I invested enough time and energy in this online course to meet/exceed course requirements.
   2. The structure of the content in this online course was easy to follow.
   3. I spent more time working on this online course than I have spent working on a comparable in-person course worth the same number of credits.
   4. Appropriate technical assistance was readily available.
   5. I was satisfied with the regular interactions with instructor
   6. I was satisfied with the regular interactions between classmates
   7. I was satisfied with the information made available prior to the course (course requirements, pre-requisites, technical skills, fees, etc.)

Online Course Experience

9. 1. What did you like most about your experience in this online course?
   2. What did you like least about your experience in this online course?

Instructors also have the option of writing their own survey questions to add to this course evaluation. Consequently, you may use this evaluation to get answers to questions that may help you fine-tune specific areas of your course. However, the Office of Institutional Research cautions faculty against making the course evaluation survey too lengthy. Note also that some departments or programs allow faculty to make changes on their own, and others prefer to see the additional questions beforehand and give approval, so check with your chair about your department’s practice.

Quality Matters Peer Review

As part of this seminar, your course will go through a Quality Matters Peer Review in which three Boise State faculty will review your course after it has been taught for at least one semester. During this review, they will evaluate how well the course design meets all of the Quality Matters Standards. These standards are discussed throughout this seminar, so they will not be listed here. However, you should be aware that this peer review can also serve as a source of information for your overall course evaluation plan.

The Course Evaluation Process

In addition to the regular course evaluations and the Quality Matters Peer Review, you may wish to focus attention on evaluating certain qualities or components of your course. You may plan this type of evaluation by using the following steps to collect and use data about your course.

Step 01. Specify the evaluation goals or the questions you want to answer.

Step 02. Prepare the data collection instruments and obtain any necessary resources.

Step 03. Collect data during course delivery.

Step 04. Analyze the data.

Step 05. Revise the course based on results from data analysis.

Step 06. Share the evaluation results and course revisions with stakeholders.

Step 07. Return to Step 01 to refine or revise the evaluation goals and begin the course evaluation process again.

This is an iterative process that may be repeated each semester the course is taught. By following these steps, you will be able to draft a Course Evaluation Plan that helps you practice continual course improvement.

Formative Evaluation Instruments

To get the most informative results while you are teaching, we suggest you use more than one data collection technique and several measures to obtain a clearer picture of how well a course (or program) is working. Remember that we are trying to provide information and feedback to an instructor about his or her teaching and about the course for the purpose of learning, developing, and improving the course as the course is being taught. Multiple methods are available to evaluate an online course, including the following:

• Mid-Semester Assessment Process (MAP) (a Boise State University service – more on this below)
• Midterm evaluation surveys
• Module surveys
• Student self-assessments
• Student observations/participation statistics
• Student-experience testing (e.g., usability testing of navigation)
• A sampling of student work
• Instructor self-assessments
• Faculty peer evaluations
• Focus groups
• Interviews
• Student journals
• Course completion rates

Mid-Semester Assessment Process (MAP)

The Mid-Semester Assessment Process (MAP) for Online Courses  s a service offered by eCampus Center and mirrors the MAP for courses taught on campus provided by the Center for Teaching and Learning. A MAP is a qualitative evaluation in which students respond anonymously to two questions:

1. What about this course and its instruction helps your learning?
2. What about this course and its instruction presents a barrier to your learning?

Students are temporarily put into a new Blackboard course with only a discussion board. The students talk about the questions as a group with light moderation to keep them on task, then the results go through a qualitative analysis. Two or more reviewers code responses for themes, which are shared with you in consultation. Because the evaluation is performed in the middle of the course, there is an opportunity to make changes during the course itself, in addition to keeping the ideas in mind for the next time it is taught.

The MAP process can be a more rigorous process compared to a survey, in fact, results might give you ideas for future surveys. The goal of the process is to help you gain insight into the course, things that are going well, and what you might think about in terms of change. Go to the website MAP for Online Courses page to get more details.

Student Self-Assessments

There are at least three good reasons for using self-assessments in an online course.

1. Self-assessments provide instructors and students with a simple way to track (a) the completion of assigned tasks and/or (b) progress made toward mastery of the learning objectives.
2. Self-assessments promote learning by providing opportunities for students to reflect on, discuss, or defend their knowledge and skills (Merrill, 2009).
3. Self-assessments can help students develop the meta-cognitive skills needed to self-regulate their own learning. This includes the self-monitoring of thinking, motivation, and behavior (Bercher, 2012).

Because self-assessments provide so many benefits to students and instructors, we encourage you to talk with your instructional design consultant about the ways in which you may integrate self-assessments into your course.

Test and Survey Statistics

You can view statistical information about test and survey results from the Grade Center in Blackboard. For example, you can view what percentage of your students chose each multiple choice answer for one of your tests, identify persistent misconceptions and adjust your instruction, accordingly. A regular component of teaching this seminar is our weekly review of feedback provided by faculty in the module evaluations. To learn more, see Test and Survey Results provided by Blackboard Inc.

Midterm and Module Survey Tool Options

If you choose to build a survey, note that there are two primary tools in use at Boise State.

1. Surveys may be delivered with Blackboard’s built-in Survey tool, found under Tests, Surveys, and Pools.

Your instructional design consultant can build a pool of module and midterm survey questions in the Survey area of your Master course shell.

The main advantage of using a Blackboard survey is that students stay in the course to take it. Surveys are tracked in the Gradebook as complete/incomplete. Survey feedback (“Attempt Statistics”) is also accessed by instructors via the Gradebook. Blackboard survey responses are anonymous, but the system will let you know who has completed a survey, in case you are offering incentives.

2. You may prefer to use a software program called Qualtrics which the university licenses (free to you).

Qualtrics offers added flexibility in creating questions and has other robust reporting and design features.

Your Instructional Design Consultant can help you develop your survey questions in Qualtrics. You can then import the developed survey into your own Qualtrics account (the questions may still be modified). Once the survey questions are developed, data collection must be activated in order to obtain the web link that you provide to students. Clicking the link takes students to the Qualtrics site, where they will complete the survey.

Survey feedback is accessed via the Qualtrics account where the survey was activated. As there is no way to identify who has taken the survey, you will not know which students have completed the survey and which have not. Some instructors encourage survey completion with a point incentive for everyone in the class based on 80% of the class completing the survey.

Increasing Student Participation In Your Course Evaluations

If you are making an effort to elicit feedback, you certainly want to get students to participate! The Boise State Office of Institutional Research (n.d.) suggests several factors that can influence the response rates on Bronco CourseEvals and other measures:

• Faculty enthusiasm — If you do not show that students’ input matters to you, they are less likely to be motivated to give it. Let your students know why you are asking for their perspectives and how you plan to use the results.One Boise State instructor told us she always makes at least one change in her courses based on the mid-course evaluation, and then she reminds students of that change when it comes time to do the end-of-course evaluation.

“The key is to talk with your students about the importance you attach to course evaluations, why they are important to you, and examples of how you have used the feedback to improve your classes in the past. Students feel they are wasting their time filling out course evaluations if they believe the faculty member will not even look at them. This is a point made widely in the research literature, and experience bears out here at Boise State.” (Office of Institutional Research, n.d.)

• Incentives —Consider providing extra credit for participation in the evaluation process based on the overall response rate; for example, give 5 points to everyone if the class reaches 80% participation. Another incentive could be tied to students’ grades; perhaps offer to release final grades early to students who complete the course evaluation.
• Reminders — Based on experience from prior surveys at Boise State, students typically need four reminders (in the absence of incentives or other extra efforts) to get participation in the Bronco CourseEvals above 70 percent.
• Potential bias — Might a student’s ethnicity make him predisposed to ignore the survey? Maybe her age would make her more likely to participate? The Institutional Research office is gathering data to determine whether any demographic variables, including GPA, total credits, and gender, have an effect on student participation.

We suggest that you provide multiple means of informing the students about the evaluation opportunities, perhaps in the following ways:

• Course announcement — If the evaluation is housed within Blackboard, use the Course Link feature within the Announcement to point students to the right location. If you are using the university Bronco CourseEval survey, students will receive the link in an email; make sure to post that link within Blackboard as well.
• Module checklist — Post information about both formative and summative evaluation opportunities within the module checklist when the evaluation will take place.
• Link within a module — Consider providing a link/reminder within the actual module, again using the Course Link feature if appropriate.

Who usually participates in course evaluations? Weimer (2012) summarizes a research report by Samer Kherfi titled Whose Opinion Is It Anyway? Determinants of Participation in Student Evaluation of Teaching which suggests that end of course evaluations are biased in favor of the course and instructor.

Optional Evaluation Resources

Your instructional design consultant can help you to develop the tools to evaluate your course’s effectiveness, efficiency and appeal. The information and resources below provide a rich resource of evaluation ideas that you may use if you choose to create something fresh. You may wish to start with the evaluation tools provided by your instructional design consultant, then if you have time, add to or modify those tools using these optional resources, as needed.

Evaluation Goals for Online Course Design

You can establish goals for evaluation in regard to the design of your course by thinking about the aspects of course design that we have explored in this seminar as well as the Quality Matters Standards. Many of these goals will be evaluated by Boise State colleagues during the Quality Matters Peer Review. We’ve discussed the value of instructor-student interaction, for example. An evaluation goal may be “To assess the perceived quality and value of instructor feedback”. Lynch (2001) cites Moore and Kearsley’s 12 general principles to consider when evaluating any course design. Some of these are assessed through student feedback, others through analysis of course components:

1. Good structure — Course materials must be well defined and display internal consistency among different parts of the course.
2. Clear objectives — Identify suitable learning experiences and subsequent evaluation.
3. Small unit — The content and course organization should be presented in small units, preferably that correspond to a single instructional objective or learning activity.
4. Planned participation — Opportunities for student interaction should be embedded throughout the course materials.
5. Completeness — Extensive commentary or examples should be provided.
6. Repetition — Important ideas are reinforced to compensate for distractions and memory limitations.
7. Synthesis — Important ideas are woven together (usually in summaries).
8. Stimulation — Materials capture and hold the attention of students through varied formats, content, or guest participation.
9. Variety — Format and media variety are present to appeal to student interests, backgrounds, and learning styles.
10. Open-ended — Assignments, examples, and problems allow students to adapt the content to their own situations.
11. Feedback — Regular feedback is provided on assignments and student progress in the course.
12. Continuous evaluation — Effectiveness of materials, media, and instructional strategies are routinely assessed using a variety of methods.

Evaluation Goals for Online Course Delivery

The focus of this seminar has been on course design and development. Course evaluation feedback will also reflect how well a course is delivered. (In addition to this seminar, eCampus offers a six week Teaching Online Seminar where participants explore aspects of course delivery that influence students’ experience of course quality). In any case, to evaluate course delivery goals, consider input from Savery (2005), who encourages online instructors to be VOCAL:

Visible
Organized
Compassionate
Analytical
Leader-by-Example

Chickering and Gamson’s well-known seven principles of good practice in undergraduate education (1987) can also serve as a framework from which to evaluate online course delivery. Good practice:

1. Encourages contacts between students and faculty.
2. Develops reciprocity and cooperation among students.
3. Encourages active learning.
4. Gives prompt feedback.
5. Emphasizes time on task.
6. Communicates high expectations.
7. Respects diverse talents and ways of learning.

Examples

Online Course Evaluation Survey and Instructor Self-Assessment and Satisfaction Survey – Examples from Lawrence Technological University written for face-to-face but adaptable for online

Designing and Delivering Effective Online Nursing Courses – An article by Bangert and Easterby that includes the survey instruments used to assess the effectiveness of course design and delivery based on Chickering and Gamson’s seven principles of effective teaching (see Table 1)

Formative Class Evaluation (PDF download) and End-of-Semester Survey (PDF download) – From David Jaffee of the University of North Florida, examples created for the traditional classroom but adaptable for online

Surveys in Blackboard  – Link to Blackboard survey information
Qualtrics – Link to Boise State Qualtrics survey information