Supplemental Types of Evaluation

Table of Contents | Previous Section | Next Section


While end of term student questionnaires continue to be the predominant type of instructor evaluation in most departments, various instructional needs often warrant the inclusion of additional evaluation strategies. The usefulness of an evaluation strategy depends on the desired level of detail necessary and the timeframe within which the data are required.The following section offers a range of techniques that vary from informal feedback strategies to help address current term teaching improvement, to more elaborate processes that garner in depth evaluative data for ongoing improvement.

Informal Feedback Strategies

Questioning

—A very simple tool for checking effective teaching is to incorporate specific questions within a lesson to gauge student understanding of the material. For example, an instructor may ask students to verbally answer a question similar to one that will be asked on an exam. This tool is more useful than simply asking if students have any questions because students who are confused may not be able to articulate their questions. Moreover, some students may falsely believe they understand the lesson and not ask questions. Checking for understanding within a lesson helps the instructor discover students’ level of learning and to make adjustments during the lesson itself.

Classroom Response Systems—

A problem with simple questioning is that an instructor generally will get a response from only one or two students rather than the entire class. This problem can be resolved with a few strategies that fall under the Classroom Response umbrella.

The first strategy is the easiest to implement. An instructor asks a multiple choice question or makes an agree/disagree statement about the material.Students indicate by the position of their thumb whether they believe the answer is A (upright), B (sideways), or C (downward) or Agree (upright) or Disagree (downward).The instructor can then quickly look around the room to determine how many students have the correct answer.

The second strategy involves the use of colored index cards.Its method is identical to the first strategy except that the instructor is using color coded cards for the responses.The advantage of using colored index cards is that they are easier to see than thumbs.

The third strategy involves the use of hand-held remote controls (“clickers”) to measure student responses.The technology is linked to software in a computer—either a laptop or a classroom computer—and can keep a record of student responses. Many instructors use this technology by imbedding the question into their presentation software. Both the instructor and students receive immediate feedback to the responses. In addition to the recordkeeping aspect of this system, a primary advantage of clickers is student anonymity in their responses in the classroom. A major disadvantage is the cost and performance reliability of the clickers themselves.

Open Class Discussion

—This technique can be used either during the class session or by monitoring student online discussion.By asking discussion questions that require critical thought, instructors are able to gauge students’ understanding of the lesson material and whether they are making necessary connections to other course material.Many times students believe they know the material but their misunderstandings are revealed during discussion.

Minute Paper

—This evaluation tool is done at the end of class several times during the quarter.It derives its name from the fact that students spend no more than one minute answering any number of questions. The instructor reads the responses before the next class meeting and responds appropriately. Examples of questions asked are:
  • What was the most important thing you learned during class?
  • What unanswered questions do you have?
  • What was the muddiest point for you?
  • At what point this week were you most engaged as a learner?
  • Can you summarize today’s lesson in one sentence? If so, please summarize it.
  • What has been most helpful to you this week in learning the course material?

Index Card

—A variation on the Minute Paper is for the instructor to write the responses to the following questions on a 3 x 5” index card following a lesson:“What worked? What didn’t work? What are some ideas for changing the lesson?”The 3 x 5 card limits the amount of information than can be written down and serves as a reminder to write down ideas but to only spend a few minutes writing them down. Attach the card to the lesson notes to serve as a reminder the next time the lesson is taught.

Course Exams and Assignments

—Student success on course exams and assignments are a powerful data source on teaching effectiveness. A short questionnaire at the end of exams can ask students to identify which questions were the most difficult to answer and why they were difficult. A pattern may develop that can be used to make changes. Additionally, an instructor may ask students to critique assignments. Questions instructors may ask are:
  • Were instructions clear?
  • Did the assignment help students learn course material?
  • Were the expectations reasonable for the time-frame?
  • How many hours were devoted to completing the assignment?

Mid-quarter evaluation

—An effective way of gauging student learning and satisfaction is via anonymous mid-quarter evaluations. The evaluations can take a variety of forms. A simple survey asking students to describe what is working, what is not working, and suggestions for change can be conducted via paper-pencil or online. Many of the course management systems have tools that allow anonymous feedback. Instructors need to check with their system’s administrator to find out how to do it. Many instructors provide 15-25 minutes of class time to a neutral party for the purpose of getting feedback from students. A more formal method is to use the same forms that are used for course evaluations. One thing to note is that even if course changes cannot be made during the quarter the evaluation takes place, mid-quarter evaluations allow instructors to engage in dialogue with their students regarding the teaching-learning process and students will feel more positive toward the instructor.

Suggested Readings

Angelo, Thomas A., and K. Patricia Cross. 1993. Classroom Assessment Techniques: A Handbook for College Teachers, Second Edition. San Francisco, CA: Jossey-Bass Inc.

Davis, B.G. 1993. Tools for Teaching. San Francisco, CA: Jossey-Bass.

Duncan, D. (2005). Clickers in the Classroom. Upper Saddle River, NJ: Pearson Education.

top


Data From Students: Interviews with Students


A procedure using interviews with students is an excellent technique for obtaining rich, in-depth information about student reaction to courses and instructors. Two methods have proven useful. The first involves an interview with a group of students. The second procedure uses a series of interviews with single students.

The procedure is simply to have two or more colleagues (either in an instructor’s department or from other departments) interview a group of students from a current course. Alternatively, a group of former students may be interviewed. The interview usually takes no longer than 30 minutes, and a brief two or three page report is completed by the interviewers. A group interview requires some planning, but it is not a difficult technique to use and it can yield valuable information. An instructor may request group interviews when he or she wants some candid student feedback at midterm, or after a course is completed. A department may also want to use this technique to get information for personnel actions such as tenure decisions, accelerations, or special teaching awards.

The interviewers


Usually colleagues from the instructor’s department or other departments do the interviews. It is best to use two interviewers, one to ask all the questions, the second to record the responses. Alternatively, the Faculty Consultation service offer consultation regarding the training of interviewers.


Conference with instructor


The interviewers should meet the instructor to learn the course characteristics, goals, instructor’s concerns and make arrangements for the interview. This meeting should decide the issues to be stressed in the interview, such as course objectives, organization, workload, instructor skills, instructor-student relationships, students’ attitudes, and the like. The instructor may have special concerns that should be considered in structuring the interview. Decisions should be made about whether the report should be written and/or oral and who should receive copies.

Constructing the interview schedule


After meeting with the instructor, the interviewers should formulate some of the questions for the interview. Table 1.2 provides suggested questions. Issues frequently arise during an interview which suggests a line of questioning not anticipated in the interview schedule. Probing students’ comments may sometimes be more useful and appropriate than asking all the questions on the schedule.
Table 1.2

Sample Interview Schedule for a Focus Group with Students



The instructor of this class has asked for your feedback on his
/her teaching of the class. Your comments will be treated with
the strictest confidence. Suggestions from this focus group will
be summarized and relayed anonymously to the instructor to improve
his/her teaching. Please take a few minutes to answer the
questions below before we discuss them as a group. Be as specific
as possible, giving examples to illustrate your points. The more
constructive you are, and the more suggestions you can give, the
more you will help your instructor to improve, both in this and
in other classes. Thank you in advance for your time.

1. Do you feel that your instructor is well-organized in this
class? (Please explain your answer with an example. You might
wish to comment on time management, presentation of concepts,
or clarity of explanations)

2. Do you find it easy to identify the main points from each
class? (Please comment with reference to ways the instructor
helps you take notes, or uses other methods to summarize or
emphasize main points.)

3. Do the exams/quizzes relate well to the class material?
(Please comment on whether questions are fair, and reflect
concepts taught in class.)

4. Is the feedback you receive on assignments helpful? How might
feedback you receive be improved to help you to learn better?

5. Please identify key strengths your instructor could build on
to improve his/her teaching.

6. Please identify any barriers which prevent you learning in this
class. Wherever possible please suggest specific ways in which
the instructor could help you to overcome these barriers.

7. How appropriate was the instructor’s use of technology in this
class? How did the use of technology enhance your learning
experience, if at all?

Interviewing the Students


A convenient time for the interview is the last 20-30 minutes of a class period. Unless class time is used for the interview, it is difficult to get a representative group. In conducting the interview, one interviewer should concentrate on asking the questions, the other on recording the answers and comments. Taping interviews is not recommended because of the problem of maintaining confidentiality.
The instructor should briefly discuss the purpose of the interview with the class before the interviewers arrive. After introducing the interviewers, the instructor should leave.

Group Size


The size of the group is an important variable. In a small group (less than 12-15), the interviewers can more easily probe for in-depth information. With a larger group, the number of students who want to comment makes in-depth coverage more difficult. You can divide the class group into smaller groups of about five and select one person from each group to act as a recorder and one as a spokesperson. If you are dividing the group up, then first have the smaller groups meet individually and arrive at a consensus on the predetermined questions. Then after 10 minutes of discussion time, have each spokesperson report one response to each of the questions.

Alternatively, to get a representative sample of students from large classes, the interviewers may want to use the instructor’s course list and/or grade book to select a small number of students to participate in the interview. In larger classes, students may feel more comfortable writing down their responses before participating in a group discussion.

Suggested Readings

Arreola, Raoul. 2000. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.

top


Data from Peers


While peer review of teaching may take many forms, at UCLA it most often involves class observation. Classroom visitation is a form of evaluation strongly supported by faculty as a useful source of information. When colleague observation is undertaken for instructional improvement, the most important considerations in establishing systematic and fair procedures are:

Number and timing of visits


In courses taught exclusively by the lecture method, at least two visits by each colleague evaluator are advisable. If the instructor employs a variety of teaching strategies (such as lecture, discussion, student presentations, or role playing), it becomes very difficult to choose one or two class sessions that would be typical or would give a balanced picture of the instructor’s teaching. In some small classes, the presence of an observer may be more distracting than in a larger class, and frequent observations by several colleagues during a single term might be problematic. The number and timing of visits should probably be worked out between colleague evaluators and the faculty member being evaluated, to assure an adequate evaluation with minimal disruption.


Explicit, appropriate criteria and guidelines


A set of explicit criteria by which colleague observers are asked to judge the quality of teaching will make the evaluations much more reliable and the evaluations made by different colleagues more comparable. For colleagues observing strictly for the purpose of evaluation, the criteria help to guide the observations. For colleagues who have ongoing contact and observation, they help to summarize the impressions developed over numerous observations. The number of criteria should be kept small and appropriate to the type of teaching done in the department. The format may consist of open-ended questions, or rating scales, or a combination of these. Criteria should reflect aspects of teaching on which there is broad departmental consensus and for which faculty observers would be in the best position to provide information. For example, faculty observers might be asked how well prepared the instructor was for the class session, but should not be asked to comment on the instructor’s accessibility to students outside of class if this has not been observed. Colleagues may reasonably be asked to comment on the instructor’s coverage of a topic or on the appropriateness of the teaching strategy, but should not be asked to evaluate student motivation or satisfaction, which can only be inferred at best. Comments about actual student participation, however, would be appropriate.

Special teaching situations


Colleague observation and evaluation of clinical teaching present problems analogous to those which arise with small classes. Clinical teaching takes place most often in a one-to-one or small group context. In this case, the presence of several colleague observers would be intrusive and might significantly disrupt the teaching and practice situation. On the other hand, many clinical settings provide a natural situation for colleague observation. Indeed, colleagues often work side by side in clinical settings and frequently observe one another’s teaching. Observation for evaluation should take advantage of these opportunities in much the same manner as evaluation based on observation in team-taught courses. There should be some indication as to the content, frequency, and length of the observations on which the evaluation is based. Where such natural observation is not possible and special visits for colleague observation-evaluation are needed, only one observer should have an opportunity to make several observations over a period of time. The development of criteria for the evaluation of clinical teaching should follow the guidelines for regular courses, although the specific items would be different and would be sensitive to the nature and purpose of clinical teaching. For example, in medicine, the observer might be asked to comment on the instructor’s integration of biomedical theory and clinical management or on the actual demonstration of a procedure or technique.

Constructive feedback


Feedback to the faculty member is an important consideration in designing departmental peer review procedures. For evaluations to be useful for the improvement of teaching, feedback and discussion are essential, yet this may present certain problems of confidentiality when colleague evaluation is conducted as part of the personnel process. Sensitive implementation Many instructors are understandably anxious about peer evaluation. Departments implementing new systems of colleague observations should be sensitive to the problems and insecurities among faculty that will inevitably arise. The suggestions summarized above are also useful for instructors to keep in mind when observing their TAs in discussion or laboratory sections and writing up their evaluations.

Suggested Readings:



Arreola, Raoul. (2000). Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.

Seldin, P. et al. (1999). Changing Practices in Evaluation Teaching: a Practical Guide to Improved Faculty Performance and Promotion/Tenure Decisions. Bolton, MA: Anker Publishing Company, Inc.


top


Data from Oneself


In conducting a self-review of a course, faculty members may wish to compare their own pre-course objectives or expectations with perceived post-course outcomes. A model whereby the instructor assesses the abilities and knowledge of students before and after the introduction of an innovation or improvement effort is especially useful for evaluating new courses or courses with significant changes in content or structure. Often, however, instructors may not have planned for a self-evaluation before the course began; the self-evaluation can then look only at course outcomes.

When faculty members are interested in examining their own teaching behavior, rather than course outcomes, they can follow an end-of-course format similar to that used by students. An instructor might use the same form and complete it from the point of view of self-perceived behavior. Alternatively, they might benefit from completing the student form from the perspective of what they expect, on the average, that students will say; instructors might complete a self-evaluation form at the same time that students complete their evaluations and then compare results. Also, instructors can arrange to videotape a class and then observe their performance, focusing on particular teaching skills of interest. While this latter technique can be extremely valuable, it is usually best achieved with the help of a faculty consultant, who can help the instructor to focus on the key elements. Individuals looking at videotapes of themselves often are biased towards seeing only the negative elements.

Faculty should expect that their self-review in most cases will be more favorable than reviews by students. If these self-evaluations are included as part of the dossier (see “Documenting teaching using a teaching portfolio or dossier” section below), instructors may wish to comment on any deficiencies noted or discrepancies between self and student evaluations. The comment should not attempt to defend one’s self-review as being more accurate than that of the students, or to discourse on every aspect of the course. Rather it should be considered as additional information that can assist reviewers in interpreting student data or in understanding how the self-review contributed to course changes or modifications in teaching style.

Suggested Readings:

Arreola, Raoul. (2000). Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.

Seldin, P. et al. (1999). Changing Practices in Evaluation Teaching: a Practical Guide to Improved Faculty Performance and Promotion/Tenure Decisions. Bolton, MA: Anker Publishing Company, Inc.

top


Table of Contents | Previous Section | Next Section