Data From Students: Systematic Student Evaluations
Table of Contents | Previous Section | Next Section
The Student Evaluation System at UCLA
The Office of Instructional Development's Evaluation of Instruction Program (EIP) helps assess and improve teaching at UCLA by providing instructor evaluation services for instructors and TAs across the campus. At the end of each quarter, instructors have the opportunity to solicit formal evaluations from students enrolled in their classes. EIP distributes, collects, and processes evaluation forms via a network of departmental coordinators. Interested instructors should first contact their department evaluation coordinator, or may reach the centralized Evaluation of Instruction Program at extension 56939 (http://www.oid.ucla.edu/eip).
EIP's Standard evaluation forms, which cover most teaching situations for faculty and TAs, were designed in consultation with faculty committees, national experts on assessment, and recommendations from UCLA and System-wide surveys of faculty, TAs and students. While most departments on campus use the standard campus-wide forms, a few departments and schools have devised their own standard forms that are distributed and administered by EIP. The Faculty Consultation Service can work with departments and individuals to develop questionnaires for special needs.
Policy on Data
The Evaluation of Instruction Program seeks to provide as secure an environment for data as it can. The forms are stored for processing in a physically safeguarded location. Data are compiled centrally only for the numerical scores. In addition to individual instructor scores, larger collective comparison data (e.g. for Divisions, or Schools) are also calculated. These compiled individual and comparison data are returned to Department Chairs along with the original student forms. Each Department may devise its own particular policy on how data are presented, distributed, or made openly available. Most often, the forms are returned to the instructor so that the written comments may be read firsthand. Some departments choose to transcribe the comments for small seminars to protect the identity of the students. The numerical data are provided by the Evaluation of Instruction Program in electronic format, and include both departmental scores and appropriate comparison data. The Evaluation of Instruction Program does not release individual instructor data except to the Department or to the individual instructor. External requests for such data are referred to the Departments. External requests for comparison data (without any individuals identified) are generally granted.
Retaining Data
Most departments develop a reliable system for storing numerical data from teaching evaluations. It is less common for departments to retain written comments. Because such data are often used when compiling dossiers for personnel decisions, faculty should be careful to keep copies of their own evaluations. Even normally reliable systems sometimes have unexplainable lapses, and it can be extremely difficult (if not impossible) to re-establish such data after the fact. In addition, it may be useful to annotate records with information that might provide insight into any anomalous results (e.g. “the class was scheduled into a room that was frequently impacted by construction noise,” or “it was my first attempt to develop group projects and they did not work the way I had hoped.”)
Some departments retain only overall ratings, and again, instructors would be better advised to keep data which encompass all the individual items on the form. Such information can often expand on understanding why certain classes may have been rated higher or lower.
Most evidence of teaching data is used within a period of six years after the time it was collected.Based on actual requests for re-establishing older records, eight years would provide a more certain time frame.Comparison data, such as to other instructors in the department or to the overall University mean, should likewise be kept in order to provide bases for comparison, should later disputes arise.Faculty who are nominated by their departments for teaching awards also find some of the written comments useful in documenting what students find particularly compelling about their teaching.
Disputed Data
In the infrequent situation that the integrity of the data are disputed (e.g. if forms are intercepted by the instructor or the chair before processing, or considerably more forms are returned than the number of students enrolled in the course) what facts are known are forwarded to the Committee on Teaching for any further consideration of action. Other Senate Committees may become involved as appropriate.
topStudent Questionnaire Administration Procedures
Since student ratings are sensitive to a wide variety of situational factors, it is very important that questionnaire administration procedures are consistent. The goal is to get candid feedback focused on instructional issues from as many students in the course as possible. The following general guidelines for collecting student ratings may help instructors achieve this goal:Since student ratings are sensitive to a wide variety of situational factors, it is very important that questionnaire administration procedures are consistent. The goal is to get candid feedback focused on instructional issues from as many students in the course as possible. The following general guidelines for collecting student ratings may help instructors achieve this goal: Protect student Anonymity
It is important that students not be identified in any way with their evaluations, otherwise less than candid responses are likely. In particular, students fear that an adverse rating might negatively impact their course grade. It is, of course, difficult to maintain confidentiality of student raters in small classes and individual study courses. There is no simple solution to this problem. One option is to have the departmental coordinator type up the responses in such cases.
Have a third party administer evaluations
In order to protect anonymity, evaluation
questionnaires should not be administered by instructors or TAs. Rather, a
responsible student or the department evaluation coordinator should be
appointed to collect the completed questionnaires and deliver them to the
department office.
Time questionnaire administration appropriately
Ratings should be collected during the last two
weeks of the quarter. Students should be forewarned that evaluations will be
done on a certain date so that they will be in class and will be prepared.
Administration of evaluations at the final exam or other test is not
recommended.
Emphasize the importance of evaluation
It is advisable to give students some context for
the evaluation, especially for first year students. It is useful for them to
know that the department and instructor value their comments, and the use to
which they will be put. Distributing questionnaires at the beginning of the
class period and allowing sufficient time for students to complete them, all
contribute to the sense of importance placed upon a student’s opinion, and are
hence likely to produce more constructive results.
Suggested Readings:
Arreola, Raoul. 2000. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.
Davis, B.G. (1993). Tools for Teaching. San Francisco, CA: Jossey-Bass
topInterpreting Quantitative Student Evaluations for a Course
Several weeks after the end of each quarter,
instructors will receive their students’ responses to the formal
questionnaires, along with a summary sheet of statistical information,
including mean, median, and standard deviation of the questionnaire responses.
Instructors should keep the following points in mind when interpreting the
results:
Procedures
Were reasonable procedures (such as suggested in
the preceding section) used to collect the ratings?
Sample size/response rate
How
many students in the class completed the questionnaires? Responses from less
than two-thirds of the class should be viewed with caution. The minimum number
of student raters in a class should be 8 to 10; a sample of 15 or more students
increases reliability and generalizability of the results. It should be noted,
however, that even in such “small sample” situations the qualitative comments
may still be extremely valuable.
Making comparisons
The
average ratings can be interpreted according to an absolute scale or relative
to the ratings of other courses and instructors. For example, a mean rating of
6.5 on a 9-point scale for overall course evaluation may seem “above average.”
Taking them at their word, students rated this course as adequate. It may be,
however, that 70 percent of similar courses are rated above 6.5. Thus, relative
to other courses, the 6.5 rating was in the lower third. Which interpretation
is correct? The answer is that both interpretations are useful. The course was
judged positively, but students were not particularly excited by it.
The Campus Context
In making comparisons, it may be helpful to
consider the campus context. Means do vary considerably between departments and
divisions, or between different kinds of courses. Departments are, therefore,
encouraged to keep records regarding their own norms.
Variability of responses
The variability of student responses is important
diagnostic information. For example, consider an average course rating of 7 on
a 9-point scale with a small standard deviation, say of 1.0 or so. This means
that most students rated the course as 6, 7, or 8. On the other hand, the same
average rating with a larger standard deviation, say 3.4, indicates a greater
range of ratings that may suggest problems with the course. It is also
important to look at the total distribution of ratings. For example, there are
sometimes bimodal distributions of ratings in which a large group of students really
liked the course and another large group of students disliked it. This could
indicate two different ability or interest groups in the class, which would be
worth exploring further for future iterations of the course.
The importance of other variables
Although student response to a
course is important for evaluation and teaching improvement, it should not be
used as the only measure of teaching. Student ratings are affected by many
variables including course content, amount of work, expected grades, class
size, and students’ own needs and attitudes.
Suggested Readings:
Arreola, Raoul. 2000. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.
Cashin, William E. Student Ratings of Teaching: Recommendations for Use. (IDEA Paper No. 22). Manhattan: Kansas State University, Center for Faculty Evaluation and Development, 1990.
Cashin, William E. Student Ratings of Teaching: The Research Revisited. (IDEA Paper No. 32). Manhattan: Kansas State University, Center for Faculty Evaluation and Development, 1995.
Davis, B.G. (1993). Tools for Teaching. San Francisco, CA: Jossey-Bass
topInterpreting Quantitative Student Evaluations for Personnel Decisions
Faculty members are often concerned about how evaluation results will be used by their departments in administrative personnel decision-making. Although details of the process differ among departments, the substantial body of research literature on the use of student course and instructor evaluations suggests that the following practices be observed: Guidelines for administration and explanations of how ratings will be used should be consistent and well-publicized. Small differences between scores should be kept in perspective. Multiple sets of student ratings should be used in administrative decision-making. Global ratings (i.e., overall ratings of the instructor) are more often used than specific items (such as the instructor’s organization or communication skills) for making personnel decisions. While this may be the most convenient measure, decision-makers might note that global ratings are also those most likely to reflect personal bias on the part of students. Ratings for any single instructor or course should be considered in conjunction with university, college, division, department, and even specific course norms. Multiple sources of information should be used in administrative decision-making. In other words, numerical ratings should be only one piece of the larger picture.
Suggested Readings
Arreola, Raoul. 2000. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.
Cashin, William E. Student Ratings of Teaching: Recommendations for Use. (IDEA Paper No. 22). Manhattan: Kansas State University, Center for Faculty Evaluation and Development, 1990.
Cashin, William E. Student Ratings of Teaching: The Research Revisited. (IDEA Paper No. 32). Manhattan: Kansas State University, Center for Faculty Evaluation and Development, 1995.
Davis, B.G. 1993. Tools for Teaching. San Francisco, CA: Jossey-Bass.
top
Further Help for Faculty Concerned about Student Evaluations
Perhaps the most common concern that faculty
members express about the evaluation process is that students do not take the
evaluations seriously and that students are not aware of the gravity of their
input into the tenure and merit review process. Research and experience show
that instructors who openly announce to students that they themselves take
student input seriously are usually the recipients of the most constructive
comments.
The vast body of research on student ratings that has accumulated over the years also suggests that student ratings correlate highly with the ratings conducted by one’s own colleagues and even, in some instances, with self-ratings. The high correlation holds up, however, only when students are asked to judge selected aspects of teaching. Students as consumers are well-equipped to assess the clarity of presentation, to comment on organizational matters, to rate the quality of student/instructor interaction, and even to assess their own learning to some extent. Students are not, however, the best judges of whether what they are learning is current, whether the instructor’s perspective is biased, or even whether the selection of course material was appropriate for the achievement of course goals.
Instructors occasionally deride quarterly evaluations because they believe that students cannot make accurate judgments regarding a course or instructor until the students have been away from the course, or even the university, for several years. While it has proven to be very difficult for researchers to obtain a representative sample in longitudinal follow-up studies, the research shows that, in general, alumni who had been out of school for five to ten years rated instructors much the same as did the students currently enrolled. Current research also provides little substantiation for the widely held belief that only warm, friendly, humorous instructors win the “ratings game.” Most students are quite able to discriminate between glossy presentations with little substance and less flashy lectures with relevant content.
If a faculty member’s number one concern about evaluation of instruction results is how they will be used in personnel decision-making, the number one concern among students is that their feedback will not be acted upon. It is, therefore, crucial that having conducted any type of feedback activity with students, instructors be seen to respond to the results. This may not be as easy as it sounds, since bimodal distributions, for example, can make obvious courses of action elusive.
It is important for faculty—particularly first-time faculty—to remember that some students can be insensitive or may lack the maturity necessary to deliver constructive criticism in a non-threatening manner. At times, their comments are overstated and short on diplomacy. While such comments can be very discouraging, if they come from only a few students, they represent only an extreme point of view. However, if such comments come from a majority of the students, advice from a trusted peer or from an objective consultant might be useful. Even if painful, they may contain insight into teaching issues that can be addressed – but again, only if they present a cogent argument, not just a personal attack.
Suggested Readings
Arreola, Raoul. (2000). Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA: Anker Publishing Co., Inc.
Davis, B.G. 1993. Tools for Teaching. San Francisco, CA: Jossey-Bass
Chen, Y. & Hoshower, L.B. 2003. Student evaluation of teaching effectiveness: an assessment of student perception and motivation. Assessment and Evaluation in Higher Education 28(1): 71-88.
McKeachie, W. and Svinicki, M. 2006. McKeachie’s Teaching Tips. Boston, MA: Houghton Mifflin Company.
top
Consulation
Faculty are well advised to seek consultation when deciphering their teaching evaluations.Quite often an outside perspective offers insight to evaluation data that may not be apparent to the recipient of the evaluation.Speaking with peers or mentors within one’s department or at other universities may help discern new approaches to instructional improvement.
Instructors who wish for advice outside of a departmental context are welcome to contact the Office of Instructional Development’s Faculty Consultation service. The office offers consultation to faculty on all aspects of teaching and learning. The objective is to help them reflect on their classroom practice and suggest strategies for improvement. Faculty Consultation services take the form of individual consultation by appointment, workshops for groups of interested faculty, and seminars in college teaching tailored for individual schools/departments. Professional consultation is given based on an analysis of some initial inquiry, or on the provision of such data as a classroom video taped by Audio Visual Services or a set of evaluation results administered through the Evaluation of Instruction Program (EIP). All consultations are strictly confidential and participation by faculty is entirely voluntary.
Suggested Readings
Brinko, K.T. 1991. "The interactions of teaching improvement." In M. Theall & Franklin (Eds.), Effective practices for improving teaching. New Directions for Teaching and Learning, No.48. San Francisco, CA: Jossey-Bass.
Cohen, P.A. 1980. "Effectiveness of student rating feedback for improving college instruction: a meta-analysis of findings." Research of Higher Education, 13 (4), 321-341.
Seldin, P. et al. 1999. Changing Practices in Evaluation Teaching: a Practical Guide to
Improved Faculty Performance and Promotion/Tenure Decisions. Bolton, MA:
Anker Publishing Company, Inc.
ShareThis
