Data From Students: Systematic Student Evaluations

Table of Contents | Previous Section | Next Section

The Student Evaluation System at UCLAThe Office of Instructional Development's Evaluation of Instruction Program (EIP) helps assess and improve teaching at UCLA by providing instructor evaluation services for instructors and TAs across the campus. At the end of each quarter, instructors have the opportunity to solicit formal evaluations from students enrolled in their classes. EIP distributes, collects, and processes evaluation forms via a network of departmental coordinators. Interested instructors should first contact their department evaluation coordinator, or may reach the centralized Evaluation of Instruction Program at extension 56939 (http://www.oid.ucla.edu/eip).EIP's Standard evaluation forms, which cover most teaching situations for faculty and TAs, were designed in consultation with faculty committees, national experts on assessment, and recommendations from UCLA and System-wide surveys of faculty, TAs and students. While most departments on campus use the standard campus-wide forms, a few departments and schools have devised their own standard forms that are distributed and administered by EIP. The Faculty Consultation Service can work with departments and individuals to develop questionnaires for special needs.Policy on DataThe Evaluation of Instruction Program seeks
to provide as secure an environment for data as it can. The forms are stored for processing in a
physically safeguarded location. Data
are compiled centrally only for the numerical scores. In addition to individual
instructor scores, larger collective comparison data (e.g. for Divisions, or
Schools) are also calculated. These
compiled individual and comparison data are returned to Department Chairs along
with the original student forms. Each Department may devise its own particular
policy on how data are presented, distributed, or made openly available. Most often, the forms are returned to the
instructor so that the written comments may be read firsthand. Some departments choose to transcribe the
comments for small seminars to protect the identity of the students. The numerical data are provided by the
Evaluation of Instruction Program in electronic format, and include both
departmental scores and appropriate comparison data. The Evaluation of Instruction Program does
not release individual instructor data except to the Department or to the
individual instructor. External requests for such data are referred to the
Departments. External requests for
comparison data (without any individuals identified) are generally granted.Retaining DataMost departments
develop a reliable system for storing numerical data from teaching
evaluations. It is less common for
departments to retain written comments. Because such data are often used when compiling dossiers for personnel
decisions, faculty should be careful to keep copies of their own
evaluations. Even normally reliable
systems sometimes have unexplainable lapses, and it can be extremely difficult
(if not impossible) to re-establish such data after the fact. In addition, it may be useful to annotate
records with information that might provide insight into any anomalous results
(e.g. “the class was scheduled into a room that was frequently impacted by
construction noise,” or “it was my first attempt to develop group projects and
they did not work the way I had hoped.”)

Some departments
retain only overall ratings, and again, instructors would be better advised to
keep data which encompass all the individual items on the form. Such information can often expand on
understanding why certain classes may have been rated higher or lower.

Most evidence of
teaching data is used within a period of six years after the time it was collected.Based on actual requests for re-establishing
older records, eight years would provide a more certain time frame.Comparison data, such as to other instructors
in the department or to the overall University mean, should likewise be kept in
order to provide bases for comparison, should later disputes arise.Faculty who are nominated by their
departments for teaching awards also find some of the written comments useful
in documenting what students find particularly compelling about their teaching.

Disputed Data

In the infrequent situation that the
integrity of the data are disputed (e.g. if forms are intercepted by the
instructor or the chair before processing, or considerably more forms are
returned than the number of students enrolled in the course) what facts are
known are forwarded to the Committee on Teaching for any further consideration
of action. Other Senate Committees may
become involved as appropriate.

top

Student Questionnaire Administration ProceduresSince student ratings are sensitive to a wide variety of
situational factors, it is very important that questionnaire administration
procedures are consistent. The goal is to get candid feedback focused on
instructional issues from as many students in the course as possible. The
following general guidelines for collecting student ratings may help
instructors achieve this goal:Since student ratings are sensitive to a wide
variety of situational factors, it is very important that questionnaire
administration procedures are consistent. The goal is to get candid feedback
focused on instructional issues from as many students in the course as
possible. The following general guidelines for collecting student ratings may
help instructors achieve this goal:

Protect student Anonymity

It is important that students not be identified in
any way with their evaluations, otherwise less than candid responses are
likely. In particular, students fear that an adverse rating might negatively
impact their course grade. It is, of course, difficult to maintain
confidentiality of student raters in small classes and individual study
courses. There is no simple solution to this problem. One option is to have the
departmental coordinator type up the responses in such cases.

Have a third party administer evaluationsIn order to protect anonymity, evaluation
questionnaires should not be administered by instructors or TAs. Rather, a
responsible student or the department evaluation coordinator should be
appointed to collect the completed questionnaires and deliver them to the
department office.

Time questionnaire administration appropriatelyRatings should be collected during the last two
weeks of the quarter. Students should be forewarned that evaluations will be
done on a certain date so that they will be in class and will be prepared.
Administration of evaluations at the final exam or other test is not
recommended.

Emphasize the importance of evaluationIt is advisable to give students some context for
the evaluation, especially for first year students. It is useful for them to
know that the department and instructor value their comments, and the use to
which they will be put. Distributing questionnaires at the beginning of the
class period and allowing sufficient time for students to complete them, all
contribute to the sense of importance placed upon a student’s opinion, and are
hence likely to produce more constructive results.

Suggested Readings:
Arreola, Raoul. 2000. Developing a
Comprehensive Faculty Evaluation System: A Handbook for College Faculty and
Administrators on Designing and Operating a Comprehensive Faculty Evaluation
System. Bolton, MA: Anker Publishing Co., Inc.

Davis,
B.G. (1993). Tools for Teaching. San Francisco, CA:
Jossey-Bass

top

Interpreting Quantitative Student Evaluations for a CourseSeveral weeks after the end of each quarter,

instructors will receive their students’ responses to the formal
questionnaires, along with a summary sheet of statistical information,
including mean, median, and standard deviation of the questionnaire responses.
Instructors should keep the following points in mind when interpreting the
results:

ProceduresWere reasonable procedures (such as suggested in
the preceding section) used to collect the ratings?

Sample size/response rateHow
many students in the class completed the questionnaires? Responses from less
than two-thirds of the class should be viewed with caution. The minimum number
of student raters in a class should be 8 to 10; a sample of 15 or more students
increases reliability and generalizability of the results. It should be noted,
however, that even in such “small sample” situations the qualitative comments
may still be extremely valuable.

Making comparisonsThe
average ratings can be interpreted according to an absolute scale or relative
to the ratings of other courses and instructors. For example, a mean rating of
6.5 on a 9-point scale for overall course evaluation may seem “above average.”
Taking them at their word, students rated this course as adequate. It may be,
however, that 70 percent of similar courses are rated above 6.5. Thus, relative
to other courses, the 6.5 rating was in the lower third. Which interpretation
is correct? The answer is that both interpretations are useful. The course was
judged positively, but students were not particularly excited by it.

The Campus ContextIn making comparisons, it may be helpful to
consider the campus context. Means do vary considerably between departments and
divisions, or between different kinds of courses. Departments are, therefore,
encouraged to keep records regarding their own norms.

Variability of responsesThe variability of student responses is important
diagnostic information. For example, consider an average course rating of 7 on
a 9-point scale with a small standard deviation, say of 1.0 or so. This means
that most students rated the course as 6, 7, or 8. On the other hand, the same
average rating with a larger standard deviation, say 3.4, indicates a greater
range of ratings that may suggest problems with the course. It is also
important to look at the total distribution of ratings. For example, there are
sometimes bimodal distributions of ratings in which a large group of students really
liked the course and another large group of students disliked it. This could
indicate two different ability or interest groups in the class, which would be
worth exploring further for future iterations of the course.

The importance of other variablesAlthough student response to a
course is important for evaluation and teaching improvement, it should not be
used as the only measure of teaching. Student ratings are affected by many
variables including course content, amount of work, expected grades, class
size, and students’ own needs and attitudes.

Suggested
Readings:

Arreola, Raoul. 2000. Developing a Comprehensive
Faculty Evaluation System: A Handbook for College Faculty and Administrators on
Designing and Operating a Comprehensive Faculty Evaluation System. Bolton, MA:
Anker Publishing Co., Inc.

Cashin, William E. Student Ratings of Teaching:
Recommendations for Use. (IDEA Paper No. 22). Manhattan:
Kansas State University,
Center for Faculty Evaluation and Development, 1990.

Cashin, William E. Student Ratings of Teaching:
The Research Revisited. (IDEA Paper No. 32). Manhattan:
Kansas State University,
Center for Faculty Evaluation and Development, 1995.

Davis,
B.G. (1993). Tools for Teaching. San Francisco, CA:
Jossey-Bass

topInterpreting Quantitative Student Evaluations for Personnel DecisionsFaculty members are often concerned about how
evaluation results will be used by their departments in administrative
personnel decision-making. Although details of the process differ among
departments, the substantial body of research literature on the use of student
course and instructor evaluations suggests that the following practices be
observed:

Guidelines for administration and explanations
of how ratings will be used should be consistent and well-publicized.

Small differences between scores should be kept
in perspective.

Multiple sets of student ratings should be used
in administrative decision-making.

Global ratings (i.e., overall ratings of the
instructor) are more often used than specific items (such as the instructor’s
organization or communication skills) for making personnel decisions. While
this may be the most convenient measure, decision-makers might note that global
ratings are also those most likely to reflect personal bias on the part of students.

Ratings for any single instructor or course
should be considered in conjunction with university, college, division,
department, and even specific course norms.

Multiple sources of information should be used
in administrative decision-making. In other words, numerical ratings should be
only one piece of the larger picture.

Suggested Readings

Arreola, Raoul. 2000. Developing a
Comprehensive Faculty Evaluation System: A Handbook for College Faculty and
Administrators on Designing and Operating a Comprehensive Faculty Evaluation
System. Bolton, MA: Anker Publishing Co., Inc.

Cashin, William E. Student Ratings of Teaching:
Recommendations for Use. (IDEA Paper No. 22). Manhattan: Kansas State University,
Center for Faculty Evaluation and Development, 1990.

Cashin, William E. Student Ratings of Teaching:
The Research Revisited. (IDEA Paper No. 32). Manhattan: Kansas State University,
Center for Faculty Evaluation and Development, 1995.

Davis,
B.G. 1993. Tools for Teaching. San Francisco, CA:
Jossey-Bass.topFurther Help for Faculty Concerned about Student EvaluationsPerhaps the most common concern that faculty
members express about the evaluation process is that students do not take the
evaluations seriously and that students are not aware of the gravity of their
input into the tenure and merit review process. Research and experience show
that instructors who openly announce to students that they themselves take
student input seriously are usually the recipients of the most constructive
comments.

The vast body of research on student ratings that
has accumulated over the years also suggests that student ratings correlate
highly with the ratings conducted by one’s own colleagues and even, in some
instances, with self-ratings. The high correlation holds up, however, only when
students are asked to judge selected aspects of teaching. Students as consumers
are well-equipped to assess the clarity of presentation, to comment on
organizational matters, to rate the quality of student/instructor interaction,
and even to assess their own learning to some extent. Students are not,
however, the best judges of whether what they are learning is current, whether
the instructor’s perspective is biased, or even whether the selection of course
material was appropriate for the achievement of course goals.

Instructors occasionally deride quarterly
evaluations because they believe that students cannot make accurate judgments
regarding a course or instructor until the students have been away from the
course, or even the university, for several years. While it has proven to be
very difficult for researchers to obtain a representative sample in
longitudinal follow-up studies, the research shows that, in general, alumni who
had been out of school for five to ten years rated instructors much the same as
did the students currently enrolled. Current research also provides little
substantiation for the widely held belief that only warm, friendly, humorous
instructors win the “ratings game.” Most students are quite able to
discriminate between glossy presentations with little substance and less flashy
lectures with relevant content.

If a faculty member’s number one concern about
evaluation of instruction results is how they will be used in personnel
decision-making, the number one concern among students is that their feedback
will not be acted upon. It is, therefore, crucial that having conducted any
type of feedback activity with students, instructors be seen to respond to the
results. This may not be as easy as it sounds, since bimodal distributions, for
example, can make obvious courses of action elusive.

It is important
for faculty—particularly first-time faculty—to remember that some students can
be insensitive or may lack the maturity necessary to deliver constructive
criticism in a non-threatening manner. At times, their comments are overstated and short on diplomacy. While such comments can be very discouraging,
if they come from only a few students, they represent only an extreme point of
view. However, if such comments come
from a majority of the students, advice from a trusted peer or from an
objective consultant might be useful. Even
if painful, they may contain insight into teaching issues that can be addressed
– but again, only if they present a cogent argument, not just a personal
attack.

Suggested Readings

Arreola, Raoul. (2000). Developing a
Comprehensive Faculty Evaluation System: A Handbook for College Faculty and
Administrators on Designing and Operating a Comprehensive Faculty Evaluation
System. Bolton, MA: Anker Publishing Co., Inc.

Davis,
B.G. 1993. Tools for Teaching. San Francisco, CA:
Jossey-Bass

Chen, Y. & Hoshower, L.B.
2003. Student evaluation of teaching effectiveness: an assessment of student
perception and motivation. Assessment and
Evaluation in Higher Education 28(1): 71-88.

McKeachie, W. and Svinicki, M. 2006. McKeachie’s Teaching Tips. Boston, MA:
Houghton Mifflin Company.

topConsulationFaculty are well advised to seek consultation when
deciphering their teaching evaluations.Quite often an outside perspective offers insight to evaluation data
that may not be apparent to the recipient of the evaluation.Speaking with peers or mentors within one’s
department or at other universities may help discern new approaches to instructional
improvement.

Instructors who wish for advice outside of a
departmental context are welcome to contact the Office of Instructional
Development’s Faculty Consultation service. The office offers consultation to
faculty on all aspects of teaching and learning. The objective is to help them
reflect on their classroom practice and suggest strategies for improvement.
Faculty Consultation services take the form of individual consultation by
appointment, workshops for groups of interested faculty, and seminars in
college teaching tailored for individual schools/departments. Professional
consultation is given based on an analysis of some initial inquiry, or on the
provision of such data as a classroom video taped by Audio Visual Services or a
set of evaluation results administered through the Evaluation of Instruction
Program (EIP). All consultations are strictly confidential and participation by
faculty is entirely voluntary.

Suggested Readings

Brinko, K.T. 1991. "The interactions of teaching
improvement." In M. Theall & Franklin (Eds.), Effective practices for
improving teaching. New Directions for Teaching and Learning, No.48. San Francisco, CA:
Jossey-Bass.

Cohen, P.A. 1980. "Effectiveness of student rating
feedback for improving college instruction: a meta-analysis of findings." Research of Higher Education, 13 (4),
321-341.

Seldin, P. et al. 1999. Changing Practices in Evaluation Teaching: a Practical Guide to
Improved Faculty Performance and Promotion/Tenure Decisions. Bolton, MA:
Anker Publishing Company, Inc.

top

Table of Contents | Previous Section | Next Section