WILLIAM J. KAISER
What matters most to you in your teaching?
How are you using technology as a tool to achieve your teaching goals?
How have your students responded to your use of technology?
What new goals do you have for using technology in teaching?
How could the University better facilitate the use of technology in instruction?
Assessing student performance
Knowledge Mapper (software)
Real-time Assessment of Student's Understanding
It’s not enough for them to attend class, do well on exams and homework, and learn basic principles; we want them to get excited about engineering, and to leave the course with a positive feeling about their future in electrical engineering, computer science, or other engineering field they are pursuing.
The Electrical Engineering major includes a series of gateway courses, the first being EE 10, Circuit Analysis I. It’s often the first time most undergraduates encounter circuits and engineering design principles. They learn not only circuit concepts, but also how to represent a circuit with a set of mathematical equations, how to analyze those equations, and then draw another circuit based on the results. A typical class size is anywhere from 120 to 170 students, and it’s an intense course—it proceeds quickly.
The principles in EE 10 are pervasive in engineering, and they represent an engineering mindset that is important to understand. The combination of skills required to work the problems assigned is really very unique. Because this course is essential in preparing students for the next sequence, our two fundamental goals are to teach basic engineering knowledge and to develop their engineering problem-solving skills. Those goals are matched by the need to motivate the students. It’s not enough for them to attend class, do well on exams and homework, and learn basic principles; we want them to get excited about engineering, and to leave the course with a positive feeling about their future in electrical engineering.
We find the biggest challenge is the limitation students feel in understanding the graphical representation of a mathematical concept—the circuit. Since the classes are large, there’s also a challenge in making sure that everyone gets their questions answered. Unfortunately, students do not always appreciate the fact that an inadequate or vague understanding of these concepts can cause them very serious problems in the near future. Rather than ask questions and reveal what they don’t know, they tend to think that they can catch up, and that the material will sink in after further reading. This can result in a significant gap between what the students understand and the instructor’s perception of that comprehension.
As instructors, we can’t always know what students understand, because we do not obtain enough real-time feedback. All students, to some degree, feel conflicted about participating—underrepresented students in particular, because they feel more acutely the social pressure to appear knowledgeable. People fall behind, and may not realize how behind they are until they get a poor score on a midterm, which can lead to general demoralization. In large classrooms, it’s extremely difficult to get accurate assessment feedback that doesn’t involve slowing the class down with quizzes and other things that are disruptive of flow. We wanted a better way of assessing how students are doing from minute to minute in a lecture, so we developed the concept of providing a real-time private access channel for students, so that they can use a wireless notebook to communicate to the instructor during the lecture. Allowing them to do this privately and anonymously lets students participate comfortably without revealing their individual difficulties with the material. The key inspiration for this came from Dean Stephen Jacobsen and Rick Ainsworth, the Director of the Center for Excellence in Engineering and Diversity (CEED), and Tina Guidry, also of the CEED. We’ve also been working with Greg Chung at the Graduate School of Education, who is a member of the CRESST program. We are indebted to Greg for his expert advice and guidance. Greg has developed a series of assessment tools and methods; and he has also created and managed a series of assessment surveys that provide us with feedback on our method itself. Greg’s group and ours have also been funded by the National Science Foundation to pursue our new systems.
We refer to the protocol as individualized, interactive instruction (3I). This method is still based on an instructor teaching—because that’s very, very good for students —we just add a networked component so that communication with the lecturer becomes a two-way street. It works in the following manner: I, as an instructor, present material. I then present a problem that all the students can see, via the interface, on their personal wireless notebook screens. They then begin solving a problem, and I’m able to see the responses, in real time, of every student—every keystroke that’s entered is revealed. I don’t know which student is providing which response; from my perspective, they’re simply a list of numbers and this preserves the anonymous nature.
It provides an incredible degree of insight. As students respond to a problem, I’m able to see characteristic errors that the entire class is making, and I can review material to address a particular weakness . I’ve found that in typically just one problem cycle of about 10 minutes, a given weakness can be completely eliminated. In a typical 3I session, I’ll work through 6 different problem types, starting from very simple to very complex. At the beginning, students might have only two-thirds success on the initial problem, and at the end of the session, virtually all students are solving the hardest problem.
I think it’s even better than the feedback that instructors get during one-on-one office hours. Office hours are a great way to resolve comprehension problems specific to one student, but in that context you don’t generally ask the student to work a problem on their own, because it can make them uncomfortable. It’s not anonymous and puts them under pressure. In a 3I session, all students work the problem. There’s a high response rate from everyone—people are very engaged and no one is falling asleep.
We use software developed by the Educational Testing Service called Discourse. In addition to the problem-response interface, there is also an instant-messaging feature. Students can send the instructor a question, and the instructor can reply back to that student, or all students—that might be one way in which I would send back the solution to everyone, instead of using the blackboard. It’s well adapted to any sort of ASCII-text input, and we’ve never had to instruct anyone in its use. I think students are all so adept at instant messaging and e-mail that this just comes naturally. I do tailor the problems somewhat—we can’t for example, ask questions that require graphical responses. And there’s a limit to how many students the instructor can track at once—it works quite well for up to 20 notebook connections and we have recently managed a 55 student session comfortably. We are embarking on research to extend the ability of the interface to interpret student responses, to assist the instructor and enable much larger scales. That’s a very, very important research goal over the coming year.
The system uses client-server technology—the instructor operates the server. The laptops are loaned out to the participants for each session. We are indebted to Hewlett-Packard for donating wireless notebooks, and for providing key support that’s been just fantastic for us. Mario Gerla in Computer Science and Rajit Gadh in Mechanical and Aerospace Engineering have collaborated in the development of the wireless networking technologies we use. We’re not yet at the point of distributing the software for students to install on their own notebooks and PDAs. The trouble is, you can’t rely on every student to have a notebook available with the appropriate wireless interface. If you could depend upon everyone having wireless, you could use this in every classroom, and I’m really convinced this would be revolutionary.
Currently, 3I is offered as an additional voluntary recitation section. It’s given us the opportunity to assess our success. Greg Chung’s project has been to study the instructor-student interaction in detail, and to interview students and interpret their perception of the benefits of 3I. He developed a technology called the Knowledge Mapper that allows us to correlate students’ understanding of specific problem-solving concepts with performance on exams. Our goal, of course, is to provide everyone in the classroom environment with 3I, and use it to build student collaboration, student study groups, and so on.
The response has been very positive, both in the results of our research, and in the anecdotal comments from the students. I think it has had a positive impact on students’ view of the interaction and the instruction process. A hidden benefit, of course, is how the instructor feels—I think almost everyone will find that they can make use of the feedback.
The system can potentially be applied in other instructional contexts. I run large evening review sessions prior to exams, and would like to implement it there. If we have continued success with this, I think it could be applied more widely. The interface for more freeform entry needs to be advanced, but still, in almost any course, questions can be adapted to this plain text entry. Even if you have students examine four or five answers in a graphical or other format presented on the board, then make a selection and defend their choice in writing, the ASCII interface can still provide valuable feedback while we develop the technology for more interactive, general-purpose displays.
We would be thrilled if other departments in other disciplines could make use of this. I think it would map effectively into all science instruction, and with some effort it could move into other curricula. We have the opportunity to extend the technology, to standardize this real-time assessment, so that as instructors we can gauge the effectiveness of our instructional techniques. Instructor self-assessment should be private and voluntary, of course—I want to be free to run my class as I see fit, ensure that other faculty can continue to do so, because diversity is essential for the overall health of our academic approach. But providing more access to information would generally be welcomed by everyone.
Oral Interview, May 2004, updated March 2005