Data Research and Development Center and IERI Research Cmmunity header

The IERI Research Community :: Projects

Learning-Oriented Dialogs in Cognitive Tutors: Toward a Scalable Solution to Performance Orientation

Vincent Aleven

Albert Corbett, Carolyn Rosť


Background: The purpose of the project is to investigate issues related to engagement with learning technologies, specifically targeting math for urban middle school children. Increasing engagement with learning materials, specifically with this at risk population, has the potential to address the racial achievement gap and ultimately increase the likelihood that minority and low income students will achieve enough success with math to go on to math and science based majors. We begin with Cognitive Tutors as a tried and true foundation, seeking to verify the generality of previous findings with Cognitive Tutors in urban settings with other age groups as well as ultimately to increase it's effectiveness with this specific population.

Purpose: The work under this grant comprises two separate strands of research. One strand concerns the evaluation of a four-year middle-school Cognitive Tutor math course sequence in an urban schooling environment, by comparing it against a more traditional math curriculum. This study is set to begin in August of 2006, in four participating schools in the Cleveland Municipal School district. The second strand is focused on the design and evaluation of interventions that promote a learning orientation during work with Cognitive Tutors for middle-school math. Two studies were carried out in middle schools in the Pittsburgh area.

Study 1 (Support for Collaborative Learning): Under the current grant, we ran a study to explore the relationship between system feedback, perceived inter-dependence, and learning. There is a considerable amount of evidence from the collaborative learning literature that fostering a sense of inter-dependence between students increases identification with the learning community, long-term interest in the subject matter, and learning.

Study 2 (Gaming the System): Some students who use intelligent tutoring systems have been observed to game the system, by rapidly clicking through multiple help messaged (drilling down or help abuse ) or by rapid systematic guessing (help avoidance ). In a classroom study we introduced a real-time gaming detector into the Cognitive Tutor along with an intervention designed (a) to decrease harmful gaming behaviors, and (b) when harmful gaming behaviors persist, to alleviate their negative effects with additional learning activities. This intervention consisted of an animated agent that communicated the student's frequency of gaming to the student and teacher through expressions of positive and negative emotion, and which presented supplementary learning exercises on exactly those problem steps which the student gamed.

Setting: Study 1 took place at Perrysville Elementary School with two classrooms of 15 6th-grade students each. Study 2 was conducted during the spring of 2005 in Chartiers Valley Middle School and North Hills Junior High School in southwestern Pennsylvania.

This 5 year project began in 10/2004 and will end in 9/2009. The main longitudinal study that is funded under this project will begin in August 2006. This study will involve about 240 six-graders, assigned to two conditions, a Cognitive Tutor condition and a traditional math condition. We will follow this cohort of students for three years, as they progress through grades 6, 7, and 8. At the end of each school year, a two-day end-of-year evaluation will be conducted. We will also follow a second cohort of students, which will start a year later. Other smaller studies have already been conducted, and will be conducted throughout the duration of the project.

Research Design:
Study 1: We contrasted two feedback conditions as a within-subject manipulation. In one condition, students received the kinds of guidance that Cognitive Tutors typically provide, namely, immediate feedback on solution steps and on-demand hints. In another condition, students received no feedback from the system during problem solving but received a fully worked-out example after submitting their solution. These examples contained all of the instruction that was contained in the hints that were available in the other condition at the students' request.

We had 15 pairs, 10 of which were mixed-gender pairs. Pairs were arranged and assigned randomly to conditions by their instructor. Our hypothesis was that typical Cognitive Tutor hints and feedback might interfere with collaboration since students might feel that it is easier and more reliable to get help from the computer rather than from their partner. We measured perceived inter-dependence by means of a questionnaire administered after the first lab day (so that students would have only experienced one of the two conditions by then). We gave students a pretest, two quizzes (one after each lab day), and a retention test.

Study 2: Half the students were randomly assigned to the experimental condition and the other half to the control condition. All students viewed an introduction to the instructional topic (scatterplots), completed a pretest, engaged in Cognitive Tutor scatterplot problem-solving activities, and completed a posttest. In the experimental condition, the gaming detector and intervention was included in the Cognitive Tutor unit and in the control condition they were not included. Systematic human observation was employed to measure students' gaming frequency and Cognitive Tutor log files recorded the frequency of gaming interventions. The population of interest is students enrolled in middle-level mathematics or pre-algebra courses. The study sample included the students enrolled in two Cognitive Tutor Seventh-Grade Math Classes at Chartiers Valley Middle School and the students enrolled in three Ninth-Grade Cognitive Tutor Pre-Algebra classes at North Hills Junior High School.

In the control condition, students solved the same set of problems in a Cognitive Tutor unit on scatterplots as students in the experimental condition. However, the Cognitive Tutor in the control condition did not include the gaming detector and intervention which was present in the experimental condition. Learning outcomes were measured by comparing student accuracy on scatterplot problem-solving pretests and posttests. The tests were developed by the project; with multiple items that were counterbalanced across pretests and posttests. Gaming frequency measures were compiled from systematic classroom observations. Tutor invention frequencies were compiled from the Cognitive Tutor log files.

Study 1: We found a significant condition by gender interaction on perceived inter-dependence based on the questionnaire (p < .05), and a marginal condition by gender interaction consistent with this on the quizzes (p < .08). The consistent pattern was that girls perceived more help and learned more in the delayed condition whereas boys were the opposite. It appears that the delayed condition magnifies a typical pattern found in mixed-gender collaborative learning for math and science based topics where boys assume the role of help giver and girls assume the role of help receiver. Our preliminary analysis of the chat logs collected during the study suggests, however, that students in this age group are not adept at offering instruction to one another. We are still exploring the reasons for the gender by condition interaction. We did not observe this interaction on the retention test. Instead, on the retention test there was only a significant pre- to post-test gain in the delayed feedback condition (this was over the whole population, not gender specific).

Study 2: The real-time gaming detector and gaming interventions were successful both in decreasing gaming and alleviating its effects on learning. In this study, gaming behaviors decreased by half when the gaming intervention was employed. In addition, those students in the experimental condition who received a substantial number of additional learning activities (triggered by the harmful gaming detector) scored lower than the rest of the class on the pretest, but caught up to the rest of the class on the posttest. This result is quite different from the pattern observed in previous studies or in the control condition, in which students who engage in frequent harmful gaming start behind the rest of the class on the pretest and fall further behind by the posttest.

We combined the data from this study with results of earlier classroom studies to conduct a meta-analysis across three mathematics topics (data analysis, 3-D geometry, and percents) involving 261 students in 16 classes over 3 years, to further develop our understanding of gaming. We replicated the results that there are two distinguishable categories of gaming, harmful gaming that is associated with poorer learning outcomes and non-harmful gaming that is not associated with adverse learning outcomes. We confirmed that (a) gaming frequency as detected by human classroom observations is reliably associated with poorer learning outcomes, and (b) harmful gaming as detected by the online algorithm is more strongly associated with poorer learning outcomes, while (c) non-harmful gaming as detected by the online algorithm is not reliably associated with poorer learning outcomes. We found that these categories of harmful and non-harmful gaming are chiefly distinguished by the timing rather than the form of the gaming behaviors. Students who game in the non-harmful fashion tend to game problem-solving skills that they seem to know, but avoid executing, while students who are harmed tend to game skills they have not yet learned. Note that this would be a difficult distinction for a human observer to detect.

Study 1
Gweon, G., Ros , C. P., Zaiss, Z., & Carey, R. (2006). Providing Support for Adaptive Scripting in an On-Line Collaborative Learning Environment, Proceedings of CHI 06: ACM conference on human factors in computer systems. New York: ACM Press.

Gweon, G., Ros , C. P., Carey, R., Zaiss, Z. (2005). Towards Data Driven Design of a Peer Collaborative Agent, Proceedings of AI in Education 05.

Gweon, G., Ros , C. P., Carey, R., Zaiss, Z. (2005). Exploring the Effectiveness of Mixed-Language Peer Problem Solving Interactions, Proceedings of the AIED 2005 Workshop on Mixed Language Explanations in Learning Environments.

Study 2
Baker, R.S., Corbett, A.T. and Koedinger, K.R. (in press). Responding to problem behaviors in Cognitive Tutors: Towards educational systems which support all students. National Association for the Dually Diagnosed Bulletin.

Baker, R.S., Corbett, A.T., Koedingr, K., Evenson, S., Roll, I., Wagner, A., Naim, M., Raspat, J. Baker, D., and Beck, J. (in press). Adapting to when students game an intelligent tutoring system. Proceedings of the 8th International Conference on Intelligent Tutoring System, ITS 2006.

Baker, R.S., Corbett, A.T., Koedinger, K.R. and Roll, I (in press). Generalizing detection of gaming the system across a tutoring curriculum. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006.

Baker, R.S., Corbett, A.T., Koedinger, K.R. and Roll, I (2005). Detecting when students game the system, across tutor subjects and classroom cohorts. User Modeling: Proceedings of the Tenth International Conference, UM 2005, 220-224.

Baker, R.S., Roll, I., Corbett, A.T. and Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? Proceedings of AIED 2005: The 12th International Conference on Artificial Intelligence and Education, 57-64.

Roll, I., Baker, R., Aleven, V., McLaren, B., Koedinger, K., (2005). "Modeling Students' Metacognitive Errors in Two Intelligent Tutoring Systems." User Modeling: Proceedings of the Tenth International Conference, UM 2005, 367-376.

You can learn more about this project by visiting