Data Research and Development Center and IERI Research Cmmunity header

The IERI Research Community :: Projects

Scaling Up an Assessment-Driven Intervention Using the Internet and Hand-Held Computers

PRINCIPAL INVESTIGATOR:
Barbara R. Foorman

CO-INVESTIGATORS:
Kristi Santi, Larry Berger, David Francis

CATEGORIES:
Reading

PROJECT OVERVIEW:
Background: Under the Reading First component of No Child Left Behind, primary-grade teachers are flooded with assessment data. Our IERI study examines technological conditions and teacher support conditions that potentially moderate teachers' use of data to inform instruction. The technological conditions are use of a handheld device, web entry, or paper and pencil to capture assessment data. The teacher support conditions are web mentoring, human mentoring, or no mentoring.

Purpose: The purpose of this IERI Phase II project was to scale up quality assessment-driven intervention using the Internet and handheld computers in collaboration with a commercial partner, Wireless Generation. The assessment was the Texas Primary Reading Inventory (TPRI) and Tejas LEE (TJL) used in 96% of Texas schools and in many other states.

Intervention: Year 1 was devoted to working with Wireless Generation to build the mClass: TPRI and mClass:Tejas LEE, translate our paper-based intervention content to the web, and pilot these electronic systems with K-2 teachers. Years 2-3 were devoted to the randomized study of the effects of technological systems and teacher support. Year 4 followed up on the important question of whether the timing of kindergarten assessment influenced teachers' use of data to inform instruction by randomly assigning kindergarten teachers to administer the TPRI and Tejas LEE at the beginning of the year or the middle of the year.

Setting: The project runs for 5 years, starting September 19, 2002 and ending September 18, 2007. The Year 1 pilot was conducted in 7 Texas schools. The randomized studies of Years 2-4 were conducted in 255 Texas elementary schools (3 urban districts and 5 different rural regions).

Research Design: The primary methodology is randomized experiments. Two cohorts of data in the 2003-2004 and 2004-2005 school years allow us to analyze data longitudinally. Also included are classroom observations and teacher surveys. During the Year 1 pilot, descriptive data, interviews, and videography were employed. Statistical modeling techniques are being employed to analyze data as we enter Year 5.

In Years 2-4 the randomized study took place in 255 Texas elementary schools (3 urban districts and 5 different rural regions) with approximately 2115 teachers and 46,530 students in kindergarten, first, and second grades. The "business as usual" procedure for early reading assessments in Texas has been the teacher collecting data with paper and pencil versions of the TPRI and Tejas LEE and working alone to interpret the data. The comparison conditions we introduced were two electronic data collection techniques (web entry and handheld) and two additional types of teacher support (web mentor and human mentor).

Outcomes consist of composite scores from the TPRI and Tejas LEE: graphophonemic composite (from phonological awareness and letter substitution tasks) in Grade 1; word reading composite and fluency scores (words correct per minute on grade-level passage) in Grades 1 and 2. Covariates related to students are: initial graphophonemic composite (letter-sounds and phonological awareness) in kindergarten; and initial graphophonemic composite in Grade 1 (phonological awareness and letter substitution tasks). Additional covariates related to the design are: location (urban vs. rural), technological medium (paper, web entry, handheld), and mentoring (urban only). Missing data are imputed using SAS Proc MI and analyses are run in SAS Proc Mixed.

Findings: Preliminary analyses indicate that initial graphophonemic scores in kindergarten and the interaction of kindergarten classroom mean and student deviation from this mean predict word reading and fluency scores at the end of Grade 1. Once student and classroom performance were accounted for, there were no effects of assessment conditions (i.e., intervention support and administration format). On average, ICCs were roughly 10-15% at the classroom and school levels. Analyses of Grade 2 outcomes predicted by Grade 1 word reading are in progress. 2005-2006 data are being cleaned and analyses regarding the question of timing of kindergarten assessment will be conducted in the fall.

PROJECT PUBLICATIONS:
Foorman, B.R., Santi, K., & Berger, L. (in press). Scaling assessment-driven instruction using the Internet and handheld computers. In B. Schneider & S. McDonald (Eds.), Scale-up in education, vol. 1: Practice. Lanham, MD: Rowan & Littlefield Publishers, Inc.

ON THE WEB:
The IERI study website may be viewed at https://www.mclasshome.com/wgen/Login.do. Use the following username and password:
Username: ksanti9
Password: awe83on