Data Research and Development Center and IERI Research Cmmunity header

The IERI Research Community :: Projects

Phase-I Study of the Effects of Professional Development and Long-term Support on Curriculum Implementation and Scaling Up

PRINCIPAL INVESTIGATOR:
Paul Brandon

CO-INVESTIGATOR:
Donald B. Young

CATEGORIES:
Science

PROJECT OVERVIEW:
Background: Little conclusive evidence from randomized studies exists about variation in program implementation, student achievement, or long-term program scale-up due to characteristics of K 12 teacher professional development (PD) such as the duration of PD institutes or of PD follow-up in the form of call centers, on-line assistance, multi-media support, and so forth. Curriculum researchers and program evaluators at the University of Hawai'i at Manoa prepared to study these characteristics in the project entitled, Phase-I Study of the Effects of Professional Development and Long-term Support on Program Implementation and Scaling Up. The project addressed middle-school inquiry-based science, with a focus on the Foundational Approaches in Science Teaching (FAST) program, an award-winning, constructivist, inquiry-based, three-year middle-school science program that has been used in 36 states and 10 foreign countries.

Purpose: The purpose of the project was to prepare for an eventual randomized experiment by (a) developing an alternative to the traditional two-week PD institute that is required of all FAST teachers and (b) preparing and validating the instrumentation for the randomized experiment.

Setting: Questionnaire and log validation data were collected on a sample of middle-school inquiry science teachers nationwide, and observation and assessment validation data were collected from a sample of teachers and classrooms in Hawai'i.

Research Design: An alternative version of PD for the first year of FAST (Physical Science, Ecology, and Relational Studies), including a one-week teacher institute; a one-semester online course; and a multimedia DVD-ROM, with text, still photos, videos of FAST teachers, and narrated animations, was developed. Four instruments were developed, including (a) a class-room observation protocol for measuring the quality of teacher questioning; (b) a teacher log for reporting the implementation of lessons as they were conducted; (c) a teacher questionnaire for collecting self-reports on implementation, program context, and teacher demographics; and (d) a student attitude survey, student multiple-choice and extended-response test, and student performance assessment. Observation validation data were collected through videotaping 105 classroom periods of 19 inquiry-based teachers on four of the Hawaiian islands. Questionnaire validation data were collected on a sample of 79 inquiry-based teachers nationwide who mirrored closely the national population of K 12 teachers, and log validation data were collected from a subset of 66 of the questionnaire respondents. Assessment validation data were collected on 428 children in 10 classrooms.

Findings: Evidence for the validity of data collected with the instruments was shown in several ways. Evidence for content validity was provided by the documentation on the extensive instrument development procedures. The extent to which information that was elicited in responses to log and questionnaire items addressed the intent of the items was identified in think-aloud protocols for the log and questionnaire. The extent to which the questionnaire and log subscale results performed as expected was shown in test-retest analyses, internal consistency analyses, generalizability theory analyses, factor analyses, multi-trait/multi-method correlational analyses, and criterion-related correlational and regression analyses. Observation data validity was shown in correlations with assessment results and was supported by interrater agreement analyses.

The validation data collected during the study were analyzed to glean their implications for middle-school inquiry-based science. The analyses addressed the theoretical model underlying the development of the instrumentation. The findings showed that (a) sampled teachers generally adhered to key aspects of inquiry-based science but implemented fewer inquiry-based lessons than expected; (b) younger and less-experienced teachers implemented more inquiry lessons than older, experienced teachers; (c) inquiry-based science classroom implementation was correlated with contextual factors that suggest high-performing schools;(d) teacher demographics were not correlated with student achievement; and (e) high-quality teacher questioning in inquiry-based classrooms was strongly correlated with mean student achievement test scores.
Citations: As of the date of the preparation of this description, technical reports and articles for publications were being prepared.

PROJECT PUBLICATIONS:
As of the date of the preparation of this description, technical reports and articles for publications were being prepared. Past and forthcoming conference presentations stemming from the project will be available at /www.hawaii.edu/crdg/programs/evaluation/. Papers presented or scheduled to date include:

Ayala, C. C. (2005, April). Developing student outcome measures frameworks. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

Ayala, C. A., & Brandon, P. R. (2006, November). The relationships among middle-school inquiry-based science student achievement and affect: Results of structural equation modeling analyses. Paper presented at the annual meeting of the American Evaluation Association, Portland, OR.

Ayala, C. A., Brandon, P. R., & Taum, A. K. H. (2005, October). Development and validation of the Inquiry Science Student Assessment Suite. Paper presented at the meeting of the American Evaluation Association, Toronto.

Brandon, P. R. (2005, April). The merit of a study comparing two versions of inquiry science professional development. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

Brandon, P. R., & Taum, A. K. H. (2005, April). Instrument development for a study comparing two versions of inquiry science professional development. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

Brandon, P. R., & Taum, A. K. H. (2005, October). Development and validation of the Inquiry Science Teacher Log and the Inquiry Science Teacher Questionnaire. Paper presented at the meeting of the American Evaluation Association, Toronto.

Brandon, P. R., Taum, A. H. K., Ayala, C. A., Young, D. B, & Pottenger, F. M. (2006, July). The implementation and outcomes of the Foundational Approaches in Science Teaching Program: Results of an American pilot study. Paper presented at the annual meeting of the Pacific Circle Consortium, Mexico City.

Brandon, P. R., Taum, A. H. K., Ayala, C. A., Young, D. B, & Pottenger, F. M. (2006, November). Findings about the implementation and outcomes of inquiry-based science in middle-school classrooms. Paper presented at the annual meeting of the American Evaluation Association, Portland, OR.

Gray, M. E., Nguyen, T. T., & Speitel, T. W. (2005, April). Developing and implementing an alternative version of FAST professional development. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

Lawton, B. (2005, April). The differential effects of two versions of professional development on teachers self-efficacy. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.

Lawton, B. (2005, October). The differential effects of two versions of professional development on teachers self-efficacy after one year of program implementation. Paper presented at the annual meeting of the American Evaluation Association, Toronto, Canada.

Taum, A. K. H., & Brandon, P. R. (2005, April). Coding teachers in inquiry science classrooms using the Inquiry Science Observation Guide. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

Taum, A. K. H., & Brandon, P. R. (2005, October). The development of the Inquiry Science Observation Code Sheet. Paper presented at the annual meeting of the American Evaluation Association, Toronto.

Taum, A. K. H., & Brandon, P. R. (2006, April). The iterative process of developing an inquiry science classroom observation protocol. Paper presented at the annual meeting of the American Educational Research Association. San Francisco.

Taum, A. K. H., & Brandon, P. R. (2006, November). Some limitations of using classroom observation data on teachers implementation middle school inquiry-based science programs. Paper presented at the annual meeting of the American Evaluation Association, Portland, OR.

ON THE WEB:
You can learn more about this project by visiting http://www.hawaii.edu/crdg/programs/evaluation.html.