Data Research and Development Center and IERI Research Cmmunity header

The IERI Research Community :: Projects

IERI: A System of Principled Assessment Designs for Inquiry: Implementation Study

PRINCIPAL INVESTIGATOR:
Geneva Haertel

CO-INVESTIGATORS:
Robert Mislevy, Nancy Songer, Kathy Long, Mark Wilson

CATEGORIES:
Science

PROJECT OVERVIEW:
Background: Principled Assessment Designs for Inquiry (PADI) draws on new understandings in cognitive psychology, research on science inquiry, and recent advances in measurement theory and technology to create a conceptual framework and supporting software tools for use in the design of assessments, especially those that measure inquiry processes.

Purpose: PADI seeks to provide a practical, theory-based approach to the design of high-quality assessments of science inquiry by developing: (1) a technology-based online system of reusable assessment-task templates, organized around schemas of inquiry identified in research drawn from studies of cognitive psychology and science education; (2) an organized set of assessment resources that include design patterns, exemplar templates, and task specifications; (3) rubrics for recognizing and evaluating evidence of inquiry skills; (4) examples of reverse-engineered and forward-engineered assessment tasks that conform to the principles of evidence-centered design and were produced in the context of the BioKIDS and FOSS science curricula projects; and (5) a scoring and a calibration engine, as well as a data management and reporting tool (i.e., Gradebook) that support the design and use of complex assessments and their statistical models.

Intervention: The PADI Project was intended to develop a set of assessment resources that could be used to support the design of evidence-centered assessments in the domain of science inquiry. In fact, the PADI assessment system including its design component (design patterns, task templates, and task specifications) and its scoring and calibration engines can be used to support the design of assessments in: (a) any content domain, (b) based on any theory of learning (e.g., cognitivist, behaviorist, or socio-cultural) and (c) serving any assessment purpose (e.g., diagnosis of gaps in learning, benchmark examinations, summative examinations). The conceptual framework that underlies the PADI assessment system is based on the tenets of evidence-centered design and includes the articulation of student, evidence and task models. These three models specify the assessment argument and establish the claims and warrants that are required in such a design process. While the PADI system provides tools and resources that are useful in any domain, the project has focused on assessing science inquiry at the middle school level. The design patterns, task templates and task specifications that were developed in this area can themselves be used for this purpose, and further guide the development of new inquiry-focused design patterns, task templates and task specifications. PADI resources can be used to develop assessments to measure the effects of science inquiry curriculum.

Setting: There were two science curriculum development projects that served as implementation sites for the PADI Project. One was the Full Option Science System (FOSS), located at the Lawrence Hall of Science at the University of California, Berkeley. As part of the internal evaluation of the PADI assessment design system, 24 classrooms of 4th, 5th, and 6th school students will be tested using assessments developed from the PADI assessment system in the Fall of 2006. These students, who will be in FOSS science classrooms will be from the following school districts: ESD 112 in Vancouver, Washington; Madison School District in Phoenix, Arizona; and Madison Metropolitan School District in Madison, WI. The second PADI implementation site is associated with the BioKIDS Project at the University of Michigan. In that setting, 400 students from 20 classrooms were tested as part of the internal evaluation of the quality of assessments that were developed using the PADI assessment system. These middle school students were located in five schools in inner-city Detroit.

Research Design: As part of the internal evaluation of the PADI assessment design system, 24 FOSS classrooms of 4th, 5th, and 6th school students, approximately 450 students, will be tested using assessments developed from the PADI assessment system in the Fall of 2006. In addition, 400 middle school students from 20 BioKIDS classrooms were tested in five inner city schools.

As part of the internal evaluation of the PADI assessment system, both the FOSS and BioKIDS implementation sites will have collected data on the middle school students performance on PADI-supported science inquiry items. BioKIDS is also collecting student performances on standardized, released science inquiry and content items from national and international science reference examinations, and statewide examinations. In addition, a selected number of students at both sites will participate in detailed cognitive task analyses on selected PADI-supported items. The PADI calibration and scoring engines will be used to estimate item parameters and student proficiencies. Information gathered from the cognitive analyses will be used to create "gold standard" estimates of student proficiencies. Confirmatory factor analyses and IRT-based analyses of item information will be conducted to compare the information about these targeted proficiencies from PADI-supported items and carefully selected standardized test items. In addition, qualitative information on the use of the design system will be collected.

Findings: The PADI Project has articulated a conceptual framework that incorporates the design, delivery, scoring and calibrating of assessment tasks and has applied this framework to assess inquiry skills in science. PADI has developed a suite of assessment resources, including an online design tool, a series of increasingly particularized templates to guide assessment design, scoring and calibration engines, and a data management and reporting tool. These tools and resources have been implemented within two curriculum development sites. The PADI Project has specified the requirements of the delivery systems that are needed to present assessment tasks and evaluate performances. The project has established a digital library of working exemplars of design patterns, task templates, and task specifications. Results from an evaluation of PADI-supported assessment tasks within the BioKIDS and FOSS implementation sites will be available 12/06.

PROJECT PUBLICATIONS:
Publications and Presentations
Bao, H., & Mislevy, R. J. (2005). Detecting local item dependence: an application of multidimensional random coefficients multinomial logit item bundle model. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Quebec.

Baxter, G. P., & Mislevy, R. J. (2005). The case for an integrated design framework for assessing science inquiry (PADI Technical Report 5). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Brecht, J., Mislevy, R. J., & Haertel, G. D. (2005). The PADI design system as a complex of epistemic games and forms. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Quebec.

Chudowsky, N., Morrison, A., Schank, P., & Hamel, L. (April, 2003). The role of design patterns in assessment design. Paper presented at the annual meeting of the American Educational Research Association, Chicago,IL.

Chudowsky, N. (April, 2003). Principled assessment designs for inquiry. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Cromley, J., & Mislevy, R. J. (2005). Task templates based on misconception research (PADI Technical Report 6). Menlo Park, CA: SRI International.

DeBarger, A. H., & Riconscente, M. M. (2005). An example-based exploration of design patterns in measurement (PADI Technical Report 8). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

DeBarger. A. H., Yumoto, F., Haertel, G., & Quellmalz, E. (2006). Reverse-engineering GLOBE classroom assessment tools. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

DeBarger. A. H., Yumoto, F., & Quellmalz, E. (2005). GLOBE: Using PADI templates as an alternative structure for specifying GLOBE investigation strategies. Paper presented at the annual meeting of the American Educational Research Association, Monteal, Quebec.

DeBarger. A. H., Yumoto, F., & Quellmalz, E. (in press). An illustration of PADI design system capabilities with GLOBE assessments (PADI Technical Report 18). Menlo Park, CA: SRI International.

Gotwals, A. W. & Songer, N. B. (2004). A systematic scheme for measuring inquiry skills across curricular units. Paper presented at the annual meeting of the American Educational Research Association. San Diego, CA.

Gotwals, A. W., & Songer, N. B. (2006). Cognitive predictions: BioKIDS implementation of the PADI assessment system (PADI Technical Report 10). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Haertel, G.D. & Mislevy, R. (April, 2003). Overview of the Principled Assessment Design for Inquiry (PADI) Project. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Haertel, G. (2004). Design patterns for assessing science inquiry. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.

Haertel, G. (2004). Conceptualizing Scale-Up: Multidisciplinary Perspectives -- Discussant: PADI perspective on development of a technology to support assessment design. Invited address. IERI PI Meeting. Washington, DC

Hamel, L. (2005). Gradebook. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Quebec.

Hamel, L., Mislevy, R. J., & Kennedy, C. A. (2006). A guide to the PADI gradebook (PADI Technical Report 12). Menlo Park, CA: SRI International.

Hamel. L., & Schank, P. (2005). Participatory, example-based data modeling in PADI (PADI Technical Report 4). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Hamel, L., & Schank, P. (2005). A wizard to create task templates in the PADI design system. Paper presented at the annual meeting of American Educational Research Association. Montreal, Quebec.

Hamel, L., & Schank, P. (2006). A wizard for PADI assessment design (PADI Technical Report 11). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Haynie, K. C., Haertel, G. D., Lash, A. A., & Quellmalz, E. S. (2006). Reverse-engineering the NAEP floating pencil task. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.

Haynie, K. C., Haertel, G. D., Lash, A. A., Quellmalz, E. S., & DeBarger, A. H. (2006). Reverse engineering the NAEP Floating Pencil Task using the PADI design system (PADI Technical Report 16). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Haynie, K. C., Lash, A., Haertel, G. D., Quellmalz, E., & DeBarger, A. (2005). Using the PADI design system to examine the features of a NAEP performance assessment. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Quebec

Kennedy, C. A. (2005). Constructing PADI measurement models for the BEAR scoring engine (PADI Technical Report 7). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Kennedy, C., Timms, M., Long, K., & Ketchner, S., & Bluestein, D. (2005). FOSS: Using PADI to develop an online self-assessment system. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Quebec.

Kennedy, C., & Wilson, M. (2005). Models and tools for drawing inferences from student work. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Quebec.

Lash, A., Haynie, K. C., Riconscente, M. M. (2006). Reverse-engineering science inquiry tasks using PADI. Paper presented at the annual meeting of the National Council on Measurement in Education. San Francisco, CA.

Means, B. & Haertel, G. D. (May, 2002). Technology supports for assessing science inquiry. Washington, DC: National Research Council.

Mislevy, R. (July, 2002). Design patterns for assessing inquiry in science. Paper presented at invitational meeting on Technology and Assessment, Board of Testing and Assessment, Working Group. Washington DC: National Research Council.

Mislevy, R. J., Chudowsky, N., Draney, K., Fried, R., Gaffney, T., Haertel, G., et al. (2003). Design patterns for assessing science inquiry (PADI Technical Report 1). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Mislevy, R., & Haertel, G. (2004). Overview of the Principled Assessment Designs for Inquiry (PADI) project. Paper presented at the annual meeting of the American Educational Research Association. San Diego, CA.

Mislevy, R. J. & Haertel, G. D. (2005). Overview of the PADI design system. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Quebec.

Mislevy, R., Murphy, R., Wilson, M., & Kennedy, C. (2004). Next steps for the PADI project. Paper presented at the annual meeting of the American Educational Research Association. San Diego, CA.

Mislevy, R. J. & Riconscente, M. M. (2005). On layers and objects in assessment design. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Quebec.

Mislevy, R. J., & Riconscente, M. M. (in press). Evidence-centered assessment design: Layers, structures, and terminology (PADI Technical Report 9). Menlo Park, CA: SRI International.

Mislevy, R., Rosenquist, A. & Quellmalz, E. (August, 2003). How Do You Design Assessments for Inquiry in Science? Paper presented at the conference on Optimizing State and Classroom Tests: Implications of Cognitive Research.

Mislevy, R., Steinberg, L.S., Almond, R.G. & Haertel, G.D. (November, 2002). Leveraging points for improving educational assessments. Paper presented at the annual meeting of the American Educational Research Association. New Orleans, LA.

Mislevy, R. J., Steinberg, L. S., Almond, R. G., Haertel, G. D., & Penuel, W. R. (2003). Leverage points for improving educational assessment (PADI Technical Report 2). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Mislevy, R., Steinberg, L. S., Almond, R. G., Haertel, G. D., & Penuel, W. R. (2003). Improving educational assessment. In G. Haertel and B. Means (Eds.), Evaluating educational technology: Effective research designs for improving learning. New York: Teachers College Press.

Mislevy, R., Wilson, M., & Murphy, R. (April, 2003). Next Steps for the PADI Project. Paper presented at the 2003 Annual Meeting of the American Educational Research Association. Chicago, IL.

PADI Research Group (2006). Principled Assessment Designs for Inquiry (PADI): A walk through the online assessment system. Interactive symposium session at the annual meeting of the American Educational Research Association. San Francisco, CA.

Quellmalz, E. S., & Haertel, G. (in press). Assessing new literacies in science and mathematics. In D. J. Leu, Jr., J. Coiro, M. Knobel, & C. Lankshear (Eds.), Handbook of Research on New Literacies. Hillsdale, NJ: Lawrence Erlbaum.

Quellmalz, E. S. & Haertel, G. (2004). Technology supports for state science assessment systems. Paper commissioned by the National Research Council Committee on Test Design for K-12 Science Achievement. Washington, DC: National Research Council.

Quellmalz, E., Rosenquist, A., Butler Songer, N., Wenk, A., & Long, K. (April, 2003). Examples of design patterns from various curricula. Paper presented at the annual meeting of the American Educational Research Association. Chicago, IL.

Riconscente, M., & Haydel, A. (2004). The role of task templates in assessment design. Paper presented at the annual meeting of the American Educational Research Association. San Diego, CA.

Riconscente, M. M., & Mislevy, R. (2006). Reverse-engineering the mystery boxes task. Paper presented at the annual meeting of the National Council on Measurement in Education. San Francisco, CA.

Riconscente, M. M., Mislevy, R. J., Hamel, L., & PADI Research Group (2005). An introduction to task templates (PADI Technical Report 3). Menlo Park, CA: SRI International. Also available at http://padi.sri.com/publications.html

Riconscente, M., & Mislevy, R. J. (2005). Issues in school accountability: A measurement science perspective. In E. Gori & D. Vidoni, (Eds.) Collection: Institutional models in education: Legal framework and methodological aspects for a new approach to the problem of school governance. Rome, Italy: Instituto Nazionale per la Valutazione del Sistema dell Istruzione.

Rosenquist, A. & Hurst, K. (2004). Scientific modeling, inquiry, and assessment. Paper presented at the annual meeting of the American Educational Research Association. San Diego, CA.

Schank, P., & Hamel, L. (2004). Collaborative modeling: Hiding UML and promoting data examples in NEMo. Presentation at the Computer Supported Collaborative Work (CSCW) conference.

Schank, P., Hamel, L., & Kennedy, C. (2004). Technology underlying task templates in the PADI assessment design system. Paper presented at the annual meeting of the American Educational Research Association. San Diego, CA.

Seibert, G., Mislevy, R., & Hamel, L. (2006). Reverse and forward-engineering the mystery powders task. Paper presented at the annual meeting of the National Council on Measurement in Education. San Francisco, CA.

Seibert, G., Hamel, L., Haynie, K., Mislevy, R., & Bao, H. (in press). Mystery powders: An application of the PADI design system using the four process delivery system (PADI Technical Report 17). Menlo Park, CA: SRI International.

Songer, N. B. (2004). Evidence of complex reasoning in technology and science: Notes from inner-city Detroit, Michigan, USA. In V. Milutinovic (Ed.), Internet, Processing, Systems and Interdisciplinary. Belgrade, Serbia and Montenegro: Academic Mind.

Songer, N. B. (April, 2004). Persistence of inquiry: Evidence of complex reasoning among inner city middle school students. Paper presented at the annual meeting of the American Educational Research Association. San Diego, CA.

Songer, N.B. (2004). Evidence of complex reasoning among inner city middle school students. Paper presented at the American Educational Research Association annual meeting. San Diego, CA.

Songer, N. B. (2004). The BioKIDS Research Project/ Interagency Education Research Association (IERI). Invited address. Project Director's Meeting, Alexandria, VA, Sept. 9.

Songer, N. B. (2005). Longitudinal development of complex reasoning in science: Evidence from Detroit. Paper presented at the annual meeting of American Association for the Advancement of Science, Washington, DC.

Songer, N. B. (2005). Technology as a cognitive tool for e-learning in today's classrooms. Paper presented at the conference on Internet, Processing, Systems and Interdisciplinary. Pescara, Italy.

Songer, N. (2005). The invisible challenge of scholarly research. Invited address at the annual meeting of Principal Investigators of IERI Projects, Alexandria, VA.

Songer, N. B. & Gotwals, A. W., (under review), Measuring the development of complex reasoning in science. Cognition and Instruction.

Songer, N. B., & Gotwals, A. W., (2004). What constitutes evidence of complex reasoning in science? The Proceedings of the Sixth International Conference of the Learning Science (ICLS), pp. 497-504. Mahwah, NJ: Erlbaum.

Songer, N. B., & Gotwals, A. (2005). BioKIDS: The design, measurement, and modeling of complex tasks and students' reasoning with PADI. Paper presented at the annual meeting of the American Education Research Association. Montreal, Quebec.

Songer, N. B., Wenk, A., & Long, K. (2004). Examples of task templates from two inquiry curricula. Paper presented at the annual meeting of the American Education Research Association. San Diego, CA.

ON THE WEB:
You can learn more about this project by visiting the PADI Project site at http://padi.sri.com and the Scoring Engine Web site at http://bearcenter.berkeley.edu/padi/index.jsp.