Data Research and Development Center and IERI Research Cmmunity header

The IERI Research Community :: Projects

IERI: Assessments to Support the Transition to Complex Learning in Science

PRINCIPAL INVESTIGATOR:
Eva L. Baker

CO-INVESTIGATOR:
David Niemi

CATEGORIES:
Science

PROJECT OVERVIEW:
Background: It has been compellingly demonstrated that formative assessments can effectively improve student achievement and understanding (e.g., Black and Wiliam, 1998a, 1998b; Black and Harrison, in press; Black, Harrison, Lee & Wiliam, 2001), but practical constraints limit the productive use of formative assessments in classrooms. Teachers and school district staff may lack the time and expertise necessary to build or find good formative assessments, score student responses, and use those responses to draw conclusions about student learning. An even more daunting challenge is using formative assessment information to change the course of instruction when necessary; this is a challenge for teachers at all levels, but it is particularly acute for middle and high school teachers, who may be teaching as many as 6 classes and well over 200 students per day.

Purpose: To help teachers use formative assessment to improve student learning, CRESST researchers have developed and tested an on-line authoring system that can be used to build assessments of complex learning across subject areas and grade levels. The assessment system provides: 1) utilities for individual teachers, teams of teachers, or others to become designers and users of assessments that yield actionable information to guide instruction and student learning, 2) embedded content, assessment, and pedagogical knowledge to assist teachers in designing assessments and interpreting student progress, and 3) a database of assessments and assessment components.

Intervention: Two assessment tools were developed and tested in this project:

  1. The Assessment Design and Delivery System (ADDS) is a general assessment authoring and delivery system that integrates knowledge from practice, model-building, and assessment research in a suite of powerful tools intended to improve teachers assessment and instructional practices; teachers and others can use the ADDS to design and administer assessments of middle school students' understanding of big ideas in science.
  1. The Diagnoser is a web-based assessment program with pre-designed sets of questions that teachers can use as formative assessments. Based on students' responses to a series of assignments, the Diagnoser provides information on student FACETs or pieces of knowledge and strategies of reasoning about a concept that teachers can use to focus instruction on problematic areas.

    Setting: Studies using the ADDS were carried out in Los Angeles, Vista, Montebello, Riverside, Santa Monica/Malibu and San Diego Unified School Districts in southern California, Tracy Unified School District in northern California and Vancouver, Washington. Additional usability and training studies were conducted in LAUSD. The project timeline is from January 2002 to December 2006 and has proceeded as follows:

    • Year 1 (2002): Initial work focused on constructing domain (science) knowledge maps and creating assessment design frameworks. Elements of assessment models were constructed and a start was been made implementing the Assessment Design and Delivery System (ADDS).
    • Year 2 (2003): Second-year work has focused on knowledge construction and integration, constructing elements of assessment models, and the design work necessary to build the ADDS. We started creating a database of assessments and also made a start with testing the ADDS and assessments internally as well as in a usability study.
    • Year 3 (2004): In year 3 our work centered on the continuation of creating a database of assessments. We tested the ADDS internally and made revisions to the ADDS and it s assessments. Two IERI research studies were conducted to see the impact of the ADDS on teachers assessments. At the same time studies on subsets and sequences of tasks were conducted. Two pilot studies for the Center for the Assessment and Evaluation of Student Learning (CAESL) and one IERI study.
    • Year 4 (2005): During this period, our work continued to focus on creation of assessments and on refinement and testing of the ADDS. We also conducted another IERI research study to evaluate the impact of the ADDS on teachers assessments and one in which teachers gave a pretest, short answer assessments, and a posttest to their students. A start was made with the implementation into large scale use in urban school districts by CAESL in Vista, Montebello, Tracy, and San Diego.
  • Year 5 (2006): This last year will focus on more testing of the ADDS and assessments and refining both of these. We will continue with validity studies and implementation into large scale use in urban school districts. In addition we will look at the effects of teacher knowledge and experience. Lastly we will prepare the final report.

    Research Design: In an NSF-sponsored project that was intended to develop teachers' capacity to design and use assessments to support science learning, 18 participants developed approximately 50 items in each of 4 domains (electricity & magnetism, water cycle, ecosystems, and properties of matter) using the Assessment Design and Delivery System (ADDS). Four final instruments consisting of 20 selected-response and 2 open-ended items were then field tested with approximately 400 students from the same 4 school districts used to pilot the initial items. Another study on the effects of the ADDS on quality of teacher assessments involved 33 grade 7-9 science teachers from Los Angeles. Fifteen of these teachers taught life science; 10, physical science; and 8, both life and physical science. To see whether using the ADDS helps teachers to design assessments of comparable quality and difficulty on the same topic, 17 teachers were randomly assigned to a treatment and control group, 1 using the ADDS and 1 not. Finally, 14 teachers in Vancouver, WA administered the ADDS and Diagnoser assessments to their students in a study investigating the value added by each type of assessment.

    To evaluate the quality of teacher-designed assessments, a 2-dimension scoring rubric was developed to measure the cognitive demands of a task and the general task quality, including clarity of instructions. Rater reliability on these dimensions was above 0.8. In another study, items designed by teachers using the ADDS were piloted with approximately 8 teachers and 200 students in 4 California school districts. Results from the selected response items were analyzed using standard statistical models, and results from the open-ended items were scored using rubrics developed by participants as part of the item construction process. The participants constructed a final instrument based on coverage of the big ideas in each of the domains, the coverage of state standards in each of the 4 domains and the technical quality of each item. Responses from 400 students are currently being analyzed.

    Findings: Use of the ADDS caused teachers to focus more on important conceptual knowledge in a domain and on creating appropriate rubrics as part of the item development cycle, compared with teachers not using the ADDS. When teachers used the ADDS, their question prompts and scoring criteria became more coherent and tended to address critical ideas rather than seemingly unrelated facts within the domain. In brief, the ADDS improves the assessment capabilities of those using it.

    PROJECT PUBLICATIONS:
    Baker, E.L. (2001). Authoring and technology support for assessment design and interpretation. Presentation to the National Research Council, Board on Testing and Assessment, Woodshole, MA.

    Baker, E. L. (2002). Design of automated authoring systems for tests. Proceedings of Technology and assessment: Thinking ahead proceedings from a workshop." (pp. 79-89). Published of Collection: National Research Council, Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Sciences and Education.

    Baker, E. L. (2002). The promise of technology-based assessment. Presentation at the IAETE/AEL symposium, Dallas, TX.

    Baker, E. L. (2002b). Authoring and technology support for assessment design and interpretation. Presentation to the National Research Council, Board on Testing and Assessment, Woodshole, MA.

    Baker, E. L. (2002, November). The promise of technology-based assessment. Presentation at the IAETE/AEL Symposium, Dallas, TX.

    Baker, E. L. (2003). Improving accountability models by using technology-enabled knowledge systems (TEKS). Paper prepared for the Education Commission of the States, Denver, CO.

    Baker, E. L. (2003). Improving accountability models by using technology-enabled knowledge systems (TEKS). Paper prepared for the Education Commission of the States, Denver, CO.

    Baker, E. L. (2003, April). Evidence-based interventions: What then and now what? Paper presented at the annual meeting of the American Educational Research Association. In M. Constas session 48.010 PRES-59, Advancing the Scientific Investigation of Large Scale Interventions: The Interagency Education Research Initiative (Presidential Invited Session), Chicago.

    Baker, E. L. (2003, May). Pull and push assessment transformation. Presentation at The Learning Federation s Learner Modeling and Assessment Workshop, Seattle, WA.

    Baker, E. L. (2003, November). Principles for measuring scaling up. Presentation for the IERI, NORC conference Conceptualizing Scale-Up: Multidisciplinary Perspectives, Washington, DC.

    Baker, E. L. (2003, November). Transforming performance: Technology, testing, accountability and improvement. Presentation at the Syfr Corporation seminar: Technology: A Tool for Educational Innovation, San Antonio, TX.

    Baker, E. L. (2004, January). Assessment transformation: Addressing learning and transfer. Presentation to the Hawaii International Conference on Education, Honolulu, HI.

    Baker, E. L. (2004, February). Beyond gee whiz technology-minded assessment. Presentation at American Association for the Advancement of Science Annual Meeting in the session Integrating Technology with Cognitive Science to Improve Assessment and Learning, Seattle, WA.

    Baker, E. L. (2004, March). Assessing and monitoring performance across time and place. Presentation at the U.S. Department of Education Secretary s No Child Left Behind Leadership Summits Empowering Accountability and Assessment Using Technology, St. Louis, MO.

    Baker, E. L. (2004, March). Learning, assessment, and accountability: Priorities for educational reform. Presentation to the British Columbia Ministry of Education Conference Public Education in BC and the World, Victoria, BC.

    Baker, E. L. (2004, July). Evaluation and assessment. Presentation at the American Association of School Administrators Summer Institute for Rural and Suburban School System Leaders Stand Up for Public Education: Taking the Next Steps, Dana Point, CA.

    Baker, E. L. (2004, July). Research and assessment of learning outcomes. Presentation at the Hewlett Foundation Games for Improving K-12 Learning Meeting, Menlo Park, CA.

    Baker, E. L. (2004, September). Toward multipurpose test design for accountability and improvement. In L. Winters (Chair). Symposium conducted at the annual CRESST conference, Los Angeles.

    Baker, E. L. (2005). Technology and effective assessment systems. In J. L. Herman & E. H. Haertel (Eds.), Uses and misuses of data for educational accountability and improvement (NSSE Yearbook, Vol. 104, Part 2, pp. 358-378).

    Baker, E. L. (2005, March). Technology to improve assessment. Presentation at the University of New South Wales, Sydney, Australia.

    Baker, E. L. (2005, July). Improving accountability models by using technology-enabled knowledge systems (TEKS) (CSE Rep. No. 656). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

    Baker, E.L. (2005, September). Accelerating future possibilities for assessment and learning. In S. Thompson (Chair). Symposium conducted at the annual CRESST conference. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

    Baker, E. L. (2004, October). Future of assessment. Presentation at the Air Force Research Laboratory Future of Learning, Instruction, Assessment and Performance Technology Symposium, Mesa, AZ.

    Baker, E. L. (forthcoming). Principles for scaling up: choosing, measuring effects, and promoting the widespread use of educational innovation. In B. Schneider and S. McDonald (Eds.), Scale-Up in Education, Vol. 1: Principles. Lanham, MD: Rowman & Littlefield.

    Baker, E. L. & Niemi, D. (2001). Assessments to support the transition to complex learning in science. Presentation at IERI PI conference, Arlington, VA.

    Baker, E.L. & O Neil, H.F., Jr. (2003). Evaluation and research for technology: Not just playing around. Evaluation and Program Planning, 26, 169-176.

    Hunt, E. and T. Madhyastha (2005). Data mining patterns of thought. Presentation at The Twentieth National Conference on Artificial Intelligence (AAAI), Pittsburgh, PA.

    Kraus, P. (2005). Introduction to a diagnostic learning environment. Presentation at the Association of Physics Teachers (AAPT), Dallas, TX.

    Kraus, P., S. Williamson, et al. (2005). Implementing a Diagnostic Learning Environment Workshop for a consortium of School Districts around Seattle, Institute for Systems Biology, Seattle, WA.

    Lee, J. J. (2004, September). Technology tools and what they tell us: The Assessment Design and Delivery System (ADDS) and the Quality School Portfolio (QSP). Paper presented at the Annual CRESST conference. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

    Lee, J. J., & Vendlinski, T. P. (2005, April). Computer-guided assessment design. In F. C. Sloane & J. W. Pellegrino (co-Chairs), Moving technology up-design requirements for valid, effective classroom and large-scale assessment. Symposium conducted at the annual meeting of the American Educational Research Association, Montreal.

    Minstrell, J. (2004). Building on what students know. Presentation at National Science Teachers Association (NSTA), Seattle, WA.

    Minstrell, J. (2005). Facets and Facet clusters: A classroom assessment framework. Presentation at the American Educational Research Association (AERA), Montreal.

    Minstrell, J. and P. Kraus (2005). Guided inquiry in the science classroom. In M. S. Donovan and J. D. Bransford (Eds.), How Students Learn: History, Mathematics, and Science in the Classroom. Washington, D.C., The National Academies Press: 475-513.

    Minstrell, J., P. Kraus, et al. (2005). Using Facets and Facet clusters to design diagnostic assessments. Presentation at the American Educational Research Association (AERA), Montreal.

    Minstrell, J. E., J. Minstrell, and T. Madhyastha (2005). The Diagnoser Project demonstration. Presentation at the American Educational Research Association (AERA), Montreal.

    Niemi, D. (2002). Building a research-based authoring system. Presentation at annual CRESST advisory board meeting, Los Angeles, CA.

    Niemi, D. (2002). Ontologies to support assessment, instruction and learning. Presentation at annual CRESST conference, Los Angeles, CA.

    Niemi, D. (2003, April). Assessment design using ontologies: Linking assessments, cognitive demands, and content. Presentation at American Education Research Association, Chicago, IL.

    Niemi, D. (2005, April). Assessment objects for domain-independent and domain specific assessment. In F. C. Sloane & J. W. Pellegrino (co-Chairs), Moving technology up-design requirements for valid, effective classroom and large-scale assessment. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

    Niemi, D. (2005, September). Making evidence-based practice a reality in schools and classrooms. In R. Young (Chair). Symposium conducted at the annual CRESST conference. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

    Niemi, D., & Baker, E.L. (2005, April). Reconceiving assessment shortfalls: System requirements needed to produce learning. In F. C. Sloane & J. W. Pellegrino (co-Chairs), Moving technology up-design requirements for valid, effective classroom and large-scale assessment. Presentation at the annual meeting of the American Educational Research Association, Montreal.

    Niemi, D., & Vallone, J. (2006). Tools to improve formative assessment. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
    Niemi, D., Vendlinski, T. P., Baker, E. L., & Wang, J. (In press.) On-line tools to improve formative assessment. British Journal of Educational Technology.

    Niemi, D., Vendlinski, T. P., & Wang, J. (November, 2004). Computer guided assessment design: Assessing complex learning. Paper presented at the annual meeting of the California Educational Research Association, Pasadena, CA.

    Niemi, D., Wang, J., de Vries Adreani, L. F., Lee, J. J., & Vendlinkski, T. P. (2005, April). Computer-guided assessment design and delivery: Two demonstrations. In J. H. Earle (Chair). Symposium conducted at the annual meeting of the American Educational Research Association, Montreal.

    Vendlinski, T.P. (2006, April). Making connections: Fusing assessment results to improve inference and instruction. Keynote address at the Annual Conference of the Association of State Supervisors of Mathematics, St. Louis, Missouri.

    Vendlinski, T. P. (2005, September). What's the "Big Idea"? Presentation to the NATO Task Group on Advanced Authoring, Capabilities and Opportunities. Los Angeles, CA.

    Vendlinski, T.P. (2005, September). Teaching to the "Big Ideas": Moving beyond the standards. In S. Nagashima (Chair), Innovative Measures of Learning and Instruction. Symposium conducted at the 2005 CRESST Conference, Los Angeles, CA.

    Vendlinski, T., & Niemi, D. (2006, April). Making simulations educationally beneficial. In B. Bewley (Chair), Using Simulations for Assessment. Symposium conducted at the annual meeting of the American Educational Research Association, San Francisco, CA.

    Vendlinski, T. P., Niemi, D., & Baker, E. L. (in press). Objects and templates in authoring problem-solving assessments. In E. L. Baker, J. Dickieson, W. Wulfeck, & H. F. O Neil (Eds.), Assessment of problem solving using simulations. Mahwah, NJ: Erlbaum.

    Vendlinski, T., Niemi, D., & Wang, J. (2005). Learning assessment by designing assessments: An On-line formative assessment design tool. In C. Crawford, R. Carlsen, I. Gibson, K. McFerrin, J. Price, & R. Weber (Eds.) (2005). Proceedings of Society for Information Technology and Teacher Education International Conference 2005 (pp. 228-240). Norfolk, VA: AACE.

    Vendlinski, T.P., Niemi, D.M., Wang, J., & Monempour, S. (2005). Improving formative assessment practice with educational information technology. In F. Malpica, F. Welsch, A. Tremante, & J. Lawler (Eds.), The 3rd International Conference on Education and Information Systems: Technologies and Applications: Vol. 1 (pp. 361 - 366). Orlando, FL: International Institute of Informatics and Systematics.

    Wang, J., & Vendlinski, T. (2006, April). Online assessment within an ontological framework. Presentation at the annual meeting of the American Educational Research Association, San Francisco, CA.

    ON THE WEB:
    You can learn more about this project by visiting the ADDS website at http://adds.cse.ucla.edu/ and the DIAGNOSER website at http://www.diagnoser.com/diagnoser/.