| | Home | About BEAR | Events | Research Projects | Publications | Archives | Measurement | Contact BEAR | | |||||||||||
|
http://bear.berkeley.edu | |||||||||||
|
B E A R C e n t e r | |||||||||||
|
Berkeley Evaluation & Assessment Research Center
| ||||||||||
| Research Projects
|
||
|
The Imperial versus Metric Study (IMS) Involving: Mark Wilson and Yiyu Xie (BEAR) & WESTAT The Imperial versus Metric Study (IMS) carried out jointly by BEAR and WESTAT is part of a larger study, namely PISA (Programme for International Student Assessment). PISA was coordinated by governments of participating countries, through the Organisation for Economic Co-operation and Development (OECD). A survey of the knowledge and skills of 15-year-olds in the principal industrialized countries was first conducted in 2000, and will be repeated every three years. PISA assesses literacy in reading, mathematics and science. (For more info, check out the PISA web page www.pisa.oecd.org) The overall purpose of the IMS is to
examine whether there is any support for the speculation that US
students might be relatively disadvantaged in international comparisons
of achievement by the use of metric units (e.g., meters, liters,
etc.) in items rather than the more familiar (to US students) imperial
units (e.g., feet, gallons, etc.). The PISA 2003 main study
has a focus on mathematics, therefore, matching imperial versions of
math items containing metric units have been made. Data will
be collected on both imperial and metric samples from the
|
|
|
|
Standard-Setting Using the Construct Map Involving: Mark Wilson, Karen Draney, Cathy Kennedy, Deborah Peres, Sevan Tutunciyan, Hiroyuki Yamada. (BEAR) Wilson and Draney (2002) have described a new method, called "construct mapping" for setting standards in achievement testing that takes advantage of the image of a "progress map" based on an item response model analysis of the data. This approach allows the standard setters to give interpretational meaning to potential cut-scores in terms of the responses to both multiple choice and open-ended items. It can also be used as a basis for the "propagation" of the selected cut-scores to new tests using the linking facilities of item response modeling. In the current project, we will investigate the improvement and use of this method in several ways. First, we have written a new software tool that greatly enhances the usefulness of the construct mapping approach. Second, we will use qualitative methods to investigate the best ways to help standard setters use the new tool. Third, we will apply the approach to a more complex scoring procedure involving items scored in multiple (correlated) dimensions.
|
|
|
|
Work Circles: Developing research sensitive teachers and practice-sensitive researchers. Involving: Mark Wilson, Brent Duckor (BEAR), Kathy Long (LHS), school district curriculum coordinator and/or assessment specialist, and several teachers Consultants: Paul Black and one scientist This is a 15 month applied research project to investigate the implementation and impact of a pilot implementation of an innovative group research and professional development structure called a Work Circle. It is proposed by the Lawrence Hall of Science (LHS) and BEAR, working with partner schools. A Work Circle will be established and its organization, operation, and accomplishments will be investigated over time. A Work Circle is conceived as a vital and dynamic small organizational unit, engaging specialists, practitioners, and researchers in collaborative R&D. The partners on this proposal have worked in collaborative, project-oriented environments for many years, and know that focused teamwork can produce results. Within our existing R&D teams, we have experienced how the innovative collaboration of specialists and practitioners working on problems in authentic settings is the most powerful way to achieve solutions and support the mutual professional development of highly qualified people who can bridge the gap between research and practice. This project will produce a number of products (several types of new assessments, implementation approaches, "next-steps" strategies) as well as significant professional development. Major outcomes include (1) best practices for implementation of an assessment system, based on the principles of the BEAR Assessment System; (2) Each of the types of educational professionals involved in the Work Circle (teachers, specialists and researchers) will have the opportunity (a) to improve their skills in working in multidisciplinary and multi-layered teams, and (b) to understand the different specialist roles that they might themselves decide to move towards, and to try-out some of the activities of those roles. |
|
|
Constructing a Framework for Science Assessment Systems BEAR Participants: Mark Wilson, Cathy Kennedy,
Hueying Tzou, Lydia Liu, Nathaniel Brown This project seeks to create, validate, and publish a practical and research-based framework to guide state science assessment systems in striving to meet the criteria of coherence, comprehensiveness, and continuity. Our research and development for the framework will draw on existing research and will reflect our belief in the need for appropriate usage of valid assessment systems grounded in sound cognitive theories of science learning. Our goal is to produce a framework that can be used by policy-makers, state and district education staff, and others involved in decision-making processes that measure student achievement in science. The framework will support decision-makers who need to choose a set of curriculum and assessments (both formative and summative) or develop a new assessment system to match a set of standards and curriculum. As such, the framework will be for use primarily by those involved in choosing, specifying, or designing state or district science assessments, such as State Board of Education members, State Department of Education staff, and school district staff. The framework will assist states in different ways dependent upon their needs. For states wanting to create a science assessment system, it will provide a design blueprint for the development of new assessment systems that will link standards-based, large-scale assessments to classroom instructional assessments. For other states wanting to augment assessments already in place, or wanting to evaluate assessments they intend to adopt, the framework will provide a lens through which to look at the quality of existing large-scale assessments and a method for linking them to standards, to classroom instructional assessments, and to national and international measures of science performance (e.g., TIMSS, NAEP). The framework will also provide guidance for teachersí classroom instructional and assessment practices to best maintain the linkage of effective science learning, standards, and assessment use. |
||
|
Assessing Science Knowledge (ASK) Involving: Mark Wilson, Stephen Moore, Linda Woodward, and graduate students (BEAR); Kathy Long, Linda DeLucchi, and Larry Malone (LHS); Geneva Haertel, Edys Quellmatz, and Angela Haydel (SRI); & facilitators and teachers from eight national test centers The Assessing Science Knowledge (ASK) project is designed to define, field test, and validate effective assessment tools and techniques to be used by grade 3ñ6 classroom teachers to assess, guide, and confirm student learning in science. The assessments are being conceptualized, developed, and refined using one exemplary science-education program, the Full Option Science System (FOSS). During the first two years of the project, the BEAR Center is helping with the development and refinement of the frameworks, progress maps, items, scoring guides, and other elements of the system. We are providing guidance and support in the psychometric data analysis and, where appropriate, also performing such analysis and assisting in the interpretation of the results. During the final two years of the ASK project,
the BEAR Center will perform calibration analyses of the assessment
data gathered for the items and tasks developed during the first
two years. We will assist in selecting appropriate measurement models,
use available software to produce parameter estimates for these
models, and provide assistance in making these parameter estimates
useful to the various stakeholders (e.g., by setting up GradeMap
to produce student progress maps for teachers using the FOSS system).
In addition, the BEAR Center will conduct standard setting studies
as appropriate, to select cut scores for criterion zones (qualitatively
different levels of student performance) for use on these progress
maps. |
|
|
|
PADI: Principled Assessment Designs for Inquiry Involving: Mark Wilson, Cathy Kennedy, Sevan Tutunciyan, Karen Draney (BEAR), and Kathy Long (LHS) Project Partners: Bob Mislevy (University of Maryland), Nancy Butler-Songer (University of Michigan), Geneva Haertel (SRI) The overarching purpose of the PADI project is to provide a practical framework for designing assessments of science inquiry skills. The framework will be grounded in recent developments in cognitive psychology, research on science inquiry, and advances in measurement theory and technology. Project objectives include 1) articulation of a conceptual framework for designing, delivering, and scoring complex assessment tasks that can be used to assess inquiry skills in science; 2) providing templates and related resources to assist curriculum designers in developing assessments in the same conceptual framework; 3) defining system requirements for an assessment delivery and performance evaluation system; and 4) providing a digital library of working exemplars including a sample assessment application. The BEAR team is working primarily on developing new methods of psychometric modeling to reflect the cognitive structures theorized by curriculum developers. This involves creation of software that integrates with external assessment databases for initial model testing and calibration plus ongoing dynamic estimation of student proficiencies. A primary challenge faced by the BEAR team is to provide a seamless integration of the measurement components to the data gathering and presentation components developed by the other teams. System developers will follow emerging guidelines for educational technology standards to help ensure portability of the system. Our goal is to provide useful interpretations of student learning and performance tailored to specific cognitive theories underlying diverse curricula development efforts. See http://padi.sri.com/ for more information. |
|
|
Evaluating the Validity of Teacher Licensure Decisions Involving: Pamela Moss, Principal Investigator, University of Michigan; Mark Wilson, Co-Principal Investigator, University of California, Berkeley; Ray Pecheone, Co-Principal Investigator, Stanford University; Jean Miller, Director, INTASC; Kathryn Lind, Wisconsin Department of Public Instruction; Laura Haniford, Research Associate, University of Michigan; Renee Miller, Research Associate, University of Michigan; PJ Hallam, Research Specialist, BEAR A Collaborative Effort of The University of Michigan, The University of California, Berkeley, The Interstate New Teacher Assessment and Support Consortium (INTASC), and The Wisconsin Department of Public Instruction The goal of this research program is to develop and evaluate the validity of a portfolio assessment for early career elementary teachers with potential use in the WI Master Educator certification process. Validity research for the newly developed portfolio is being undertaken by the University of Michigan (UM) and the University of California, Berkeley (UCB). The research is designed to evaluate the quality of the portfolio assessment by comparing completed portfolios to (a) qualitative case studies of teaching; (b) other indicators of preparation for teaching; and (c) gains in students' achievement. We hope to find out how useful portfolios might be as part of Wisconsin's Master Educator certification process. Portfolio development was undertaken by the Interstate New Teacher Assessment and Support Consortium (INTASC), in collaboration with the Wisconsin Department of Public Instruction (WIDPI), UM, and UCB. It is based on the Wisconsin Standards for Teacher Development and the INTASC Standards for Beginning Elementary Teachers. The goal is to evaluate the usefulness of the portfolio assessment. Data collection for all participating teachers will include: a completed portfolio, fall and spring administrations of an achievement test to their students, a content knowledge test, and a survey of their teaching preparation and practice. Teachers will receive support in the development of their portfolios. Approximately one third of participating teachers will be asked to allow 3-5 day case studies of their teaching practice including observations and interviews. For further information, contact PJ Hallam, pjhallam@berkeley.edu |
||