Every Change Is an Opportunity to Improve
With change comes opportunity. Assessment developers look for ways to ensure that their tests provide valid, consistent, and fair results. As students progress through their K-12 education, high-quality assessments that emphasize critical thinking and problem solving are needed to determine whether students are on track to be college- and career-ready by graduation. One of the critical aspects of ensuring successful implementation of anything new—including high-stakes tests—is evaluation.
Among the groups at the forefront of this challenge is the High-Quality Assessment Project (HQAP), an initiative led by Education First and supported by a coalition of national foundations. The HQAP supports state-based advocacy, communications, and policy work to help ensure successful transitions to new assessments.HQAP sponsors the Center for Assessment to develop methodologies and procedures that can be used to evaluate summative assessments designed to measure college and career readiness. The Center for Assessment has developed a framework that can be used by the evaluation team, as well as assessment developers and measurement professionals who want to improve the quality of assessments. (See the Center of Assessment’s “A Guide to Evaluating College- and Career-Ready Assessments.”)The framework is based on the Criteria for Procuring and Evaluating High-Quality Assessments that have been established by the Council of Chief State School Officers (CCSSO). These criteria—as well as any criteria for evaluating assessments—focus both on content and delivery.
CCSSO Categories of Criteria for High-Quality Assessments
A. Meet Overall Assessment Goals and Ensure Technical Quality
B. Align to Standards – English Language Arts/Literacy
C. Align to Standards – Mathematics
D. Yield Valuable Reports on Student Progress and Performance
E. Adhere to Best Practices in Test Administration
It’s Complicated
Evaluation of summative assessments is highly complex. Assessments have different desired purposes, outcomes, and expectations. How the results are to be used and the stakes that are involved may vary significantly. These factors must be taken into consideration when developing an evaluation strategy.
Because of the subjectivity involved in evaluating assessments, having a clearly defined framework is essential. Evaluator training is the key to ensuring that all evaluators have the same understanding of the criterion and evidence to be included in the evaluation.
Phases of the Test Characteristics Evaluation Methodology
Phase 1: Preparation, including the collection of evidence, establishment of timelines, and identification of the evaluation team.
Phase 2: Evaluator Training and Independent Evaluator Review.
Phase 3: Group Discussion and Summary of Evaluation Findings.
Phase 4: Report Generation and Approval.
The Criterion Evaluation Framework developed by the Center for Assessment expands upon and draws connections among the criteria established by the CCSSO. The goal is to operationalize criteria in a manner that supports a fair, transparent, and reliable evaluation. To this end, the Center has articulated a set of claims—defined as statements about procedures, materials, reports, and data that underlie each CCSSO criterion—as well as statements that describe features and characteristics that should be reflected in the evidence used to evaluate each criterion. Comments are included to aid reviewers to judge the quality of evidence within the context of the specific assessment program.
Finally, for an evaluation to be effective, results need to be communicated to all interested parties—including educators and assessment developers. Just as having clearly delineated standards helps ensure that educational materials are designed to reach consensus goals, having clearly delineated criteria for assessment can help to ensure that the developers of these tests reach their goals. A partnership among the parties all along the continuum can help to ensure not only that assessments are fair and that results are valid, but also that any changes in assessments will bring about desired educational goals.
