By Gary A. Troia and Rachel Marias
Three major policy streams – standards, accountability, and assessment – have converged in recent years. Standards-based reform argues that top-down control of student expectations and accountability for meeting those expectations (as measured through high-stakes testing) will improve student achievement. Before adoption of the Common Core State Standards (CCSS), states had varied standards and purportedly aligned assessments to evaluate the impact of such reform. The researchers examined how well individual state standards were aligned with their associated assessments prior to CCSS adoption. To be more specific, the researchers performed a content analysis of a selection of state standards and their assessments for writing using a multi-theoretical framework for writing and determined the degree of alignment (cell-by-cell matches in content) between them for each state.
Methods: How did we study alignment?
Content of writing standards and assessments were mapped onto a common coding framework (similar to the procedure used by the Survey of Enacted Curriculum (e.g., Porter, 2002)). This framework incorporated writing process, writing context, writing purposes, writing components, writing metacognition and knowledge, and writing motivation. Trained raters independently coded the standards and assessments using the coding framework. For example, the standard, “Create multiple-paragraph narrative compositions: Establish and develop a situation or plot, describe the setting, present the ending” was assigned codes for text length demands, narrative purpose, and canonical narrative content. Alignment indices (averaged across grade bands) were used to quantify the degree of exact match between the content of standards and that of assessments for each state. The index ranges from 0 to 1, with 1 indicating perfect alignment.
Results: What did we find?
When looking at the visual representation of the data in the following map, it is apparent that overall there is generally not a high degree of alignment; the most well aligned state elementary (grades 3-5) writing standards align with the corresponding writing assessment only about two-third to three-quarters of the time. This means that teachers may be focusing their instructional efforts on writing expectations that are not assessed or, of greater concern, that students are being tested on writing knowledge and skills that are likely not taught.
For the 27 states for which data were available, the average alignment index for grades 3-5 is 0.41 and grades 6-8 is 0.40. The majority of states had at least one set of standards and assessments (grades 3-5 or 6-8) with average alignment (based on the alignments obtained with this sample). States that are colored white exhibit average alignment at both grade bands (5 states) while states with light blue shading (8 states) have one grade band with average alignment and the other above average. States with light red shading (7 states) have one grade band with average alignment and the other below average. Darker blue states (3 states) have above average alignment across grade bands and darker red states (3 states) have below average alignment across grade bands. One state, VA (shaded purple), exhibited a large discrepancy in alignment between the two grade bands examined.
Discussion: What does this all mean?
It is important to note that no state showed very strong alignment at all grade levels: at best, there was only 75% overlap between the content of writing standards and assessments. However, the alignment index does not indicate whether it is the set of standards or the assessment materials that contain different content, so this is an inherent limitation of this type of content measure. Weak alignment is especially troubling when a test assesses content not taught because it is excluded from standards. This may result in writing test scores that are disproportionately lower in poorly aligned states/grades than in those where the content of standards and assessments are well aligned. The new Common Core standards and aligned assessments will have to guard against such misalignment.
Note: This research was supported in part by grant #R305A100040 from the U.S. Department of Education, Institute of Education Sciences, to Michigan State University. Statements do not necessarily reflect the positions or policies of this agency, and no official endorsement by it should be inferred.
Gary A. Troia is an associate professor of special education. His research interests include the connections between oral language and literacy in typical and atypical learners, writing assessment and instruction, and teacher professional development in literacy. His recent work involves examining alignment between states’ content standards and assessment frameworks in writing and how alignment between these influences writing outcomes and enables students to meet postsecondary writing expectations.
Contact Dr. Troia: firstname.lastname@example.org