Making Classroom Assessments Reliable and Valid. Robert J. Marzano

      chapter 1

      Discussing the Classroom Assessment Paradigm for Validity

      Validity is certainly the first order of business when researchers or educators design CAs. The concept of validity has evolved over the years into a multifaceted construct. As mentioned previously, the initial conception of a test’s validity was that it measures what it purports to measure. As Henry E. Garrett (1937) notes, “the fidelity with which [a test] measures what it purports to measure” (p. 324) is the hallmark of its validity. By the 1950s, though, important distinctions emerged about the nature and function of validity. Samuel Messick (1993) explains that since the early 1950s, validity has been thought of as involving three major types: (1) criterion-related validity, (2) construct validity, and (3) content validity.

      While the three types of validity have unique qualities, these distinctions are made more complex by virtue of the fact that one can examine validity from two perspectives. John D. Hathcoat (2013) explains that these perspectives are (1) the instrumental perspective and (2) the argument-based perspective. Validity in general—and the three different types in particular—look quite different depending on the perspective. This is a central theme of this chapter, and I make a case for the argument-based perspective as superior, particularly as it relates to CAs. The chapter also covers the following topics.

      ■ Standards as the basis of CA validity

      ■ Dimensionality

      ■ Measurement topics and proficiency scales

      ■ The rise of learning progressions

      ■ The structure of proficiency scales

      ■ The school’s role in criterion-related validity

      ■ The nature of parallel assessments

      ■ The measurement process

      I begin by discussing the instrumental perspective and its treatment of the three types of validity.

The Instrumental Perspective

      The instrumental perspective focuses on the test itself. According to Hathcoat (2013), this has been the traditional perspective in measurement theory: a specific test is deemed valid to one degree or another. All three types of validity, then, are considered aspects of a specific test that has been or is being developed within the instrumental perspective. A test possesses certain degrees of the three types of validity.

      For quite some time, measurement experts have warned that the instrumental perspective invites misinterpretations of assessments. For example, in his article “Measurement 101: Some Fundamentals Revisited,” Frisbie (2005) provides concrete examples of the dangers of a literal adherence to an instrumental perspective. About validity, he notes, “Validity is not about instruments themselves, but it is about score interpretations and uses” (p. 22). In effect, Frisbie notes that it is technically inaccurate to refer to the validity of a particular test. Instead, discussion should focus on the valid use or interpretation of the scores from a particular test. To illustrate the lack of adherence to this principle, he offers examples of inaccurate statements about validity from published tests:

      1. “You can help ensure that the test will be valid and equitable for all students.” (From an examiner’s manual for a statewide assessment program, 2005)

      2. “Evidence of test validity … should be made publicly available.” (From a major publication of a prominent testing organization, 2002)

      3. “In the assessment realm, this is referred to as the validity of the test.” (From an introductory assessment textbook, 2005)

      4. “[Test name] has proven itself in use for more than 50 years as a … valid test.” (From the website of a prominent test publisher, 2005)

      5. “Such efforts represent the cornerstone of test validity.” (From the technical manual of a prominent achievement test, 2003). (Frisbie, 2005, p. 22)

      Challenges like that presented by Frisbie notwithstanding, the instrumental perspective still dominates.

      Criterion-related validity, construct validity, and content validity contain certain requirements if a test is deemed valid from the instrumental perspective. To establish criterion-related validity for an assessment from the instrumental perspective, a researcher typically computes a correlation coefficient between the newly developed test and some other assessment considered to already be a valid measure of the topic. This second assessment is referred to as the criterion measure; hence the term criterion-related validity. A test is considered valid for any criterion it predicts accurately (Guilford, 1946).

      The major problem with criterion-related validity is that it is difficult in some cases to identify an appropriate criterion measure. Citing the work of Roderick M. Chisholm (1973), Hathcoat (2013) exemplifies the criterion problem using the ability to determine the quality of apples:

      If we wish to identify apple quality then we need a criterion to distinguish “good” apples from “bad” apples. We may choose to sort apples into different piles based upon their color, though any criterion is adequate for this example. The problem arises whenever we ask whether our criterion worked in that color actually separated good apples from bad apples. How can we investigate our criterion without already knowing something about which apples are good and bad? (pp. 2–3)

      In effect, identifying an appropriate criterion measure renders criterion-related validity very difficult for test designers in general and for classroom teachers in particular.

      Construct validity became prominent about halfway through the 1900s. According to Hathcoat (2013), a seminal article in 1955 by Lee J. Cronbach and Paul E. Meehl led to a focus on construct validity. Hathcoat (2013) notes that “Cronbach and Meehl were concerned about situations wherein a target domain and/or a relevant criterion remained ambiguous” (p. 3).

      Cronbach and Meehl (1955) were saying that construct validity must be established for any type of content for which it is difficult to find a criterion measure (as cited in Hathcoat, 2013). For example, where it is rather easy to find a criterion measure for content like fifth-grade geometry, it is quite difficult to find criterion measures for content like students’ abilities to apply knowledge in unique situations or students’ abilities to make good decisions. Any instrument designed to measure these topics must establish construct validity evidence of what such an ability entails.

      In the middle of the 20th century, around the same time Cronbach and Meehl (1955) established the need for construct validity, statistical procedures became readily available that allowed psychometricians to induce the nature of an otherwise ambiguous construct. One such statistical procedure is factor analysis, which mathematically provides evidence that specific items on a test measure the same construct. (For a technical discussion of factor analysis, see Kline, 1994.) This type of analysis is also beyond the resources of the typical classroom teacher. In fact, from the perspective of the classroom teacher, construct validity is probably more a function of the standard that he or she is trying to assess than the assessment he or she is designing. For example, assume a teacher is trying to design an assessment for the following standard: “Students will be able to work effectively in cooperative teams.” Construct validity would address the extent to which this standard represents a definable set of knowledge and skill—something that could actually be taught and measured.

      For criterion-related validity and construct validity, the classroom teacher has few, if any, resources to address them. However, the classroom teacher can address content validity, which basically reaffirms the early definition of validity—the test measures what it is purported to measure. For the classroom teacher, this simply involves ensuring that the CA addresses the content in the standard that is the focus of instruction and assessment.

      In summary, from the instrumental perspective, the classroom teacher has limited or no control over two of the three types of validity associated with CAs he or she is designing. However, from the argument-based perspective, the teacher has some control over all three types of validity.

The Argument-Based Perspective

      The