In the Caribbean, have we really moved past early understandings of validity?

Can we question the value of the unitary conceptualization of validity?

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425.



Construct validity functions as a black hole from which nothing can escape: Once a question gets labeled as a problem of construct validity, its difficulty is considered superhuman and its solution beyond a mortal’s ken. Validity theorists have themselves contributed to this situation by stating that validation research is a “never-ending process”; (e.g., Messick, 1988), which, at most, returns a “degree of validity”; (Cronbach & Meehl, 1955; Messick, 1989), but can by its very nature never yield a definitive answer to the question whether a test measures a certain attribute or not. This effectively amounts to a mystification of the problem, and discourages researchers to address it. In addition, this stance must be fundamentally ill-conceived for the simple reason that no physicists are currently involved in the “never-ending process”; of figuring out whether meter sticks really measure length, or are trying to estimate their “degree of validity”;; nevertheless, meter sticks are doing fine. So why should “construct validity”; be such an enormous problem in psychology?

The general idea seems to be based on the conviction (taken from the philosophy of science, and especially the work of Popper, 1959) that all scientific theories are by their nature “conjectures that have not yet been refuted”;; i.e., tentative and provisionally accepted working hypotheses. Whether one subscribes to this idea or not, it is evident that it cannot be specifically relevant for the problem of validity, because this view concerns not just validity, but every scientific hypothesis, and, by implication, applies to every psychometric hypothesis. Thus, if validity is problematic for this particular reason, then so are reliability, unidimensionality, internal consistency, continuity, measurement invariance, and all other properties of tests, test scores, and theoretical attributes, as well as all the relations between these properties that one could possibly imagine. But this is thoroughly uninformative; it merely teaches us that scientific research is difficult, and that we hardly ever know anything for sure. While this may be an important fact of life, it has no special bearing on the problem of test validity and most certainly cannot be used to justify the aura of intractability that surrounds the problem of “construct validity.”



Can we develop argument-based validation for high stakes public examinations?

(see Kane, M. (2004). Certification testing as an illustration of argument-based validation. Measurement, 2(3), 135-170.)