Defining Fairness

Fairness is a very complicated, ambiguous, value-laden, and multi-faceted issue. As such a variety of definitions have been proposed to clarify its nature. According to Webster’s Ninth New Collegiate Dictionary (1988), fairness means, being free from having favor toward either or any side. However, this definition captures onoy a few dimensions.

Despite its ambiguity, fairness may be considered as an essential quality of a test. In this content and for this purpose, it "has been broadly defined as equitable treatment of all testtakers during the testing process, absence of measurement bias, equitable access to the constructs being measured, and justifiable validity of test score interpretation for the intended purpose(s)"

Banerjee, H. L. (2016). Test fairness in second language assessment. Studies in Applied Linguistics and TESOL, 16(1).

Michelle Boyer has an interesting blog o the difficulty of defining fairness. She wrote

We take for granted that we know what is fair and what is not, but fairness means different things to different people, in different contexts, at different times. Fairness is amorphous – often defined using related, but non-equivalent, terms like equity, justice, unbiased, and the like. We accept these definitions, and yet each of those terms is just as difficult to define. Such ambiguity inevitably leads to circumstances in which what we label as fair is based on our own value systems. To the extent that we do not share the same values, determining what is fair can be more than just difficult. It can be an outright dilemma. Because there really is no universally-fair outcome, we are left with fairness arguments that can be sources of deep divisions. We might even imagine some extreme examples where individuals or communities hold values that represent a threat to others, or that restrict access to certain benefits in unreasonable ways.

Conceptualizing the relationship between fairness and validity

 Three distinct perspectives according to  Xioaming Xi

View 1: Fairness as an independent test quality
Fairness is characterized as a test quality that is separate from validity. However some aspects of validity may be highlighted.

Examples of such an approach is found in The Code of Fair Testing Practices in Education (Joint Committee on Testing Practices, 1998, 2004) and the Standards for Fairness and Quality by Educational Testing Service (ETS, 2002)

Code of Fair Testing Practices

'The Code of Fair Testing Practices in Education (Code) is a guide for professionals in fulfilling their obligation to provide and use tests that are fair to all test takers regardless of age, gender, disability, race, ethnicity, national origin, religion, sexual orientation,linguistic background, or other personal characteristics. Fairness is a primary consideration in all aspects of testing. Careful standardization of tests and administration conditions helps to ensure that all test takers are given a comparable opportunity to demonstrate what they know and how they can perform in the area being tested. Fairness implies that every test taker has the opportunity to prepare for the test and is informed about the general nature and content of the test, as appropriate to the purpose of the test. Fairness also extends to the accurate reporting of individual and group test results. Fairness is not an isolated concept, but must be considered in all aspects of the testing process.'

The Code provides guidance separately for test developers and test users in four critical areas:
A. Developing and Selecting Appropriate Tests
B. Administering and Scoring Tests
C. Reporting and Interpreting Test Results
D. Informing Test Takers

Study the Code

 ETS Standards

The purposes of the ETS Standards for Quality and Fairness (henceforth the SQF) are to help Educational Testing Service design, develop, and deliver technically sound, fair, accessible, and useful products and services, and to help auditors evaluate those products and services. Additionally, the SQF is a publicly available document to help current and prospective clients, test takers, policymakers, score users, collaborating organizations, and others understand the requirements for the quality and fairness of ETS products and services.

The SQF is designed to provide policy-level guidance to ETS staff. The individual standards within the document are put into practice through the use of detailed guidelines, standard operating procedures, work rules, checklists, and so forth.

Study the ETS Code

View 2 – Fairness as an all-encompassing test quality



This view privileges  test fairness by conceptualzing it as an umbrella term which incorporates and goes beyond validity. Thus a test has to  be valid to be fair.

Kunnan's work on language testing reflects this view. His 2000 work deveoloped fairness as a three faceted concept - validity, access, and justice incorporating moral, legal, and philosophical aspects. Hois 2004 work focuses upon fairness in access and administration but also highlights validity, absence of bias, and social consequences.

Kunnan's 2020 work is useful in stressing the need for broader philosophical assumptions. He examines the past models for evaluating language assessments-the standards approach , the usefulness approach, and the argument use argument. He finds fault with each and of them and then he notes-

While both approaches provide ways for researchers to conduct evaluations, they have a weakness, and that is they generally lack an articulated philosophical grounding. This lack of philosophical grounding can be seen in the Standards approach in which why the listed standards are important and not others is not articulated. In the Argument approach, what aspects are to be included as claims and warrants is left the assessment developer with the evaluator following them which is a critical problem. To remedy this situation, I am proposing an Ethics-based approach to assessment evaluation. The framework that implements the approach harnesses the dual concepts of fair assessments and just institutions leading to the Principle of Fairness and Principle of Justice, respectively.

 To a large extent fairness reviews and item bias investigations pay attention to social and political considerations outside he psychometric issues.

View 3: Fairness linked directly to validity

Building on earlier work in the 1970s, Willingham and Cole (1997) conceived of fairness as comparable validity for all individuals and groups. Their fairness fromework considers - (1) comparability of opportunity for examinees to demonstrate relevant proficiency, (2)comparable assessment exercises and scores, and (3) comparable treatment of examinees in test interpretation and use.

Fairness issues should not be considered in isolation but should be related to the variables aspects of impact. For Wilingham (1999) comparable vality is a fundamental principle to be considered at each stage of the test development process.

The 2014 standards also sees fairness primarily as a validity issue, saying

Fairness is a fundamental validity issue and requires attention throughout all stages of test developmentand use. In previous versions of the Standards, fairness and the assessment of individuals from specific subgroups of test takers, such as individuals with disabilities and individuals with diverse linguistic and cultural backgrounds, were presented in separate chapters. In the current version of the Standards, these issues are presentedin a single chapter to emphasize that fairness to all individuals in the intended population of test takers is an overriding, foundational concern, and that common principles apply in responding to test-taker characteristics that could interfere with the validity of test score interpretation. This is not to say that the response to test-taker characteristics is the same for individuals from diverse subgroups such as those defined by race, ethnicity, gender, culture, language, age, disability or socioeconomic status, but rather that these responses should be sensitive to individual characteristics that otherwise would compromise validity.

Fairness as a philosophical orientation

 Measurement models provide only one dimension to the issue of fairness and cannot completely capture the full complexity of the construct. Philsophical perspectives add to the analysis and have been incorporated by several theorists recently.

Zwick and Dorans (2016) explored three perspectives-Aristotle, Robert Nozick and John Rawls.

Aristotle

This perspective holds that individuals should be honored and reqarded for their virtues and accomlishments instead of receiving goods solely because of their ancestry. Applied to testing, we regard some tests as serving a purpose in determining whether an individual deserves an honour scholarship or award. This is broadly Aristotelian justice, focusing upon merit, fitness and purpose. It is questionable whether a test can provide an estimate of merit without considering other factors such as opportunity to learn.

Robert Nozick

He proposed the libertarian approach which is focused on protecting the freedom of the individual to engage in acquisitions and transfers of goods. For him justice is wrapped up in the idea of self ownership (promoted by John Locke) . We own ourselves and therefore what we produced is also our own. Nozick felt that the activities of the state should be restricted to proteting life, liberty, property, and contract.

Applied to testing the libertarian position holds that test takers have a right to take advantage of opportunities afforded them. They are at liberty to use their wealth and other resources to obtain an advantage. The testing agency should also not be restricted in its use of test scores. The liberterian position is that inequality is not in itself unjust if there were no improper acquisition or transfer or property.

John Rawls

Rawls proposed a theory of distributive justice based upon a hypothetical situation (people from different aspects of society are brought together roles and rules from behind a veil of ignorance).

Two principles arise.

1. "Each person is to have an equal right to the most extensive total system of equal basic liberties compatible with a similar system of liberty for all". (greatest equal liberty principle.)

2. "Social and economic inequalities are to be arranged so that they are both:

(a) to the greatest benefit of the least advantaged, consistent with the just savings principle (the difference principle)

(b) attached to offices and positions open to all under conditions of fair equality of opportunity." (the equal opportunity principle)



Issues in TEST Fairness IN TTO to consider

How should the fairness of a selection/ placement system such as the 11+ be studied?

What psychometric models and philosophical perspectives should be applied?

Exercise-1000 page argumentative essay

Write on the topic-But is it fair-11+ selection and placement in Trinidad and Tobago



Zwick, R. (2019). Fairness in measurement and selection: Statistical, philosophical, and public perspectives. Educational Measurement: Issues and Practice, 38(4), 34-41.



Stobart, G. (2005). Fairness in multicultural assessment systems. Assessment in Education: Principles, Policy & Practice, 12(3), 275-287.



Fairness chapter in Fourth Edition of Educational Measurement

Zachary Stein describes the difference between efficiency-oriented testing and justice-oriented testing.

Elsewhere he wrote

Measures and standards are implicated in social justice because they quite literally structure our lives in profound ways, impacting the ways that we understand ourselves and the social world, and the nature of reality itself. They quite literally give shape to the physical environments we inhabit and the temporal durations that constitute the rhythms of our lives. They make some things possible and others impossible; they reveal certain aspects of reality, while concealing others. Measures and standards facilitate cooperation and trust at a distance and across cultures, while they also enable complex institutional processes that are often exclusionary, exploitative, and oppressive. They constitute what the great moral theorist John Rawls would call, basic structures (Rawls, 1971; Stein, 2014). Basic structures are social structures we enter into by virtue of entering into society at all. We are born into institutions and other social inventions and infrastructures that are not of our choosing and they shape our life prospects from day one. Because measurement infrastructures are basic structures they are intrinsically implicated in issues of social justice.

Read here