'Re-conceptualising Validity in High Stakes Testing' - Seminar by Professor Barry O'Sullivan (British Council)

Over the past six decades we have moved from thinking about validity in terms of the test, to thinking about it in terms of the impact the test has on the individual and on society (test consequence). Recently, a swing back to a more test-focused approach has emerged. One reason for this return to a more traditional approach is the failure of assessment theoreticians to adequately deal with the concept of test consequence. This is not hugely unexpected since the its proponent (Messick) himself pulled back from his original position over the course of his writing on the subject, so that his later works take an essentially traditional view. While it has been generally accepted that test consequence
is important, the degree to which this is the case and the way in which
it might impact on test development and validation has been debated and, more
recently, challenged. Like others, my position on the topic has changed
over the past number of years, from one of rejection (i.e. seeing the concept of
‘consequential validity’ is itself as an error), to one of slightly more acceptance
(consequence is somehow important to all aspects of test development
and validation). In this presentation, I will argue that the key understanding how test consequence can be operationalised in development
and validation models is to focus on test stakeholders. By considering stakeholder groups from the
conceptualisation stage of development we can essentially building consequence into test design. We can also postulate a clear a priori and a posteriori role for consequence within the development and validation model, allowing us to view consequence as a source of validation evidence. Acknowledging the importance of stakeholder groups to test development brings with it the equally important concept of how to more appropriately communicate validation results to these audiences.