SMABS 2004 Jena University
SMABS 2004 Home Organization About Jena Sponsors Links Imprint / Contact SMABS Home

European Association of Methodology

Department of methodology and evaluation research

Jena University

Contributions: Abstract

An IRT-based comparison of an item- with an examinee-centered method for standard setting

Daniƫl van Nijlen Rianne Janssen
University of Leuven

In item-centered methods for standard setting, judges indicate for each item the expected level of performance of a minimally competent student (MCS). The item judgements are combined into a cutoff on the test score scale. In examinee-centered methods, judges classify the examinees into different categories of mastery, such as masters versus nonmasters. A cutoff score is derived from a model that predicts category membership on the basis of the examinees' test performance in both groups.

In the present study, the standards derived from an item-centered and an examinee-centered method are compared on a test assessing the Flemish curriculum standards in biology for primary education. Using the revised Angoff method, primary school teachers indicated for each test item whether a MCS should be able to solve it correctly. Using the contrasting-group method, the same teachers also classified the pupils of their class into masters and nonmasters.

The two methods are compared using item response theory (IRT). Using the two-parameter IRT solution for the test, the judgements of the revised Angoff method can be transformed into an estimate of the location of the MCS on the ability scale (Van der Linden, 1982). A logistic regression of the estimated abilties of the students on their dichotomous classification by their teacher leads to a cutoff on the ability scale as well (Longford, 1996). A unified IRT model for the analysis of both methods of standard setting will be presented. The model is estimated in a Bayesian way using data augmented Gibbs sampling.