SMABS 2004 Jena University
SMABS 2004 Home Organization About Jena Sponsors Links Imprint / Contact SMABS Home

European Association of Methodology

Department of methodology and evaluation research

Jena University

Contributions: Abstract

Online calibration of pretest items with computerized adaptive tests of variable length

Ivailo Partchev
University of Jena
Germany

Online calibration is an economical approach to the replenishment of item pools that obviates the necessity of a separate calibration and scaling study. New items are `seeded' among the operational items of a CAT, and responses from the CAT session are used to both estimate ability and calibrate the new items.

Much of the literature on online calibration deals with CAT of fixed length, say 30 operational and 10 pretest items. We have investigated the performance of two methods based on the EM algorithm, OEM and MEM (Ban et al., 2000), in CAT of predetermined precision and variable length - on the average, much smaller than 30 items. It appears that the OEM method does not work in that situation.

Under the MEM method, the precision of calibration for any item can depend on whatever items are calibrated concurrently with it. However, the detrimental influence of a bad pretest item can be expected to subside as the number of pretest items increases.

We have undertaken a large simulation study to investigate the joint influence of the number of pretest items and sample size on the precision in online calibration. Based on that, we search for an optimal strategy in the online calibration of pretest items for the item pools of interest.

References

Jae-Chun Ban, Bradley A. Hanson, Tianyou Wang, Qing Yi and Deborah J. Harris. A comparative study of online pretest item calibration / scaling methods in CAT. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, April 2000