SMABS 2004 Jena University
SMABS 2004 Home Organization About Jena Sponsors Links Contact SMABS Home

European Association of Methodology

Department of methodology and evaluation research

Jena University

© Webmaster

Contributions: Abstract

Bias reduction of Warm's weighted likelihood estimation in computerized adaptive tests with polytomous items

Otto B. Walter Matthias Rose
Humboldt University Berlin

An essential step in computerized adaptive tests (CAT) is the computation of an ability estimate. Warm's weighted likelihood estimation (WLE) deserves particular attention because this approach was designed to reduce the first-order bias term of maximum likelihood estimation (MLE).

We investigated the properties of bias reduction of WLE in comparison to expected a posteriori (EAP) estimation in a simulation study with our CAT-engine for two CATs for depression and anxiety. The tests were developed from responses from 3,270 (Depression-CAT) and 2,348 (Anxiety-CAT) psychosomatic patients. Item response curves were analyzed according to Muraki's generalized partial credit model. Item banks consisted of 64 and 50 polytomous items, respectively.

For each 0.25-interval of the latent trait between -3.5 and +3.5 we generated the responses of N = 100 fictitious patients (simulees) with known ability levels. CAT scores based on WLE and EAP estimation were compared with the ability levels of the simulees. Simulated runs of the Anxiety-CAT for both estimation methods were generated from real responses of N = 1,010 patients. CAT scores were compared to N(0,1) standardized scores achieved in the State-Trait-Anxiety-Inventory (STAI).

In CAT scores based on simulated responses, WLE yielded markedly lower bias than the EAP estimator for ability levels < -1.0 and > 1.0. Simulated runs of the Anxiety-CAT based on real responses showed very high correlations ranging from .88 to .94 between N(0,1) standardized scores and CAT scores for both estimation methods. However, the deviation from standardized STAI values for ability levels < -1.0 and > 1.0 was significantly lower for WLE scores.

Our results show that WLE reduces bias in CAT with polytomous items. Thus, the application of WLE is recommended in conditions where unbiased estimates are needed. This is particularly true when cut-off scores are set on the ability scale, or when a comparison of CAT scores and paper-and-pencil tests is intended.