SMABS 2004 Jena University
SMABS 2004 Home Organization About Jena Sponsors Links Imprint / Contact SMABS Home

European Association of Methodology

Department of methodology and evaluation research

Jena University

Contributions: Abstract

Assessing the appropriateness of specifications in LLTM weight matrices

Johannes Hartig
German Institute for International Educational Research

The Linear Logistic Test Model (LLTM) allows to directly incorporate the contribution of item features to item difficulties into an IRT model. In educational assessment this provides the possibility to specify theory-driven measurement models for student competencies and to anchor scales not only with the difficulties of single items, but with the difficulties of task characteristics. A priori assumptions about which task characteristics contribute to which item difficulties are collected in the weight matrix W. Several statistics can be used to assess the appropriateness of these assumptions: one possibility is to inspect the global model fit and fit indices at the level of single model parameters. Additionally, the extent to which the LLTM reproduces the item and person parameters of a Rasch model with freely estimated item parameters can be evaluated.

Simulation studies with ConQuest were conducted to test the sensitivity of various statistics for misspecifications in W. While likelihood ratio tests based on the overall deviance reliably indicate decreases in model fit when misspecifications are built in W, the weighted averages of squared residuals that should indicate fit at the parameter level are not sensitive to misspecifications. The similarity of the population variances estimated for the LLTM and the unconstrained Rasch model turns out to be sensitive to misspecified weight matrices to some extent; the correlation between LLTM-implicit and freely estimated item difficulties proves to be highly sensitive for misspecifications.

Additionally, the plausibility of the estimated parameter values can be used to evaluate the appropriateness of the assumptions modelled in W. The application of the LLTM and the use of the various methods for model evaluation in educational testing are briefly illustrated with empirical data from a test for listening comprehension in English as an foreign language administered in a representative German study.