SMABS 2004 Jena University
SMABS 2004 Home Organization About Jena Sponsors Links Imprint / Contact SMABS Home

European Association of Methodology

Department of methodology and evaluation research

Jena University

Contributions: Abstract

An exploratory multi-group approach for carving out differential performance profiles within an IRT metric

AndrĂ© A. Rupp
University of Ottawa
Canada

Determining the degree to which a given measurement process and inferences based thereon are generalizable requires an understanding of how assessment items function differently across multiple examinee populations and multiple measurement conditions. If parameters are identical in different populations or across different conditions one speaks of parameter invariance, which is typically cited as one of the major advantages of a latent variable approach for data calibration such as item response theory (e.g., Embretson & Reise, 2000). As research in differential item or bundle functioning (DIF and DBF respectively) and item parameter drift has shown, however, parameter invariance is a theoretical property that often does not hold in practice. Therefore, the potential for and impact of a lack of invariance needs to be empirically assessed as a statistical hypothesis for multiple populations or conditions of interest (e.g., Meredith, 1993).

This research extends current investigations on DIF and DBF in two major directions. First, it utilizes examinee groupings based on complex attitudinal and background profiles, rather than groupings based on coarse proxy variables such as gender, to statistically define multiple examinee groups for which DBF analyses are conducted. That is, DBF is utilized not solely as an outcome of interest in its own right but, more specifically, as a metric to carve out differential performance profiles (see Klieme & Baumert, 2001). Second, the methodology allows one to simultaneously quantify the relative performance of multiple examinee groups in a 2-dimensional model space.

The approach employs current methods from functional data analysis (Ramsay & Silverman, 1997, 2002) and is exploratory in nature; however, junctures at which strengths from multiple component approaches can be borrowed are discussed to ensure a higher degree of generalizability. Practical results are provided using multiple-choice items in mathematics from the 1999 TIMSS study (Mullis et al., 2001).