SMABS 2004 Jena University
SMABS 2004 Home Organization About Jena Sponsors Links Imprint / Contact SMABS Home

European Association of Methodology

Department of methodology and evaluation research

Jena University

Contributions: Abstract

A method for determining intra-assessor inconsistency in setting performance standards

Hans J. Vos
University Of Twente
The Netherlands

A common task for assessors working in the field of HRD is setting performance standards on assessments composed of complex, multiply scored performance-based exercises. Intra-assessor inconsistency arises when assessors specify performance standards which are incompatible with each other and, consequently, imply different standards.

The purpose of this paper is to propose a method for analyzing intra-assessor inconsistency by comparing assessor's subjective performance standards using the extended Angoff method (Hambleton, 1995) with those obtained under an IRT model for polytomously ordered items. More specifically, the well-known two-parameter generalized partial credit model (GPCM) for rating scales (Muraki, 1992) will be used as a polytomous IRT model. The proposed method will be demonstrated to an empirical study in which rating scores are obtained by providing assessors with hypothetical profiles on several dimensions of the complex tasks to be performed by employees in an Assessment Center.

The technique of generating systematically profile scores is borrowed from Judgmental Policy Capturing (JPC), a technique well-known in the field of, for instance, personnel selection in industrial psychology. Fitting the profile scores to an underlying multiple regression model, it will also be investigated to what extent the assessors are capable to assign relative weights to the several dimensions of the tasks as part of the extended Angoff method for subjective performance standard setting.


Hambleton, R.K. (1995). Using an extended Angoff procedure to set standards on complex performance assessments. Applied Measurement in Education, 8, 41-56.

Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.