ENBIS-17 in Naples

9 – 14 September 2017; Naples (Italy) Abstract submission: 21 November 2016 – 10 May 2017

Assessing Assessors (by Means of Binary Test Design and Analysis)

12 September 2017, 15:40 – 16:00


Submitted by
Emil Bashkansky
Emil Bashkansky (ORT Braude College of Engineering), Vladimir Turetsky (ORT Braude College of Engineering)
A method for evaluating proficiency skills (scoring) of assessors (men, laboratory, method etc.) conducting binary tests is proposed. The method is based on the scale-invariant item response model proposed by the authors in their earlier publications.
We consider the case where the assessors under proficiency evaluation conduct the same test consisting of a set of binary test items presenting different, but unknown beforehand levels of difficulty (known or unknown beforehand). When trying to detect a particular property of objects under test, we need to evaluate/compare both the intrinsic abilities of the participating assessors and the level of difficulty of these objects/test items (if they are unknown beforehand). We assume that the responses to different test items do not affect one another and discuss how to get and interpret the most likely estimates/scores. The contribution of placebo in the test effectiveness will also be discussed. The method is illustrated by the example of 28 medical laboratories proficiency testing. We propose, as a criterion for screening for ‘‘bad’’ appraisers, to use the probability of successful detection of the studied property by test item/s having a certain, predetermined or defined posteriori level of difficulty.
View paper

Return to programme