Investigating examiner consistency in criterion-based assessment using rasch measurement

Abu Kasim, N.H. (2006) Investigating examiner consistency in criterion-based assessment using rasch measurement. Journal of Dental Research.

Preview

PDF Investigating_Examiner_Consistency_in_Criterion.pdf Download (27kB)

Abstract

In the implementation of criterion-based assessment some disparities in the interpretation of the assessment criteria have been observed.Therefore it is essential that a standardization exercise be carried out to ensure equitable and fair assessment. Objectives: The purpose of the study was to evaluate inter-examiner variability and examiners' internal consistency in applying a criterion-based assessment. Method: The subjects consisted of 9 full-time staff of the Department of Conservative Dentistry, University of Malaya, 4 part-time staff and 2 postgraduate students. One buccally-approached and one palatally-approached Class III cavity preparation were assessed according to a set of criteria developed by the department. Both preparations were cut by undergraduates in their preclinical year of study. Examiners rated each cavity on a 5-point scale (1=Redo, 2=Modification required, 3=Pass with minor mistakes, 4=Pass, and 5=Good). Ratings were analyzed based on the Many-facet Rasch model using its computer application, Facets, version 3.48.0. Results: Varying levels of examiner severity were found ranging from -1.99 logits to 0.18 logit. However, the majority of the examiners fall between -1.04 logits and -0.31 logit. The percentage of exact agreement was 45.5. Several examiners were found to be inconsistent in their ratings as indicated by the Infit and Outfit mean-square statistics. Results also revealed range restrictions amongst some examiners. With respect to the assessment criteria, some were misfitting; these include (1) shape of cavity outline, (2) bevel width, (3) bevel thickness, (4) preservation of contact point and (5) placement of labial wall. On the other hand, several criteria demonstrated high agreement amongst examiners. These were (1) 90o cavo-surface angle, (2) rounded internal line angle, and (3) contour of axial wall. Conclusions: This study has highlighted the need for standardization of assessment criteria intepretation and better staff training in the assessment of preclinical students' operative skills.

Item Type:	Article
Funders:	UNSPECIFIED
Additional Information:	Associate Prof. Dr. Noor Hayaty Binti Abu Kasim Department of Conservative Dentistry, Faculty of Dentistry, University of Malaya, 50603 Kuala Lumpur, MALAYSIA
Uncontrolled Keywords:	Investigating; Examiner; Consistency; Criterion-Based; Assessment; Rasch; Measurement; Self-Repair; Culture-Conditions; Composite Resin; Dental Pulp Stem Cell; Functionally Graded Design; Multi Layered Post; Functionally Graded Dental Post; Soft Skills; Clinical Pairing; Dental Pulp Stromal Cells; Long-Term Expansion
Subjects:	R Medicine R Medicine > RK Dentistry
Divisions:	Faculty of Dentistry > Dept of Conservative Dentistry
Depositing User:	Associate Prof. Dr. Noor Hayaty Abu Kasim
Date Deposited:	10 Feb 2012 01:39
Last Modified:	10 Feb 2012 01:39
URI:	http://eprints.um.edu.my/id/eprint/2589

Actions (login required)

View Item