Items in knowledge and ability tests are traditionally scored dichotomously based on their accuracy. This paper also considers using confidence level (CL) in response accuracy as additional information about responder’s item and test scores. We analyse theoretical and conceptual advantages, as well as methodological problems of incorporating these measures in psychometric analysis of test results and evaluation/ interpretation/formation of total scores. The theoretical frameworks on which integration of CL and response accuracy is based on are the concept of metacognition and the dispositional measures of self-concept and self-efficacy (Stankov & Lee, 2008). Empirical results demonstrated here are based on data collected with the computer version of the non-verbal cognitive ability test (Test S) on 168 students. The analyses were performed on both item and test score intraindividual and interindividual level. On the item score level, the average CL is related to item difficulty, and for items with higher discriminative validity the difference in CL between correct and false answers is more pronounced. Regression analysis indicated that 69.1% of CL averaged across all responses could be explained, with metacognitive assessment of one’s score and the actual number of correct responses being the best predictors. The latent structure of the matrix of CLs across items indicates the dominance of one general factor, as well as high reliability of the average CL of .96. Differential weighting with confidence level results with higher reliability and homogeneity of the test. The results indicate the interesting potential of incorporating CL with response accuracy when analysing cognitive abilities tests.

confidence level, response accuracy, item analysis