Validating item response processes in digital competence assessment through eye-tracking techniques

Thumbnail Image
Files
JBB.pdf (3.53 MB)
Identifiers
ISBN: 978-1-4503-8850-4
ISBN: 9781450388504
DOI: 10.1145/3434780.3436641
Publication date
2020-10-21
Authors
Advisors
Journal Title
Journal ISSN
Volume Title
Publisher
ACM
Citations
Google Scholar
Export
Research Projects
Organizational Units
Journal Issue
Abstract
This paper reports on an exploratory study with the aim to validate item response processes in digital competence assessment through eye-tracking techniques. When measuring complex cognitive constructs, it is crucial to correctly design the evaluation items to trigger the intended knowledge and skills. Furthermore, to assess the validity of a test requires considering not only the content of the evaluation tasks involved in the test, but also whether examinees respond to the tasks by engaging construct-relevant response processes. The eye tracking observations helped to fill an ‘explanatory gap’ by providing data on variation in item response processes that are not captured by other sources of process data such as think aloud protocols or computer-generated log files. We proposed a set of metrics that could help test designers to validate the different item formats used in the evaluation of digital competence. The gaze data provided detailed information on test item response strategies, enabling profiling of examinee engagement and response processes associated with successful performance. There were notable differences between the participants who correctly solved the tasks and those who failed, in terms of the time spent on solving them, as well as the data on their gazes. Moreover, this included insights into response processes which contributed to the validation of the assessment criteria of each item.
Description
Publisher Copyright: © 2020 ACM.
Keywords
Eye-tracking techniques , Digital competence , Eye-tracking techniques , Digital competence , computer-based assessment , eye-tracking , response process validity , Software , Human-Computer Interaction , Computer Vision and Pattern Recognition , Computer Networks and Communications
Type
conference output
Citation
Bartolomé , J , Garaizar , P & Bastida , L 2020 , Validating item response processes in digital competence assessment through eye-tracking techniques . in F J Garcia-Penalvo (ed.) , unknown . ACM International Conference Proceeding Series , ACM , pp. 738-746 , 8th International Conference on Technological Ecosystems for Enhancing Multiculturality, TEEM 2020 , Virtual, Online , Spain , 21/10/20 . https://doi.org/10.1145/3434780.3436641
conference