SPEAKERS

Winner of the Kathleen Tattersall New Assessment Researcher Award 2022: Understanding Technology-Based Assessments that use Multimedia Stimuli

Dr Paula Lehane is an Assistant Professor in the School of Inclusive and Special  Education. A graduate of the B.Ed in  Education and Psychology programme at Mary Immaculate College Limerick, Paula started her career as a primary school teacher in a developing school in Dublin. While working as a primary school teacher, she gained extensive experience in the areas of digital education, literacy, assessment and inclusive education.

During her time as a primary school teacher, Paula also completed a Graduate Diploma in Special Educational Needs (GradDip SEN) and a Masters in Additional Support Needs (M.Ed. ASN) with University College Dublin.

Paula subsequently worked as a research assistant with DCU’s Centre for Assessment Research, Policy & Practice in Education (CARPE), where her work focused on various issues in assessment and test development in educational and workplace settings. She received funding from the Irish Research Council (IRC) to complete her PhD in the field of digital tests and assessments for post-primary learners.  It is hoped that the findings of this research will support the effective design and deployment of computer-based exams within post-primary education systems.

 Technology-Based Assessments (TBAs) use items that employ a broad array of interactive, dynamic or static stimuli e.g. simulations, animations, text-image. Although it is assumed that these features can make TBAs more authentic and effective, their impact on test-taker performance and behaviour has yet to be fully clarified.

This research investigated the extent to which the use of different multimedia stimuli can affect test-taker performance and behaviour using a mixed methods approach. Guided by four main research questions, an experiment was conducted with 251 Irish post-primary students using an animated and text-image version of the same TBA of scientific literacy. Eye movement and interview data were also collected from subsets of these students (n=32 and n=12 respectively) to determine how differing multimedia stimuli can affect test-taker attentional behaviour. A second study involving 24 test-takers completing a series of simulation-type items was also undertaken. Eye movement, interview and test-score data were gathered to provide insight into test-taker engagement with these items.

The results indicated that, overall, there was no significant difference in test-taker performance when identical items used animated or text-image stimuli. However, items with dynamic stimuli often had higher discrimination indices indicating that these items were better at distinguishing between those with high and low levels of knowledge. Eye movement data also revealed that dynamic item stimuli encouraged longer average fixation durations on the response area of an item. An examination of the data relating to test-taker performance and behaviour for simulation-type items found that there was a weak to moderate relationship between task performance and time-to-first-fixation on relevant information/areas. 

Education systems around the world are now attempting to devise their own TBAs for their terminal post-primary exams e.g. New Zealand, Ireland. It is hoped that the findings of this research will act as a resource for those who wish to use TBAs in this manner. In particular, insights into test-takers’ eye movements may help to support more appropriate inferences from test scores.