WORKSHOPS

Workshop 1:

How hard can it be? The practice and challenge of validation: Issues around how best to provide evidence for assessment validity, reliability and fairness

Abstract

The primacy of validity as measurement concept has been consistently affirmed in the assessment literature. The responsibility for assessment providers to demonstrate robust and thorough validity evidence is, therefore, a long established expectation (Messick, 1992, p.89) as are warnings about the “potentially serious consequences” (Kane, 2009, p.61) of shirking such responsibilities. Even assessment providers that have limited resources will still have a responsibility to demonstrate the quality and validity of their assessments. This workshop is intended to make the complexities of validation theory and practice less challenging and more readily operational. The workshop broadly divides into two parts. The first establishes the importance and relevance of validation theory and practice. The second unpacks the complexities and challenges of conducting a validation exercise. Each part comprises sessions affording group activities and discussion. By sharing experiences through a collaborative workshop environment, greater insights will be drawn leading to an increased understanding of the validation process and how it might be routinely operationalised in differing contexts.

Biodata

Stuart Shaw has worked for international awarding bodies for over 20 years. From 2007 to 2021, he was Head of Research at Cambridge Assessment International Education. Stuart is particularly interested in demonstrating how educational, psychological and vocational tests seek to meet the demands of validity, reliability and fairness. Stuart has a wide range of publications in English second language assessment and educational research journals (around 150). His assessment books include: Examining Writing: Research and practice in assessing second language writing (Shaw & Weir, 2007); The IELTS Writing Assessment Revision Project: towards a revised rating scale (Shaw & Falvey, 2008); Validity in Educational and Psychological Assessment (Newton & Shaw, 2014); and Language Rich: Insights from Multilingual Schools (Shaw, Imam & Hughes, 2015). His most recent book is entitled Is Assessment Fair? (Isabel Nisbet & Stuart Shaw) published by Sage (2020). Stuart is currently working on a book with Isabel Nisbet (to be published by Routledge in early 2024) entitled: Educational Assessment in a Changing World: Lessons Learned and the Path Ahead. Stuart is a Fellow of the Chartered Institute of Educational Assessors (CIEA) and a Fellow of the Association for Educational Assessment in Europe (AEA-Europe). Stuart is an elected member of the Council of AEA-Europe and is Chair of its Scientific Programme Committee. He is an elected member of the Board of Trustees of the International Association for Educational Assessment (IAEA) and Chair of the IAEA Communications Committee. Stuart is also an affiliated lecturer at the Faculty of Education, University of Cambridge, and an Honorary Professor – Curriculum, Pedagogy and Assessment, Institute of Education, University College London (UCL). Stuart has contributed to international debates on validity and fairness (see, for example, Special Issue of Assessment in Education: Principles, Policy & Practice, 2016) and more recently at The British Educational Research Association (BERA) Presidential Roundtable (2022).

Workshop 1:

How hard can it be? The practice and challenge of validation: Issues around how best to provide evidence for assessment validity, reliability and fairness

Abstract

The primacy of validity as measurement concept has been consistently affirmed in the assessment literature. The responsibility for assessment providers to demonstrate robust and thorough validity evidence is, therefore, a long established expectation (Messick, 1992, p.89) as are warnings about the “potentially serious consequences” (Kane, 2009, p.61) of shirking such responsibilities. Even assessment providers that have limited resources will still have a responsibility to demonstrate the quality and validity of their assessments. This workshop is intended to make the complexities of validation theory and practice less challenging and more readily operational. The workshop broadly divides into two parts. The first establishes the importance and relevance of validation theory and practice. The second unpacks the complexities and challenges of conducting a validation exercise. Each part comprises sessions affording group activities and discussion. By sharing experiences through a collaborative workshop environment, greater insights will be drawn leading to an increased understanding of the validation process and how it might be routinely operationalised in differing contexts.

Biodata

Stuart Shaw has worked for international awarding bodies for over 20 years. From 2007 to 2021, he was Head of Research at Cambridge Assessment International Education. Stuart is particularly interested in demonstrating how educational, psychological and vocational tests seek to meet the demands of validity, reliability and fairness. Stuart has a wide range of publications in English second language assessment and educational research journals (around 150). His assessment books include: Examining Writing: Research and practice in assessing second language writing (Shaw & Weir, 2007); The IELTS Writing Assessment Revision Project: towards a revised rating scale (Shaw & Falvey, 2008); Validity in Educational and Psychological Assessment (Newton & Shaw, 2014); and Language Rich: Insights from Multilingual Schools (Shaw, Imam & Hughes, 2015). His most recent book is entitled Is Assessment Fair? (Isabel Nisbet & Stuart Shaw) published by Sage (2020). Stuart is currently working on a book with Isabel Nisbet (to be published by Routledge in early 2024) entitled: Educational Assessment in a Changing World: Lessons Learned and the Path Ahead. Stuart is a Fellow of the Chartered Institute of Educational Assessors (CIEA) and a Fellow of the Association for Educational Assessment in Europe (AEA-Europe). Stuart is an elected member of the Council of AEA-Europe and is Chair of its Scientific Programme Committee. He is an elected member of the Board of Trustees of the International Association for Educational Assessment (IAEA) and Chair of the IAEA Communications Committee. Stuart is also an affiliated lecturer at the Faculty of Education, University of Cambridge, and an Honorary Professor – Curriculum, Pedagogy and Assessment, Institute of Education, University College London (UCL). Stuart has contributed to international debates on validity and fairness (see, for example, Special Issue of Assessment in Education: Principles, Policy & Practice, 2016) and more recently at The British Educational Research Association (BERA) Presidential Roundtable (2022).

Workshop 2:

Artificial Intelligence as a tool for Assessment Reform

Abstract

Biographies

Stephen Bezzina is a Research Support Officer at the Department of Artificial Intelligence at the Faculty of ICT at the University of Malta. His research interests include the application of Artificial Intelligence to playful and game-informed approaches, such as game-based methodologies and gamification. He holds a Master of Science degree in Digital Education (University of Edinburgh), specialising in games and digital assessment. In the past, he has worked as an Education Officer (responsible for diversity, cooperative learning and digital literacy) within the Ministry for Education and Employment in Malta.

Prof Alexiei Dingli is a renowned AI expert and Professor at the University of Malta. With over 20 years of experience in the field, he has helped numerous companies successfully implement AI solutions. His work has been recognized as world-class by international experts, and he has received numerous awards from organizations such as the European Space Agency, the World Intellectual Property Organization, and the United Nations. In addition to his considerable peer-reviewed publications, he has also been a critical member of the Malta.AI task force, working to position Malta as a global leader in AI.

Workshop 2:

Artificial Intelligence as a tool for Assessment Reform

Abstract

Biographies

Stephen Bezzina is a Research Support Officer at the Department of Artificial Intelligence at the Faculty of ICT at the University of Malta. His research interests include the application of Artificial Intelligence to playful and game-informed approaches, such as game-based methodologies and gamification. He holds a Master of Science degree in Digital Education (University of Edinburgh), specialising in games and digital assessment. In the past, he has worked as an Education Officer (responsible for diversity, cooperative learning and digital literacy) within the Ministry for Education and Employment in Malta.

Prof Alexiei Dingli is a renowned AI expert and Professor at the University of Malta. With over 20 years of experience in the field, he has helped numerous companies successfully implement AI solutions. His work has been recognized as world-class by international experts, and he has received numerous awards from organizations such as the European Space Agency, the World Intellectual Property Organization, and the United Nations. In addition to his considerable peer-reviewed publications, he has also been a critical member of the Malta.AI task force, working to position Malta as a global leader in AI.

Workshop 3:

Moving to fully inclusive e-assessment

Short abstract

Small group discussion will be used to explore the premise from multiple perspectives, drawing on experiences from across the group. The presenters will provide an informative introduction to various key areas, through short presentations, and participants will investigate some of these areas in more detail for themselves during the workshop and share findings with the group.

This workshop is led by members of the AEA-Europe eAssessment and Inclusive Assessment SIGs. We aim to provide participants with a basic understanding of inclusion considerations to inform everyday practice and/or opportunity to extend existing understanding.
No prior experience of e-assessment or inclusive assessment is needed.

Presenters’ Bios

Helen Claydon is an experienced assessment developer and leader, having worked in assessment for over 25 years. She has led a range of projects developing summative and formative assessments for ages from 6 years-old to adult. Most notably she has led assessment development for national programmes in the United Kingdom, including the Scottish National Standardised Assessments, the KS1 and KS2 National Curriculum Tests in mathematics (SATs) and the Professional Skills Tests for Prospective Teachers. Her experience covers all aspects of the assessment development process, including assessment design, item writing and review, trialling, review of item and test statistics, test construction, standard setting and standards maintenance.
Helen is currently an assessment consultant, undertaking qualification and assessment design, development and review work for organisations such as the Association for Project Management (APM), GL Assessment, MH&A and Qualifications Wales.
Helen holds a Master’s degree in Education. She is a Steering Committee Member for AEA-Europe’s eAssessment Special Interest Group and a Fellow of the Chartered Institute of Educational Assessors. Helen helped to launch the annual International e-Assessment Awards in 2016 and has been a judge for the awards each year since then.

Caroline Jongkamp is currently a senior consultant for international projects at Cito, the Netherlands. She is experienced as a professional test developer in economics, financial accounting, and ICT. At Cito, she was responsible for the implementation of computer-based testing in Dutch secondary school leaving exams. She is an experienced change manager involved in change processes on item banking and computer-based testing. Caroline holds an MSc in Econometrics with a specialization in Operations Research.
Caroline is a Steering Committee member for AEA-Europe’s eAssessment Special Interest Group.

Irini-Renika Papakammenou holds a BA in English literature with English language from the university of North Wales, Bangor and an MSc in Teaching English to Speakers of other Languages (TESOL) from the University of Stirling. She has done a PhD in Linguistics with specialization in Language Testing and Assessment at the University of Cyprus. She has received an award for her PhD thesis as the best postgraduate thesis of the year. She has also published scientific papers on language testing and assessment and alternative assessment techniques in international books and journals. She has presented in numerous local and international conferences, and she has delivered training courses. She is a member of language teaching and assessment societies and research groups. Her research interests include language testing and assessment, curricula, material design (for face-to-face and online classrooms), classroom practices and teacher education and development.

Irini has been involved in EFL teaching and learning for the past 21 years. During her 21-year career she has contributed to the field in a number of different capacities such as English Language Teacher, Exam Preparation Teacher (all exams including IELTS and TOEFL), Teacher Trainer, Oral Examiner, Speaker and Researcher. She is the owner of a private institute of foreign languages, she owns a distance learning platform and leads distance learning EFL examination programs. She has created innovative classroom and online materials which have been presented in international conferences and have been published. Irini has recently received the Global Teacher Award 2020.

Thomais Rousoulioti, PhD in Applied Linguistics, works as a special teaching staff at Aristotle University of Thessaloniki, Greece. During the years 2010-2017 she worked at Department of Support and Promotion of the Greek Language of the Centre for the Greek Language, Greece, where she was involved in the design and implementation of research programs regarding the teaching and assessment of the Greek language as a second/foreign language but also with the design and editing of tests for the assessment of participants in the examinations for the Certification of Attainment in Greek.

She has also worked for the Hellenic Open University and the University of Nicosia teaching in online distance Ma programs. Her research interests include the teaching and assessment of language proficiency in multilingual settings, the design of teaching materials, adult education, distance education and inclusive assessment. She is the coordinator of a postgraduate course for the assessment in Greek as a second/foreign language and the coordinator of the module Students’ Assessment in the training program Routes of the Centre for the Greek Language which is the sole representative of Greece at the European Federation of National Institutions for Language (http://efnil.org/). She is a member of the SIG Steering Committee on Inclusive Assessment in Education (AEA-Europe), the "Psifis" laboratory of Aristotle University of Thessaloniki, Greece, the Hellenic Society of Applied Linguistics, EALTA, ALTE and OsloMet's EnA research team.

Workshop 3:

Moving to fully inclusive e-assessment

Short abstract

Small group discussion will be used to explore the premise from multiple perspectives, drawing on experiences from across the group. The presenters will provide an informative introduction to various key areas, through short presentations, and participants will investigate some of these areas in more detail for themselves during the workshop and share findings with the group.

This workshop is led by members of the AEA-Europe eAssessment and Inclusive Assessment SIGs. We aim to provide participants with a basic understanding of inclusion considerations to inform everyday practice and/or opportunity to extend existing understanding.
No prior experience of e-assessment or inclusive assessment is needed.

Presenters’ Bios

Helen Claydon is an experienced assessment developer and leader, having worked in assessment for over 25 years. She has led a range of projects developing summative and formative assessments for ages from 6 years-old to adult. Most notably she has led assessment development for national programmes in the United Kingdom, including the Scottish National Standardised Assessments, the KS1 and KS2 National Curriculum Tests in mathematics (SATs) and the Professional Skills Tests for Prospective Teachers. Her experience covers all aspects of the assessment development process, including assessment design, item writing and review, trialling, review of item and test statistics, test construction, standard setting and standards maintenance.
Helen is currently an assessment consultant, undertaking qualification and assessment design, development and review work for organisations such as the Association for Project Management (APM), GL Assessment, MH&A and Qualifications Wales.
Helen holds a Master’s degree in Education. She is a Steering Committee Member for AEA-Europe’s eAssessment Special Interest Group and a Fellow of the Chartered Institute of Educational Assessors. Helen helped to launch the annual International e-Assessment Awards in 2016 and has been a judge for the awards each year since then.

Caroline Jongkamp is currently a senior consultant for international projects at Cito, the Netherlands. She is experienced as a professional test developer in economics, financial accounting, and ICT. At Cito, she was responsible for the implementation of computer-based testing in Dutch secondary school leaving exams. She is an experienced change manager involved in change processes on item banking and computer-based testing. Caroline holds an MSc in Econometrics with a specialization in Operations Research.
Caroline is a Steering Committee member for AEA-Europe’s eAssessment Special Interest Group.

Irini-Renika Papakammenou holds a BA in English literature with English language from the university of North Wales, Bangor and an MSc in Teaching English to Speakers of other Languages (TESOL) from the University of Stirling. She has done a PhD in Linguistics with specialization in Language Testing and Assessment at the University of Cyprus. She has received an award for her PhD thesis as the best postgraduate thesis of the year. She has also published scientific papers on language testing and assessment and alternative assessment techniques in international books and journals. She has presented in numerous local and international conferences, and she has delivered training courses. She is a member of language teaching and assessment societies and research groups. Her research interests include language testing and assessment, curricula, material design (for face-to-face and online classrooms), classroom practices and teacher education and development.

Irini has been involved in EFL teaching and learning for the past 21 years. During her 21-year career she has contributed to the field in a number of different capacities such as English Language Teacher, Exam Preparation Teacher (all exams including IELTS and TOEFL), Teacher Trainer, Oral Examiner, Speaker and Researcher. She is the owner of a private institute of foreign languages, she owns a distance learning platform and leads distance learning EFL examination programs. She has created innovative classroom and online materials which have been presented in international conferences and have been published. Irini has recently received the Global Teacher Award 2020.

Thomais Rousoulioti, PhD in Applied Linguistics, works as a special teaching staff at Aristotle University of Thessaloniki, Greece. During the years 2010-2017 she worked at Department of Support and Promotion of the Greek Language of the Centre for the Greek Language, Greece, where she was involved in the design and implementation of research programs regarding the teaching and assessment of the Greek language as a second/foreign language but also with the design and editing of tests for the assessment of participants in the examinations for the Certification of Attainment in Greek.

She has also worked for the Hellenic Open University and the University of Nicosia teaching in online distance Ma programs. Her research interests include the teaching and assessment of language proficiency in multilingual settings, the design of teaching materials, adult education, distance education and inclusive assessment. She is the coordinator of a postgraduate course for the assessment in Greek as a second/foreign language and the coordinator of the module Students’ Assessment in the training program Routes of the Centre for the Greek Language which is the sole representative of Greece at the European Federation of National Institutions for Language (http://efnil.org/). She is a member of the SIG Steering Committee on Inclusive Assessment in Education (AEA-Europe), the "Psifis" laboratory of Aristotle University of Thessaloniki, Greece, the Hellenic Society of Applied Linguistics, EALTA, ALTE and OsloMet's EnA research team.

Workshop 4:

Responding as assessment professionals to calls for reform

Presenters

Dr. Eleanor Andressen has particular interests in technical and vocational education and training, adult and lifelong learning, and assessment cultures (with a focus on TVET and compulsory national examinations). She is in her third year as a judge for the e-Assessment Association awards and is a peer reviewer for the Higher Education and Work-based Learning journal. She is the current chair of the Steering Group for the AEA Europe Assessment Cultures special interest group.

Dr. Andrew Watts taught in secondary schools in England, and then in Singapore, where he joined a team of specialist inspectors which focused on curriculum development and in-service training. When he returned to England he worked for the Cambridge school examination board (then Cambridge Assessment) and set up the Cambridge Assessment Network, whose aim is to provide support for assessment professionals internationally.

Henning Fjørtoft is professor of Norwegian didactics at NTNU, The Norwegian University of Science and Technology, in Trondheim. His research interests include literacy education, classroom assessment, and professional development in schools. He has been involved in assessment research in Scandinavia, New Zealand, and Ethiopia, and serves on the editorial boards of Assessment in Education: Principles, Policy & Practice, Educational Assessment, Evaluation and Accountability and Teachers and Teaching: Theory and Practice.

Workshop outline

Workshop 4:

Responding as assessment professionals to calls for reform

Presenters

Dr. Eleanor Andressen has particular interests in technical and vocational education and training, adult and lifelong learning, and assessment cultures (with a focus on TVET and compulsory national examinations). She is in her third year as a judge for the e-Assessment Association awards and is a peer reviewer for the Higher Education and Work-based Learning journal. She is the current chair of the Steering Group for the AEA Europe Assessment Cultures special interest group.

Dr. Andrew Watts taught in secondary schools in England, and then in Singapore, where he joined a team of specialist inspectors which focused on curriculum development and in-service training. When he returned to England he worked for the Cambridge school examination board (then Cambridge Assessment) and set up the Cambridge Assessment Network, whose aim is to provide support for assessment professionals internationally.

Henning Fjørtoft is professor of Norwegian didactics at NTNU, The Norwegian University of Science and Technology, in Trondheim. His research interests include literacy education, classroom assessment, and professional development in schools. He has been involved in assessment research in Scandinavia, New Zealand, and Ethiopia, and serves on the editorial boards of Assessment in Education: Principles, Policy & Practice, Educational Assessment, Evaluation and Accountability and Teachers and Teaching: Theory and Practice.

Workshop outline

Workshop 5:

(Re)design YOUR assessment! Designing assessment tasks with evidence-centered design

Abstract

Biodata

Michiel Waltman (1979) is an educational researcher at CitoLab, Cito’s research and innovation division. Michiel worked as a high school teacher, lecturer at Fontys University of Applied Sciences and University of Twente and as an assessment expert at Cito’s division of central examination. Michiel is especially interested in design and validation of assessment instruments for complex constructs like critical thinking and citizenship development. His expertise is in making validation and assessment design approaches (e.g. evidence-centered design and argument-based approach to validation) accessible and meaningful for educational practitioners. He published in educational practitioners’ journals and books and contributed to conferences.

Sebastiaan de Klerk (1986) is an educational researcher at CitoLab. CitoLab is the research and innovation division of Cito. Sebastiaan received his PhD from Twente University in 2016 for research on the design and validation of simulation-based assessment (SBA) in vocational education. In his PhD research, he focused on the measurement of procedural skills in SBA. The key element in his PhD research was to design and validate an assessment for a Dutch security profession. To that extent, evidence-centered design was used for the design. In 2016, during the AEA-Europe conference on Cyprus, he was awarded the Kathleen Tattersall New Assessment Researcher Award for his work on the design and validation of SBA. Sebastiaan also worked (2017 – 2022) as the treasurer of the Dutch Association for Educational Assessment (NVE). At CitoLab, Sebastiaan’s main focus is on research on the influence of technology on assessment. Sebastiaan has multiple scientific publications and presented at numerous conferences.

Workshop 5:

(Re)design YOUR assessment! Designing assessment tasks with evidence-centered design

Abstract

Biodata

Michiel Waltman (1979) is an educational researcher at CitoLab, Cito’s research and innovation division. Michiel worked as a high school teacher, lecturer at Fontys University of Applied Sciences and University of Twente and as an assessment expert at Cito’s division of central examination. Michiel is especially interested in design and validation of assessment instruments for complex constructs like critical thinking and citizenship development. His expertise is in making validation and assessment design approaches (e.g. evidence-centered design and argument-based approach to validation) accessible and meaningful for educational practitioners. He published in educational practitioners’ journals and books and contributed to conferences.

Sebastiaan de Klerk (1986) is an educational researcher at CitoLab. CitoLab is the research and innovation division of Cito. Sebastiaan received his PhD from Twente University in 2016 for research on the design and validation of simulation-based assessment (SBA) in vocational education. In his PhD research, he focused on the measurement of procedural skills in SBA. The key element in his PhD research was to design and validate an assessment for a Dutch security profession. To that extent, evidence-centered design was used for the design. In 2016, during the AEA-Europe conference on Cyprus, he was awarded the Kathleen Tattersall New Assessment Researcher Award for his work on the design and validation of SBA. Sebastiaan also worked (2017 – 2022) as the treasurer of the Dutch Association for Educational Assessment (NVE). At CitoLab, Sebastiaan’s main focus is on research on the influence of technology on assessment. Sebastiaan has multiple scientific publications and presented at numerous conferences.