Search
64 resources
-
Christopher Ormerod|Feb 23rd, 2022|preprintChristopher OrmerodFeb 23rd, 2022
We investigate the effectiveness of ensembles of pretrained transformer-based language models on short answer questions using the Kaggle Automated Short Answer Scoring dataset. We fine-tune a collection of popular small, base, and large pretrained transformer-based language models, and train one feature-base model on the dataset with the aim of testing ensembles of these models. We used an early stopping mechanism and hyperparameter optimization in training. We observe that generally that...
-
Okan Bulut, Alexander MacIntosh, Cole Wa...|Dec 1st, 2022|bookSectionOkan Bulut, Alexander MacIntosh, Cole Wa...Dec 1st, 2022
-
The Promises and Challenges of Artificial Intelligence for Teachers: a Systematic Review of ResearchIsmail Celik, Muhterem Dindar, Hanni Muu...|Jul 1st, 2022|journalArticleIsmail Celik, Muhterem Dindar, Hanni Muu...Jul 1st, 2022
Abstract This study provides an overview of research on teachers’ use of artificial intelligence (AI) applications and machine learning methods to analyze teachers’ data. Our analysis showed that AI offers teachers several opportunities for improved planning (e.g., by defining students’ needs and familiarizing teachers with such needs), implementation (e.g., through immediate feedback and teacher intervention), and assessment (e.g., through automated essay scoring) of their...
-
David W. Dorsey, Hillary R. Michaels|Sep 1st, 2022|journalArticleDavid W. Dorsey, Hillary R. MichaelsSep 1st, 2022
-
David W. Dorsey, Hillary R. Michaels|Sep 1st, 2022|journalArticleDavid W. Dorsey, Hillary R. MichaelsSep 1st, 2022
Abstract We have dramatically advanced our ability to create rich, complex, and effective assessments across a range of uses through technology advancement. Artificial Intelligence (AI) enabled assessments represent one such area of advancement—one that has captured our collective interest and imagination. Scientists and practitioners within the domains of organizational and workforce assessment have increasingly used AI in assessment, and its use is now becoming more common in...
-
Nigel Fernandez, Aritra Ghosh, Naiming L...|Dec 1st, 2022|preprintNigel Fernandez, Aritra Ghosh, Naiming L...Dec 1st, 2022
Automated scoring of open-ended student responses has the potential to significantly reduce human grader effort. Recent advances in automated scoring often leverage textual representations based on pre-trained language models such as BERT and GPT as input to scoring models. Most existing approaches train a separate model for each item/question, which is suitable for scenarios such as essay scoring where items can be quite different from one another. However, these approaches have two...
-
Steve Ferrara, Saed Qunbar|Sep 1st, 2022|journalArticleSteve Ferrara, Saed QunbarSep 1st, 2022
Abstract In this article, we argue that automated scoring engines should be transparent and construct relevant—that is, as much as is currently feasible. Many current automated scoring engines cannot achieve high degrees of scoring accuracy without allowing in some features that may not be easily explained and understood and may not be obviously and directly relevant to the target assessment construct. We address the current limitations on evidence and validity arguments for...
-
Yong He, Shumin Jing, Y Lu|Dec 1st, 2022|conferencePaperYong He, Shumin Jing, Y LuDec 1st, 2022
-
A. Corinne Huggins‐Manley, Brandon M. Bo...|Sep 1st, 2022|journalArticleA. Corinne Huggins‐Manley, Brandon M. Bo...Sep 1st, 2022
Abstract The field of educational measurement places validity and fairness as central concepts of assessment quality. Prior research has proposed embedding fairness arguments within argument‐based validity processes, particularly when fairness is conceived as comparability in assessment properties across groups. However, we argue that a more flexible approach to fairness arguments that occurs outside of and complementary to validity arguments is required to address many of the...
-
Matthew S. Johnson, Xiang Liu, Daniel F....|Sep 1st, 2022|journalArticleMatthew S. Johnson, Xiang Liu, Daniel F....Sep 1st, 2022
-
Susan Lottridge, Mackenzie Young|Dec 1st, 2022|conferencePaperSusan Lottridge, Mackenzie YoungDec 1st, 2022
The use of automated scoring (AS) of constructed responses has become increasingly common in k - 12 formative, interim, and summative assessment programs. AS has been shown to perform well in essay writing, reading comprehension, and mathematics. However, less is known about how automated scoring engines perform for key subgroups such as gender, race/ethnicity, English proficiency status, disability status, and economic status. Bias evaluations have focused primarily on mean score...
-
Christopher Ormerod|Dec 1st, 2022|journalArticleChristopher OrmerodDec 1st, 2022
We introduce a regression-based framework to explore the dependence that global features have on score predictions from pretrained transformer-based language models used for Automated Essay Scoring (AES). We demonstrate that neural networks use approximations of rubric-relevant global features to determine a score prediction. By considering linear models on the hidden states, we can approximate global features and measure their importance to score predictions. This study uses DeBERTa models...
-
Maria Mercedes Rodrigo, Noburu Matsuda, ...|Dec 1st, 2022|bookMaria Mercedes Rodrigo, Noburu Matsuda, ...Dec 1st, 2022
-
Shiki Sato, Yosuke Kishinami, Hiroaki Su...|Dec 1st, 2022|conferencePaperShiki Sato, Yosuke Kishinami, Hiroaki Su...Dec 1st, 2022
Automation of dialogue system evaluation is a driving force for the efficient development of dialogue systems. This paper introduces the bipartite-play method, a dialogue collection method for automating dialogue system evaluation. It addresses the limitations of existing dialogue collection methods: (i) inability to compare with systems that are not publicly available, and (ii) vulnerability to cheating by intentionally selecting systems to be compared. Experimental results show that the...
-
Zachari Swiecki, Hassan Khosravi, Guanli...|Dec 1st, 2022|journalArticleZachari Swiecki, Hassan Khosravi, Guanli...Dec 1st, 2022
-
Shunya Takano, Osamu Ichikawa|Dec 1st, 2022|conferencePaperShunya Takano, Osamu IchikawaDec 1st, 2022
-
Ruben van Genugten, Daniel L Schacter|Dec 1st, 2022|journalArticleRuben van Genugten, Daniel L SchacterDec 1st, 2022
-
Kafeng Wang, Pengyang Wang, Chengzhong x...|Dec 1st, 2022|journalArticleKafeng Wang, Pengyang Wang, Chengzhong x...Dec 1st, 2022
Automated Feature Engineering (AFE) refers to automatically generate and select optimal feature sets for downstream tasks, which has achieved great success in real-world applications. Current AFE methods mainly focus on improving the effectiveness of the produced features, but ignoring the low-efficiency issue for large-scale deployment. Therefore, in this work, we propose a generic framework to improve the efficiency of AFE. Specifically, we construct the AFE pipeline based on reinforcement...
-
Zichao Wang, Jakob Valdez, Debshila Basu...|Dec 1st, 2022|bookSectionZichao Wang, Jakob Valdez, Debshila Basu...Dec 1st, 2022
-
Zichao Wang, Jakob Valdez, Debshila Basu...|Dec 1st, 2022|bookSectionZichao Wang, Jakob Valdez, Debshila Basu...Dec 1st, 2022