app/docs/METHODOLOGY.md at main · weval-org/app - weval-org (2025)
Contribute to weval-org/app development by creating an account on GitHub.
<标题>: app/docs/METHODOLOGY.md 在 main · weval-org/app Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks - Patwardhan, Tejal; Dias, Rachel; Proehl, Elizabeth; Kim, Grace; Wang, Michele; Watkins, Olivia; Fishman, Simón Posada; Aljubeh, Marwan; Thacker, Phoebe; Fauconnet, Laurance; Kim, Natalie S.; Chao, Patrick; et al. (arXiv, 2025)
We introduce GDPval, a benchmark evaluating AI model capabilities on real-world economically valuable tasks. GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP (Gross Domestic Product). Tasks are constructed from the representative work of industry professionals with an average of 14 years of experience. We find that frontier model performance on GDPval is improving roughly linearly over time, and that the current best frontier models are approaching industry experts in deliverable quality. We analyze the potential for frontier models, when paired with human oversight, to perform GDPval tasks cheaper and faster than unaided experts. We also demonstrate that increased reasoning effort, increased task context, and increased scaffolding improves model performance on GDPval. Finally, we open-source a gold subset of 220 tasks and provide a public automated grading service at evals.openai.com to facilitate future research in understanding real-world model capabilities.
arXiv:2510.04374 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computers and Society; Computer Science - Machine Learning

What past education technology failures can teach us about the future of AI in schools - Reich, Justin (2025)
It can take years to collect evidence that shows effective uses of new technologies in schools. Unfortunately, early guesses sometimes go seriously wrong.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

How does artificial intelligence compare to human feedback? A meta-analysis of performance, feedback perception, and learning dispositions - Kaliisa, Rogers; Misiejuk, Kamila; López-Pernas, Sonsoles; Saqr, Mohammed (Educational Psychology, 2025)
This exploratory meta-analysis synthesises current research on the effectiveness of Artificial Intelligence (AI)-generated feedback compared to traditional human-provided feedback. Drawing on 41 studies involving a total of 4813 students, the findings reveal no statistically significant differences in learning performance between students who received AI-generated feedback and those who received human-provided feedback. The pooled effect size was small and statistically insignificant (Hedge’s g = 0.25, CI [−0.11; 0.60]), indicating that AI feedback is potentially as effective as human feedback. A separate meta-analysis focusing exclusively on studies in the domain of language and writing confirmed similar findings, with high heterogeneity persisting (I2 = 95%). The study further explored differences in feedback perception and found a small, negative, and statistically insignificant effect size (Hedge’s g = −0.20, CI [−0.67; 0.27]). The study advocates for a hybrid approach, leveraging the scalability of AI while retaining the deep, empathetic, and contextual features of human feedback.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: Artificial intelligence-feedback; human-feedback; learning analytics; meta-analysis

Fluid Language Model Benchmarking - Hofmann, Valentin; Heineman, David; Magnusson, Ian; Lo, Kyle; Dodge, Jesse; Sap, Maarten; Koh, Pang Wei; Wang, Chun; Hajishirzi, Hannaneh; Smith, Noah A. (arXiv, 2025)
Language model (LM) benchmarking faces several challenges: comprehensive evaluations are costly, benchmarks often fail to measure the intended capabilities, and evaluation quality can degrade due to labeling errors and benchmark saturation. Although various strategies have been proposed to mitigate these issues, they tend to address individual aspects in isolation, neglecting broader questions about overall evaluation quality. Here, we introduce FLUID BENCHMARKING, a new evaluation approach that advances LM benchmarking across multiple dimensions. Inspired by psychometrics, FLUID BENCHMARKING is based on the insight that the relative value of benchmark items depends on an LM’s capability level, suggesting that evaluation should adapt to each LM. Methodologically, FLUID BENCHMARKING estimates an item response model based on existing LM evaluation results and uses the inferred quantities to select evaluation items dynamically, similar to computerized adaptive testing in education. In our experiments, we compare FLUID BENCHMARKING against the common practice of random item sampling as well as more sophisticated baselines, including alternative methods grounded in item response theory. We examine four dimensions—efficiency, validity, variance, and saturation—and find that FLUID BENCHMARKING achieves superior performance in all of them (e.g., higher validity and less variance on MMLU with fifty times fewer items). Our analysis shows that the two components of FLUID BENCHMARKING have distinct effects: item response theory, used to map performance into a latent ability space, increases validity, while dynamic item selection reduces variance. Overall, our results suggest that LM benchmarking can be substantially improved by moving beyond static evaluation.
arXiv:2509.11106 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Instructional Agents: LLM Agents on Automated Course Material Generation for Teaching Faculties - Yao, Huaiyuan; Xu, Wanpeng; Turnau, Justin; Kellam, Nadia; Wei, Hua (arXiv, 2025)
Preparing high-quality instructional materials remains a labor-intensive process that often requires extensive coordination among teaching faculty, instructional designers, and teaching assistants. In this work, we present Instructional Agents, a multi-agent large language model (LLM) framework designed to automate end-to-end course material generation, including syllabus creation, lecture scripts, LaTeX-based slides, and assessments. Unlike existing AI-assisted educational tools that focus on isolated tasks, Instructional Agents simulates role-based collaboration among educational agents to produce cohesive and pedagogically aligned content. The system operates in four modes: Autonomous, Catalog-Guided, Feedback-Guided, and Full Co-Pilot mode, enabling flexible control over the degree of human involvement. We evaluate Instructional Agents across five university-level computer science courses and show that it produces high-quality instructional materials while significantly reducing development time and human workload. By supporting institutions with limited instructional design capacity, Instructional Agents provides a scalable and cost-effective framework to democratize access to high-quality education, particularly in underserved or resource-constrained settings.
arXiv:2508.19611 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

An Early Review of Generative Language Models in Automated Writing Evaluation: Advancements, Challenges, and Future Directions for Automated Essay Scoring and Feedback Generation - Huang, Yue; Palermo, Corey; Liu, Ruitao; He, Yong (Chinese/English Journal of Educational Measurement and Evaluation | 教育测量与评估双语期刊, 2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z

Leveraging LLMs to Assess Tutor Moves in Real-Life Dialogues: A Feasibility Study - Thomas, Danielle R.; Borchers, Conrad; Lin, Jionghao; Kakarla, Sanjit; Bhushan, Shambhavi; Gatz, Erin; Gupta, Shivang; Abboud, Ralph; Koedinger, Kenneth R. (arXiv, 2025)
Tutoring improves student achievement, but identifying and studying what tutoring actions are most associated with student learning at scale based on audio transcriptions is an open research problem. This present study investigates the feasibility and scalability of using generative AI to identify and evaluate specific tutor moves in real-life math tutoring. We analyze 50 randomly selected transcripts of college-student remote tutors assisting middle school students in mathematics. Using GPT-4, GPT-4o, GPT-4-turbo, Gemini-1.5-pro, and LearnLM, we assess tutors’ application of two tutor skills: delivering effective praise and responding to student math errors. All models reliably detected relevant situations, for example, tutors providing praise to students (94-98% accuracy) and a student making a math error (82-88% accuracy) and effectively evaluated the tutors’ adherence to tutoring best practices, aligning closely with human judgments (83-89% and 73-77%, respectively). We propose a cost-effective prompting strategy and discuss practical implications for using large language models to support scalable assessment in authentic settings. This work further contributes LLM prompts to support reproducibility and research in AI-supported learning.
arXiv:2506.17410 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: Computer Science - Computation and Language; Computer Science - Computers and Society

Delving Into the Psychology of Machines: Exploring the Structure of Self-Regulated Learning via LLM-Generated Survey Responses - Vogelsmeier, Leonie V. D. E.; Oliveira, Eduardo; Misiejuk, Kamila; López-Pernas, Sonsoles; Saqr, Mohammed (arXiv, 2025)
Large language models (LLMs) offer the potential to simulate human-like responses and behaviors, creating new opportunities for psychological science. In the context of self-regulated learning (SRL), if LLMs can reliably simulate survey responses at scale and speed, they could be used to test intervention scenarios, refine theoretical models, augment sparse datasets, and represent hard-to-reach populations. However, the validity of LLM-generated survey responses remains uncertain, with limited research focused on SRL and existing studies beyond SRL yielding mixed results. Therefore, in this study, we examined LLM-generated responses to the 44-item Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich \& De Groot, 1990), a widely used instrument assessing students' learning strategies and academic motivation. Particularly, we used the LLMs GPT-4o, Claude 3.7 Sonnet, Gemini 2 Flash, LLaMA 3.1-8B, and Mistral Large. We analyzed item distributions, the psychological network of the theoretical SRL dimensions, and psychometric validity based on the latent factor structure. Our results suggest that Gemini 2 Flash was the most promising LLM, showing considerable sampling variability and producing underlying dimensions and theoretical relationships that align with prior theory and empirical findings. At the same time, we observed discrepancies and limitations, underscoring both the potential and current constraints of using LLMs for simulating psychological survey data and applying it in educational contexts.
arXiv:2506.13384 [cs] <标题>: 深入机器心理学：通过大语言模型生成的调查问卷回答探索自我调节学习的结构 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computers and Society; Statistics - Methodology; Statistics - Other Statistics

Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects - Schwartz, Reva; Chowdhury, Rumman; Kundu, Akash; Frase, Heather; Fadaee, Marzieh; David, Tom; Waters, Gabriella; Taik, Afaf; Briggs, Morgan; Hall, Patrick; Jain, Shomik; Yee, Kyra; et al. (2025)
Conventional AI evaluation approaches concentrated within the AI stack exhibit systemic limitations for exploring, navigating and resolving the human and societal factors that play out in real world deployment such as in education, finance, healthcare, and employment sectors. AI capability evaluations can capture detail about first-order effects, such as whether immediate system outputs are accurate, or contain toxic, biased or stereotypical content, but AI's second-order effects, i.e. any long-term outcomes and consequences that may result from AI use in the real world, have become a significant area of interest as the technology becomes embedded in our daily lives. These secondary effects can include shifts in user behavior, societal, cultural and economic ramifications, workforce transformations, and long-term downstream impacts that may result from a broad and growing set of risks. This position paper argues that measuring the indirect and secondary effects of AI will require expansion beyond static, single-turn approaches conducted in silico to include testing paradigms that can capture what actually materializes when people use AI technology in context. Specifically, we describe the need for data and methods that can facilitate contextual awareness and enable downstream interpretation and decision making about AI's secondary effects, and recommend requirements for a new ecosystem.
arXiv:2505.18893 [cs] <标题>: 现实检验：理解人工智能在现实世界影响需要新的评估生态系统 <AI Smry>: It is argued that measuring the indirect and secondary effects of AI will require expansion beyond static, single-turn approaches conducted in silico to include testing paradigms that can capture what actually materializes when people use AI technology in context. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.338Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computers and Society

From Chalkboards to Chatbots : Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria - Dikoru, Federico Hernan Tiberti, Maria Rebeca Barron Rodriguez, Federico Alfredo Manolio, Wuraola Mosuro, Eliot Jolomi, Martin Elias De Simone (2025)
From Chalkboards to Chatbots : Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria (English)
<标题>: 从黑板到聊天机器人：评估生成性人工智能对尼日利亚学习成果的影响 Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Extending Minds with Generative AI - Clark, Andy (Nature Communications, 2025)
As human-AI collaborations become the norm, we should remind ourselves that it is our basic nature to build hybrid thinking systems – ones that fluidly incorporate non-biological resources. Recognizing this invites us to change the way we think about both the threats and promises of the coming age.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: Intelligence; Interdisciplinary studies

Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs) - Geathers, Jadon; Hicke, Yann; Chan, Colleen; Rajashekar, Niroop; Sewell, Justin; Cornes, Susannah; Kizilcec, Rene F.; Shung, Dennis (arXiv, 2025)
Objective Structured Clinical Examinations (OSCEs) are widely used to assess medical students' communication skills, but scoring interview-based assessments is time-consuming and potentially subject to human bias. This study explored the potential of large language models (LLMs) to automate OSCE evaluations using the Master Interview Rating Scale (MIRS). We compared the performance of four state-of-the-art LLMs (GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro) in evaluating OSCE transcripts across all 28 items of the MIRS under the conditions of zero-shot, chain-of-thought (CoT), few-shot, and multi-step prompting. The models were benchmarked against a dataset of 10 OSCE cases with 174 expert consensus scores available. Model performance was measured using three accuracy metrics (exact, off-by-one, thresholded). Averaging across all MIRS items and OSCE cases, LLMs performed with low exact accuracy (0.27 to 0.44), and moderate to high off-by-one accuracy (0.67 to 0.87) and thresholded accuracy (0.75 to 0.88). A zero temperature parameter ensured high intra-rater reliability ({\alpha} = 0.98 for GPT-4o). CoT, few-shot, and multi-step techniques proved valuable when tailored to specific assessment items. The performance was consistent across MIRS items, independent of encounter phases and communication domains. We demonstrated the feasibility of AI-assisted OSCE evaluation and provided benchmarking of multiple LLMs across multiple prompt techniques. Our work provides a baseline performance assessment for LLMs that lays a foundation for future research into automated assessment of clinical communication skills.
arXiv:2501.13957 [cs] <标题>: 在客观结构化临床考试（OSCEs）中对生成式人工智能进行医学生面试评分的基准测试 <AI Smry>: The feasibility of AI-assisted OSCE evaluation is demonstrated and benchmarking of multiple LLMs across multiple prompt techniques is provided, providing a baseline performance assessment for LLMs that lays a foundation for future research into automated assessment of clinical communication skills. Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.004Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis - Wang, Jin; Fan, Wenxiang (Humanities and Social Sciences Communications, 2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems - Yarmohammadtoosky, Sahar; Zhou, Yiyun; Yaneva, Victoria; Baldwin, Peter; Rezayi, Saed; Clauser, Brian; Harikeo, Polina (arXiv, 2025)
This study examines vulnerabilities in transformer-based automated short-answer grading systems used in medical education, with a focus on how these systems can be manipulated through adversarial gaming strategies. Our research identifies three main types of gaming strategies that exploit the system's weaknesses, potentially leading to false positives. To counteract these vulnerabilities, we implement several adversarial training methods designed to enhance the systems' robustness. Our results indicate that these methods significantly reduce the susceptibility of grading systems to such manipulations, especially when combined with ensemble techniques like majority voting and ridge regression, which further improve the system's defense against sophisticated adversarial inputs. Additionally, employing large language models such as GPT-4 with varied prompting techniques has shown promise in recognizing and scoring gaming strategies effectively. The findings underscore the importance of continuous improvements in AI-driven educational tools to ensure their reliability and fairness in high-stakes settings.
arXiv:2505.00061 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language; Computer Science - Cryptography and Security

AI as Normal Technology (Knight First Amendment Institute, 2025)
<标题>: AI as Normal 技术 Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Anthropic Education Report: How University Students Use Claude (2025)
AI systems are no longer just specialized research tools: they’re everyday academic companions. As AIs integrate more deeply into educational environments, we need to consider important questions about learning, assessment, and skill development. Until now, most discussions have relied on surveys and controlled experiments rather than direct evidence of how students naturally integrate AI into their academic work in real settings.
<标题>: 人类学教育报告：大学生如何使用Claude Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Using Rasch Modeling to Validate AI Scoring Models - McGee, Steven; Hwang, Yaurie; Kelleigh, Willow (The Learning Partnership, 2025)
<标题>: 使用Rasch建模验证人工智能评分模型 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms - Razavi, Pooya; Powers, Sonya J. (arXiv, 2025)
Estimating item difficulty through field-testing is often resource-intensive and time-consuming. As such, there is strong motivation to develop methods that can predict item difficulty at scale using only the item content. Large Language Models (LLMs) represent a new frontier for this goal. The present research examines the feasibility of using an LLM to predict item difficulty for K-5 mathematics and reading assessment items (N = 5170). Two estimation approaches were implemented: (a) a direct estimation method that prompted the LLM to assign a single difficulty rating to each item, and (b) a feature-based strategy where the LLM extracted multiple cognitive and linguistic features, which were then used in ensemble tree-based models (random forests and gradient boosting) to predict difficulty. Overall, direct LLM estimates showed moderate to strong correlations with true item difficulties. However, their accuracy varied by grade level, often performing worse for early grades. In contrast, the feature-based method yielded stronger predictive accuracy, with correlations as high as r = 0.87 and lower error estimates compared to both direct LLM predictions and baseline regressors. These findings highlight the promise of LLMs in streamlining item development and reducing reliance on extensive field testing and underscore the importance of structured feature extraction. We provide a seven-step workflow for testing professionals who would want to implement a similar item difficulty estimation approach with their item pool.
arXiv:2504.08804 [cs] <标题>: 使用大型语言模型和基于树的机器学习算法估算题目难度 <AI Smry>: The present research examines the feasibility of using an LLM to predict item difficulty for K-5 mathematics and reading assessment items and provides a seven-step workflow for testing professionals who would want to implement a similar item difficulty estimation approach with their item pool. Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Computation and Language; Computer Science - Computers and Society; Computer Science - Machine Learning

Fine-Tuning GPT-3.5-Turbo for Automatic Feedback Generation - Mazzullo, Elisabetta; Bulut, Okan; Walsh, Cole; Sitarenios, Gill; MacIntosh, Alexander (Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing, 2025)
<标题>: 对 GPT-3.5-Turbo 进行微调以生成自动反馈 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Are LLMs (Really) Ideological? An IRT-based Analysis and Alignment Tool for Perceived Socio-Economic Bias in LLMs - Wachter, Jasmin; Radloff, Michael; Smolej, Maja; Kinder-Kurlanda, Katharina (arXiv, 2025)
We introduce an Item Response Theory (IRT)-based framework to detect and quantify socioeconomic bias in large language models (LLMs) without relying on subjective human judgments. Unlike traditional methods, IRT accounts for item difficulty, improving ideological bias estimation. We fine-tune two LLM families (Meta-LLaMa 3.2-1B-Instruct and Chat- GPT 3.5) to represent distinct ideological positions and introduce a two-stage approach: (1) modeling response avoidance and (2) estimating perceived bias in answered responses. Our results show that off-the-shelf LLMs often avoid ideological engagement rather than exhibit bias, challenging prior claims of partisanship. This empirically validated framework enhances AI alignment research and promotes fairer AI governance.
arXiv:2503.13149 [cs] <标题>: 大型语言模型（真的）有意识形态倾向吗？一种基于项目反应理论的分析与校准工具，用于感知大型语言模型中的社会经济偏见 <AI Smry>: An Item Response Theory (IRT)-based framework to detect and quantify socioeconomic bias in large language models (LLMs) without relying on subjective human judgments is introduced, challenging prior claims of partisanship. Read_Status: New Read_Status_Date: 2026-01-26T11:32:30.842Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Computers and Society

Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models (2025)
<标题>: Vibe-Eval：用于测量多模态语言模型进度的硬评估套件 Read_Status: New Read_Status_Date: 2026-01-26T11:33:01.187Z

Qualitative Coding with GPT-4: Where it Works Better - Liu, Xiner; Zambrano, Andrés; Baker, Ryan; Barany, Amanda; Ocumpaugh, Jaclyn; Zhang, Jiayi; Pankiewicz, Maciej; Nasiar, Nidhi; Wei, Zhanlan (Journal of Learning Analytics, 2025)
This study explores the potential of the large language model GPT-4 as an automated tool for qualitative data analysis by educational researchers, exploring which techniques are most successful for different types of constructs. Specifically, we assess three different prompt engineering strategies-Zero-shot, Few-shot, and Few-shot with contextual information-as well as the use of embeddings. We do so in the context of qualitatively coding three distinct educational datasets: Algebra I semi-personalized tutoring session transcripts, student observations in a game-based learning environment, and debugging behaviors in an introductory programming course. We evaluated each approach's performance based on its inter-rater agreement with human coders and explored how different methods vary in effectiveness depending on a construct's degree of clarity, concreteness, objectivity, granularity, and specificity. Our findings suggest that while GPT-4 can code a broad range of constructs, no single method consistently outperforms the others, and the selection of a particular method should be tailored to the specific properties of the construct and context being analyzed. We also found that the constructs that GPT-4 has the most difficulty with are the same constructs than human coders find more difficult to reach inter-rater reliability on. Notes for Practice (research paper)  GPT-4 can be used to code qualitative data for educationally-relevant constructs.  Using embeddings and examples can improve agreement with humans. Examples are more useful for constructs that are more difficult to define.  Constructs that human beings find difficult to agree on are also difficult for GPT-4.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.206Z

Turnitin Clarity: Bringing transparency to the student writing process | Turnitin (2025)
Find out how Turnitin Clarity brings transparency and integrity insights into the student writing process in Turnitin Feedback Studio
Read_Status: New Read_Status_Date: 2026-01-26T11:33:01.187Z

Assessing and alleviating state anxiety in large language models (npj Digital Medicine, 2025)
<AI Smry>: It is found that traumatic narratives increased Chat-GPT-4’s reported anxiety while mindfulness-based exercises reduced it, though not to baseline, which suggests managing LLMs’ “emotional states” can foster safer and more ethical human-AI interactions. <标题>: 评估和缓解大型语言模型中的状态焦虑 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs - Xiao, Changrong; Ma, Wenxing; Song, Qingping; Xu, Sean Xin; Zhang, Kunpeng; Wang, Yufang; Fu, Qi (arXiv, 2025)
Receiving timely and personalized feedback is essential for second-language learners, especially when human instructors are unavailable. This study explores the effectiveness of Large Language Models (LLMs), including both proprietary and open-source models, for Automated Essay Scoring (AES). Through extensive experiments with public and private datasets, we find that while LLMs do not surpass conventional state-of-the-art (SOTA) grading models in performance, they exhibit notable consistency, generalizability, and explainability. We propose an open-source LLM-based AES system, inspired by the dual-process theory. Our system offers accurate grading and high-quality feedback, at least comparable to that of fine-tuned proprietary LLMs, in addition to its ability to alleviate misgrading. Furthermore, we conduct human-AI co-grading experiments with both novice and expert graders. We find that our system not only automates the grading process but also enhances the performance and efficiency of human graders, particularly for essays where the model has lower confidence. These results highlight the potential of LLMs to facilitate effective human-AI collaboration in the educational context, potentially transforming learning experiences through AI-generated feedback.
arXiv:2401.06431 [cs] <标题>: 人类与人工智能协作作文评分：基于大语言模型的双流程框架 <AI Smry>: This study explores the effectiveness of Large Language Models, including both proprietary and open-source models, for Automated Essay Scoring (AES), and proposes an open-source LLM-based AES system, inspired by the dual-process theory. Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.004Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs - Xiao, Changrong; Ma, Wenxing; Song, Qingping; Xu, Sean Xin; Zhang, Kunpeng; Wang, Yufang; Fu, Qi (arXiv, 2025)
Receiving timely and personalized feedback is essential for second-language learners, especially when human instructors are unavailable. This study explores the effectiveness of Large Language Models (LLMs), including both proprietary and open-source models, for Automated Essay Scoring (AES). Through extensive experiments with public and private datasets, we find that while LLMs do not surpass conventional state-of-the-art (SOTA) grading models in performance, they exhibit notable consistency, generalizability, and explainability. We propose an open-source LLM-based AES system, inspired by the dual-process theory. Our system offers accurate grading and high-quality feedback, at least comparable to that of fine-tuned proprietary LLMs, in addition to its ability to alleviate misgrading. Furthermore, we conduct human-AI co-grading experiments with both novice and expert graders. We find that our system not only automates the grading process but also enhances the performance and efficiency of human graders, particularly for essays where the model has lower confidence. These results highlight the potential of LLMs to facilitate effective human-AI collaboration in the educational context, potentially transforming learning experiences through AI-generated feedback.
arXiv:2401.06431 [cs] <标题>: 人类与人工智能协作作文评分：基于大语言模型的双流程框架 <AI Smry>: This study explores the effectiveness of Large Language Models, including both proprietary and open-source models, for Automated Essay Scoring (AES), and proposes an open-source LLM-based AES system, inspired by the dual-process theory. Read_Status: New Read_Status_Date: 2026-01-26T11:32:31.459Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

The Human Complimentary Usage of AI and ML for Fair and Unbiased Educational Assessments - Suna, Hayri Eren; Özer, Mahmut (Chinese/English Journal of Educational Measurement and Evaluation, 2025)
In recent years, artificial intelligence (AI) and machine learning (ML) algorithms have played an influential role in advancing educational assessment. As a means of improving equal opportunities in education, assessing students' learning deficiencies and developing personalized learning suggestions is considered an important aspect. Furthermore, big data-based algorithms play an increasing role in assessing students' cognitive and social-emotional development and conducting research on school-based monitoring. A number of these developments, on the one hand, provide valuable feedback to both students and educational authorities regarding assessment of student growth, and on the other hand, they raise the issue of bias in the algorithms. As we experience numerous examples of AI- and ML- algorithms that provide biased results and reproduce existing inequalities based on training data inadequacies. Studies indicate that relevant algorithms can lead biased inferences about a number of factors, such as gender, race, ethnicity, socioeconomic status, and migration. For this reason, various approaches and methods have been developed, including sample weight adjustments, bias attenuation methods, fairness through unawareness, adversarial learning, and participatory management, so that AI- and ML- algorithms can deliver more fair and valid outcomes. In this paper, we examine the challenges that lead AI and ML to produce biased results in evaluating cognitive and social-emotional skills of educational students, the recent methods and approaches to examine and debias the algorithmic bias as well as the policies and precautions for achieving more unbiased and valid results. We anticipate that insights from this study will increase awareness of objective and valid assessment of educational skills via AI- and ML- algorithms.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Use of AI (GPT-4)-generated multiple-choice questions for the examination of surgical subspecialty residents: Report of feasibility and psychometric analysis - Kim, Jin Kyu (Justin); Chua, Michael; Lorenzo, Armando; Rickard, Mandy; Andreacchi, Laura; Kim, Michael; Cheung, Douglas; Krakowsky, Yonah; Lee, Jason Y. (Canadian Urological Association Journal, 2025)
Introduction: Multiple-choice questions (MCQs) are essential in medical education and widely used by licensing bodies. They are traditionally created with intensive human effort to ensure validity. Recent advances in AI, particularly large language models (LLMs), offer the potential to streamline this process. This study aimed to develop and test a GPT-4 model with customized instructions for generating MCQs to assess urology residents. Methods: A GPT-4 model was embedded using guidelines from medical licensing bodies and reference materials specific to urology. This model was tasked with generating MCQs designed to mimic the format and content of the 2023 urology examination outlined by the Royal College of Physicians and Surgeons of Canada (RCPSC). Following generation, a selection of MCQs underwent expert review for validity and suitability. Results: From an initial set of 123 generated MCQs, 60 were chosen for inclusion in an exam administered to 15 urology residents at the University of Toronto. The exam results demonstrated a general increasing performance with level of training cohorts, suggesting the MCQs’ ability to effectively discriminate knowledge levels among residents. The majority (33/60) of the questions had discriminatory value that appeared acceptable (discriminatory index 0.20.4) or excellent (discriminatory index >0.4). Conclusions: This study highlights AI-driven models like GPT-4 as efficient tools to aid with MCQ generation in medical education assessments. By automating MCQ creation while maintaining quality standards, AI can expedite processes. Future research should focus on refining AI applications in education to optimize assessments and enhance medical training and certification outcomes.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.078Z

Leveraging LLM respondents for item evaluation: A psychometric analysis - Liu, Yunting; Bhandari, Shreya; Pardos, Zachary A. (British Journal of Educational Technology, 2025)
Effective educational measurement relies heavily on the curation of well‐designed item pools. However, item calibration is time consuming and costly, requiring a sufficient number of respondents to estimate the psychometric properties of items. In this study, we explore the potential of six different large language models (LLMs; GPT‐3.5, GPT‐4, Llama 2, Llama 3, Gemini‐Pro and Cohere Command R Plus) to generate responses with psychometric properties comparable to those of human respondents. Results indicate that some LLMs exhibit proficiency in College Algebra that is similar to or exceeds that of college students. However, we find the LLMs used in this study to have narrow proficiency distributions, limiting their ability to fully mimic the variability observed in human respondents, but that an ensemble of LLMs can better approximate the broader ability distribution typical of college students. Utilizing item response theory, the item parameters calibrated by LLM respondents have high correlations (eg, >0.8 for GPT‐3.5) with their human calibrated counterparts. Several augmentation strategies are evaluated for their relative performance, with resampling methods proving most effective, enhancing the Spearman correlation from 0.89 (human only) to 0.93 (augmented human).
<标题>: 利用 LLM 受访者进行题目评估：一项心理测量分析 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z

Leveraging LLM respondents for item evaluation: A psychometric analysis - Liu, Yunting; Bhandari, Shreya; Pardos, Zachary A. (British Journal of Educational Technology, 2025)
Effective educational measurement relies heavily on the curation of well‐designed item pools. However, item calibration is time consuming and costly, requiring a sufficient number of respondents to estimate the psychometric properties of items. In this study, we explore the potential of six different large language models (LLMs; GPT‐3.5, GPT‐4, Llama 2, Llama 3, Gemini‐Pro and Cohere Command R Plus) to generate responses with psychometric properties comparable to those of human respondents. Results indicate that some LLMs exhibit proficiency in College Algebra that is similar to or exceeds that of college students. However, we find the LLMs used in this study to have narrow proficiency distributions, limiting their ability to fully mimic the variability observed in human respondents, but that an ensemble of LLMs can better approximate the broader ability distribution typical of college students. Utilizing item response theory, the item parameters calibrated by LLM respondents have high correlations (eg, >0.8 for GPT‐3.5) with their human calibrated counterparts. Several augmentation strategies are evaluated for their relative performance, with resampling methods proving most effective, enhancing the Spearman correlation from 0.89 (human only) to 0.93 (augmented human).
<标题>: 利用 LLM 受访者进行题目评估：一项心理测量分析 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring - Wu, Xuansheng; Saraf, Padmaja Pravin; Lee, Gyeonggeon; Latif, Ehsan; Liu, Ninghao; Zhai, Xiaoming (arXiv, 2025)
Large language models (LLMs) have demonstrated strong potential in performing automatic scoring for constructed response assessments. While constructed responses graded by humans are usually based on given grading rubrics, the methods by which LLMs assign scores remain largely unclear. It is also uncertain how closely AI's scoring process mirrors that of humans or if it adheres to the same grading criteria. To address this gap, this paper uncovers the grading rubrics that LLMs used to score students' written responses to science tasks and their alignment with human scores. We also examine whether enhancing the alignments can improve scoring accuracy. Specifically, we prompt LLMs to generate analytic rubrics that they use to assign scores and study the alignment gap with human grading rubrics. Based on a series of experiments with various configurations of LLM settings, we reveal a notable alignment gap between human and LLM graders. While LLMs can adapt quickly to scoring tasks, they often resort to shortcuts, bypassing deeper logical reasoning expected in human grading. We found that incorporating high-quality analytical rubrics designed to reflect human grading logic can mitigate this gap and enhance LLMs' scoring accuracy. These results underscore the need for a nuanced approach when applying LLMs in science education and highlight the importance of aligning LLM outputs with human expectations to ensure efficient and accurate automatic scoring.
arXiv:2407.18328 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language; Computer Science - Computers and Society

Improve LLM-based Automatic Essay Scoring with Linguistic Features - Hou, Zhaoyi Joey; Ciuba, Alejandro; Li, Xiang Lorraine (arXiv, 2025)
Automatic Essay Scoring (AES) assigns scores to student essays, reducing the grading workload for instructors. Developing a scoring system capable of handling essays across diverse prompts is challenging due to the ﬂexibility and diverse nature of the writing task. Existing methods typically fall into two categories: supervised feature-based approaches and large language model (LLM)-based methods. Supervised feature-based approaches often achieve higher performance but require resource-intensive training. In contrast, LLMbased methods are computationally eﬃcient during inference but tend to suﬀer from lower performance. This paper combines these approaches by incorporating linguistic features into LLM-based scoring. Experimental results show that this hybrid method outperforms baseline models for both in-domain and out-of-domain writing prompts1.
arXiv:2502.09497 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Finding Words Associated with DIF: Predicting Differential Item Functioning using LLMs and Explainable AI - Maeda, Hotaka; Lu, Yikai (arXiv, 2025)
We fine-tuned and compared several encoder-based Transformer large language models (LLM) to predict differential item functioning (DIF) from the item text. We then applied explainable artificial intelligence (XAI) methods to these models to identify specific words associated with DIF. The data included 42,180 items designed for English language arts and mathematics summative state assessments among students in grades 3 to 11. Prediction $R2$ ranged from .04 to .32 among eight focal and reference group pairs. Our findings suggest that many words associated with DIF reflect minor sub-domains included in the test blueprint by design, rather than construct-irrelevant item content that should be removed from assessments. This may explain why qualitative reviews of DIF items often yield confusing or inconclusive results. Our approach can be used to screen words associated with DIF during the item-writing process for immediate revision, or help review traditional DIF analysis results by highlighting key words in the text. Extensions of this research can enhance the fairness of assessment programs, especially those that lack resources to build high-quality items, and among smaller subpopulations where we do not have sufficient sample sizes for traditional DIF analyses.
arXiv:2502.07017 [cs] <标题>: 寻找与 DIF 相关的词语：使用大型语言模型和可解释人工智能预测差异项目功能 <AI Smry>: The findings suggest that many words associated with DIF reflect minor sub-domains included in the test blueprint by design, rather than construct-irrelevant item content that should be removed from assessments, which may explain why qualitative reviews of DIF items often yield confusing or inconclusive results. Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.807Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Algorithmic bias in educational systems: Examining the impact of AI-driven decision making in modern education - Obed Boateng; Bright Boateng (World Journal of Advanced Research and Reviews, 2025)
The increasing integration of artificial intelligence and algorithmic systems in educational settings has raised critical concerns about their impact on educational equity. This paper examines the manifestation and implications of algorithmic bias across various educational domains, including admissions processes, assessment systems, and learning management platforms. Through analysis of current research and studies, we investigate how these biases can perpetuate or exacerbate existing educational disparities, particularly affecting students from marginalized communities. The study reveals that algorithmic bias in education operates through multiple channels, from data collection and algorithm design to implementation practices and institutional policies. Our findings indicate that biased algorithms can significantly impact students' educational trajectories, creating new forms of systemic barriers in education. We propose a comprehensive framework for addressing these challenges, combining technical solutions with policy reforms and institutional guidelines. This research contributes to the growing discourse on ethical AI in education and provides practical strategies for creating more equitable educational systems in an increasingly digitized world.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions - Huang, Lei; Yu, Weijiang; Ma, Weitao; Zhong, Weihong; Feng, Zhangyin; Wang, Haotian; Chen, Qianglong; Peng, Weihua; Feng, Xiaocheng; Qin, Bing; Liu, Ting (arXiv, 2025)
The emergence of large language models (LLMs) has marked a significant breakthrough in natural language processing (NLP), fueling a paradigm shift in information acquisition. Nevertheless, LLMs are prone to hallucination, generating plausible yet nonfactual content. This phenomenon raises significant concerns over the reliability of LLMs in real-world information retrieval (IR) systems and has attracted intensive research to detect and mitigate such hallucinations. Given the open-ended general-purpose attributes inherent to LLMs, LLM hallucinations present distinct challenges that diverge from prior task-specific models. This divergence highlights the urgency for a nuanced understanding and comprehensive overview of recent advances in LLM hallucinations. In this survey, we begin with an innovative taxonomy of hallucination in the era of LLM and then delve into the factors contributing to hallucinations. Subsequently, we present a thorough overview of hallucination detection methods and benchmarks. Our discussion then transfers to representative methodologies for mitigating LLM hallucinations. Additionally, we delve into the current limitations faced by retrieval-augmented LLMs in combating hallucinations, offering insights for developing more robust IR systems. Finally, we highlight the promising research directions on LLM hallucinations, including hallucination in large vision-language models and understanding of knowledge boundaries in LLM hallucinations.
arXiv:2311.05232 [cs] <标题>: 关于大型语言模型幻觉的调研：原理、分类、挑战与未解问题 <AI Smry>: A thorough overview of hallucination detection methods and benchmarks is presented and the promising research directions on LLM hallucinations are highlighted, including hallucination in large vision-language models and understanding of knowledge boundaries in LLM hallucinations. Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: /unread; Computer Science - Computation and Language

Auto-Evaluation: A Critical Measure in Driving Improvements in Quality and Safety of AI-Generated Lesson Resources - Clark, Hannah-Beth; Dowland, Margaux; Benton, Laura; Budai, Reka; Keskin, Ibrahim Kaan; Searle, Emma; Gregory, Matthew; Hodierne, Mark; Roberts, John (The AI + Open Education Initiative, 2025)
Designing AI tools for use in educational settings presents distinct challenges; the need for accuracy is heightened, safety is imperative and pedagogical rigor is crucial.As a publicly funded body in the UK, Oak National Academy is in a unique position to innovate within this field as we have a comprehensive curriculum of approximately 13,000 open education resources (OER) for all National Curriculum subjects, designed and quality-assured by expert, human teachers. This has provided the corpus of content needed for building a high-quality AI-powered lesson planning tool, Aila, that is free to use and, therefore, accessible to all teachers across the country. Furthermore, using our evidence-informed curriculum principles, we have codified and exemplified each component of lesson design. To assess the quality of lessons produced by Aila at scale, we have developed an AI-powered auto-evaluation agent, facilitating informed improvements to enhance output quality. Through comparisons between human and auto-evaluations, we have begun to refine this agent further to increase its accuracy, measured by its alignment with an expert human evaluator. In this paper we present this iterative evaluation process through an illustrative case study focused on one quality benchmark - the level of challenge within multiple-choice quizzes. We also explore the contribution that this may make to similar projects and the wider sector.Author Note: Correspondence concerning this article should be addressed to Hannah-Beth Clark. Email: hannah-beth.clark@thenational.academy <mailto:hannah-beth.clark@thenational.academy>
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z

AI and the future of education: Disruptions, dilemmas and directions (UNESCO, 2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

Toward Subtrait-Level Model Explainability in Automated Writing Evaluation - Andrade-Lotero, Alejandro; Becker, Lee; Southerland, Joshua; Hellman, Scott; Behan, Bradley (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Showcasing a STEM Conversation-based Assessment with AI Agents - Andrews-Todd, Jessica; Graf, Edith Aurora; Forsyth, Carol; Cisterna, Dante; Song, Yi; Lee, Jeremy (2025)
<标题>: 展示基于对话的STEM评估与人工智能代理 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Investigating Different Prompt Engineering Strategies Across Different Content Areas - Araneda, Sergio (2025)
Presenters: _:n14546 <标题>: 在不同内容领域研究不同的提示工程策略 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Generative-AI automatic item generation for large- scale language assessments - Attali, Yigal; Runge, Andrew; Church, Jacqueline; Park, Yena; Goodwin, Sarah (2025)
Presenters: _:n14557 <标题>: 用于大规模语言评估的生成式人工智能自动题目生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Realistic Data Augmentation for Automated Scoring Engine Training - Barber, Justin O.; Wolfe, Edward W. (2025)
<标题>: 用于自动评分引擎训练的真实数据增强 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Automated Item Generation for Reading Assessments - Bezirhan, Ummugul; Wry; von Davier, Mattias (2025)
Presenters: _:n14819 <标题>: 用于阅读评估的自动化题目生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Automated Scoring of Student Essays for Diagnostic Assessments - Bryer, Jason M.; Lui, Angela L.; Fraser, Anthony; Ferris, Julia (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Automated Scoring of Student Essays for Diagnostic Assessments - Bryer, Jason M.; Lui, Angela L.; Fraser, Anthony; Ferris, Julia (2025)
Presenters: _:n14792 <标题>: 用于诊断性评估的学生作文自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Extending Human and AI Annotations beyond Argumentative Writing: A Model Transfer Inquiry - Burkhardt, Amy; Woolf, Sherri; Ormerod, Christopher; Kehat, Gitit; Lottridge, Susan (2025)
<标题>: 将人类和人工智能标注扩展到论证性写作之外：一种模型迁移探究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

Detection of AI-Generated Argumentative Essays - Demir, Cihan; Burkhardt, Amy; Kwako, Alexander; Schultz, Zachary; Lottridge, Susan (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Comparing LLMs in Conversational Interactions - DiCerbo, Kristen; Burleigh, Tyler; Deters, Lauren (2025)
Presenters: _:n14740 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Scoring Math Constructed Response Prompts with a Hybrid Automated Scoring Model (HASM) - Eggers-Robertson, Lisa (2025)
Presenters: _:n14485 <标题>: 使用混合自动评分模型（HASM）对数学构建性回答提示进行评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Investigating Appropriate Uses for Automated Item Generation Content - Fen, Fan; Henderek, Jerusha (2025)
<标题>: 研究自动化生成项目内容的适用用途 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

Instruction‐Tuned Large‐Language Models for Quality Control in Automatic Item Generation: A Feasibility Study - Gorgun, Guher; Bulut, Okan (Educational Measurement: Issues and Practice, 2025)
Automatic item generation may supply many items instantly and efficiently to assessment and learning environments. Yet, the evaluation of item quality persists to be a bottleneck for deploying generated items in learning and assessment settings. In this study, we investigated the utility of using large-language models, specifically Llama 3-8B, for evaluating automatically generated cloze items. The trained large-language model was able to filter out majority of good and bad items accurately. Evaluating items automatically with instruction-tuned LLMs may aid educators and test developers in understanding the quality of items created in an efficient and scalable manner. The item evaluation process with LLMs may also act as an intermediate step between item creation and field testing to reduce the cost and time associated with multiple rounds of revision.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

Predicting the Item Characteristics of Scenario-Based Assessment Items - Gorgun, Guher; Bulut, Okan (2025)
Presenters: _:n14480 <标题>: 基于情境评估题目的题目特征预测 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Mathematical Explorations in an LLM - Graf, Edith Aurora; Forsyth, Carol; Ruiz Diaz, Shona; Yan, Duanli; Jiang, Yang (2025)
Presenters: _:n14489 <标题>: 在大型语言模型中的数学探索 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Mathematical Explorations in an LLM - Graf, Edith Aurora; Forsyth, Carol; Ruiz Diaz, Shona; Yan, Duanli; Jiang, Yang (2025)
<标题>: 在大型语言模型中的数学探索 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Automated Plagiarism and AI-Influenced Speech Detection in Spoken Responses Using Linguistic and Acoustic Features - Hadi, Syed A; Bishop, Kyoungwon; Karamese, Hacer; Lee, Sooyoung (2025)
<标题>: 使用语言和声学特征对口语回答中的自动化抄袭和受AI影响的言语进行检测 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Automated Monitoring of Test-Taking Behaviors Using Process Data - Han, Suhwa; Kang, Hyeon-Ah (2025)
<标题>: 使用过程数据对考试行为进行自动化监控 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Automated Coding of Communications in Collaborative Problem-solving Tasks Using ChatGPT - Hao, Jiangang; Cui, Wenju; Kyllonen, Patrick C.; Kerzabi, Emily; Liu, Lei; Flor, Michael (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Leveraging Large Language Models and Prompt Engineering for Automated Essay Scoring - Huang, Yue; Wilson, Joshua; Palermo, Corey (2025)
Presenters: _:n14540 <标题>: 利用大型语言模型和提示工程进行自动化作文评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Validation Beyond Agreement - Hughes, Sarah R. (2025)
Presenters: _:n14395 <标题>: 超越协议的验证 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

EXPLORING HUMAN AND AUTOMATED SCORING SYSTEM AGREEMENT FOR SHORT ANSWER QUESTIONS - Iaccarino, Stephanie; Clauser, Brian E.; Harik, Polina; Baldwin, Peter; Zhou, Yiyun; Kane, Michael (2025)
Presenters: _:n14625 <标题>: 探索人类与自动评分系统在简答题上的一致性 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Enhancing Systematic Reviews with LLMs - Kaptur, Danielle; Huang, Yue; Ji, Xuejun Ryan; Guo, Yanhui; Kaptur, Bradley (2025)
Presenters: _:n14673 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Designing a Pipeline for Item Development using NLP, LLMs & Gen-AI - Karelitz, Tzur; Levi, Effi; Sarig, Joni; Shor, Yael; Shillo, Roi (2025)
Presenters: _:n14718 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Incorporating NLP features into Transformer-Based AES Models - Kim, YoungKoung; Moses, Tim (2025)
<标题>: 将自然语言处理功能整合到基于Transformer的自动评分模型中 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Augmenting LLMs with Knowledge Seeds for Clinical Question Generation - Kong, Lingchen; Muntean, William; Betts, Joe (2025)
Presenters: _:n14825 <标题>: 利用知识种子增强大型语言模型以生成临床问题 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Human Raters and Automated Scores: Hybrid Scoring for the Best of Both Worlds - Kwako, Alexander; Lottridge, Susan; Ormerod, Chris (2025)
<标题>: 人工评分与自动评分：兼得两者优势的混合评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

TOWARDS RELIABLE GENERATION OF CLINICAL CHART ITEMS: A KNOWLEDGE-INFUSED COUNTERFACTUAL APPROACH - Le, Jiaxuan; Rezayi, Saed; Baldwin, Peter; Harik, Polina; Yaneva, Victoria (2025)
Presenters: _:n14412 <标题>: 迈向临床图表条目的可靠生成：一种知识驱动的反事实方法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Performance of Six BERT Models in Detecting AI-Generated Texts - Lee, Selene (2025)
<标题>: 六种 BERT 模型在检测 AI 生成文本中的表现 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

Assessing the Representativeness of LLM-Generated Item Responses Using Latent Class Analysis - Lu, Yining; Huang, Yuan; Kanopka, Klint (2025)
Presenters: _:n14848 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Developing Parallel Forms for Sentence Reading Efficiency Using LLM-based Item Response Simulator - Ma, Wanjing Anya; Zelikman, Eric; Tran, Jasmine E.; Domingue, Ben; Haber, Nick; Yeatman, Jason D. (2025)
Presenters: _:n14709 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

AI-Generated Medical Certification Items Using LLMs - Makinde, Henry (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Bridging AI and Educational Measurement: A Systematic Review and Curriculum Reform - Makinde, Henry; Adegoke, Hope (2025)
<标题>: 人工智能与教育测量的桥梁：系统综述与课程改革 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Aligning Automation with Expertise: A Better Approach to Assessment Evaluation - Moore, Steven (2025)
Presenters: _:n14876 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Using Large Language Models for Automatic Item Generation: Development and Validation on the example of TIMSS Grade 4 - Muszyński, Marek; Plisiecki, Hubert; Żółtak, Tomasz; Pokropek, Artur (2025)
Presenters: _:n14399 <标题>: 使用大型语言模型进行自动题目生成：以 TIMSS 四年级为例的开发与验证 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Improving Zero-Shot Classification of Alerting Content Essays with Automatically Generated Descriptions - Nam, Sungjin (2025)
Presenters: _:n14565 <标题>: 通过自动生成的描述改进警示内容论文的零样本分类 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

An Evaluation of Automated Item Scoring in the Context of Inter-Rater Reliability - Nolan, Kate; Kang, Youngsoon; Liu, Ran (2025)
Presenters: _:n14870 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Generative Language Models for Argumentation Annotation - North, Kai; Ormerod, Christopher (2025)
<标题>: 用于论证标注的生成式语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Exploring Generative AI Assisted Item Development in Mathematics: A Mixed-Method Approach - Osaro, Sarah; Song, Dan; Welch, Catherine; Dunbar, Stephen; Hazen, Tim (2025)
Presenters: _:n14640 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Senem-Ai: Leveraging Llms for Student Behavior Simulation in Virtual Learning Environments - Pentangelo, Viviana; Turco, Luigi; Lambiase, Stefano; Gravino, Carmine; Palomba, Fabio (SSRN, 2025)
SENEM-AI is a 3D virtual environment-based tool designed to enhance teaching and presentation skills by leveraging immersive simulations. Built upon the SENEM platform, it integrates virtual students powered by LLaMA with distinct personalities that simulate realistic classroom interactions. Educators can refine their communication strategies by reacting to dynamically generated questions. Preliminary evaluations highlighted its usability and potential impact, with participants valuing the immersive experience and engagement. SENEMAI represents a novel approach to supporting educators through accessible technology, paving the way for further research into AI-driven teaching aids and training environments in virtual settings.
Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z

Leveraging Large Language Models for Automated Scoring of Constructed Response Mathematics Items - Perkoff, E. Margaret (2025)
<标题>: 利用大型语言模型对构建性回答数学题进行自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Examining the Effects of AIG Item Variance on Equating - Phenow, Aurore Yang (2025)
Presenters: _:n14662 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Using Multi-label Classification Neural Network to Detect Intersectional DIF with Small Sample Sizes - Quan, Yale; Wang, Chun (2025)
This study introduces InterDIFNet, a multilabel classification neural network for detecting intersectional differential item functioning (DIF) in educational and psychological assessments, with a focus on small sample sizes. Unlike traditional marginal DIF methods, which often fail to capture the effects of intersecting identities and require large samples, InterDIFNet models uniform and non-uniform DIF across multiple intersectional groups simultaneously. The method utilizes an optimized thresholding procedure to balance power and Type I error control. A Monte Carlo simulation compared InterDIFNet to the Truncated Lasso Penalty (TLP) test and other intersectional DIF methods across varying sample sizes, numbers of groups, and DIF prevalence rates. Results show that when trained using TLP features, InterDIFNet consistently achieved higher power than TLP while maintaining comparable Type I error control, particularly in scenarios with three or more intersectional groups. An empirical application to real assessment data further demonstrated the method’s practical utility. InterDIFNet provides a scalable, data-driven solution for identifying intersectional DIF across multiple small sample groups.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: ❓ Multiple DOI

Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms - Razavi, Pooya; Powers, Sonya J. (2025)
<标题>: 使用大型语言模型和基于树的机器学习算法估算题目难度 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Automatic Detection of Inauthentic, Templated Responses in English Language Assessments - Samant, Yashad; Becker, Lee; Hellman, Scott; Behan, Bradley; Hughes, Sarah; Southerland, Joshua (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Automatic Detection of Inauthentic, Templated Responses in English Language Assessments - Samant, Yashad; Becker, Lee; Hellman, Scott; Behan, Bradley; Hughes, Sarah; Southerland, Joshua (2025)
Presenters: _:n14757 <标题>: 英语语言评估中非真实、模板化回答的自动检测 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Assessing Writing Using ChatGPT to Score Essays and Short-Form Constructed Responses - Shermis, Mark D. (2025)
<标题>: 使用 ChatGPT 评估写作：评分论文和简短构建式回答 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Explorations in using student solution pathways and large language models for enhanced mathematical learning - Stacho, Ladislav; Graf, Edith Aurora; Zapata-Rivera, Diego (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Exploring Automated Scoring Using Topics and Predefined Features in Hybrid Attention-Based Model - Tang, Cheng; Engelhard, George; Xiong, Jiawei (2025)
Presenters: _:n14648 <标题>: 在混合注意力模型中使用主题和预定义特征进行自动评分的探索 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Evaluating the Content Quality of Items Written With and Without Artificial Intelligence - Tran, Peter; ., Ketan; Sireci, Stephen G. (2025)
Presenters: _:n14634 <标题>: 评估使用人工智能与未使用人工智能撰写的内容质量 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Automated Scoring in TIMSS and PIRLS Using Large Language Models - Von Davier, Matthias (2025)
Presenters: _:n14806 <标题>: 在 TIMSS 和 PIRLS 中使用大语言模型进行自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

Evaluating Large Language Models in Automated Online Comment Review - Wang, Ting; Du, Ying; Hoeve, Karen (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Evaluating Large Language Models in Automated Online Comment Review - Wang, Ting; Du, Ying; Hoeve, Karen (2025)
Presenters: _:n14689 <标题>: 在自动化在线评论审查中评估大型语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

AI-generated Classification Rubrics in the Process of Item Validation or Computer, is this a good test item? - Wang, Yu; Gopalakrishnan, Madhu; Bergner, Yoav (2025)
Presenters: _:n14903 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Using Generated Rubrics to Provide a Window into Item Evaluation with Multi-Agent LLMs - Wang, Yu; Gopalakrishnan, Madhumitha; Bergner, Yoav (2025)
<标题>: 使用生成的评分量表提供多智能体大语言模型项目评估的视角 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

A Framework to Guide the Development and Evaluation of AI-based Formative Assessment Systems - Wilson, Joshua; Palermo, Corey; Deane, Paul; Ormerod, Christopher; Shermis, Mark D. (2025)
Presenters: _:n14895 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Applying Cross-Prompt (Generic) Automated Scoring Models to a Large-Scale Writing Assessment - Wood, Scott W. (2025)
Presenters: _:n14866 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z

Applying Cross-Prompt (Generic) Automated Scoring Models to a Large-Scale Writing Assessment - Wood, Scott W. (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Leveraging Large Language Models with Auxiliary Information to Improve Automated Scoring of Science Items - Xue, Mingfeng; Wu, Yi-Fang (2025)
<标题>: 利用大型语言模型和辅助信息改进科学题目的自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

A Framework for Evaluating Automated Conventions Feedback - Young, Mackenzie; Burkhardt, Amy; Wilson, Quinell; Lottridge, Susan (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Assessing the Accuracy of Automated Writing Evaluation in Predicting ELA Proficiency for Middle-grade ELL and non-ELL Students - Zhang, Fan; Wilson, Joshua (2025)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Instruction‐Tuned Large‐Language Models for Quality Control in Automatic Item Generation: A Feasibility Study - Gorgun, Guher; Bulut, Okan (Educational Measurement: Issues and Practice, 2024)
Automatic item generation may supply many items instantly and efficiently to assessment and learning environments. Yet, the evaluation of item quality persists to be a bottleneck for deploying generated items in learning and assessment settings. In this study, we investigated the utility of using large‐language models, specifically Llama 3‐8B, for evaluating automatically generated cloze items. The trained large‐language model was able to filter out majority of good and bad items accurately. Evaluating items automatically with instruction‐tuned LLMs may aid educators and test developers in understanding the quality of items created in an efficient and scalable manner. The item evaluation process with LLMs may also act as an intermediate step between item creation and field testing to reduce the cost and time associated with multiple rounds of revision.
<标题>: 用于自动题目生成质量控制的指令调优大型语言模型：可行性研究 <AI Smry>: The utility of using large‐language models, specifically Llama 3‐8B, for evaluating automatically generated cloze items was investigated and the trained large‐language model was able to filter out majority of good and bad items accurately. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z

Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance - Fan, Yizhou; Tang, Luzhen; Le, Huixiao; Shen, Kejie; Tan, Shufang; Zhao, Yueying; Shen, Yuan; Li, Xinyu; Gašević, Dragan (British Journal of Educational Technology, 2024)
With the continuous development of technological and educational innovation, learners nowadays can obtain a variety of supports from agents such as teachers, peers, education technologies, and recently, generative artificial intelligence such as ChatGPT. In particular, there has been a surge of academic interest in human‐AI collaboration and hybrid intelligence in learning. The concept of hybrid intelligence is still at a nascent stage, and how learners can benefit from a symbiotic relationship with various agents such as AI, human experts and intelligent learning systems is still unknown. The emerging concept of hybrid intelligence also lacks deep insights and understanding of the mechanisms and consequences of hybrid human‐AI learning based on strong empirical research. In order to address this gap, we conducted a randomised experimental study and compared learners' motivations, self‐regulated learning processes and learning performances on a writing task among different groups who had support from different agents, that is, ChatGPT (also referred to as the AI group), chat with a human expert, writing analytics tools, and no extra tool. A total of 117 university students were recruited, and their multi‐channel learning, performance and motivation data were collected and analysed. The results revealed that: (1) learners who received different learning support showed no difference in post‐task intrinsic motivation; (2) there were significant differences in the frequency and sequences of the self‐regulated learning processes among groups; (3) ChatGPT group outperformed in the essay score improvement but their knowledge gain and transfer were not significantly different. Our research found that in the absence of differences in motivation, learners with different supports still exhibited different self‐regulated learning processes, ultimately leading to differentiated performance. What is particularly noteworthy is that AI technologies such as ChatGPT may promote learners' dependence on technology and potentially trigger “metacognitive laziness”. In conclusion, understanding and leveraging the respective strengths and weaknesses of different agents in learning is critical in the field of future hybrid intelligence.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/bjet.13544 <AI Smry>: A randomised experimental study compared learners' motivations, self‐regulated learning processes and learning performances among different agent groups (AI, human expert and checklist tools) and found that AI technologies such as ChatGPT may promote learners' dependence on technology and potentially trigger metacognitive "laziness". <标题>: 警惕元认知惰性：生成式人工智能对学习动机、过程和表现的影响 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: ChatGPT; experimental study; generative AI; hybrid intelligence; learning analytics

Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise - Wang, Rose E; Ribeiro, Ana T; Robinson, Carly D; Loeb, Susanna; Demszky, Dora (2024)
<标题>: 导师协助智能助手：一种用于扩展实时专业知识的人机结合方法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: empirical; fairness; math; tutoring; validity

A generative AI-driven interactive listening assessment task - Runge, Andrew; Attali, Yigal; LaFlair, Geoffrey T.; Park, Yena; Church, Jacqueline (Frontiers in Artificial Intelligence, 2024)
Introduction Assessments of interactional competence have traditionally been limited in large-scale language assessments. The listening portion suffers from construct underrepresentation, whereas the speaking portion suffers from limited task formats such as in-person interviews or role plays. Human-delivered tasks are challenging to administer at large scales, while automated assessments are typically very narrow in their assessment of the construct because they have carried over the limitations of traditional paper-based tasks to digital formats. However, computer-based assessments do allow for more interactive, automatically administered tasks, but come with increased complexity in task creation. Large language models present new opportunities for enhanced automated item generation (AIG) processes that can create complex content types and tasks at scale that support richer assessments. Methods This paper describes the use of such methods to generate content at scale for an interactive listening measure of interactional competence for the Duolingo English Test (DET), a large-scale, high-stakes test of English proficiency. The Interactive Listening task assesses test takers’ ability to participate in a full conversation, resulting in a more authentic assessment of interactive listening ability than prior automated assessments by positing comprehension and interaction as purposes of listening. Results and discussion The results of a pilot of 713 tasks with hundreds of responses per task, along with the results of human review, demonstrate the feasibility of a human-in-the-loop, generative AI-driven approach for automatic creation of complex educational assessments at scale.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Whose ChatGPT? Unveiling Real-World Educational Inequalities Introduced by Large Language Models - Yu, Renzhe; Xu, Zhen; CH-Wang, Sky; Arum, Richard (arXiv, 2024)
The universal availability of ChatGPT and other similar tools since late 2022 has prompted tremendous public excitement and experimental effort about the potential of large language models (LLMs) to improve learning experience and outcomes, especially for learners from disadvantaged backgrounds. However, little research has systematically examined the real-world impacts of LLM availability on educational equity beyond theoretical projections and controlled studies of innovative LLM applications. To depict trends of post-LLM inequalities, we analyze 1,140,328 academic writing submissions from 16,791 college students across 2,391 courses between 2021 and 2024 at a public, minority-serving institution in the US. We find that students' overall writing quality gradually increased following the availability of LLMs and that the writing quality gaps between linguistically advantaged and disadvantaged students became increasingly narrower. However, this equitizing effect was more concentrated on students with higher socioeconomic status. These findings shed light on the digital divides in the era of LLMs and raise questions about the equity benefits of LLMs in early stages and highlight the need for researchers and practitioners on developing responsible practices to improve educational equity through LLMs.
arXiv:2410.22282 [cs] <标题>: 谁的ChatGPT？揭示大型语言模型引入的现实教育不平等 Read_Status: New Read_Status_Date: 2026-01-26T11:32:23.432Z
Keywords: Computer Science - Computers and Society

Pre-Trained Language Models for Text Generation: A Survey - Li, Junyi; Tang, Tianyi; Zhao, Wayne Xin; Nie, Jian-Yun; Wen, Ji-Rong (ACM Computing Surveys, 2024)
Text Generation aims to produce plausible and readable text in human language from input data. The resurgence of deep learning has greatly advanced this field, in particular, with the help of neural generation models based on pre-trained language models (PLMs). Text generation based on PLMs is viewed as a promising approach in both academia and industry. In this article, we provide a survey on the utilization of PLMs in text generation. We begin with introducing two key aspects of applying PLMs to text generation: (1) how to design an effective PLM to serve as the generation model; and (2) how to effectively optimize PLMs given the reference text and to ensure that the generated texts satisfy special text properties. Then, we show the major challenges that have arisen in these aspects, as well as possible solutions for them. We also include a summary of various useful resources and typical text generation applications based on PLMs. Finally, we highlight the future research directions which will further improve these PLMs for text generation. This comprehensive survey is intended to help researchers interested in text generation problems to learn the core concepts, the main techniques and the latest developments in this area based on PLMs.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Foreword by Dragan Gašević (2024)
<标题>: 序言作者：Dragan Gašević Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

The Write & Improve Corpus 2024: Error-annotated and CEFR-labelled essays by learners of English - Nicholls, Diane; Caines, Andrew; Buttery, Paula (Apollo - University of Cambridge Repository, 2024)
We present a new annotated corpus of written learner English, derived from essays submitted to the learning platform Write & Improve (W&I). Users of W&I are presented with automated scoring and feedback on grammatical errors, and are encouraged to act on their error feedback, submitting multiple versions of their essays for any given prompt. We build the corpus on this interplay between users and prompts, collecting sets of essays submitted by users for a selected list of 50 popular prompts. The prompts include 20 aimed at beginner learners of English, 20 aimed at intermediate learners, and 10 at advanced learners. This distribution reflects the greater use of W&I by beginner and intermediate learners of English. We ensured that the prompts were not likely to elicit personal information and covered a broad range of tasks and topics. This list of prompts enabled us to identify 5050 essay sets written by 766 users, forming the basis for the Write & Improve Corpus, which is being made available for non-commercial use on the ELiT website. We describe the steps we took to ensure the corpus contains appropriate texts, does not include personal information, and will come with annotations relating to Common European Framework of Reference (CEFR) level and grammatical errors. All essays were submitted between 2020 and 2022 by registered users of W&I who have supplied their first language (L1) in an optional questionnaire. In total, there are more than 23K essays containing more than 3.5 million word tokens. The final versions of each essay set amount to 762K word tokens. There are 22 different L1s in the corpus, with the most common being Spanish, Portuguese, Japanese, Arabic and Vietnamese. We present some descriptive statistics for the corpus, and consider some research use cases.
<标题>: 《写作与改进语料库 2024》：由英语学习者撰写的带错误标注和 CEFR 等级标注的作文 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: corpus linguistics; education technology; language learning; machine learning; natural language processing

ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review - Kıyak, Yavuz Selim; Emekli, Emre (Postgraduate Medical Journal, 2024)
Abstract ChatGPT’s role in creating multiple-choice questions (MCQs) is growing but the validity of these artificial-intelligence-generated questions is unclear. This literature review was conducted to address the urgent need for understanding the application of ChatGPT in generating MCQs for medical education. Following the database search and screening of 1920 studies, we found 23 relevant studies. We extracted the prompts for MCQ generation and assessed the validity evidence of MCQs. The findings showed that prompts varied, including referencing specific exam styles and adopting specific personas, which align with recommended prompt engineering tactics. The validity evidence covered various domains, showing mixed accuracy rates, with some studies indicating comparable quality to human-written questions, and others highlighting differences in difficulty and discrimination levels, alongside a significant reduction in question creation time. Despite its efficiency, we highlight the necessity of careful review and suggest a need for further research to optimize the use of ChatGPT in question generation. Main messages Ensure high-quality outputs by utilizing well-designed prompts; medical educators should prioritize the use of detailed, clear ChatGPT prompts when generating MCQs. Avoid using ChatGPT-generated MCQs directly in examinations without thorough review to prevent inaccuracies and ensure relevance. Leverage ChatGPT’s potential to streamline the test development process, enhancing efficiency without compromising quality.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Opportunities and challenges of using generative AI to personalize educational assessment - Arslan, Burcu; Lehman, Blair; Tenison, Caitlin; Sparks, Jesse R.; López, Alexis A.; Gu, Lin; Zapata-Rivera, Diego (Frontiers in Artificial Intelligence, 2024)
In line with the positive effects of personalized learning, personalized assessments are expected to maximize learner motivation and engagement, allowing learners to show what they truly know and can do. Considering the advances in Generative Artificial Intelligence (GenAI), in this perspective article, we elaborate on the opportunities of integrating GenAI into personalized educational assessments to maximize learner engagement, performance, and access. We also draw attention to the challenges of integrating GenAI into personalized educational assessments regarding its potential risks to the assessment’s core values of validity, reliability, and fairness. Finally, we discuss possible solutions and future directions.
<标题>: 使用生成式人工智能个性化教育评估的机遇与挑战 <AI Smry>: The opportunities and challenges of integrating GenAI into personalized educational assessments regarding its potential risks to the assessment’s core values of validity, reliability, and fairness are elaborated. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Field-Testing Multiple-Choice Questions With AI Examinees: English Grammar Items - Maeda, Hotaka (Educational and Psychological Measurement, 2024)
Field-testing is an essential yet often resource-intensive step in the development of high-quality educational assessments. I introduce an innovative method for field-testing newly written exam items by substituting human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability θ, and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was .82 (bias = 0.00, root mean squared error = 0.18). The study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach did not achieve the level of accuracy obtainable with human examinee response data. If further refined, potential resource savings in transitioning from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low-quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank. Example Python code from this study is available on Github: https://github.com/hotakamaeda/ai_field_testing1
<标题>: 使用人工智能考生进行选择题实地测试：英语语法题目 <AI Smry>: An innovative method for field-testing newly written exam items by substituting human examinees with artificially intelligent (AI) examinees is introduced, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z

Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise - Wang, Rose E.; Ribeiro, Ana T.; Robinson, Carly D.; Loeb, Susanna; Demszky, Dora (arXiv, 2024)
Generative AI, particularly Language Models (LMs), has the potential to transform real-world domains with societal impact, particularly where access to experts is limited. For example, in education, training novice educators with expert guidance is important for effectiveness but expensive, creating significant barriers to improving education quality at scale. This challenge disproportionately harms students from under-served communities, who stand to gain the most from high-quality education. We introduce Tutor CoPilot, a novel Human-AI approach that leverages a model of expert thinking to provide expert-like guidance to tutors as they tutor. This study is the first randomized controlled trial of a Human-AI system in live tutoring, involving 900 tutors and 1,800 K-12 students from historically under-served communities. Following a preregistered analysis plan, we find that students working with tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more likely to master topics (p<0.01). Notably, students of lower-rated tutors experienced the greatest benefit, improving mastery by 9 p.p. We find that Tutor CoPilot costs only $20 per-tutor annually. We analyze 550,000+ messages using classifiers to identify pedagogical strategies, and find that tutors with access to Tutor CoPilot are more likely to use high-quality strategies to foster student understanding (e.g., asking guiding questions) and less likely to give away the answer to the student. Tutor interviews highlight how Tutor CoPilot's guidance helps tutors to respond to student needs, though they flag issues in Tutor CoPilot, such as generating suggestions that are not grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates how Human-AI systems can scale expertise in real-world domains, bridge gaps in skills and create a future where high-quality education is accessible to all students.
arXiv:2410.03017 [cs] <标题>: 导师协助智能助手：一种用于扩展实时专业知识的人机结合方法 <AI Smry>: This study is the first randomized controlled trial of a Human-AI system in live tutoring, involving 900 tutors and 1,800 K-12 students from historically under-served communities, and demonstrates how Human-AI systems can scale expertise in real-world domains, bridge gaps in skills and create a future where high-quality education is accessible to all students. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.338Z
Keywords: Computer Science - Computation and Language

The influence of ChatGPT on student engagement: A systematic review and future research agenda - Lo, Chung Kwan; Hew, Khe Foon; Jong, Morris Siu-yung (Computers & Education, 2024)
ChatGPT, a state-of-the-art artificial intelligence (AI) chatbot, has gained considerable attention as a transformative yet controversial tool for enhancing teaching and learning experiences. Several reviews and numerous articles have been written about harnessing ChatGPT in education since its release on November 30, 2022. Besides summarising its strengths, weaknesses, opportunities, and threats (SWOT) as identified in previous systematic reviews of ChatGPT research, this systematic review aims to develop a new understanding of its influence on student engagement by synthesising the existing related research using a three-dimensional framework comprising behavioural, emotional, and cognitive aspects. We searched relevant databases and included 72 empirical studies published within one year of ChatGPT's initial release. The findings reveal robust but narrowly focused evidence related to behavioural engagement (i.e., work with ChatGPT) and disengagement (i.e., academic dishonesty). The evidence related to the emotional aspect is mixed, with instances of both engagement (e.g., satisfaction and interest/fun) and disengagement (e.g., disappointment and worry/anxiety). There is broad but weak evidence regarding cognitive engagement (e.g., increased understanding and positive self-perception) and disengagement (e.g., reduced critical thinking and overreliance). Our review uncovers several under-explored indicators of student engagement, pointing to the need for further research. Specifically, future studies could focus on students' study habits and attendance (behavioural engagement), social interaction (emotional engagement), and self-regulation and critical thinking (cognitive engagement) in ChatGPT-supported learning environments.
<标题>: ChatGPT对学生参与度的影响：系统综述与未来研究议程 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z
Keywords: AI: ChatGPT; Educational Area: Student Learning; Evidence: Measured Student Behaviour; Evidence: Self-reported Student Behaviour/ Attitude; Location: Asia; Research Design: Systematic Review; Research Method: Qualitative; Source: Open Source

Detecting and assessing AI-generated and human-produced texts: The case of second language writing teachers - Nguyen, Loc; Barrot, Jessie S. (Assessing Writing, 2024)
<标题>: 检测和评估人工智能生成与人类创作的文本：以第二语言写作教师为例 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

AI hallucination: towards a comprehensive classification of distorted information in artificial intelligence-generated content - Sun, Yujie; Sheng, Dongfang; Zhou, Zihan; Wu, Yifei (Humanities and Social Sciences Communications, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Generative Psychometrics via AI-GENIE: Automatic Item Generation and Validation via Network-Integrated Evaluation - Russell-Lasalandra, Lara Lee; Christensen, Alexander P.; Golino, Hudson (PsyArXiv, 2024)
The rapid advancement of artificial intelligence (AI), particularly large language models (LLMs), has introduced powerful tools for various research domains, including psychological scale development. This study presents a fully automated method to efficiently generate and select high-quality, non-redundant items for psychological assessments using LLMs and network psychometrics. Our approach called, Automatic Item Generation and Validation via Network-Integrated Evaluation (AI-GENIE), reduces reliance on expert intervention by integrating generative AI with the latest network psychometric techniques. The efficacy of AI-GENIE was evaluated through Monte Carlo simulations using the Mixtral, Gemma 2, Llama 3, GPT 3.5, and GPT 4o models to generate item pools that mimic Big Five personality assessment. The results demonstrated improvement in item selection efficiency, with overall average increases of 9.78-17.80 in normalized mutual information in the final item pool across all models. After, each model in AI-GENIE generated a Big Five inventory that was administered to independent, representative samples (N = 1000 each) in the U.S. The empirical results show that the items produced across all models were diverse, theoretically consistent, and structurally stable. Taken together, these findings demonstrate that AI-GENIE is a highly effective tool to automate and streamline scale development and validation process.
<标题>: 通过 AI-GENIE 的生成心理测量学：基于网络集成评估的自动题目生成与验证 Read_Status: New Read_Status_Date: 2026-01-26T11:32:30.842Z

Evaluation frameworks | Ai2 (2024)
A collection of Ai2’s evaluation frameworks and benchmarks, open and accessible to compare like-for-like outcomes.
<标题>: 评估框架 | Ai2 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Research on public attitudes towards the use of AI in education (2024)
<标题>: 关于公众对在教育中使用人工智能的态度的研究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Teachers to get more trustworthy AI tech as generative tools learn from new bank of lesson plans and curriculums, helping them mark homework and save time (2024)
The UK government announced a new project today that will enhance AI's ability to assist teachers in marking work and planning lessons.
<标题>: 随着生成式工具从新的课程和教案库中学习，教师将获得更可信赖的人工智能技术，帮助他们批改作业并节省时间 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Responsible AI for Test Equity and Quality: The Duolingo English Test as a Case Study - Burstein, Jill; LaFlair, Geoffrey T.; Yancey, Kevin; von Davier, Alina A.; Dotan, Ravit (2024)
Artificial intelligence (AI) creates opportunities for assessments, such as efficiencies for item generation and scoring of spoken and written responses. At the same time, it poses risks (such as bias in AI-generated item content). Responsible AI (RAI) practices aim to mitigate risks associated with AI. This chapter addresses the critical role of RAI practices in achieving test quality (appropriateness of test score inferences), and test equity (fairness to all test takers). To illustrate, the chapter presents a case study using the Duolingo English Test (DET), an AI-powered, high-stakes English language assessment. The chapter discusses the DET RAI standards, their development and their relationship to domain-agnostic RAI principles. Further, it provides examples of specific RAI practices, showing how these practices meaningfully address the ethical principles of validity and reliability, fairness, privacy and security, and transparency and accountability standards to ensure test equity and quality.
arXiv:2409.07476 [cs] <标题>: 负责任的人工智能在测试公平性与质量中的应用：以多邻国英语测试为案例研究 <AI Smry>: The chapter provides examples of specific RAI practices, showing how these practices meaningfully address the ethical principles of validity and reliability, fairness, privacy and security, and transparency and accountability standards to ensure test equity and quality. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.338Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Computers and Society

The Llama 3 Herd of Models - Dubey, Abhimanyu; Jauhri, Abhinav; Pandey, Abhinav; Kadian, Abhishek; Al-Dahle, Ahmad; Letman, Aiesha; Mathur, Akhil; Schelten, Alan; Yang, Amy; Fan, Angela; Goyal, Anirudh; Hartshorn, Anthony; et al. (arXiv, 2024)
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.
arXiv:2407.21783 [cs] <AI Smry>: It is found that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks, and performs competitively with the state-of-the-art on image, video, and speech recognition tasks. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Computer Vision and Pattern Recognition

Towards Bidirectional Human-AI Alignment: A Systematic Review for Clarifications, Framework, and Future Directions - Shen, Hua; Knearem, Tiffany; Ghosh, Reshmi; Alkiek, Kenan; Krishna, Kundan; Liu, Yachuan; Ma, Ziqiao; Petridis, Savvas; Peng, Yi-Hao; Qiwei, Li; Rakshit, Sushrita; Si, Chenglei; et al. (arXiv, 2024)
Recent advancements in general-purpose AI have highlighted the importance of guiding AI systems towards the intended goals, ethical principles, and values of individuals and groups, a concept broadly recognized as alignment. However, the lack of clarified definitions and scopes of human-AI alignment poses a significant obstacle, hampering collaborative efforts across research domains to achieve this alignment. In particular, ML- and philosophy-oriented alignment research often views AI alignment as a static, unidirectional process (i.e., aiming to ensure that AI systems' objectives match humans) rather than an ongoing, mutual alignment problem. This perspective largely neglects the long-term interaction and dynamic changes of alignment. To understand these gaps, we introduce a systematic review of over 400 papers published between 2019 and January 2024, spanning multiple domains such as Human-Computer Interaction (HCI), Natural Language Processing (NLP), Machine Learning (ML). We characterize, define and scope human-AI alignment. From this, we present a conceptual framework of "Bidirectional Human-AI Alignment" to organize the literature from a human-centered perspective. This framework encompasses both 1) conventional studies of aligning AI to humans that ensures AI produces the intended outcomes determined by humans, and 2) a proposed concept of aligning humans to AI, which aims to help individuals and society adjust to AI advancements both cognitively and behaviorally. Additionally, we articulate the key findings derived from literature analysis, including literature gaps and trends, human values, and interaction techniques. To pave the way for future studies, we envision three key challenges and give recommendations for future research.
arXiv:2406.09264 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Human-Computer Interaction

Why does AI being good at math matter? | MIT Technology Review (2024)
<标题>: 人工智能擅长数学为什么重要？ | 麻省理工科技评论 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.518Z
Keywords: /unread

AI Chatbots in Classrooms: Promoting Grit Characteristics in Students - Akeyama, Hinako; Yang, Alan T; Yazdi, Amirhessam (2024)
This paper explores the role of Artificial Intelligence in education (AIEd) with an emphasis on the use of chatbots to promote students’ learning outcomes and grit characteristics. This paper proposes a framework derived from the deliberate practice and technology acceptance model (TAM). Understanding the importance of teachers’ perspectives and support of students, this framework presents a means to empower students’ learning in a non-threatening yet productive manner while captivating the grit personality which is predicted to be the key component in long-term success. Additionally, the pilot study proposes a future research design that can analyze the effectiveness of the presented framework. By addressing these challenges, this paper contributes to the ongoing discourse on the effective implementation of AIEd in the educational sphere.
<标题>: 课堂中的人工智能聊天机器人：促进学生坚毅品质的发展 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z
Keywords: AI: ChatGPT; AI: Chatbot; Educational Area: Student Learning; Evidence: Self-reported Student Behaviour/ Attitude; Evidence: Self-reported Teacher Behaviour/ Attitude; Location: Americas; Research Design: Other Research Design; Research Method: Qualitative; Source: Open Source

Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification - Liu (2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z

The Impact of AI on Computer Science Education – Communications of the ACM (2024)
<标题>: 人工智能对计算机科学教育的影响——《美国计算机协会通讯》 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Field-testing multiple-choice questions with AI examinees: English grammar items - Maeda, Hotaka (2024)
Abstract Field-testing is a necessary but resource-intensive step in the development of high-quality educational assessments. I present an innovative method for field-testing newly written exam items by replacing human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability θ , and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was .82 (bias = 0.00, root-mean-squared-error = 0.18). The simulation study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach still fell short of the accuracy of analyses that can be achieved with human examinee response data. If further refined, potential resource savings in transition from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank. Example Python code from this study is available on Github: https://github.com/hotakamaeda/ai_field_testing1
Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z

FlexEval: a customizable tool for chatbot performance evaluation and dialogue analysis - Christie, S; Moreau-Pernet, Baptiste; Tian, Yu; Whitmer, John (2024)
Large language models (LLMs) are increasingly being deployed in user-facing applications in educational settings. Deployed applications often augment LLMs with fine-tuning, custom system prompts, and moderation layers to achieve particular goals. However, the behaviors of LLM-powered systems are difficult to guarantee, and most existing evaluations focus instead on the performance of unmodified 'foun-dation' models. Tools for evaluating such deployed systems are currently sparse, inflexible, or difficult to use. In this paper , we introduce an open-source tool called FlexEval. Flex-Eval extends OpenAI Evals to allow developers to construct customized, comprehensive automated evaluations of both pre-production and live conversational systems. FlexEval runs locally and can be easily modified to meet the needs of application developers. Developers can evaluate new LLM applications by creating function-based or machine-graded metrics and obtaining results for chat completions or entire conversations. To illustrate FlexEval's utility, we share two use-cases involving content moderation and utterance classification. We built FlexEval to lower the effort required to implement automated testing and evaluation of LLM applications. The code is available on GitHub 1 .
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

A.I. Can Write Poetry, but It Struggles With Math - Lohr, Steve (The New York Times, 2024)
A.I.’s math problem reflects how much the new technology is a break with computing’s past.
<标题>: 人工智能可以写诗，但在数学方面却很吃力 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z
Keywords: /unread; Artificial Intelligence; Calculators; ChatGPT; Computers and the Internet; Education (K-12); Khan Academy; Mathematics; OpenAI Labs

SyllabusQA: A Course Logistics Question Answering Dataset - Fernandez, Nigel; Scarlatos, Alexander; Lan, Andrew (arXiv, 2024)
Automated teaching assistants and chatbots have significant potential to reduce the workload of human instructors, especially for logistics-related question answering, which is important to students yet repetitive for instructors. However, due to privacy concerns, there is a lack of publicly available datasets. We introduce SyllabusQA, an open-source dataset with 63 real course syllabi covering 36 majors, containing 5,078 open-ended course logistics-related question-answer pairs that are diverse in both question types and answer formats. Since many logistics-related questions contain critical information like the date of an exam, it is important to evaluate the factuality of answers. We benchmark several strong baselines on this task, from large language model prompting to retrieval-augmented generation. We introduce Fact-QA, an LLM-based (GPT-4) evaluation metric to evaluate the factuality of predicted answers. We find that despite performing close to humans on traditional metrics of textual similarity, there remains a significant gap between automated approaches and humans in terms of fact precision.
arXiv:2403.14666 [cs] <标题>: SyllabusQA: 一份课程后勤问题回答数据集 <AI Smry>: SyllabusQA, an open-source dataset with 63 real course syllabi covering 36 majors, containing 5,078 open-ended course logistics-related question-answer pairs that are diverse in both question types and answer formats is introduced. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language; Computer Science - Computers and Society; Computer Science - Information Retrieval; Computer Science - Machine Learning

Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models - He-Yueya, Joy; Ma, Wanjing Anya; Gandhi, Kanishk; Domingue, Benjamin W.; Brunskill, Emma; Goodman, Noah D. (arXiv, 2024)
Language models (LMs) are increasingly used to simulate human-like responses in scenarios where accurately mimicking a population's behavior can guide decision-making, such as in developing educational materials and designing public policies. The objective of these simulations is for LMs to capture the variations in human responses, rather than merely providing the expected correct answers. Prior work has shown that LMs often generate unrealistically accurate responses, but there are no established metrics to quantify how closely the knowledge distribution of LMs aligns with that of humans. To address this, we introduce "psychometric alignment," a metric that measures the extent to which LMs reflect human knowledge distribution. Assessing this alignment involves collecting responses from both LMs and humans to the same set of test items and using Item Response Theory to analyze the differences in item functioning between the groups. We demonstrate that our metric can capture important variations in populations that traditional metrics, like differences in accuracy, fail to capture. We apply this metric to assess existing LMs for their alignment with human knowledge distributions across three real-world domains. We find significant misalignment between LMs and human populations, though using persona-based prompts can improve alignment. Interestingly, smaller LMs tend to achieve greater psychometric alignment than larger LMs. Further, training LMs on human response data from the target distribution enhances their psychometric alignment on unseen test items, but the effectiveness of such training varies across domains.
arXiv:2407.15645 [cs] <标题>: 心理测量对齐：通过语言模型捕捉人类知识分布 <AI Smry>: This work introduces "psychometric alignment," a metric that measures the extent to which LMs reflect human knowledge distribution, and applies this metric to assess existing LMs for their alignment with human knowledge distributions across three real-world domains. Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Automated Scoring of Constructed Response Items in Math Assessment Using Large Language Models - Morris, Wesley; Holmes, Langdon; Choi, Joon Suh; Crossley, Scott (International Journal of Artificial Intelligence in Education, 2024)
Recent developments in the field of artificial intelligence allow for improved performance in the automated assessment of extended response items in mathematics, potentially allowing for the scoring of these items cheaply and at scale. This study details the grand prize-winning approach to developing large language models (LLMs) to automatically score the ten items in the National Assessment of Educational Progress (NAEP) Math Scoring Challenge. The approach uses extensive preprocessing that balanced the class labels for each item. This was done by identifying and filtering over-represented classes using a classifier trained on document-term matrices and data augmentation of under-represented classes using a generative pre-trained large language model (Grammarly’s Coedit-XL; Raheja et al., 2023). We also use input modification schemes that were hand-crafted to each item type and included information from parts of the multi-step math problem students had to solve. Finally, we finetune several pre-trained large language models on the modified input for each individual item in the NAEP automated math scoring challenge, with DeBERTa (He et al., 2021a) showing the best performance. This approach achieved human-like agreement (less than QWK 0.05 difference from human–human agreement) on nine out of the ten items in a held-out test set.
<标题>: 使用大型语言模型对数学评估中的构建式回答题进行自动评分 <AI Smry>: This study details the grand prize-winning approach to developing large language models (LLMs) to automatically score the ten items in the National Assessment of Educational Progress (NAEP) Math Scoring Challenge, with DeBERTa (He et al., 2021a) showing the best performance. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z

Automated Assessment in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses - Sami Baral; Eamon Worden; Wen-Chiang Lim; Zhuang Luo; Christopher Santorelli; Ashish Gurung (International Educational Data Mining Society, 2024)
The effectiveness of feedback in enhancing learning outcomes is well documented within Educational Data Mining (EDM). Various prior research have explored methodologies to enhance the effectiveness of feedback to students in various ways. Recent developments in Large Language Models (LLMs) have extended their utility in enhancing automated feedback systems. This study aims to explore the potential of LLMs in facilitating automated feedback in math education in the form of numeric assessment scores. We examine the effectiveness of LLMs in evaluating student responses and scoring the responses by comparing 3 different models: Llama, SBERT-Canberra, and GPT4 model. The evaluation requires the model to provide a quantitative score on the student's responses to open-ended math problems. We employ Mistral, a version of Llama catered to math, and fine-tune this model for evaluating student responses by leveraging a dataset of student responses and teacher-provided scores for middle-school math problems. A similar approach was taken for training the SBERT-Canberra model, while the GPT4 model used a zero-shot learning approach. We evaluate and compare the models' performance in scoring accuracy. This study aims to further the ongoing development of automated assessment and feedback systems and outline potential future directions for leveraging generative LLMs in building automated feedback systems.
<标题>: 数学教育中的自动化评估：针对开放式回答的大型语言模型比较分析 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z

Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation - Lu, Xinyi; Wang, Xu (Proceedings of the Eleventh ACM Conference on Learning @ Scale, 2024)
Evaluating the quality of automatically generated question items has been a long standing challenge. In this paper, we leverage LLMs to simulate student profiles and generate responses to multiple-choice questions (MCQs). The generative students' responses to MCQs can further support question item evaluation. We propose Generative Students, a prompt architecture designed based on the KLI framework. A generative student profile is a function of the list of knowledge components the student has mastered, has confusion about or has no evidence of knowledge of. We instantiate the Generative Students concept on the subject domain of heuristic evaluation. We created 45 generative students using GPT-4 and had them respond to 20 MCQs. We found that the generative students produced logical and believable responses that were aligned with their profiles. We then compared the generative students' responses to real students' responses on the same set of MCQs and found a high correlation. Moreover, there was considerable overlap in the difficult questions identified by generative students and real students. A subsequent case study demonstrated that an instructor could improve question quality based on the signals provided by Generative Students.
arXiv:2405.11591 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Human-Computer Interaction

LLM Based Math Tutoring: Challenges and Dataset - Miller, Pepper; DiCerbo, Kristen (OSF, 2024)
Large Language Models (LLMs) face documented challenges in solving mathematical problems. While substantial work has been done to quantify and improve LLMs’ abilities to solve static math problems, evaluating their performance in real-time math tutoring scenarios presents distinct challenges that remain underexplored. This paper specifically addresses the accuracy of LLMs in performing math correctly while tutoring students. It highlights the unique difficulties of this context, classifies types of interactions students may have with an LLM, presents a dataset, Conversation-Based Math Tutoring Accuracy Dataset (CoMTA Dataset), for evaluating the mathematical accuracy in tutoring scenarios, and discusses techniques to address these issues. Additionally, it evaluates the mathematical accuracy of a range of models in LLM-based tutoring.
<标题>: 基于大型语言模型的数学辅导：挑战与数据集 Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.807Z
Keywords: AI tutors; GPT-4; Large language models; Math tutoring

Automated Text Scoring in the Age of Generative AI for the GPU-poor - Ormerod, Christopher Michael; Kwako, Alexander (arXiv, 2024)
Current research on generative language models (GLMs) for automated text scoring (ATS) has focused almost exclusively on querying proprietary models via Application Programming Interfaces (APIs). Yet such practices raise issues around transparency and security, and these methods offer little in the way of efficiency or customizability. With the recent proliferation of smaller, open-source models, there is the option to explore GLMs with computers equipped with modest, consumer-grade hardware, that is, for the "GPU poor." In this study, we analyze the performance and efficiency of open-source, small-scale GLMs for ATS. Results show that GLMs can be fine-tuned to achieve adequate, though not state-of-the-art, performance. In addition to ATS, we take small steps towards analyzing models' capacity for generating feedback by prompting GLMs to explain their scores. Model-generated feedback shows promise, but requires more rigorous evaluation focused on targeted use cases.
arXiv:2407.01873 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

EvaluMate: Using AI to support students’ feedback provision in peer assessment for writing - Guo, Kai (Assessing Writing, 2024)
Peer feedback plays an important role in promoting learning in the writing classroom. However, providing high-quality feedback can be demanding for student reviewers. To address this challenge, this article proposes an AI-enhanced approach to peer feedback provision. I introduce EvaluMate, a newly developed online peer review system that leverages ChatGPT, a large language model (LLM), to scaffold student reviewers’ feedback generation. I discuss the design and functionality of EvaluMate, highlighting its affordances in supporting student reviewers’ provision of comments on peers’ essays. I also address the system’s limitations and propose potential solutions. Furthermore, I recommend future research on students’ engagement with this learning approach and its impact on learning outcomes. By presenting EvaluMate, I aim to inspire researchers and practitioners to explore the potential of AI technology in the teaching, learning, and assessment of writing.
<标题>: EvaluMate：利用人工智能支持学生在写作同伴互评中的反馈提供 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: AI: ChatGPT; Educational Area: Assessment Tools & Processes; Evidence: Improve a process; Evidence: Saving Teacher Time; Location: Asia; Research Design: Case Study; Research Method: Qualitative; Source: Closed Source

Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments - Tang, Xiaoyi; Chen, Hongwei; Lin, Daoyu; Li, Kexin (Heliyon, 2024)
<标题>: 利用大型语言模型进行多维写作评估：与人类判断的一致性及可靠性 <AI Smry>: Results indicate that prompt engineering significantly affects the reliability of LLMs, with GPT-4 showing marked improvement over GPT-3.5 and Claude 2, achieving 112% and 114% increase in scoring accuracy under the criteria and sample-referenced justification prompt. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges - Bulut, Okan; Beiting-Parrish, Maggie; Casabianca, Jodi M.; Slater, Sharon C.; Jiao, Hong; Song, Dan; Ormerod, Christopher M.; Fabiyi, Deborah Gbemisola; Ivan, Rodica; Walsh, Cole; Rios, Oscar; Wilson, Joshua; et al. (2024)
The integration of artificial intelligence (AI) in educational measurement has revolutionized assessment methods, enabling automated scoring, rapid content analysis, and personalized feedback through machine learning and natural language processing. These advancements provide timely, consistent feedback and valuable insights into student performance, thereby enhancing the assessment experience. However, the deployment of AI in education also raises significant ethical concerns regarding validity, reliability, transparency, fairness, and equity. Issues such as algorithmic bias and the opacity of AI decision-making processes pose risks of perpetuating inequalities and affecting assessment outcomes. Responding to these concerns, various stakeholders, including educators, policymakers, and organizations, have developed guidelines to ensure ethical AI use in education. The National Council of Measurement in Education's Special Interest Group on AI in Measurement and Education (AIME) also focuses on establishing ethical standards and advancing research in this area. In this paper, a diverse group of AIME members examines the ethical implications of AI-powered tools in educational measurement, explores significant challenges such as automation bias and environmental impact, and proposes solutions to ensure AI's responsible and effective use in education.
arXiv:2406.18900 [cs] <AI Smry>: A diverse group of AIME members examines the ethical implications of AI-powered tools in educational measurement, explores significant challenges such as automation bias and environmental impact, and proposes solutions to ensure AI's responsible and effective use in education. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computers and Society

PlagBench: Exploring the Duality of Large Language Models in Plagiarism Generation and Detection - Lee, Jooyoung; Agrawal, Toshini; Uchendu, Adaku; Le, Thai; Chen, Jinghui; Lee, Dongwon (arXiv, 2024)
Recent literature has highlighted potential risks to academic integrity associated with large language models (LLMs), as they can memorize parts of training instances and reproduce them in the generated texts without proper attribution. In addition, given their capabilities in generating high-quality texts, plagiarists can exploit LLMs to generate realistic paraphrases or summaries indistinguishable from original work. In response to possible malicious use of LLMs in plagiarism, we introduce PlagBench, a comprehensive dataset consisting of 46.5K synthetic plagiarism cases generated using three instruction-tuned LLMs across three writing domains. The quality of PlagBench is ensured through fine-grained automatic evaluation for each type of plagiarism, complemented by human annotation. We then leverage our proposed dataset to evaluate the plagiarism detection performance of five modern LLMs and three specialized plagiarism checkers. Our findings reveal that GPT-3.5 tends to generates paraphrases and summaries of higher quality compared to Llama2 and GPT-4. Despite LLMs' weak performance in summary plagiarism identification, they can surpass current commercial plagiarism detectors. Overall, our results highlight the potential of LLMs to serve as robust plagiarism detection tools.
arXiv:2406.16288 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language

Toward Personalised Learning Experiences: Beyond Prompt Engineering - Kruis, Joost; Pera, Maria Soledad; Napel, Zoë Ten; Landoni, Monica; Murgia, Emiliana; Huibers, Theo; Feskens, Remco (Proceedings of the 23rd Annual ACM Interaction Design and Children Conference, 2024)
<标题>: 迈向个性化学习体验：超越提示工程 <AI Smry>: A framework would ensure that assessment item generation technology goes beyond the current one-dimensional approach by incorporating multifaceted, adaptable perspectives that consider the variegated landscape of learners’ needs, subject matter complexities, and pedagogical goals. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: Educational Area: Assessment Tools & Processes; Evidence: Improve a process; Location: Europe; Research Design: Qualitative Exploratory; Research Method: Qualitative; Source: Open Source

Assessing the relationship between anxiety and the adoption of Artificial Intelligence tools among mathematics preservice teachers - Falebita, Oluwanife Segun (Interdisciplinary Journal of Education Research, 2024)
Many revelations have been made about the revolution that artificial intelligence (AI) has brought to the education sector, including the opening of opportunities for personalised instruction, boosting the quality of content developed by teachers while preparing for lessons, and improving the quality of classroom evaluations. Despite the many benefits of AI adoption, there have been concerns and apprehensions about its use in the educational sector. A survey was conducted to investigate the relationship between AI anxiety and the adoption of artificial intelligence tools among mathematics preservice teachers who are university undergraduates studying mathematics education in Ekiti State, Nigeria. The study sample consisted of 129 mathematics preservice teachers selected through purposive sampling. The AI anxiety scale and AI adoption scale were used for data collection after being tested for reliability. The data collected through the scales were analysed using descriptive and inferential statistics. The findings of the study revealed that the mathematics preservice teachers had a high level of AI anxiety and adopted AI at a moderate level. The study further showed that there is a significant weak relationship between mathematics preservice teachers' AI-Anxiety and AI adoption. Also, there is no significant gender difference in mathematics preservice teachers' AI anxiety and AI adoption. Based on the findings of the study, it was recommended that teacher education programs include AI and digital literacy in the curriculum to prepare students for the seamless integration of AI. Additionally, targeted interventions should be implemented to reduce the anxiety exhibited by preservice teachers.
<标题>: 评估焦虑与数学预备教师采纳人工智能工具之间的关系 <AI Smry>: It was recommended that teacher education programs include AI and digital literacy in the curriculum to prepare students for the seamless integration of AI. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z
Keywords: Educational Area: Pre-service Teacher Education; Evidence: Self-reported Teacher Behaviour/ Attitude; Location: Africa; Research Design: Qualitative Exploratory; Research Method: Qualitative; Source: Open Source

Beyond Answers: Large Language Model-Powered Tutoring System in Physics Education for Deep Learning and Precise Understanding - Jiang, Zhoumingju; Jiang, Mengjun (arXiv, 2024)
The integration of artificial intelligence (AI) in education has shown significant promise, yet the effective personalization of learning, particularly in physics education, remains a challenge. This paper proposes Physics-STAR, a framework for large language model (LLM)- powered tutoring system designed to address this gap by providing personalized and adaptive learning experiences for high school students. Our study evaluates Physics-STAR against traditional teacher-led lectures and generic LLM tutoring through a controlled experiment with 12 high school sophomores. Results showed that Physics-STAR increased students' average scores and efficiency on conceptual, computational, and on informational questions. In particular, students' average scores on complex information problems increased by 100% and their efficiency increased by 5.95%. By facilitating step-by-step guidance and reflective learning, Physics-STAR helps students develop critical thinking skills and a robust comprehension of abstract concepts. The findings underscore the potential of AI-driven personalized tutoring systems to transform physics education. As LLM continues to advance, the future of student-centered AI in education looks promising, with the potential to significantly improve learning outcomes and efficiency.
arXiv:2406.10934 [physics] <标题>: 超越答案：基于大型语言模型的物理教育辅导系统，实现深度学习与精准理解 <AI Smry>: By facilitating step-by-step guidance and reflective learning, Physics-STAR helps students develop critical thinking skills and a robust comprehension of abstract concepts, and underscores the potential of AI-driven personalized tutoring systems to transform physics education. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.338Z
Keywords: Educational Area: Assessment Tools & Processes; Evidence: Student Learning Outcomes; Location: Asia; Research Design: Quasi-experimental design; Research Method: Quantitative; Source: Open Source

Unleashing the transformers: NLP models detect AI writing in education - Campino, José (Journal of Computers in Education, 2024)
Artificial Intelligence (AI) has witnessed widespread application across diverse domains, with education being a prominent focus for enhancing learning outcomes and tailoring educational approaches. Transformer models, exemplified by BERT, have demonstrated remarkable efficacy in Natural Language Processing (NLP) tasks. This research scrutinizes the current landscape of AI in education, emphasizing the utilization of transformer models. Specifically, the research delves into the influence of AI tools facilitating text generation through input prompts, with a notable instance being the GPT-4 model developed by OpenAI. The study employs pre-trained transformer models to discern whether a given text originates from AI or human sources. Notably, BERT emerges as the most effective model, fine-tuned using a dataset comprising authored by humans and those generated by AI. The outcomes reveal a heightened accuracy in distinguishing AI-generated text. These findings bear significance for the educational realm, suggesting that while endorsing the use of such tools for learning, vigilance is warranted to identify potential misuse or instances where students should independently develop their reasoning skills. Nevertheless, ethical considerations must be paramount when employing such methodologies. We have highlighted vulnerabilities concerning the potential bias of AI models towards non-native English speakers, stemming from possible deficiencies in vocabulary and grammatical structure. Additionally, users must ensure that there is no complete reliance on these systems to ascertain students' performance. Further research is imperative to unleash the full potential of AI in education and address ethical considerations tied to its application.
<标题>: 释放变压器：NLP 模型在教育中检测 AI 写作 <AI Smry>: Vital vulnerabilities concerning the potential bias of AI models towards non-native English speakers are highlighted, stemming from possible deficiencies in vocabulary and grammatical structure, to unleash the full potential of AI in education and address ethical considerations tied to its application. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: AI: ChatGPT; Educational Area: Assessment Tools & Processes; Evidence: Improve a process; Location: Europe; Research Design: Other Research Design; Research Method: Quantitative; Source: Open Source

The Impact of Artificial Intelligence (AI) Technology on Students' Social Relations - Puteri, Septiani Amanda; Saputri, Yulia; Kurniati, Yarni (BICC Proceedings, 2024)
In an era where artificial intelligence (AI) technology is increasingly penetrating various aspects of life, including education, it is important to understand its impact on students' social relationships. These changes raise questions about how interactions between students and teachers are affected by AI Technology. This research aims to investigate the impact or artificial intelligence on students' social relationships in an educational environment. The focus is on understanding changes in interactions between students, student-teacher interaction, as well as broader implications for social dynamics inside the classroom and outside the classroom. The research method used is a literature review which uses data collections sources that are relevant to this research, which can be in books, megazines and other print media, or can be obtained from photos and videos. The research results show that artificial intelligence technology has a complex impact on students' social relationships. Although it can improve efficiency and learning outcomes, AI technology can also reduce social inequality, and create excessive dependence on technology. Educators and policymakers need to carefully consider the use of AI technology in education to ensure that positive impacts are maximized, while negative impacts are minimized.
<标题>: 人工智能（AI）技术对学生社交关系的影响 <AI Smry>: The research results show that artificial intelligence technology has a complex impact on students' social relationships, although it can improve efficiency and learning outcomes, AI technology can also reduce social inequality, and create excessive dependence on technology. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z
Keywords: Educational Area: Student Learning; Evidence: Self-reported Student Behaviour/ Attitude; Location: Asia; Research Design: Non-systematic Review; Research Method: Qualitative; Source: Open Source

Advancing the In-Class Dialogic Quality: Developing an Artificial Intelligence-Supported Framework for Classroom Dialogue Analysis - Li, Xian; Han, Guangxin; Fang, Bei; He, Juhou (The Asia-Pacific Education Researcher, 2024)
The development of artificial intelligence (AI) significantly improves the effectiveness of classroom dialogue systems, but their integration into the learning environment remains challenging. To address this gap, this research presents a framework for automatic intelligent dialogue analysis, intending to promote high-quality classroom dialogue and facilitate teaching and learning. The proposed framework includes two main components: a dialogue-oriented interactive classroom and an artificial intelligence-powered analysis system. We present a synthesis of essential principles that ought to be adhered to in the dialogue-oriented interactive classroom, as viewed through the lens of three key domains: the environment, the community and the teaching–learning. The AI system will analyse the dialogues generated from the interactive classroom. The utilization of feedback obtained from the AI system assists educators who adjust their pedagogical strategies, consequently improving the quality of classroom dialogues. Elevated-quality dialogues will reciprocally boost the performance of the AI system, engendering a sustainable improvement for the entire framework. Moreover, we also propose “Guide of AI”, a union of classroom participants and experts, which serves as the bridge between the classroom and technology to guide the operation of AI system. For the validation of the framework, we conduct an empirical study that mainly investigates the effectiveness of processed essential principles and AI systems. We select 6 pre-service teachers who are randomly divided into three groups. Three groups have different levels of involvement in AI system and each teacher gives three lessons. We record and analyse all teaching dialogue records and also use questionnaires to obtain teachers’ attitudes. The results show that timely feedback from AI system can promote the improvement of dialogue quality, which demonstrates the effectiveness of AI dialogue analysis system. In addition, the proposed essential principles also show a constructive impact.
<标题>: 提升课堂对话质量：构建支持人工智能的课堂对话分析框架 <AI Smry>: Results show that timely feedback from AI system can promote the improvement of dialogue quality, which demonstrates the effectiveness of AI dialogue analysis system and the proposed essential principles also show a constructive impact. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z
Keywords: Educational Area: Teach; Evidence: Improve a process; Location: Asia; Research Design: Interview/ focus group; Research Method: Qualitative; Source: Open Source

The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches - Abeysinghe, Bhashithe; Circi, Ruhan (arXiv, 2024)
Chatbots have been an interesting application of natural language generation since its inception. With novel transformer based Generative AI methods, building chatbots have become trivial. Chatbots which are targeted at specific domains such as medicine, psychology, and general information retrieval are implemented rapidly. This, however, should not distract from the need to evaluate the chatbot responses. Especially because the natural language generation community does not entirely agree upon how to effectively evaluate such applications. With this work we discuss the issue further with the increasingly popular LLM based evaluations and how they correlate with human evaluations. Additionally, we introduce a comprehensive factored evaluation mechanism that can be utilized in conjunction with both human and LLM-based evaluations. We present the results of an experimental evaluation conducted using this scheme in one of our chatbot implementations, and subsequently compare automated, traditional human evaluation, factored human evaluation, and factored LLM evaluation. Results show that factor based evaluation produces better insights on which aspects need to be improved in LLM applications and further strengthens the argument to use human evaluation in critical spaces where main functionality is not direct retrieval.
arXiv:2406.03339 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches - Abeysinghe, Bhashithe; Circi, Ruhan (arXiv, 2024)
Chatbots have been an interesting application of natural language generation since its inception. With novel transformer based Generative AI methods, building chatbots have become trivial. Chatbots which are targeted at specific domains such as medicine, psychology, and general information retrieval are implemented rapidly. This, however, should not distract from the need to evaluate the chatbot responses. Especially because the natural language generation community does not entirely agree upon how to effectively evaluate such applications. With this work we discuss the issue further with the increasingly popular LLM based evaluations and how they correlate with human evaluations. Additionally, we introduce a comprehensive factored evaluation mechanism that can be utilized in conjunction with both human and LLM-based evaluations. We present the results of an experimental evaluation conducted using this scheme in one of our chatbot implementations, and subsequently compare automated, traditional human evaluation, factored human evaluation, and factored LLM evaluation. Results show that factor based evaluation produces better insights on which aspects need to be improved in LLM applications and further strengthens the argument to use human evaluation in critical spaces where main functionality is not direct retrieval.
arXiv:2406.03339 [cs] <标题>: 评估大型语言模型应用的挑战：自动化、人类及大型语言模型方法的分析 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models - Adelani, David Ifeoluwa; Ojo, Jessica; Azime, Israel Abebe; Zhuang, Jian Yun; Alabi, Jesujoba O.; He, Xuanli; Ochieng, Millicent; Hooker, Sara; Bukula, Andiswa; Lee, En-Shiun Annie; Chukwuneke, Chiamaka; Buzaaba, Happy; et al. (arXiv, 2024)
Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (e.g. African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench – a human-translated benchmark dataset for 16 typologically-diverse low-resource African languages covering three tasks: natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and multi-choice knowledge-based QA~(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings~(where test sets are translated into English) across 10 open and four proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages~(such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Aya-101 only at 58\% of the best-performing proprietary model GPT-4o performance. Machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, like LLaMa 3 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.
arXiv:2406.03368 [cs] <标题>: IrokoBench：大语言模型时代的非洲语言新基准 <AI Smry>: There is a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63\% of the best-performing proprietary model GPT-4o performance. Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: Educational Area: Language of Instruction; Evidence: Improve a process; Location: Africa; Research Design: Other Research Design; Research Method: Quantitative; Source: Open Source

Learning analytics: A comparison of western, educated, industrialized, rich, and democratic (WEIRD) and non-WEIRD research | Baek | Knowledge Management & E-Learning: An International Journal (2024)
<标题>: 学习分析：西方、受过教育的、工业化的、富裕的和民主的（WEIRD）与非WEIRD研究的比较 | Baek | 《知识管理与电子学习：国际期刊》 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Applying large language models for automated essay scoring for non-native Japanese - Li, Wenchao; Liu, Haitao (Humanities and Social Sciences Communications, 2024)
Recent advancements in artificial intelligence (AI) have led to an increased use of large language models (LLMs) for language assessment tasks such as automated essay scoring (AES), automated listening tests, and automated oral proficiency assessments. The application of LLMs for AES in the context of non-native Japanese, however, remains limited. This study explores the potential of LLM-based AES by comparing the efficiency of different models, i.e. two conventional machine training technology-based methods (Jess and JWriter), two LLMs (GPT and BERT), and one Japanese local LLM (Open-Calm large model). To conduct the evaluation, a dataset consisting of 1400 story-writing scripts authored by learners with 12 different first languages was used. Statistical analysis revealed that GPT-4 outperforms Jess and JWriter, BERT, and the Japanese language-specific trained Open-Calm large model in terms of annotation accuracy and predicting learning levels. Furthermore, by comparing 18 different models that utilize various prompts, the study emphasized the significance of prompts in achieving accurate and reliable evaluations using LLMs.
<标题>: 将大型语言模型应用于非母语日语的自动作文评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

FineWeb: decanting the web for the finest text data at scale - a Hugging Face Space by HuggingFaceFW (2024)
Discover amazing ML apps made by the community
<标题>: FineWeb：在大规模下提取最优文本数据的网络工具——由 HuggingFaceFW 提供的 Hugging Face 空间 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays - Fleckenstein, Johanna; Meyer, Jennifer; Jansen, Thorben; Keller, Stefan; Köller, Olaf; Köller, Jens (Computers and Education: Artificial Intelligence, 2024)
The potential application of generative artificial intelligence (AI) in schools and universities poses great challenges, especially for the assessment…
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: /unread

Human-centred learning analytics and AI in education: A systematic literature review - Alfredo, Riordan; Echeverria, Vanessa; Jin, Yueqiao; Yan, Lixiang; Swiecki, Zachari; Gašević, Dragan; Martinez-Maldonado, Roberto (Computers and Education: Artificial Intelligence, 2024)
The rapid expansion of Learning Analytics (LA) and Artificial Intelligence in Education (AIED) offers new scalable, data-intensive systems but raises concerns about data privacy and agency. Excluding stakeholders—like students and teachers—from the design process can potentially lead to mistrust and inadequately aligned tools. Despite a shift towards human-centred design in recent LA and AIED research, there remain gaps in our understanding of the importance of human control, safety, reliability, and trustworthiness in the design and implementation of these systems. We conducted a systematic literature review to explore these concerns and gaps. We analysed 108 papers to provide insights about i) the current state of human-centred LA/AIED research; ii) the extent to which educational stakeholders have contributed to the design process of human-centred LA/AIED systems; iii) the current balance between human control and computer automation of such systems; and iv) the extent to which safety, reliability and trustworthiness have been considered in the literature. Results indicate some consideration of human control in LA/AIED system design, but limited end-user involvement in actual design. Based on these findings, we recommend: 1) carefully balancing stakeholders' involvement in designing and deploying LA/AIED systems throughout all design phases 2) actively involving target end-users, especially students, to delineate the balance between human control and automation, and 3) exploring safety, reliability, and trustworthiness as principles in future human-centred LA/AIED systems.
<标题>: 以人为本的学习分析与教育中的人工智能：系统文献综述 <AI Smry>: A systematic literature review of human-centred LA/AIED research suggests carefully balancing stakeholders' involvement in designing and deploying LA/AIED systems throughout all design phases, and actively involving target end-users to delineate the balance between human control and automation. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z
Keywords: AI: Analytics; Educational Area: Data Quality & EMIS; Evidence: Improve a process; Location: Asia; Research Design: Systematic Review; Research Method: Qualitative; Source: Open Source

LLMs in Short Answer Scoring: Limitations and Promise of Zero-Shot and Few-Shot Approaches - Chamieh, Imran; Zesch, Torsten; Giebermann, Klaus (Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), 2024)
In this work, we investigate the potential of Large Language Models (LLMs) for automated short answer scoring. We test zero-shot and few-shot settings, and compare with fine-tuned models and a supervised upper-bound, across three diverse datasets. Our results, in zero-shot and few-shot settings, show that LLMs perform poorly in these settings: LLMs have difficulty with tasks that require complex reasoning or domain-specific knowledge. While the models show promise on general knowledge tasks. The fine-tuned model come close to the supervised results but are still not feasible for application, highlighting potential overfitting issues. Overall, our study highlights the challenges and limitations of LLMs in short answer scoring and indicates that there currently seems to be no basis for applying LLMs for short answer scoring.
<标题>: 短答题评分中的大型语言模型：零样本与少样本方法的局限性与潜力 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Applying Large Language Models and Chain-of-Thought for Automatic Scoring - Lee, Gyeong-Geon; Latif, Ehsan; Wu, Xuansheng; Liu, Ninghao; Zhai, Xiaoming (arXiv, 2024)
This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1,650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics. Results indicated that few-shot (acc = .67) outperformed zero-shot learning (acc = .60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = .60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT -3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches.
arXiv:2312.03748 [cs] <标题>: 将大型语言模型与链式思维应用于自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.303Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Applying large language models and chain-of-thought for automatic scoring - Lee, Gyeong-Geon; Latif, Ehsan; Wu, Xuansheng; Liu, Ninghao; Zhai, Xiaoming (Computers and Education: Artificial Intelligence, 2024)
<标题>: 应用大型语言模型和连锁思维进行自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

A systematic review of AI education in K-12 classrooms from 2018 to 2023: Topics, strategies, and learning outcomes - Lee, Sang Joon; Kwon, Kyungbin (Computers and Education: Artificial Intelligence, 2024)
AI education aims to teach AI concepts, essential knowledge, and skills related to the fundamental ideas in AI. As AI becomes increasingly prevalent in our daily lives, schools and educators have started to recognize the importance of AI education in K-12 schools. However, there have been a limited number of studies reporting on the implementation of AI education in classrooms. This systematic review aimed to provide an overview of the current state of AI education in K-12 schools, exploring topics, instructional approaches, and learning outcomes. Twenty-five peer-reviewed journal articles published between 2018 and 2023 were selected for this systematic review. The findings highlighted that various topics were covered in K-12 AI education, including fundamental AI concepts, different types of AI, AI applications, and ethical considerations related to AI. To facilitate meaningful learning experiences, educators frequently integrated hands-on activities and project-based learning. The findings supported the benefits of AI education in enhancing students' AI literacy, problem-solving skills, and ethical reflections on AI's societal impact. Furthermore, it fostered motivation, positive attitudes toward AI, and an interest in technology while inspiring career aspirations. It is recommended to develop tailored AI curricula, instructional strategies, and appropriate tools and resources that seamlessly integrate into various subjects within the standard school curriculum.
<标题>: 2018年至2023年K-12课堂人工智能教育的系统综述：主题、策略与学习成果 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z
Keywords: Educational Area: Assessment Tools & Processes; Educational Area: Quality Teaching/Learning Materials; Educational Area: Student Learning; Evidence: Improve a process; Evidence: Student Learning Outcomes; Location: Americas; Research Design: Systematic Review; Research Method: Qualitative; Source: Open Source

Tutor vs. Automatic Focused Feedback and Grading of Student ESP Compositions in an Online Learning Environment - MartínMonje, Elena; Barcena, Elena (Journal of Research in Applied Linguistics, 2024)
<标题>: 导师与自动化针对学生ESP作文的反馈与评分在在线学习环境中的比较 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students’ text revision, motivation, and positive emotions - Meyer, Jennifer; Jansen, Thorben; Schiller, Ronja; Liebenow, Lucas W.; Steinbach, Marlene; Horbach, Andrea; Fleckenstein, Johanna (Computers and Education: Artificial Intelligence, 2024)
Writing proficiency is an essential skill for upper secondary students that can be enhanced through effective feedback. Creating feedback on writing tasks, however, is time-intensive and presents a challenge for educators, often resulting in students receiving insufficient or no feedback. The advent of text-generating large language models (LLMs) offers a promising solution, namely, automated evidence-based feedback generation. Yet, empirical evidence from randomized controlled studies about the effectiveness of LLM-generated feedback is missing. To address this issue, the current study compared the effectiveness of LLM-generated feedback to no feedback. A sample of N = 459 upper secondary students of English as a foreign language wrote an argumentative essay. Students in the experimental group were asked to revise their text according to feedback that was generated using the LLM GPT-3.5-turbo. The control group revised their essays without receiving feedback. We assessed improvement in the revision using automated essay scoring. The results showed that LLM-generated feedback increased revision performance (d = .19) and task motivation (d = 0.36). Moreover, it increased positive emotions (d = 0.34) compared to revising without feedback. The findings highlight that using LLMs allows to create timely feedback that can positively relate to students’ cognitive and affective-motivational outcomes. Future perspectives and the implications for research and practice of using LLM-generated feedback in intelligent tutoring systems are discussed.
<标题>: 利用大语言模型将循证反馈引入课堂：AI生成的反馈提高了中学生的文本修改、学习动机和积极情感 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z
Keywords: AI: ChatGPT; Educational Area: Assess & Review; Educational Area: Positive Learning Environment; Educational Area: Student Learning; Evidence: Gaps in Current Research; Evidence: Improve a process; Evidence: Measured Student Behaviour; Evidence: Student Learning Outcomes; Location: Europe; Research Design: Randomised Control Trial; Research Method: Quantitative; Source: Open Source; artificial intelligence

Comparing the quality of human and ChatGPT feedback of students’ writing - Steiss, Jacob; Tate, Tamara; Graham, Steve; Cruz, Jazmin; Hebert, Michael; Wang, Jiali; Moon, Youngsun; Tseng, Waverly; Warschauer, Mark; Olson, Carol Booth (Learning and Instruction, 2024)
Structured Abstract Background Offering students formative feedback on their writing is an effective way to facilitate writing development. Recent advances in AI (i.e., ChatGPT) may function as an automated writing evaluation tool, increasing the amount of feedback students receive and diminishing the burden on teachers to provide frequent feedback to large classes. Aims We examined the ability of generative AI (ChatGPT) to provide formative feedback. We compared the quality of human and AI feedback by scoring the feedback each provided on secondary student essays. We scored the degree to which feedback (a) was criteria-based, (b) provided clear directions for improvement, (c) was accurate, (d) prioritized essential features, and (e) used a supportive tone. Sample 200 pieces of human-generated formative feedback and 200 pieces of AI-generated formative feedback for the same essays. Methods We examined whether ChatGPT and human feedback differed in quality for the whole sample, for compositions that differed in overall quality, and for native English speakers and English learners by comparing descriptive statistics and effect sizes. Results Human raters were better at providing high-quality feedback to students in all categories other than criteria-based. AI and humans showed differences in feedback quality based on essay quality. Feedback did not vary by language status for humans or AI. Conclusion Well-trained evaluators provided higher quality feedback than ChatGPT. Considering the ease of generating feedback through ChatGPT and its overall quality, generative AI may still be useful in some contexts, particularly in formative early drafts or instances where a well-trained educator is unavailable.
<标题>: 比较人类与 ChatGPT 对学生写作反馈的质量 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z
Keywords: AI: ChatGPT; Educational Area: Assessment Tools & Processes; Educational Area: Student Learning; Evidence: Improve a process; Evidence: Saving Teacher Time; Location: Americas; Research Design: Randomised Control Trial; Research Method: Quantitative; Source: Open Source

Using convolutional neural networks to automatically score eight TIMSS 2019 graphical response items - Tyack, Lillian; Khorramdel, Lale; von Davier, Matthias (Computers and Education: Artificial Intelligence, 2024)
International large-scale assessments (ILSAs) have used graphical response-based items to measure student ability for decades, but they have yet to implement automated scoring of these responses and instead rely on human scoring alone. To investigate how scores provided by machine algorithms compare to those provided by human raters, we applied convolutional neural networks (CNNs) to classify image-based responses from eight Timss 2019 items. Our results show that the most accurate CNN models classified over 99% of the image responses into the appropriate scoring category for dichotomous items and almost 98% for one trichotomous item. Additionally, during the modeling process, the CNNs correctly classified numerous image responses that human raters had scored incorrectly. For most items, the number of incorrectly human-scored responses exceeded the average number of responses misclassified by the most accurate models. These results suggest that automated scoring using CNNs is comparable to, and in many cases more accurate, than human raters, even across a wide variety of graphing tasks. This paper argues that the machine learning procedure explored could be implemented in ILSAs as a verification method to improve the accuracy and consistency of graphical response item scores. In lieu of additional human raters, ILSAs could implement CNN-based automated scoring to provide a second set of scores, thus reducing the workload and costs associated with human scoring.
<标题>: 使用卷积神经网络自动评分八个 TIMSS 2019 图形回答题 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z
Keywords: Educational Area: Assessment Tools & Processes; Evidence: Improve a process; Location: Americas; Research Design: Other Research Design; Research Method: Quantitative; Source: Open Source

Using convolutional neural networks to automatically score eight TIMSS 2019 graphical response items - Tyack, Lillian; Khorramdel, Lale; Von Davier, Matthias (Computers and Education: Artificial Intelligence, 2024)
<标题>: 使用卷积神经网络自动评分八个 TIMSS 2019 图形作答题 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

AI-Powered Education: Exploring the Potential of Personalised Learning for Students' Needs in Indonesia Education - Faresta, Rangga Alif (Path of Science, 2024)
Artificial intelligence (AI) stands out as a relatively new yet rapidly expanding technological tool that is transforming the field of education. This paper examines the potential of Artificial Intelligence to assist students and teachers in personalised learning. The research methodology employed for this study is a literature review, providing an overview of the current knowledge on practical AI applications for personalised learning and insights into methodological developments in this research field. Based on the literature, the results of this research demonstrate that personalised learning effectively accommodates students' learning preferences and enhances academic performance. Therefore, it could significantly benefit students by accommodating their learning pace and style. In the Indonesian education system context, the integration of AI for personalised learning is already included in the Indonesia Artificial Intelligence National Plan framework. To support this plan, the authors employed a Personalized Learning Plan (PLP) to integrate AI into educational settings practically. However, a challenge in integrating AI into education for personalised learning is that course or class designers often pay insufficient attention to creating content that develops practical skills. This neglect of pedagogical and technical aspects has led students and teachers to perceive these systems as unresponsive to their learning preferences, fostering a sense of pessimism.
Number: 5 <标题>: 人工智能驱动的教育：探索为印度尼西亚学生需求量身定制学习的潜力 <AI Smry>: Based on the literature, the results of this research demonstrate that personalised learning effectively accommodates students' learning preferences and enhances academic performance. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.151Z
Keywords: Educational Area: Student Learning; Evidence: Student Learning Outcomes; Location: Asia; Research Design: Non-systematic Review; Research Method: Qualitative; Source: Open Source

Intelligent tutoring effects on induced emotions and cognitive load of learning-disabled learners - Dutt, Sarthika; Ahuja, Neelu Jyothi (Disability and Rehabilitation: Assistive Technology, 2024)
This study addresses the learning requirements of learners with learning difficulties by monitoring their learning experience in an Intelligent Tutoring System. Intelligent Tutoring Systems were developed to enrich the teaching-learning process. In the present work, the interface is designed and developed utilizing the potential of Artificial Intelligence to meet their individual needs. Designing an online learning platform for a learners with learning difficulties requires consideration of their learning problems and preferences. The interface was developed focusing on all the requirements of the LD learners. The objective of the present study is to monitor the learning experience in the form of induced emotions and cognitive load of the learners to determine the impact of learning. 83 learners were observed during various stage of learning. The results obtained through the Support Vector machine (SVM) classification technique showed the positive attitude towards intelligent tutoring. The analysis revealed that a total of 0.23% of learners were positively induced. Their learning experience was positive and effective. The cognition load on learners was minimum with single-mode instruction and least disturbed. The system was improved based on preference feedback on design features. This helps in improving content design and creating device independent and responsive visual design. The fatigue effect analysis on cognitive load implied that multiple modes of instruction increased drowsiness. Single mode of instruction have a positive impact on the learning process and it reduces the cognitive load of the learners.Implications for RehabilitationThe user interface designed and developed for learners with Dyslexia, Dysgraphia and Dyscalculia has learning disabled-friendly features. These can be used to create a device-independent and responsive design.Learning experience is monitored along with the impact on cognitive load of the learners.The research helps in understanding the stimulation and response of learners with learning disability for different learning conditions.Most existing learning systems are limited to non-learning-disabled learners. The ITS developed during research presents a Universal learning design helpful for all learners with and without learning disability. The user interface designed and developed for learners with Dyslexia, Dysgraphia and Dyscalculia has learning disabled-friendly features. These can be used to create a device-independent and responsive design. Learning experience is monitored along with the impact on cognitive load of the learners. The research helps in understanding the stimulation and response of learners with learning disability for different learning conditions. Most existing learning systems are limited to non-learning-disabled learners. The ITS developed during research presents a Universal learning design helpful for all learners with and without learning disability.
_eprint: https://doi.org/10.1080/17483107.2024.2357685 <标题>: 智能辅导对学习障碍学生诱发情绪和认知负荷的影响 <AI Smry>: This study addresses the learning requirements of learners with learning difficulties by monitoring their learning experience in an Intelligent Tutoring System using the potential of Artificial Intelligence to meet their individual needs. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z
Keywords: Educational Area: Inclusion; Educational Area: Student Learning; Evidence: Self-reported Student Behaviour/ Attitude; Location: Asia; Research Design: Interview/ focus group; Research Method: Qualitative; Source: Closed Source

The Future of Child Development in the AI Era. Cross-Disciplinary Perspectives Between AI and Child Development Experts - Neugnot-Cerioli, Mathilde; Laurenty, Olga Muss (arXiv, 2024)
This report explores the potential implications of rapidly integrating Artificial Intelligence (AI) applications into children's environments. The introduction of AI in our daily lives necessitates scrutiny considering the significant role of the environment in shaping cognition, socio-emotional skills, and behaviors, especially during the first 25 years of cerebral development. As AI becomes prevalent in educational and leisure activities, it will significantly modify the experiences of children and adolescents, presenting both challenges and opportunities for their developmental trajectories. This analysis was informed by consulting with 15 experts from pertinent disciplines (AI, product development, child development, and neurosciences), along with a comprehensive review of scientific literature on children development and child-technology interactions. Overall, AI experts anticipate that AI will transform leisure activities, revolutionize education, and redefine human-machine interactions. While AI offers substantial benefits in fostering interactive engagement, it also poses risks that require careful considerations, especially during sensitive developmental periods. The report advocates for proactive international collaboration across multiple disciplines and increased research into how technological innovations affect child development. Such efforts are crucial for designing a sustainable and ethical future for the next generation through specific child-centered regulations, and helping to educate all potential stakeholders (regulators, developers, parents and educators, children) about responsible AI use and its potential impacts on child development.
arXiv:2405.19275 [cs] <标题>: 人工智能时代的儿童发展未来：人工智能与儿童发展专家的跨学科视角 <AI Smry>: Overall, AI experts anticipate that AI will transform leisure activities, revolutionize education, and redefine human-machine interactions, and offer substantial benefits in fostering interactive engagement, but poses risks that require careful considerations, especially during sensitive developmental periods. Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: Educational Area: Student Learning; Evidence: Self-reported Student Behaviour/ Attitude; Location: Americas; Research Design: Other Research Design; Research Method: No research methods used; Source: Open Source

ChatGPT in education: benefits and challenges of ChatGPT for mathematics and science teaching practices: International Journal of Mathematical Education in Science and Technology: Vol 0, No 0 - Taani, Osama; Alabidi, Suzan (International Journal of Mathematical Education in Science and Technology, 2024)
This study investigates the utilisation and perceptions of artificial intelligence (AI) applications among mathematics and science teachers in enhancing students’ learning experience in mathematics classrooms. One prominent AI application this research seeks to explore, given its recent rise in popularity and usage, is ChatGPT. The study aims to explore the frequency and purpose of ChatGPT usage, its potential for student engagement, and its implementation’s perceived effectiveness, benefits, and challenges. A sample of 109 respondents was surveyed to gather insights into the usage patterns and perceptions of ChatGPT. The findings shed light on how teachers utilise ChatGPT, including generating examples, assessing difficulty, providing explanations, supporting problem-solving, and test preparation. Additionally, the study reveals that ChatGPT shows promise in enhancing student engagement and understanding. However, respondents highlighted challenges such as accuracy, connectivity, and language limitations. These findings contribute to our understanding of the role of AI technologies, specifically ChatGPT, in mathematics and science educational settings and provide valuable insights for the effective integration of AI in teaching and learning practices in various course contents and topics.
<标题>: 教育中的 ChatGPT：ChatGPT 在数学和科学教学实践中的优势与挑战：《国际数学教育科学与技术期刊》，第0卷，第0期 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: AI: ChatGPT; Educational Area: Quality Teaching/Learning Materials; Educational Area: Teach; Evidence: Self-reported Teacher Behaviour/ Attitude; Location: Asia; Research Design: Qualitative Exploratory; Research Method: Qualitative; Source: Closed Source; ⛔ No DOI found

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (2024)
<标题>: 扩展单义性：从 Claude 3 十四行诗中提取可解释特征 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.531Z
Keywords: /unread

Artificial Intelligence (AI) Literacy, An Investment for Enhancing Educators Skills in AI Powered Primary Schools in Nigeria - Ewa, Moses Apie PhD (International Journal of Research and Innovation in Social Science, 2024)
A descriptive survey was conducted to investigate whether an investment in artificial intelligence (AI) literacy predicts the enhancement of educators’ skills for them to be effective in an AI powered environment in state primary schools in Nigeria. Three research questions and null hypotheses were formulated at .05 level of significance for the study. 1000 teachers were recruited from across five schools located in five geopolitical zones of the country using cluster sampling. Artificial intelligence literacy and educators’ skills enhancement questionnaire (AILESEQ) was deployed to generate data. Based on the human capital development theory, data was analysed via the simple linear regression. Findings suggest that knowledge of AI, use of AI devices and understanding the ethical issues about AI technologies significantly predict the enhancement of educators’ skills in AI powered primary schools in Nigeria. It is recommended that government should organise training workshops on AI literacy for educators in schools; the policy on education needs review to produce a framework for the introduction of AI in education (AIIE) to benefit the teachers; the curriculum of teacher education has to be revised to feature AIIE; AI literacy should be a requirement for recruiting new teachers; schools have to produce guidelines for the application of AI devices; government can create a demonstration school to trial the AI initiative; government can commit one percent of value added tax to fund AIIE; awareness campaigns should be organised regularly via various media to educate relevant stakeholders about the benefits of AIIE.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.102Z
Keywords: Educational Area: Continuous Professional Development; Evidence: Self-reported Teacher Behaviour/ Attitude; Location: Africa; Research Design: Non-experimental design; Research Method: Qualitative; Source: Open Source; ⛔ No DOI found

Towards a Unified Framework for Evaluating Explanations - Pinto, Juan D.; Paquette, Luc (arXiv, 2024)
The challenge of creating interpretable models has been taken up by two main research communities: ML researchers primarily focused on lower-level explainability methods that suit the needs of engineers, and HCI researchers who have more heavily emphasized user-centered approaches often based on participatory design methods. This paper reviews how these communities have evaluated interpretability, identifying overlaps and semantic misalignments. We propose moving towards a unified framework of evaluation criteria and lay the groundwork for such a framework by articulating the relationships between existing criteria. We argue that explanations serve as mediators between models and stakeholders, whether for intrinsically interpretable models or opaque black-box models analyzed via post-hoc techniques. We further argue that useful explanations require both faithfulness and intelligibility. Explanation plausibility is a prerequisite for intelligibility, while stability is a prerequisite for explanation faithfulness. We illustrate these criteria, as well as specific evaluation methods, using examples from an ongoing study of an interpretable neural network for predicting a particular learner behavior.
arXiv:2405.14016 [cs] <标题>: 迈向评估解释的统一框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Machine Learning

Towards a Unified Framework for Evaluating Explanations - Pinto, Juan D.; Paquette, Luc (arXiv, 2024)
The challenge of creating interpretable models has been taken up by two main research communities: ML researchers primarily focused on lower-level explainability methods that suit the needs of engineers, and HCI researchers who have more heavily emphasized user-centered approaches often based on participatory design methods. This paper reviews how these communities have evaluated interpretability, identifying overlaps and semantic misalignments. We propose moving towards a unified framework of evaluation criteria and lay the groundwork for such a framework by articulating the relationships between existing criteria. We argue that explanations serve as mediators between models and stakeholders, whether for intrinsically interpretable models or opaque black-box models analyzed via post-hoc techniques. We further argue that useful explanations require both faithfulness and intelligibility. Explanation plausibility is a prerequisite for intelligibility, while stability is a prerequisite for explanation faithfulness. We illustrate these criteria, as well as specific evaluation methods, using examples from an ongoing study of an interpretable neural network for predicting a particular learner behavior.
arXiv:2405.14016 [cs] <标题>: 迈向评估解释的统一框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:30.842Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Machine Learning

Towards a Unified Framework for Evaluating Explanations - Pinto, Juan D.; Paquette, Luc (arXiv, 2024)
The challenge of creating interpretable models has been taken up by two main research communities: ML researchers primarily focused on lower-level explainability methods that suit the needs of engineers, and HCI researchers who have more heavily emphasized user-centered approaches often based on participatory design methods. This paper reviews how these communities have evaluated interpretability, identifying overlaps and semantic misalignments. We propose moving towards a unified framework of evaluation criteria and lay the groundwork for such a framework by articulating the relationships between existing criteria. We argue that explanations serve as mediators between models and stakeholders, whether for intrinsically interpretable models or opaque black-box models analyzed via post-hoc techniques. We further argue that useful explanations require both faithfulness and intelligibility. Explanation plausibility is a prerequisite for intelligibility, while stability is a prerequisite for explanation faithfulness. We illustrate these criteria, as well as specific evaluation methods, using examples from an ongoing study of an interpretable neural network for predicting a particular learner behavior.
arXiv:2405.14016 [cs] <标题>: 迈向评估解释的统一框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Machine Learning

A.I.’s Black Boxes Just Got a Little Less Mysterious - Roose, Kevin (The New York Times, 2024)
Researchers at the A.I. company Anthropic claim to have found clues about the inner workings of large language models, possibly helping to prevent their misuse and to curb their potential threats.
<标题>: 人工智能的黑箱变得不那么神秘了 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread; Anthropic AI LLC; Artificial Intelligence; Computers and the Internet; OpenAI Labs; Research

The United States Artificial Intelligence Safety Institute: Vision, Mission, and Strategic Goals (2024)
This summary provides an overview of the U.S.
<标题>: 美国人工智能安全研究所：愿景、使命与战略目标 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach (Google) - Jurenka, Irina; Kunesch, Markus; McKee, Kevin; Gillick, Daniel; Zhu, Shaojian; Wiltberger, Sara; Phal, Shubham Milind; Hermann, Katherine; Kasenberg, Daniel; Bhoopchand, Avishkar; Anand, Ankit; Pîslar, Miruna; et al. (2024)
<标题>: 面向教育的生成式人工智能负责任发展：一种以评估为驱动的方法（谷歌） Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: AI: ChatGPT; Educational Area: Student Learning; Evidence: Saving Teacher Time; Location: Americas; Research Design: Case Study; Research Design: Other Research Design; Research Method: Mixed Methods; Source: Open Source

Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach - Jurenka, Irina; Kunesch, Markus; McKee, Kevin; Gillick, Daniel; Zhu, Shaojian; Wiltberger, Sara; Phal, Shubham Milind; Hermann, Katherine; Kasenberg, Daniel; Bhoopchand, Avishkar; Anand, Ankit; Pîslar, Miruna; et al. (Google, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning - Kim, Jisu; Lee, Juhwan (arXiv, 2024)
The rapid advancement of Large Language Models (LLMs) has improved text understanding and generation but poses challenges in computational resources. This study proposes a curriculum learning-inspired, data-centric training strategy that begins with simpler tasks and progresses to more complex ones, using criteria such as prompt length, attention scores, and loss values to structure the training data. Experiments with Mistral-7B (Jiang et al., 2023) and Gemma-7B (Team et al., 2024) models demonstrate that curriculum learning slightly improves performance compared to traditional random data shuffling. Notably, we observed that sorting data based on our proposed attention criteria generally led to better performance. This approach offers a sustainable method to enhance LLM performance without increasing model size or dataset volume, addressing scalability challenges in LLM training.
arXiv:2405.07490 [cs] <标题>: 战略数据排序：通过课程学习提升大型语言模型性能 <AI Smry>: A curriculum learning-inspired, data-centric training strategy that begins with simpler tasks and progresses to more complex ones, using criteria such as prompt length, attention scores, and loss values to structure the training data. Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.303Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Artificial Intelligence and Educational Measurement: Opportunities and Threats - Ho, Andrew D. (Journal of Educational and Behavioral Statistics, 2024)
I review opportunities and threats that widely accessible Artificial Intelligence (AI)-powered services present for educational statistics and measurement. Algorithmic and computational advances continue to improve approaches to item generation, scale maintenance, test security, test scoring, and score reporting. Predictable misuses of AI for these purposes will result in biased scores, construct underrepresentation, and differential impact over time. Recent efforts to develop standards for AI use in testing like those of Burstein are promising. I argue that similar efforts to develop AI standards for educational measurement will benefit from increased attention to the context of test use and explicit commitment to ongoing monitoring of bias and scale drift over time.
<AI Smry>: It is argued that similar efforts to develop AI standards for educational measurement will benefit from increased attention to the context of test use and explicit commitment to ongoing monitoring of bias and scale drift over time. <标题>: 人工智能与教育测量：机遇与威胁 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Bridging large language model disparities: Skill tagging of multilingual educational content - Kwak, Yerin; Pardos, Zachary A. (British Journal of Educational Technology, 2024)
The adoption of large language models (LLMs) in education holds much promise. However, like many technological innovations before them, adoption and access can often be inequitable from the outset, creating more divides than they bridge. In this paper, we explore the magnitude of the country and language divide in the leading open‐source and proprietary LLMs with respect to knowledge of K‐12 taxonomies in a variety of countries and their performance on tagging problem content with the appropriate skill from a taxonomy, an important task for aligning open educational resources and tutoring content with state curricula. We also experiment with approaches to narrowing the performance divide by enhancing LLM skill tagging performance across four countries (the USA, Ireland, South Korea and India–Maharashtra) for more equitable outcomes. We observe considerable performance disparities not only with non‐English languages but with English and non‐US taxonomies. Our findings demonstrate that fine‐tuning GPT‐3.5 with a few labelled examples can improve its proficiency in tagging problems with relevant skills or standards, even for countries and languages that are underrepresented during training. Furthermore, the fine‐tuning results show the potential viability of GPT as a multilingual skill classifier. Using both an open‐source model, Llama2‐13B, and a closed‐source model, GPT‐3.5, we also observe large disparities in tagging performance between the two and find that fine‐tuning and skill information in the prompt improve both, but the closed‐source model improves to a much greater extent. Our study contributes to the first empirical results on mitigating disparities across countries and languages with LLMs in an educational context.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/bjet.13465 <标题>: 弥合大型语言模型差异：多语言教育内容的技能标记 <AI Smry>: The magnitude of the country and language divide in the leading open‐source and proprietary LLMs with respect to knowledge of K‐12 taxonomies in a variety of countries and their performance on tagging problem content with the appropriate skill from a taxonomy is explored, an important task for aligning open educational resources and tutoring content with state curricula. Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.206Z
Keywords: ChatGPT; LLM; Llama2; multilingual; skill tagging

The transformative potential of Artificial Intelligence for Education - Bezzina, Stephen; Dingli, Alexiei (Networked Learning Conference, 2024)
Education today faces a range of challenges, but also unique opportunities, vis-à-vis the evolution of digital technologies. Traditional teaching methods struggle to meet the diverse needs of students, as classrooms comprise learners at different levels. The Education AI project represents an initiative aimed at integrating Artificial Intelligence (AI) into the classroom to address the diversity of students and the challenges faced by traditional teaching methods. The project's core objective is to enhance student learning outcomes by providing an adaptive and personalised learning experience, thereby making education more equitable and effective. This paper evaluates the project's impact on student learning, teacher workload, and its broader applicability in education. The study's methodology combines quantitative and qualitative data, employing a mixed-methods approach to provide a comprehensive analysis of the Education AI system's effectiveness. Preliminary findings indicate a notable improvement in student performance, particularly among low and medium-performing students, with quantitative data revealing an average improvement of 23% in assessments for low-performing students. The system's user-friendly interface, gamified elements, and immediate feedback mechanisms have been well-received, with a majority of students reporting enhanced engagement and understanding of mathematical concepts. Teachers have also reported positive experiences, highlighting the system's role in reducing their workload and facilitating personalised and adaptive teaching and learning. Qualitative feedback from interviews with teachers and students provides further insights into the system's impact, emphasising its varied and engaging question types, adaptivity, and the shift from paper-based to digital learning. The feedback also reveals a desire for expansion to other subjects and enhancements in the system's analytics for more targeted support.The Education AI project demonstrates the transformative potential of AI in the classroom, with significant improvements in student performance and positive reception from both students and teachers. The study highlights the importance of addressing technical challenges and expanding the system's capabilities for broader application. The project offers valuable insights into the future of education, where technology and human expertise converge to create a more personalised, adaptive engaging, and effective learning experience for all students. This initiative, underpinned by the transformative potential of AI for education, aims and informing and guiding future innovations in educational technology.
<标题>: 人工智能在教育中的变革潜力 <AI Smry>: The Education AI project demonstrates the transformative potential of AI in the classroom, with significant improvements in student performance and positive reception from both students and teachers, and highlights the importance of addressing technical challenges and expanding the system's capabilities for broader application. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: Educational Area: Student Learning; Evidence: Self-reported Student Behaviour/ Attitude; Evidence: Self-reported Teacher Behaviour/ Attitude; Evidence: Student Learning Outcomes; Location: Europe; Research Design: Interview/ focus group; Research Method: Mixed Methods; Source: Open Source

Learner-Oriented Tool Using Generative Language Model for Angeles Elementary School Primary Students - Guiwan, Jerald; Lacson, Leeron; Dayrit, Michaella; Esquillo, John; Leon, Jose; Guillermo, David; Parungao, Adel (2024)
In the Philippines, elementary students face a significant educational hurdle, particularly in Grade 4, where foundational competencies prove challenging to grasp. This research aims to provide a possible solution for this issue by developing and investigating the functionality of an electronic GLM-powered learner-oriented tool (EGLOT), designed to act as an educational companion that leverages a generative language model to personalize the learning experience for Grade 3 students in mathematics. EGLOT assesses students' competencies and delivers tailored learning modules based on their results. EGLOT integrates hardware, utilizing ESP32 for speech-to-text and text-to speech capabilities, along with software functionalities accessible through a teacher dashboard for monitoring and control. Through rigorous statistical analyses, including ISO 25059 and Word Error Rate (WER) assessments, EGLOT's accuracy and reliability were validated. Achieving a 91.94% accuracy rate in speech-to-text transcription, EGLOT demonstrates its ability to identify individual student weaknesses and design personalized learning modules, thereby empowering teachers with targeted interventions. This research highlights recommendations for enhancing EGLOT's functionality, such as mitigating time constraints, exploring innovative technologies, and improving user interaction through visual components. Overall, EGLOT has shown promising potential in improving Grade 3 mathematics learning outcomes. This success highlights the potential of artificial intelligence-driven (AI-driven) educational tools to personalize learning and improve educational outcomes globally.
<标题>: 面向学习者的工具：使用生成式语言模型为安吉丽斯小学学生服务 Read_Status: New Read_Status_Date: 2026-01-26T11:32:25.304Z
Keywords: Educational Area: Assessment Tools & Processes; Evidence: Self-reported Student Behaviour/ Attitude; Evidence: Self-reported Teacher Behaviour/ Attitude; Location: Asia; Research Design: Other Research Design; Research Method: Mixed Methods; Source: Open Source

Evaluating and Optimizing Educational Content with Large Language Model Judgments - He-Yueya, Joy; Goodman, Noah D.; Brunskill, Emma (arXiv, 2024)
Creating effective educational materials generally requires expensive and time-consuming studies of student learning outcomes. To overcome this barrier, one idea is to build computational models of student learning and use them to optimize instructional materials. However, it is difficult to model the cognitive processes of learning dynamics. We propose an alternative approach that uses Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. Specifically, we use GPT-3.5 to evaluate the overall effect of instructional materials on different student groups and find that it can replicate well-established educational findings such as the Expertise Reversal Effect and the Variability Effect. This demonstrates the potential of LMs as reliable evaluators of educational content. Building on this insight, we introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. We apply this approach to create math word problem worksheets aimed at maximizing student learning gains. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences. We conclude by discussing potential divergences between human and LM opinions and the resulting pitfalls of automating instructional design.
arXiv:2403.02795 [cs] <AI Smry>: This work uses GPT-3.5 to evaluate the overall effect of instructional materials on different student groups and finds that it can replicate well-established educational findings such as the Expertise Reversal Effect and the Variability Effect. <标题>: 利用大型语言模型评估和优化教育内容 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Educational Area: Quality Teaching/Learning Materials; Evidence: Saving Teacher Time; Location: Americas; Research Design: Other Research Design; Research Method: Mixed Methods; Source: Open Source

Validation of an elicited Imitation test as a measure of Korean language proficiency - Kim, Hojung; Song, Changkyung; Kim, Jiyoung; Jeong, Hyeyun; Park, Jisoo (Language Testing in Asia, 2024)
This study presents a modified version of the Korean Elicited Imitation (EI) test, designed to resemble natural spoken language, and validates its reliability as a measure of proficiency. The study assesses the correlation between average test scores and Test of Proficiency in Korean (TOPIK) levels, examining score distributions among beginner, intermediate, and advanced learner groups. Using item response theory (IRT), the study explores the influence of four key facets—learners, items, raters, and constructs—on performance evaluation. An explanatory item response modeling (EIRM) analysis identified linguistic factors impacting the EI test’s performance. Notably, the study uncovered a robust positive correlation between EI test scores and TOPIK levels. Significant score disparities were observed between beginner and intermediate, as well as beginner and advanced, learner cohorts. An IRT-based exploration of each facet revealed that item difficulty was comparatively lower in contrast to learners’ commendable performance, and raters exhibited a high degree of scoring consistency. The EIRM analysis underscores the significance of variables such as the number of syllables, vocabulary score, and content word density in influencing test performance.
<标题>: 对引出模仿测试作为韩语能力测量工具的有效性验证 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review - Khowaja, Sunder Ali; Khuwaja, Parus; Dev, Kapal; Wang, Weizheng; Nkenyereye, Lewis (Cognitive Computation, 2024)
Abstract ChatGPT is another large language model (LLM) vastly available for the consumers on their devices but due to its performance and ability to converse effectively, it has gained a huge popularity amongst research as well as industrial community. Recently, many studies have been published to show the effectiveness, efficiency, integration, and sentiments of chatGPT and other LLMs. In contrast, this study focuses on the important aspects that are mostly overlooked, i.e. sustainability, privacy, digital divide, and ethics and suggests that not only chatGPT but every subsequent entry in the category of conversational bots should undergo Sustainability, PrivAcy, Digital divide, and Ethics (SPADE) evaluation. This paper discusses in detail the issues and concerns raised over chatGPT in line with aforementioned characteristics. We also discuss the recent EU AI Act briefly in accordance with the SPADE evaluation. We support our hypothesis by some preliminary data collection and visualizations along with hypothesized facts. We also suggest mitigations and recommendations for each of the concerns. Furthermore, we also suggest some policies and recommendations for EU AI policy act concerning ethics, digital divide, and sustainability.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.078Z

A Careful Examination of Large Language Model Performance on Grade School Arithmetic - Zhang, Hugh; Da, Jeff; Lee, Dean; Robinson, Vaughn; Wu, Catherine; Song, Will; Zhao, Tiffany; Raja, Pranav; Slack, Dylan; Lyu, Qin; Hendryx, Sean; Kaplan, Russell; et al. (arXiv, 2024)
Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with several families of models (e.g., Phi and Mistral) showing evidence of systematic overfitting across almost all model sizes. At the same time, many models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show minimal signs of overfitting. Further analysis suggests a positive relationship (Spearman's r2=0.32) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that many models may have partially memorized GSM8k.
arXiv:2405.00332 [cs] <标题>: 对大语言模型在小学算术中的表现进行仔细审查 <AI Smry>: GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning, and evaluates leading open- and closed-source LLMs on GSM1k, observing accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting. Read_Status: New Read_Status_Date: 2026-01-26T11:32:22.613Z
Keywords: Educational Area: Student Learning; Evidence: Improve a process; Location: Europe; Research Design: Other Research Design; Research Method: No research methods used; Source: Open Source

Some military experts are wary of generative AI - Heath, Ryan (2024)
The Pentagon is hitting the brakes on the new technology even as business is charging forward.
<标题>: 一些军事专家对生成式人工智能持谨慎态度 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Investigating Automatic Scoring and Feedback using Large Language Models - Katuka, Gloria Ashiya; Gain, Alexander; Yu, Yen-Yun (arXiv, 2024)
Automatic grading and feedback have been long studied using traditional machine learning and deep learning techniques using language models. With the recent accessibility to high performing large language models (LLMs) like LLaMA-2, there is an opportunity to investigate the use of these LLMs for automatic grading and feedback generation. Despite the increase in performance, LLMs require significant computational resources for fine-tuning and additional specific adjustments to enhance their performance for such tasks. To address these issues, Parameter Efficient Fine-tuning (PEFT) methods, such as LoRA and QLoRA, have been adopted to decrease memory and computational requirements in model fine-tuning. This paper explores the efficacy of PEFT-based quantized models, employing classification or regression head, to fine-tune LLMs for automatically assigning continuous numerical grades to short answers and essays, as well as generating corresponding feedback. We conducted experiments on both proprietary and open-source datasets for our tasks. The results show that prediction of grade scores via finetuned LLMs are highly accurate, achieving less than 3% error in grade percentage on average. For providing graded feedback fine-tuned 4-bit quantized LLaMA-2 13B models outperform competitive base models and achieve high similarity with subject matter expert feedback in terms of high BLEU and ROUGE scores and qualitatively in terms of feedback. The findings from this study provide important insights into the impacts of the emerging capabilities of using quantization approaches to fine-tune LLMs for various downstream tasks, such as automatic short answer scoring and feedback generation at comparatively lower costs and latency.
arXiv:2405.00602 [cs] <标题>: 使用大型语言模型研究自动评分与反馈 Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.004Z
Keywords: Computer Science - Computation and Language; Computer Science - Machine Learning

Math Multiple Choice Question Generation via Human-Large Language Model Collaboration - Lee, Jaewook; Smith, Digory; Woodhead, Simon; Lan, Andrew (arXiv, 2024)
Multiple choice questions (MCQs) are a popular method for evaluating students’ knowledge due to their efficiency in administration and grading. Crafting high-quality math MCQs is a labor-intensive process that requires educators to formulate precise stems and plausible distractors. Recent advances in large language models (LLMs) have sparked interest in automating MCQ creation, but challenges persist in ensuring mathematical accuracy and addressing student errors. This paper introduces a prototype tool designed to facilitate collaboration between LLMs and educators for streamlining the math MCQ generation process. We conduct a pilot study involving math educators to investigate how the tool can help them simplify the process of crafting high-quality math MCQs. We found that while LLMs can generate well-formulated question stems, their ability to generate distractors that capture common student errors and misconceptions is limited. Nevertheless, a human-AI collaboration has the potential to enhance the efficiency and effectiveness of MCQ generation.
arXiv:2405.00864 [cs] <标题>: 通过人类与大型语言模型合作生成数学多项选择题 Read_Status: New Read_Status_Date: 2026-01-26T11:32:19.517Z
Keywords: Educational Area: Assessment Tools & Processes; Evidence: Saving Teacher Time; Location: Americas; Research Design: Other Research Design; Research Method: Mixed Methods; Source: Open Source

The Routledge international handbook of automated essay evaluation - Shermis, Mark D.; Wilson, Joshua (Routledge, 2024)
"The Routledge International Handbook of Automated Essay Evaluation (AEE) is a definitive guide at the intersection of automation, artificial intelligence, and education. This volume encapsulates the ongoing advancement of AEE, reflecting its application in both large-scale and classroom-based assessments to support teaching and learning endeavours"–
<标题>: 《劳特利奇自动化作文评估国际手册》 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z
Keywords: Artificial intelligence; Composition (Language arts); Data processing; Educational applications; Educational tests and measurements; Essay; Essays; Evaluation; Grading and marking (Students)

PARIKSHA: A Scalable, Democratic, Transparent Evaluation Platform for Assessing Indic Large Language Models - Watts, Ishaan; Gumma, Varun; Yadavalli, Aditya; Seshadri, Vivek; Manohar, Swami; Sitaram, Sunayana (2024)
Evaluation of multilingual Large Language Models (LLMs) is challenging due to a variety of factors – the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and the lack of local, cultural nuances in translated benchmarks. Hence, it is difficult to do extensive evaluation of LLMs in the multilingual […]
<标题>: PARIKSHA：一个可扩展、民主、透明的评估平台，用于评估印地语大型语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: Educational Area: Language of Instruction; Evidence: Improve a process; Location: Asia; Research Design: Other Research Design; Research Method: Mixed Methods; Source: Open Source; ⛔ No DOI found

AI, the new wingman of development (The World Bank) - Dixit, Siddharth; Gill, Indermit S (2024)
The study explores the potential benefits of implementing artificial intelligence (AI) in seven development sectors that receive significant funding from the World Bank. The study provides an overview of the challenges faced by these sectors, including agriculture, healthcare, education, finance, energy, infrastructure, and data. The findings reveal that AI can expedite the achievement of development goals in most of these sectors. The study shows that many organizations already utilize AI to address development challenges in diverse fields. AI is seen as a crop whisperer in agriculture that can help small farmers in low-income and lower-middle-income countries increase their crop productivity. AI can act as a doctor's sidekick in healthcare by assisting healthcare professionals in providing effective and efficient health services. In education, AI can revolutionize learning by providing personalized grading and feedback, helping with personalized tutoring, and customizing therapy and learning for children with autism spectrum disorder. In finance, AI can act as a humane banker by providing access to formal credit for those currently excluded. AI can act as an energy and infrastructure wizard by improving energy efficiency, traffic management, road infrastructure monitoring, power grid monitoring, and water infrastructure monitoring. Finally, AI can act as a data weaver by assisting with improving poverty statistics. The study suggests that the World Bank should promote and finance projects and organizations that use AI to tackle some of the most challenging development problems.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.102Z
Keywords: Educational Area: Assessment Tools & Processes; Educational Area: Student Learning; Evidence: Improve a process; Evidence: Saving Teacher Time; Location: Worldwide; Research Design: Other Research Design; Research Method: No research methods used; Source: Open Source

The Emergence of AI in Educational Settings: The Transformative Influence of Artificial Intelligence on Learning Process - Rachmi, Rachmi (Education Studies and Teaching Journal (EDUTECH), 2024)
<标题>: 人工智能在教育环境中的出现：人工智能对学习过程的变革性影响 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: Educational Area: Assessment Tools & Processes; Educational Area: Student Learning; Evidence: Self-reported Student Behaviour/ Attitude; Evidence: Student Learning Outcomes; Location: Asia; Research Design: Systematic Review; Research Method: Qualitative; Source: Open Source; ⛔ No DOI found

The Neglected 15%: Positive Effects of Hybrid Human-AI Tutoring Among Students with Disabilities - Thomas, Danielle R.; Gatz, Erin; Gupta, Shivang; Aleven, Vincent; Koedinger, Kenneth R. (OSF, 2024)
Incorporating human tutoring with AI holds promise for supporting diverse math learners. In the U.S., approximately 15% of students receive special education services, with limited previous research within AIED on the impact of AI-assisted learning among students with disabilities. Previous work combining human tutors and AI suggests that students with lower prior knowledge, such as lacking basic skills, exhibit greater learning gains compared to their more knowledgeable peers. Building upon this finding, we hypothesize that hybrid human-AI tutoring will have positive effects among students in inclusive settings, classrooms containing both students with and without identified disabilities. To investigate this hypothesis, we conduct a two-study quasi-experiment across two urban, low-income middle schools: one in Pennsylvania with 362 students and another in California comprised of 733 students, involving 27% and 16% of students with disabilities, respectively. Our findings indicate hybrid human-AI tutoring has positive effects on learning processes and outcomes among all students, including students with disabilities. The motivational benefits of the human tutoring treatment over a control group using solely math software show up in greater increases in practice and skill proficiency during math software use for students with disabilities than other students. In addition, the treatment yields significantly higher pre-to-post learning gains for all including students with disabilities whose relative gains trend higher and are statistically at least as high. These compelling findings support the promise of hybrid human-AI solutions and emphasize the collaborative focus among AIED to support diverse learners.
<标题>: 被忽视的15%：混合人类与人工智能辅导对残障学生的积极影响 Read_Status: New Read_Status_Date: 2026-01-26T11:32:23.432Z
Keywords: /unread; Educational Area: Inclusion; Educational Area: Student Learning; Evidence: Student Learning Outcomes; Human-AI tutoring; Learning analytics; Location: Americas; Research Design: Quasi-experimental design; Research Method: Quantitative; Source: Open Source; equity; inclusion; math; tutoring

The Ethics of Advanced AI Assistants - Gabriel, Iason; Manzini, Arianna; Keeling, Geoff; Hendricks, Lisa Anne; Rieser, Verena; Iqbal, Hasan; Tomašev, Nenad; Ktena, Ira; Kenton, Zachary; Rodriguez, Mikel; El-Sayed, Seliem; Brown, Sasha; et al. (arXiv, 2024)
This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user, across one or more domains, in line with the user's expectations. The paper starts by considering the technology itself, providing an overview of AI assistants, their technical foundations and potential range of applications. It then explores questions around AI value alignment, well-being, safety and malicious uses. Extending the circle of inquiry further, we next consider the relationship between advanced AI assistants and individual users in more detail, exploring topics such as manipulation and persuasion, anthropomorphism, appropriate relationships, trust and privacy. With this analysis in place, we consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants. Finally, we conclude by providing a range of recommendations for researchers, developers, policymakers and public stakeholders.
arXiv:2404.16244 [cs] <标题>: 高级 AI 助手的道德规范 <AI Smry>: The paper starts by considering the technology itself, providing an overview of AI assistants, their technical foundations and potential range of applications, then explores questions around AI value alignment, well-being, safety and malicious uses, and considers the deployment of advanced assistants at a societal scale. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computers and Society

geval/README.md at main · nlpyang/geval (GitHub, 2024)
Code for paper "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment" - nlpyang/geval
<标题>: geval/README.md 在主分支 · nlpyang/geval Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails - Chowdhury, Sankalan Pal; Zouhar, Vilém; Sachan, Mrinmaya (arXiv, 2024)
Large Language Models (LLMs) have found several use cases in education, ranging from automatic question generation to essay evaluation. In this paper, we explore the potential of using Large Language Models (LLMs) to author Intelligent Tutoring Systems. A common pitfall of LLMs is their straying from desired pedagogical strategies such as leaking the answer to the student, and in general, providing no guarantees. We posit that while LLMs with certain guardrails can take the place of subject experts, the overall pedagogical design still needs to be handcrafted for the best learning results. Based on this principle, we create a sample end-to-end tutoring system named MWPTutor, which uses LLMs to fill in the state space of a pre-defined finite state transducer. This approach retains the structure and the pedagogy of traditional tutoring systems that has been developed over the years by learning scientists but brings in additional flexibility of LLM-based approaches. Through a human evaluation study on two datasets based on math word problems, we show that our hybrid approach achieves a better overall tutoring score than an instructed, but otherwise free-form, GPT-4. MWPTutor is completely modular and opens up the scope for the community to improve its performance by improving individual modules or using different teaching strategies that it can follow.
arXiv:2402.09216 [cs] <标题>: AutoTutor 遇见大型语言模型：具有丰富教学法和安全防护的语言模型导师 Read_Status: New Read_Status_Date: 2026-01-26T11:32:23.432Z
Keywords: Educational Area: Student Learning; Evidence: Saving Teacher Time; Location: Europe; Research Design: Other Research Design; Research Method: Mixed Methods; Source: Open Source

The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances - Nie, Allen; Chandak, Yash; Suzara, Miroslav; Ali, Malika; Woodrow, Juliette; Peng, Matt; Sahami, Mehran; Brunskill, Emma; Piech, Chris (2024)
Large language models (LLMs) are quickly being adopted in a wide range of learning experiences, especially via ubiquitous and broadly accessible chat interfaces like ChatGPT and Copilot. This type of interface is readily available to students and teachers around the world, yet relatively little research has been done to assess the impact of such generic tools on student learning. Coding education is an interesting test case, both because LLMs have strong performance on coding tasks, and because LLM-powered support tools are rapidly becoming part of the workflow of professional software engineers. To help understand the impact of generic LLM use on coding education, we conducted a large-scale randomized control trial with 5,831 students from 146 countries in an online coding class in which we provided some students with access to a chat interface with GPT-4. We estimate positive benefits on exam performance for adopters, the students who used the tool, but over all students, the advertisement of GPT-4 led to a significant average decrease in exam participation. We observe similar decreases in other forms of course engagement. However, this decrease is modulated by the student's country of origin. Offering access to LLMs to students from low human development index countries increased their exam participation rate on average. Our results suggest there may be promising benefits to using LLMs in an introductory coding class, but also potential harms for engagement, which makes their longer term impact on student success unclear. Our work highlights the need for additional investigations to help understand the potential impact of future adoption and integration of LLMs into classrooms.
<标题>: GPT 惊喜：在大规模编程课程中提供大型语言模型聊天降低了参与度但提高了采用者的考试表现 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.303Z

In-Context Learning for Scalable and Online Hallucination Detection in RAGS - Albanese, Nicolò Cosimo (Software Engineering, 2024)
Ensuring fidelity to source documents is crucial for the responsible use of Large Language Models (LLMs) in Retrieval Augmented Generation (RAG) systems. We propose a lightweight method for real-time hallucination detection, with potential to be deployed as a model-agnostic microservice to bolster reliability. Using in-context learning, our approach evaluates response factuality at the sentence level without annotated data, promoting transparency and user trust. Compared to other prompt-based and semantic similarity baselines from recent literature, our method improves hallucination detection F1 scores by at least 11%, with consistent performance across different models. This research offers a practical solution for real-time validation of response accuracy in RAG systems, fostering responsible adoption, especially in critical domains where document fidelity is paramount.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z
Keywords: /unread

In-Context Learning for Scalable and Online Hallucination Detection in RAGS - Albanese, Nicolò Cosimo (Software Engineering, 2024)
Ensuring fidelity to source documents is crucial for the responsible use of Large Language Models (LLMs) in Retrieval Augmented Generation (RAG) systems. We propose a lightweight method for real-time hallucination detection, with potential to be deployed as a model-agnostic microservice to bolster reliability. Using in-context learning, our approach evaluates response factuality at the sentence level without annotated data, promoting transparency and user trust. Compared to other prompt-based and semantic similarity baselines from recent literature, our method improves hallucination detection F1 scores by at least 11%, with consistent performance across different models. This research offers a practical solution for real-time validation of response accuracy in RAG systems, fostering responsible adoption, especially in critical domains where document fidelity is paramount.
<标题>: 在上下文学习中进行可扩展和在线的RAGS幻觉检测 <AI Smry>: This research offers a practical solution for real-time validation of response accuracy in RAG systems, fostering responsible adoption, especially in critical domains where document fidelity is paramount. Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Using Artificial Intelligence Tools in K-12 Classrooms - Diliberti, Melissa Kay; Schwartz, Heather L; Doan, Sy; Shapiro, Anna; Rainey, Lydia R; Lake, Robin J (RAND, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

Laissez-Faire Harms: Algorithmic Biases in Generative Language Models - Shieh, Evan; Vassel, Faye-Marie; Sugimoto, Cassidy; Monroe-White, Thema (arXiv, 2024)
The rapid deployment of generative language models (LMs) has raised concerns about social biases affecting the well-being of diverse consumers. The extant literature on generative LMs has primarily examined bias via explicit identity prompting. However, prior research on bias in earlier language-based technology platforms, including search engines, has shown that discrimination can occur even when identity terms are not specified explicitly. Studies of bias in LM responses to open-ended prompts (where identity classifications are left unspecified) are lacking and have not yet been grounded in end-consumer harms. Here, we advance studies of generative LM bias by considering a broader set of natural use cases via open-ended prompting. In this "laissez-faire" setting, we find that synthetically generated texts from five of the most pervasive LMs (ChatGPT3.5, ChatGPT4, Claude2.0, Llama2, and PaLM2) perpetuate harms of omission, subordination, and stereotyping for minoritized individuals with intersectional race, gender, and/or sexual orientation identities (AI/AN, Asian, Black, Latine, MENA, NH/PI, Female, Non-binary, Queer). We find widespread evidence of bias to an extent that such individuals are hundreds to thousands of times more likely to encounter LM-generated outputs that portray their identities in a subordinated manner compared to representative or empowering portrayals. We also document a prevalence of stereotypes (e.g. perpetual foreigner) in LM-generated outputs that are known to trigger psychological harms that disproportionately affect minoritized individuals. These include stereotype threat, which leads to impaired cognitive performance and increased negative self-perception. Our findings highlight the urgent need to protect consumers from discriminatory harms caused by language models and invest in critical AI education programs tailored towards empowering diverse consumers.
arXiv:2404.07475 [cs] <标题>: 放任政策的危害：生成式语言模型中的算法偏差 Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.602Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Computers and Society; Computer Science - Machine Learning

Can Large Language Models Automatically Score Proficiency of Written Essays? - Mansour, Watheq; Albatarni, Salam; Eltanbouly, Sohaila; Elsayed, Tamer (arXiv, 2024)
Although several methods were proposed to address the problem of automated essay scoring (AES) in the last 50 years, there is still much to desire in terms of effectiveness. Large Language Models (LLMs) are transformer-based models that demonstrate extraordinary capabilities on various tasks. In this paper, we test the ability of LLMs, given their powerful linguistic knowledge, to analyze and effectively score written essays. We experimented with two popular LLMs, namely ChatGPT and Llama. We aim to check if these models can do this task and, if so, how their performance is positioned among the state-of-the-art (SOTA) models across two levels, holistically and per individual writing trait. We utilized prompt-engineering tactics in designing four different prompts to bring their maximum potential to this task. Our experiments conducted on the ASAP dataset revealed several interesting observations. First, choosing the right prompt depends highly on the model and nature of the task. Second, the two LLMs exhibited comparable average performance in AES, with a slight advantage for ChatGPT. Finally, despite the performance gap between the two LLMs and SOTA models in terms of predictions, they provide feedback to enhance the quality of the essays, which can potentially help both teachers and students.
arXiv:2403.06149 [cs] <标题>: 大型语言模型能否自动评估书面文章的语言能力？ Read_Status: New Read_Status_Date: 2026-01-26T11:32:28.375Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence (First Monday, 2024)
The stated goal of many organizations in the field of artificial intelligence (AI) is to develop artificial general intelligence (AGI), an imagined system with more intelligence than anything we have ever seen. Without seriously questioning whether such a system can and should be built, researchers are working to create “safe AGI” that is “beneficial for all of humanity.” We argue that, unlike systems with specific applications which can be evaluated following standard engineering principles, undefined systems like “AGI” cannot be appropriately tested for safety. Why, then, is building AGI often framed as an unquestioned goal in the field of AI? In this paper, we argue that the normative framework that motivates much of this goal is rooted in the Anglo-American eugenics tradition of the twentieth century. As a result, many of the very same discriminatory attitudes that animated eugenicists in the past (e.g., racism, xenophobia, classism, ableism, and sexism) remain widespread within the movement to build AGI, resulting in systems that harm marginalized groups and centralize power, while using the language of “safety” and “benefiting humanity” to evade accountability. We conclude by urging researchers to work on defined tasks for which we can develop safety protocols, rather than attempting to build a presumably all-knowing system such as AGI.
<AI Smry>: It is argued that the normative framework that motivates much of this goal is rooted in the Anglo-American eugenics tradition of the twentieth century, resulting in systems that harm marginalized groups and centralize power, while using the language of “safety” and “benefiting humanity” to evade accountability. <标题>: TESCREAL 套餐：优生学与通过通用人工智能实现乌托邦的承诺 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

Combining machine translation and automated scoring in international large-scale assessments - Jung, Ji Yoon; Tyack, Lillian; von Davier, Matthias (Large-scale Assessments in Education, 2024)
Artificial intelligence (AI) is rapidly changing communication and technology-driven content creation and is also being used more frequently in education. Despite these advancements, AI-powered automated scoring in international large-scale assessments (ILSAs) remains largely unexplored due to the scoring challenges associated with processing large amounts of multilingual responses. However, due to their low-stakes nature, ILSAs are an ideal ground for innovations and exploring new methodologies.
<标题>: 在国际大规模评估中结合机器翻译与自动评分 <AI Smry>: This study highlights that automated scoring integrated with the recent machine translation holds great promise for consistent and resource-efficient scoring in ILSAs and shows comparable performance to human scoring. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: AI: ChatGPT; Educational Area: Assessment Tools & Processes; Evidence: Gaps in Current Research; Evidence: Improve a process; Location: Asia; Location: Europe; Research Design: Non-experimental design; Research Method: Quantitative; Source: Open Source

Combining machine translation and automated scoring in international large-scale assessments - Jung, Ji Yoon; Tyack, Lillian; Von Davier, Matthias (Large-scale Assessments in Education, 2024)
Artificial intelligence (AI) is rapidly changing communication and technology-driven content creation and is also being used more frequently in education. Despite these advancements, AI-powered automated scoring in international large-scale assessments (ILSAs) remains largely unexplored due to the scoring challenges associated with processing large amounts of multilingual responses. However, due to their low-stakes nature, ILSAs are an ideal ground for innovations and exploring new methodologies.
<标题>: 在国际大规模评估中结合机器翻译与自动评分 <AI Smry>: This study highlights that automated scoring integrated with the recent machine translation holds great promise for consistent and resource-efficient scoring in ILSAs and shows comparable performance to human scoring. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Brainy: An Innovative Context-Aware Generative AI Engine for Education - Nahas, Elie; Diab, Paul Barakat; Atechian, Talar (The Southeast Asian Conference on Education 2024: Official Conference Proceedings, 2024)
In the realm of education, the integration of Generative AI tools within learning platforms has transformed teaching and learning paradigms. This paper introduces Brainy, a context-aware AI engine integrated into the Augmental Adaptive Learning platform, which implements the concept of Differentiated Instruction. At its core, Brainy employs Generative AI Models, leveraging a variety of Language Learning Models (LLMs), including OpenAI. This integration enables Brainy to generate personalized course materials and assessments tailored to individual learner needs, in addition to offering multiple learning paths. By considering learning objectives, prior knowledge, and learner proficiency levels, Brainy ensures an adaptive learning experience that promotes equity in education. It refines content and learning activities, adjusts to accommodate each learner’s learning style and progression and offers feedback on assessments using a supportive educational approach, including clear explanations and analogies that address individual learning needs. This integration not only elevates the learning experience for learners but also empowers Instructors to have complete control to review the AI-generated content, ensuring an alignment with educational goals and principles and a balance between personalization and standardization, essential to maintain integrity and effectiveness of the educational process. In summary, Brainy offers differentiation in terms of content, process, and result assessment, tailoring content to individual learner needs, adapting the learning process to diverse learning styles, and providing personalized feedback that enhances the learning experience for all learners. The implementation results confirm Brainy's personalized learning approach. However, experiments and data analysis are in perspective to better understand the solution's performance in terms of efficacy and accuracy. This involves conducting a more in-depth assessment to evaluate the impact of the solution on learners' learning outcomes.
<标题>: Brainy：面向教育的创新上下文感知生成式人工智能引擎 <AI Smry>: Brainy, a context-aware AI engine integrated into the Augmental Adaptive Learning platform, which implements the concept of Differentiated Instruction, offers differentiation in terms of content, process, and result assessment, tailoring content to individual learner needs, adapting the learning process to diverse learning styles, and providing personalized feedback that enhances the learning experience for all learners. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: Educational Area: Assessment Tools & Processes; Educational Area: Quality Teaching/Learning Materials; Educational Area: Student Learning; Evidence: Improve a process; Location: Americas; Location: Asia; Research Design: Case Study; Research Method: Qualitative; Source: Open Source

“We’re changing the system with this one”: Black students using critical race algorithmic literacies to subvert and survive AI-mediated racism in school - Tanksley, Tiera Chante (English Teaching: Practice & Critique, 2024)
Purpose This paper aims to center the experiences of three cohorts ( n = 40) of Black high school students who participated in a critical race technology course that exposed anti-blackness as the organizing logic and default setting of digital and artificially intelligent technology. This paper centers the voices, experiences and technological innovations of the students, and in doing so, introduces a new type of digital literacy: critical race algorithmic literacy. Design/methodology/approach Data for this study include student interviews (called “talk backs”), journal reflections and final technology presentations. Findings Broadly, the data suggests that critical race algorithmic literacies prepare Black students to critically read the algorithmic word (e.g. data, code, machine learning models, etc.) so that they can not only resist and survive, but also rebuild and reimagine the algorithmic world. Originality/value While critical race media literacy draws upon critical race theory in education – a theorization of race, and a critique of white supremacy and multiculturalism in schools – critical race algorithmic literacy is rooted in critical race technology theory, which is a theorization of blackness as a technology and a critique of algorithmic anti-blackness as the organizing logic of schools and AI systems.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

How Tech Giants Cut Corners to Harvest Data for A.I. - Metz, Cade; Kang, Cecilia; Frenkel, Sheera; Thompson, Stuart A.; Grant, Nico (The New York Times, 2024)
OpenAI, Google and Meta ignored corporate policies, altered their own rules and discussed skirting copyright law as they sought online information to train their newest artificial intelligence systems.
<标题>: 科技巨头如何偷工减料以收集人工智能数据 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread; Altman; Samuel H; Artificial Intelligence; Books and Literature; Brockman; Greg; ChatGPT; Computers and the Internet; Copyrights and Copyright Violations; Corporate Social Responsibility; Data-Mining and Database Marketing; Facebook Inc; Google Inc; Instagram Inc; Meta Platforms Inc; Mobile Applications; OpenAI Labs; Privacy; Social Media; Video Recordings; Downloads and Streaming; YouTube.com; Zuckerberg; Mark E; audio-neutral-informative

Designing Child-Centric AI Learning Environments: Insights from LLM-Enhanced Creative Project-Based Learning - Zha, Siyu; Qiao, Yuehan; Hu, Qingyu; Li, Zhongsheng; Gong, Jiangtao; Xu, Yingqing (arXiv, 2024)
Project-based learning (PBL) is an instructional method that is very helpful in nurturing students' creativity, but it requires significant time and energy from both students and teachers. Large language models (LLMs) have been proven to assist in creative tasks, yet much controversy exists regarding their role in fostering creativity. This paper explores the potential of LLMs in PBL settings, with a special focus on fostering creativity. We began with an exploratory study involving 12 middle school students and identified five design considerations for LLM applications in PBL. Building on this, we developed an LLM-empowered, 48-hour PBL program and conducted an instructional experiment with 31 middle school students. Our results indicated that LLMs can enhance every stage of PBL. Additionally, we also discovered ambivalent perspectives among students and mentors toward LLM usage. Furthermore, we explored the challenge and design implications of integrating LLMs into PBL and reflected on the program. By bridging AI advancements into educational practice, our work aims to inspire further discourse and investigation into harnessing AI's potential in child-centric educational settings.
arXiv:2403.16159 [cs] <标题>: 以儿童为中心的人工智能学习环境设计：基于大型语言模型增强的创意项目式学习的见解 <AI Smry>: The results indicated that LLMs can enhance every stage of PBL, and bridging AI advancements into educational practice aims to inspire further discourse and investigation into harnessing AI's potential in child-centric educational settings. Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Educational Area: Quality Teaching/Learning Materials; Educational Area: Student Learning; Evidence: Improve a process; Evidence: Saving Teacher Time; Location: Americas; Location: Asia; Research Design: Interview/ focus group; Research Design: Qualitative Exploratory; Research Method: Mixed Methods; Source: Open Source

Artificial Intelligence and Curriculum Implementation in Public Secondary Schools of Federal Capital Territory, Abuja, Nigeria - Mohammed, Ahmed; Bashir, Umar Faiza; Ismail, Abubakar Abdullahi (Innovative: International Multidisciplinary Journal of Applied Technology (2995-486X), 2024)
The study assessed the impact of artificial intelligence on curriculum implementation in public secondary schools in Federal Capital territory, Abuja, Nigeria. The research design used for the study is descriptive survey. The population of the study comprises of the all the teachers in public secondary schools in FCT. The sample for the study is 320 respondents. The researcher formulated a questionnaire titled Artificial Intelligence on Curriculum Implementation Questionnaire (AICIQ). The instrument was subjected to face and content validation using two lecturers from Educational Management department university Abuja. Reliability of the instrument was determined using test retest method. The first and second results obtained from the instrument were correlated and it yielded a coefficient of 0.86 which is high enough for the study. Descriptive statistics was used to analyze data collected for the research questions and hypothesis formulated for the study. The findings of the study revealed that artificial intelligence aided the implementation of school curriculum in the public secondary schools in FCT. The findings also revealed that shortage of Al facilities, poor implementation of ICT policies, high cost of Al, unstable power supply, poor internet service, high cost of maintenance and poor funding of Al programme are the problems militating against deployment of Al for effective curriculum implementation in public secondary schools in Federal Capital Territory, Abuja, Nigeria. The findings finally, established that there is positive relationship between artificial intelligences deployment and effective curriculum implementation in the secondary schools. Based on this, the study recommended that the government should increase funding of public secondary schools in FCT to enable adequate provision of Al facilities in all the public secondary schools to support effective curriculum implementation in the schools.
Number: 2 <标题>: 人工智能与尼日利亚联邦首都区阿布贾公立中学的课程实施 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: Educational Area: Quality Curriculum; Evidence: Improve a process; Location: Africa; Research Design: Other Research Design; Research Method: Qualitative; Source: Open Source; ⛔ No DOI found

Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT - Hou, Ruikun; Fütterer, Tim; Bühler, Babette; Bozkir, Efe; Gerjets, Peter; Trautwein, Ulrich; Kasneci, Enkelejda (arXiv, 2024)
Classroom observation protocols standardize the assessment of teaching effectiveness and facilitate comprehension of classroom interactions. Whereas these protocols offer teachers specific feedback on their teaching practices, the manual coding by human raters is resource-intensive and often unreliable. This has sparked interest in developing AI-driven, cost-effective methods for automating such holistic coding. Our work explores a multimodal approach to automatically estimating encouragement and warmth in classrooms, a key component of the Global Teaching Insights (GTI) study's observation protocol. To this end, we employed facial and speech emotion recognition with sentiment analysis to extract interpretable features from video, audio, and transcript data. The prediction task involved both classification and regression methods. Additionally, in light of recent large language models' remarkable text annotation capabilities, we evaluated ChatGPT's zero-shot performance on this scoring task based on transcripts. We demonstrated our approach on the GTI dataset, comprising 367 16-minute video segments from 92 authentic lesson recordings. The inferences of GPT-4 and the best-trained model yielded correlations of r = .341 and r = .441 with human ratings, respectively. Combining estimates from both models through averaging, an ensemble approach achieved a correlation of r = .513, comparable to human inter-rater reliability. Our model explanation analysis indicated that text sentiment features were the primary contributors to the trained model's decisions. Moreover, GPT-4 could deliver logical and concrete reasoning as potential teacher guidelines. Our findings provide insights into using advanced, multimodal techniques for automated classroom observation, aiming to foster teacher training through frequent and valuable feedback.
arXiv:2404.15310 [cs] <标题>: 利用多模态情感特征和ChatGPT进行课堂鼓励与温暖的自动评估 Read_Status: New Read_Status_Date: 2026-01-26T11:32:27.022Z
Keywords: AI: ChatGPT; Educational Area: Leadership of T&L; Educational Area: Positive Learning Environment; Educational Area: Teacher Learning; Evidence: Improve a process; Location: Europe; Research Design: Other Research Design; Research Method: Qualitative; Source: Open Source

Contextual evaluation of LLM’s performance on primary education science learning contents in the Yoruba language - Lawal, Olanrewaju; Soronnadi, Anthony; Adekanmbi, Olubayo (2024)
In the rapidly evolving era of artificial intelligence, Large Language Models (LLMs) like ChatGPT-3.5, Llama, and PaLM 2 play a pivotal role in reshaping education. Trained on diverse language data with a predominant focus on English, these models exhibit remarkable proficiency in comprehending and generating intricate human language constructs, revolutionizing educational applications. This potential has prompted exploration into personalized and enriched educational experiences, streamlining instructional design to cater to students’ needs. However, the inclusive effectiveness of LLMs in low-resource languages, like Yoruba, poses challenges. This research critically assesses the ability of LLMs, including ChatGPT-3.5, Gemini, and PaLM 2, to comprehend and generate contextually relevant science education content in Yoruba. The study, conducted across four tasks using a manually developed primary science dataset in Yoruba, reveals a comparative underperformance in various NLP tasks, emphasizing the need for language-specific and domain-specific technologies, particularly for primary science education in low-resource languages.
<标题>: 对大型语言模型在约鲁巴语小学科学学习内容上的上下文表现评估 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: AI: ChatGPT; Educational Area: Quality Teaching/Learning Materials; Evidence: Improve a process; Location: Africa; Research Design: Other Research Design; Research Method: Quantitative; Source: Open Source

Predictors of middle school students’ perceptions of automated writing evaluation - Wilson, Joshua; Zhang, Fan; Palermo, Corey; Cordero, Tania Cruz; Myers, Matthew C.; Eacker, Halley; Potter, Andrew; Coles, Jessica (Computers & Education, 2024)
This study examined middle school students' perceptions of an automated writing evaluation (AWE) system, MI Write. We summarize students' perceptions of MI Write's usability, usefulness, and desirability both quantitatively and qualitatively. We then estimate hierarchical entry regression models that account for district context, classroom climate, demographic factors (i.e., gender, special education status, limited English proficiency status, socioeconomic status, grade), students' writing-related beliefs and affect, and students' writing proficiency as predictors of students' perceptions. Controlling for districts, students reporting more optimal classroom climate also reported higher usability, usefulness, and desirability for MI Write. Also, model results revealed that eighth graders, students with limited English proficiency, and students of lower socioeconomic status perceived MI Write relatively more useable; students with lower socioeconomic status also perceived MI Write relatively more useful and desirable. Students who liked writing more and more strongly believed that writing is a recursive process viewed MI Write as more useable, useful, and desirable. Students with greater writing proficiency viewed MI Write as less useable and useful; writing proficiency was not related to desirability perceptions. We conclude with a discussion of implications and future directions.
<AI Smry>: Examination of middle school students' perceptions of an automated writing evaluation (AWE) system, MI Write, revealed that eighth graders, students with limited English proficiency, and students of lower socioeconomic status perceived MI Write relatively more useable; students with lower socioeconomic status also perceived MI Write relatively more useful and desirable. <标题>: 中学生对自动写作评估感知的预测因素 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread; Automated feedback; Automated writing evaluation; Perceptions; Social validity; Writing

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks - Xie, Yiqing; Xie, Alex; Sheth, Divyanshu; Liu, Pengfei; Fried, Daniel; Rose, Carolyn (arXiv, 2024)
To facilitate evaluation of code generation systems across diverse scenarios, we present CodeBenchGen, a framework to create scalable execution-based benchmarks that only requires light guidance from humans. Specifically, we leverage a large language model (LLM) to convert an arbitrary piece of code into an evaluation example, including test cases for execution-based evaluation. We illustrate the usefulness of our framework by creating a dataset, Exec-CSN, which includes 1,931 examples involving 293 libraries revised from code in 367 GitHub repositories taken from the CodeSearchNet dataset. To demonstrate the complexity and solvability of examples in Exec-CSN, we present a human study demonstrating that 81.3% of the examples can be solved by humans and 61% are rated as ``requires effort to solve''. We conduct code generation experiments on open-source and proprietary models and analyze the performance of both humans and models. We will release the code of both the framework and the dataset upon acceptance.
arXiv:2404.00566 [cs] <标题>: CodeBenchGen：创建可扩展的基于执行的代码生成基准 Read_Status: New Read_Status_Date: 2026-01-26T11:32:31.459Z
Keywords: /unread; Computer Science - Computation and Language; Computer Science - Software Engineering

A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science - Cohn, Clayton; Hutchins, Nicole; Le, Tuan; Biswas, Gautam (Proceedings of the AAAI Conference on Artificial Intelligence, 2024)
This paper explores the use of large language models (LLMs) to score and explain short-answer assessments in K-12 science. While existing methods can score more structured math and computer science assessments, they often do not provide explanations for the scores. Our study focuses on employing GPT-4 for automated assessment in middle school Earth Science, combining few-shot and active learning with chain-of-thought reasoning. Using a human-in-the-loop approach, we successfully score and provide meaningful explanations for formative assessment responses. A systematic analysis of our method's pros and cons sheds light on the potential for human-in-the-loop techniques to enhance automated grading for open-ended science assessments.
arXiv:2403.14565 [cs] <标题>: 使用大型语言模型的链式思维提示方法评估学生的科学形成性评估回答 <AI Smry>: This study focuses on employing GPT-4 for automated assessment in middle school Earth Science, combining few-shot and active learning with chain-of-thought reasoning and a human-in-the-loop approach. Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z
Keywords: Computer Science - Computation and Language

Towards Building a Language-Independent Speech Scoring Assessment - Gupta, Shreyansh; Unnam, Abhishek; Yadav, Kuldeep; Aggarwal, Varun (Proceedings of the AAAI Conference on Artificial Intelligence, 2024)
Automatic speech scoring is crucial in language learning, providing targeted feedback to language learners by assessing pronunciation, fluency, and other speech qualities. However, the scarcity of human-labeled data for languages beyond English poses a significant challenge in developing such systems. In this work, we propose a Language-Independent scoring approach to evaluate speech without relying on labeled data in the target language. We introduce a multilingual speech scoring system that leverages representations from the wav2vec 2.0 XLSR model and a force-alignment technique based on CTC-Segmentation to construct speech features. These features are used to train a machine learning model to predict pronunciation and fluency scores. We demonstrate the potential of our method by predicting expert ratings on a speech dataset spanning five languages - English, French, Spanish, German and Portuguese, and comparing its performance against Language-Specific models trained individually on each language, as well as a jointly-trained model on all languages. Results indicate that our approach shows promise as an initial step towards a universal language independent speech scoring.
<标题>: 迈向构建一种语言无关的语音评分评估 <AI Smry>: A multilingual speech scoring system that leverages representations from the wav2vec 2.0 XLSR model and a force-alignment technique based on CTC-Segmentation to construct speech features and shows promise as an initial step towards a universal language independent speech scoring. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Researchers at Stanford University Introduce 'pyvene': An Open-Source Python Library that Supports Intervention-Based Research on Machine Learning Models - Ganaie, Muhammad Athar (2024)
Understanding and manipulating neural models is essential in the evolving field of AI. This necessity stems from various applications, from refining models for enhanced robustness to unraveling their decision-making processes for greater interpretability. Amidst this backdrop, the Stanford University research team has introduced 'pyvene,' a groundbreaking open-source Python library that facilitates intricate interventions on PyTorch models. pyvene is ingeniously designed to overcome the limitations posed by existing tools, which often need more flexibility, extensibility, and user-friendliness. At the heart of pyvene's innovation is its configuration-based approach to interventions. This method departs from traditional, code-executed interventions, offering a more intuitive and
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Automated Scoring of Translations with BERT Models: Chinese and English Language Case Study - Cui, Yizhuo; Liang, Maocheng (Applied Sciences, 2024)
With the wide application of artificial intelligence represented by deep learning in natural language-processing tasks, the automated scoring of translations has also advanced and improved. This study aims to determine if the BERT-assist system can reliably assess translation quality and identify high-quality translations for potential recognition. It takes the Han Suyin International Translation Contest as a case study, which is a large-scale and influential translation contest in China, with a history of over 30 years. The experimental results show that the BERT-assist system is a reliable second rater for massive translations in terms of translation quality, as it can effectively sift out high-quality translations with a reliability of r = 0.9 or higher. Thus, the automated translation scoring system based on BERT can satisfactorily predict the ranking of translations according to translation quality and sift out high-quality translations potentially shortlisted for prizes.
<标题>: 使用BERT模型进行翻译自动评分：中英文案例研究 <AI Smry>: The experimental results show that the BERT-assist system is a reliable second rater for massive translations in terms of translation quality, as it can effectively sift out high-quality translations with a reliability of r = 0.9 or higher. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z

Microsoft releases its internal generative AI red teaming tool to the public (2024)
PyRIT can generate thousands of malicious prompts to test a gen AI model, and even score its response.
<标题>: 微软向公众发布其内部生成式人工智能红队测试工具 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

An Application of Text Embeddings to Support Alignment of Educational Content Standards - Butterfuss, Reese; Doran, Harold (National Council on Measurement in Education, 2024)
Large language models are increasingly used in educational and psychological measurement activities. Their rapidly evolving sophistication and ability to detect language semantics makethemviable tools to supplement subject matter experts and their reviews of large amounts of text statements, such as educational content standards. This paper presents an application of text embeddings to find relationships between different sets of educational content standards in a process termed content mapping. This content mapping process is routinely used by state education agencies and is often a requirement of the United States Department of Education peer review process. We discuss the educational measurement problem, propose a formal methodology, demonstrate an application of our proposed approach, and provide measures of its accuracy and potential to support real-world activities.
<标题>: 文本嵌入在支持教育内容标准对齐中的应用 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

AI in education is a public problem - Williamson, Ben (2024)
Photo by Mick Haupt on Unsplash Over the past year or so, a narrative that AI will inevitably transform education has become widespread. You can find it in the pronouncements of investors, tech ind…
<标题>: 人工智能在教育中是一个公共问题 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Can AI-Generated Text be Reliably Detected? - Sadasivan, Vinu Sankar; Kumar, Aounon; Balasubramanian, Sriram; Wang, Wenxiao; Feizi, Soheil (arXiv, 2024)
The unregulated use of LLMs can potentially lead to malicious consequences such as plagiarism, generating fake news, spamming, etc. Therefore, reliable detection of AI-generated text can be critical to ensure the responsible use of LLMs. Recent works attempt to tackle this problem either using certain model signatures present in the generated text outputs or by applying watermarking techniques that imprint specific patterns onto them. In this paper, we show that these detectors are not reliable in practical scenarios. In particular, we develop a recursive paraphrasing attack to apply on AI text, which can break a whole range of detectors, including the ones using the watermarking schemes as well as neural network-based detectors, zero-shot classifiers, and retrieval-based detectors. Our experiments include passages around 300 tokens in length, showing the sensitivity of the detectors even in the case of relatively long passages. We also observe that our recursive paraphrasing only degrades text quality slightly, measured via human studies, and metrics such as perplexity scores and accuracy on text benchmarks. Additionally, we show that even LLMs protected by watermarking schemes can be vulnerable against spoofing attacks aimed to mislead detectors to classify human-written text as AI-generated, potentially causing reputational damages to the developers. In particular, we show that an adversary can infer hidden AI text signatures of the LLM outputs without having white-box access to the detection method. Finally, we provide a theoretical connection between the AUROC of the best possible detector and the Total Variation distance between human and AI text distributions that can be used to study the fundamental hardness of the reliable detection problem for advanced language models. Our code is publicly available at https://github.com/vinusankars/Reliability-of-AI-text-detectors.
arXiv:2303.11156 [cs] <AI Smry>: This paper introduces recursive paraphrasing attack to stress test the robustness of AI text detectors in the presence of an attacker, and provides a theoretical framework connecting the AUROC of the best possible detector to the Total Variation distance between human and AI text distributions. <标题>: 人工智能生成的文本能被可靠地检测吗？ Read_Status: New Read_Status_Date: 2026-01-26T11:32:18.852Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations - Wang, Peiyi; Li, Lei; Shao, Zhihong; Xu, R. X.; Dai, Damai; Li, Yifei; Chen, Deli; Wu, Y.; Sui, Zhifang (arXiv, 2024)
In this paper, we present an innovative process-oriented math process reward model called \textbf{Math-Shepherd}, which assigns a reward score to each step of math problem solutions. The training of Math-Shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-Shepherd in two scenarios: 1) \textit{Verification}: Math-Shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) \textit{Reinforcement Learning}: Math-Shepherd is employed to reinforce LLMs with step-by-step Proximal Policy Optimization (PPO). With Math-Shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, the step-by-step PPO with Math-Shepherd significantly improves the accuracy of Mistral-7B (77.9\%$\to$84.1\% on GSM8K and 28.6\%$\to$33.0\% on MATH). The accuracy can be further enhanced to 89.1\% and 43.5\% on GSM8K and MATH with the verification of Math-Shepherd, respectively. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.
arXiv:2312.08935 [cs] <AI Smry>: An innovative process-oriented math process reward model called Math-Shepherd, which assigns a reward score to each step of math problem solutions, which holds significant potential for the future evolution of LLMs. <标题>: 数学牧羊人：在无需人工标注的情况下逐步验证和强化大型语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:32:21.865Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning; math

Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation - Wang, Siyuan; Long, Zhuohan; Fan, Zhihao; Wei, Zhongyu; Huang, Xuanjing (arXiv, 2024)
This paper presents a benchmark self-evolving framework to dynamically evaluate rapidly advancing Large Language Models (LLMs), aiming for a more accurate assessment of their capabilities and limitations. We utilize a multi-agent system to manipulate the context or question of original instances, reframing new evolving instances with high confidence that dynamically extend existing benchmarks. Towards a more scalable, robust and fine-grained evaluation, we implement six reframing operations to construct evolving instances testing LLMs against diverse queries, data noise and probing their problem-solving sub-abilities. With this framework, we extend benchmark datasets of four tasks. Experimental results show a general performance decline in most LLMs against their original results. This decline under our scalable and robust evaluations, alongside our fine-grained evaluation, more accurately reflect models' capabilities. Besides, our framework widens performance discrepancies both between different models and within the same model across various tasks, facilitating more informed model selection for specific tasks (Code and data are available at https://github.com/NanshineLoong/Self-Evolving-Benchmark).
arXiv:2402.11443 [cs] <标题>: 基准自我进化：用于动态大语言模型评估的多智能体框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: Computer Science - Computation and Language

Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation - Wang, Siyuan; Long, Zhuohan; Fan, Zhihao; Wei, Zhongyu; Huang, Xuanjing (arXiv, 2024)
This paper presents a benchmark self-evolving framework to dynamically evaluate rapidly advancing Large Language Models (LLMs), aiming for a more accurate assessment of their capabilities and limitations. We utilize a multi-agent system to manipulate the context or question of original instances, reframing new evolving instances with high confidence that dynamically extend existing benchmarks. Towards a more scalable, robust and fine-grained evaluation, we implement six reframing operations to construct evolving instances testing LLMs against diverse queries, data noise and probing their problem-solving sub-abilities. With this framework, we extend benchmark datasets of four tasks. Experimental results show a general performance decline in most LLMs against their original results. This decline under our scalable and robust evaluations, alongside our fine-grained evaluation, more accurately reflect models' capabilities. Besides, our framework widens performance discrepancies both between different models and within the same model across various tasks, facilitating more informed model selection for specific tasks (Code and data are available at https://github.com/NanshineLoong/Self-Evolving-Benchmark).
arXiv:2402.11443 [cs] <标题>: 基准自我进化：用于动态大语言模型评估的多智能体框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language

Using Large Language Models for Student-Code Guided Test Case Generation in Computer Science Education - Kumar, Nischal Ashok; Lan, Andrew (2024)
In computer science education, test cases are an integral part of programming assignments since they can be used as assessment items to test students' programming knowledge and provide personalized feedback on student-written code. The goal of our work is to propose a fully automated approach for test case generation that can accurately measure student knowledge, which is important for two reasons. First, manually constructing test cases requires expert knowledge and is a labor-intensive process. Second, developing test cases for students, especially those who are novice programmers, is significantly different from those oriented toward professional-level software developers. Therefore, we need an automated process for test case generation to assess student knowledge and provide feedback. In this work, we propose a large language model-based approach to automatically generate test cases and show that they are good measures of student knowledge, using a publicly available dataset that contains student-written Java code. We also discuss future research directions centered on using test cases to help students.
arXiv:2402.07081 [cs] <标题>: 在计算机科学教育中使用大型语言模型生成学生代码指导的测试用例 <AI Smry>: A large language model-based approach is proposed to automatically generate test cases and show that they are good measures of student knowledge, using a publicly available dataset that contains student-written Java code. Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.602Z
Keywords: Computer Science - Computation and Language; Computer Science - Software Engineering

Dora Demszky - Empowering Educators via Language Technology (2024)
<标题>: 多拉·德姆斯基 - 通过语言技术赋能教育工作者 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education - Doughty, Jacob; Wan, Zipiao; Bompelli, Anishka; Qayum, Jubahed; Wang, Taozhi; Zhang, Juran; Zheng, Yujia; Doyle, Aidan; Sridhar, Pragnya; Agarwal, Arav; Bogart, Christopher; Keylor, Eric; et al. (Proceedings of the 26th Australasian Computing Education Conference, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting - Kaneko, Masahiro; Bollegala, Danushka; Okazaki, Naoaki; Baldwin, Timothy (arXiv, 2024)
There exist both scalable tasks, like reading comprehension and fact-checking, where model performance improves with model size, and unscalable tasks, like arithmetic reasoning and symbolic reasoning, where model performance does not necessarily improve with model size. Large language models (LLMs) equipped with Chain-of-Thought (CoT) prompting are able to make accurate incremental predictions even on unscalable tasks. Unfortunately, despite their exceptional reasoning abilities, LLMs tend to internalize and reproduce discriminatory societal biases. Whether CoT can provide discriminatory or egalitarian rationalizations for the implicit information in unscalable tasks remains an open question. In this study, we examine the impact of LLMs' step-by-step predictions on gender bias in unscalable tasks. For this purpose, we construct a benchmark for an unscalable task where the LLM is given a list of words comprising feminine, masculine, and gendered occupational words, and is required to count the number of feminine and masculine words. In our CoT prompts, we require the LLM to explicitly indicate whether each word in the word list is a feminine or masculine before making the final predictions. With counting and handling the meaning of words, this benchmark has characteristics of both arithmetic reasoning and symbolic reasoning. Experimental results in English show that without step-by-step prediction, most LLMs make socially biased predictions, despite the task being as simple as counting words. Interestingly, CoT prompting reduces this unconscious social bias in LLMs and encourages fair predictions.
arXiv:2401.15585 [cs] <标题>: 通过链式思维提示评估大型语言模型中的性别偏见 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: Computer Science - Computation and Language

An In-Depth Review of ChatGPT’s Pros and Cons for Learning and Teaching in Education - Samala, Agariadne Dwinggo; Zhai, Xiaoming; Aoki, Kumiko; Bojic, Ljubisa; Zikic, Simona (International Journal of Interactive Mobile Technologies (iJIM), 2024)
As technology progresses, there has been an increasing interest in using Chatbot GPT (Generative Pre-trained Transformer) in education. Chatbot GPT, or ChatGPT, gained one million users within the first week of launching in November 2022 and had amassed over 100 million active users by February 2023. This type of artificial intelligence uses natural language processing to convert it into a user. This paper presents a comprehensive analysis and review of 34 articles published on ChatGPT and its potential impact on education by utilizing the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. This review analyzed various studies and articles to examine the strengths and limitations of GPT language models in education from 2018 to the present. The advantages of ChatGPT include its capacity to provide personalized and adaptive learning, instant feedback, and improved accessibility. However, there are potential drawbacks, such as the lack of emotional intelligence, the risk of overreliance on technology, and privacy concerns. This review suggests that ChatGPT has significant promise for education yet reinforces the necessity for further research and careful consideration of possible risks and limitations. Specifically, it pointed out potential invisible manipulations by instructing ChatGPT to answer educationrelated topics. The paper concludes by discussing the implications of ChatGPT for the future of education and emphasizing the need for further research in this field.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning - McNichols, Hunter; Feng, Wanyong; Lee, Jaewook; Scarlatos, Alexander; Smith, Digory; Woodhead, Simon; Lan, Andrew (arXiv, 2024)
Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and are a reliable form of assessment. An important aspect of MCQs is the distractors, i.e., incorrect options that are designed to target specific misconceptions or insufficient knowledge among students. To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers, which has limited scalability. In this work, we explore the task of automated distractor and corresponding feedback message generation in math MCQs using large language models. We establish a formulation of these two tasks and propose a simple, in-context learning-based solution. Moreover, we propose generative AI-based metrics for evaluating the quality of the feedback messages. We conduct extensive experiments on these tasks using a real-world MCQ dataset. Our findings suggest that there is a lot of room for improvement in automated distractor and feedback generation; based on these findings, we outline several directions for future work.
arXiv:2308.03234 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language

A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models - Tonmoy, S. M. Towhidul Islam; Zaman, S. M. Mehedi; Jain, Vinija; Rani, Anku; Rawte, Vipula; Chadha, Aman; Das, Amitava (arXiv, 2024)
As Large Language Models (LLMs) continue to advance in their ability to write human-like text, a key challenge remains around their tendency to hallucinate generating content that appears factual but is ungrounded. This issue of hallucination is arguably the biggest hindrance to safely deploying these powerful LLMs into real-world production systems that impact people's lives. The journey toward widespread adoption of LLMs in practical settings heavily relies on addressing and mitigating hallucinations. Unlike traditional AI systems focused on limited tasks, LLMs have been exposed to vast amounts of online text data during training. While this allows them to display impressive language fluency, it also means they are capable of extrapolating information from the biases in training data, misinterpreting ambiguous prompts, or modifying the information to align superficially with the input. This becomes hugely alarming when we rely on language generation capabilities for sensitive applications, such as summarizing medical records, financial analysis reports, etc. This paper presents a comprehensive survey of over 32 techniques developed to mitigate hallucination in LLMs. Notable among these are Retrieval Augmented Generation (Lewis et al, 2021), Knowledge Retrieval (Varshney et al,2023), CoNLI (Lei et al, 2023), and CoVe (Dhuliawala et al, 2023). Furthermore, we introduce a detailed taxonomy categorizing these methods based on various parameters, such as dataset utilization, common tasks, feedback mechanisms, and retriever types. This classification helps distinguish the diverse approaches specifically designed to tackle hallucination issues in LLMs. Additionally, we analyze the challenges and limitations inherent in these techniques, providing a solid foundation for future research in addressing hallucinations and related phenomena within the realm of LLMs.
arXiv:2401.01313 [cs] <标题>: 大型语言模型中幻觉缓解技术的综合调查 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Computation and Language

The Information Age for Education via Artificial Intelligence and Machine Learning: A Bibliometric and Systematic Literature Analysis - Abuhassna, Hassan; Awae, Fareed; Daud, Mahyudin; Almheiri, Abdulla (International Journal of Information and Education Technology, 2024)
The integration of Artificial Intelligence (AI) and Machine Learning (ML) in education is a rapidly evolving field, yet the long-term implications and actual impacts on student learning outcomes require more in-depth study. Address this gap, our study offers a novel approach combining bibliometric analysis and a Systematic Literature Review (SLR), guided by the PRISMA methodology. The first phase, a comprehensive bibliometric analysis, identified key nations, educational institutions, journals, keywords, and influential authors in the realm of AI/ML in educational settings. This phase provided a macro-level understanding of the field’s landscape, showcasing the global and interdisciplinary nature of AI/ML research in education. The subsequent phase involved a meticulous SLR of 22 select scholarly articles. This in-depth review sheds light on the current applications, emerging trends, challenges, and future directions of AI and ML in education. The findings from this dual-method approach offer a comprehensive roadmap for educators, researchers, and policymakers, underscoring the transformative potential of AI and ML in the educational sector. The review’s extensive article collection provides a deep dive into the diverse and significant impact of AI in education, highlighting its role in areas such as predicting academic success, enhancing e-learning experiences, and preparing future generations for AI’s integration in various fields like healthcare. This study not only underscores the revolutionary potential of AI in reshaping educational landscapes but also serves as a guiding framework for effectively deploying AI and ML technologies in education.
<标题>: 通过人工智能和机器学习的教育信息时代：文献计量学与系统文献分析 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: Educational Area: Assessment Tools & Processes; Educational Area: Data Quality & EMIS; Educational Area: Student Learning; Evidence: Gaps in Current Research; Location: Asia; Research Design: Systematic Review; Research Method: Qualitative; Source: Open Source

Text-based Question Difficulty Prediction: A Systematic Review of Automatic Approaches - AlKhuzaey, Samah; Grasso, Floriana; Payne, Terry R.; Tamma, Valentina (International Journal of Artificial Intelligence in Education, 2024)
Abstract Designing and constructing pedagogical tests that contain items (i.e. questions) which measure various types of skills for different levels of students equitably is a challenging task. Teachers and item writers alike need to ensure that the quality of assessment materials is consistent, if student evaluations are to be objective and effective. Assessment quality and validity are therefore heavily reliant on the quality of the items included in the test. Moreover, the notion of difficulty is an essential factor that can determine the overall quality of the items and the resulting tests.Thus, item difficulty prediction is extremely important in any pedagogical learning environment. Although difficulty is traditionally estimated either by experts or through pre-testing, such methods are criticised for being costly, time-consuming, subjective and difficult to scale, and consequently, the use of automatic approaches as proxies for these traditional methods is gaining more and more traction. In this paper, we provide a comprehensive and systematic review of methods for the priori prediction of question difficulty. The aims of this review are to: 1) provide an overview of the research community regarding the publication landscape; 2) explore the use of automatic, text-based prediction models; 3) summarise influential difficulty features; and 4) examine the performance of the prediction models. Supervised machine learning prediction models were found to be mostly used to overcome the limitations of traditional item calibration methods. Moreover, linguistic features were found to play a major role in the determination of item difficulty levels, and several syntactic and semantic features were explored by researchers in this area to explain the difficulty of pedagogical assessments. Based on these findings, a number of challenges to the item difficulty prediction community are posed, including the need for a publicly available repository of standardised data-sets and further investigation into alternative feature elicitation and prediction models.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs - Balloccu, Simone; Schmidtová, Patrícia; Lango, Mateusz; Dušek, Ondřej (arXiv, 2024)
Natural Language Processing (NLP) research is increasingly focusing on the use of Large Language Models (LLMs), with some of the most popular ones being either fully or partially closed-source. The lack of access to model details, especially regarding training data, has repeatedly raised concerns about data contamination among researchers. Several attempts have been made to address this issue, but they are limited to anecdotal evidence and trial and error. Additionally, they overlook the problem of indirect data leaking, where models are iteratively improved by using data coming from users. In this work, we conduct the first systematic analysis of work using OpenAI’s GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination. By analysing 255 papers and considering OpenAI’s data usage policy, we extensively document the amount of data leaked to these models during the first year after the model’s release. We report that these models have been globally exposed to ∼4.7M samples from 263 benchmarks. At the same time, we document a number of evaluation malpractices emerging in the reviewed papers, such as unfair or missing baseline comparisons and reproducibility issues. We release our results as a collaborative project on https://leak-llm.github.io/, where other researchers can contribute to our efforts.
arXiv:2402.03927 [cs] <标题>: 泄露、作弊、重复：封闭源大语言模型中的数据污染与评估不当行为 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Generative AI Can Harm Learning - Bastani, Hamsa; Bastani, Osbert; Sungu, Alp; Ge, Haosen; Kabakcı, Özge; Mariman, Rei (Social Science Research Network, 2024)
Generative artificial intelligence (AI) is poised to revolutionize how humans work, and has already demonstrated promise in significantly improving human productivity. However, a key remaining question is how generative AI affects learning, namely, how humans acquire new skills as they perform tasks. This kind of skill learning is critical to long-term productivity gains, especially in domains where generative AI is fallible and human experts must check its outputs. We study the impact of generative AI, specifically OpenAI's GPT-4, on human learning in the context of math classes at a high school. In a field experiment involving nearly a thousand students, we have deployed and evaluated two GPT based tutors, one that mimics a standard ChatGPT interface (called GPT Base) and one with prompts designed to safeguard learning (called GPT Tutor). These tutors comprise about 15% of the curriculum in each of three grades. Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base). That is, access to GPT-4 can harm educational outcomes. These negative learning effects are largely mitigated by the safeguards included in GPT Tutor. Our results suggest that students attempt to use GPT-4 as a "crutch" during practice problem sessions, and when successful, perform worse on their own. Thus, to maintain long-term productivity, we must be cautious when deploying generative AI to ensure humans continue to learn critical skills. * HB, OB, and AS contributed equally
Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Education; Generative AI; Human Capital Development; Human-AI Collaboration; Large Language Models

The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges - Bulut, Okan; Beiting-Parrish, Maggie; Casabianca, Jodi M.; Slater, Sharon C.; Jiao, Hong; Song, Dan; Ormerod, Christopher M.; Fabiyi, Deborah Gbemisola; Ivan, Rodica; Walsh, Cole; Rios, Oscar; Wilson, Joshua; et al. (arXiv, 2024)
The integration of artificial intelligence (AI) in educational measurement has revolutionized assessment methods, enabling automated scoring, rapid content analysis, and personalized feedback through machine learning and natural language processing. These advancements provide timely, consistent feedback and valuable insights into student performance, thereby enhancing the assessment experience. However, the deployment of AI in education also raises significant ethical concerns regarding validity, reliability, transparency, fairness, and equity. Issues such as algorithmic bias and the opacity of AI decision-making processes pose risks of perpetuating inequalities and affecting assessment outcomes. Responding to these concerns, various stakeholders, including educators, policymakers, and organizations, have developed guidelines to ensure ethical AI use in education. The National Council of Measurement in Education's Special Interest Group on AI in Measurement and Education (AIME) also focuses on establishing ethical standards and advancing research in this area. In this paper, a diverse group of AIME members examines the ethical implications of AI-powered tools in educational measurement, explores significant challenges such as automation bias and environmental impact, and proposes solutions to ensure AI's responsible and effective use in education.
Version Number: 1 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Artificial Intelligence (cs.AI); Computers and Society (cs.CY); FOS: Computer and information sciences

Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future - Chu, Zheng; Chen, Jingchang; Chen, Qianglong; Yu, Weijiang; He, Tao; Wang, Haotian; Peng, Weihua; Liu, Ming; Qin, Bing; Liu, Ting (arXiv, 2024)
Reasoning, a fundamental cognitive process integral to human intelligence, has garnered substantial interest within artificial intelligence. Notably, recent studies have revealed that chain-of-thought prompting significantly enhances LLM's reasoning capabilities, which attracts widespread attention from both academics and industry. In this paper, we systematically investigate relevant research, summarizing advanced methods through a meticulous taxonomy that offers novel perspectives. Moreover, we delve into the current frontiers and delineate the challenges and future directions, thereby shedding light on future research. Furthermore, we engage in a discussion about open questions. We hope this paper serves as an introduction for beginners and fosters future research. Resources have been made publicly available at https://github.com/zchuz/CoT-Reasoning-Survey
arXiv:2309.15402 [cs] <标题>: 探索神秘迷宫：链式思维推理综述——进展、前沿与未来 <AI Smry>: This paper systematically investigate relevant research, summarizing advanced methods through a meticulous taxonomy that offers novel perspectives, and delve into the current frontiers and delineate the challenges and future directions, thereby shedding light on future research. Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

A Large-Scale Corpus for Assessing Written Argumentation: Persuade 2.0 - Crossley, Scott Andrew; Baffour, Perpetual; Tian, Yu; Franklin, Alex; Benner, Meg; Boser, Ulrich (2024)
<标题>: 用于评估书面论证的大规模语料库：Persuade 2.0 Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.601Z

Towards a New Artificial Intelligence-based Framework for Teachers’ Online Continuous Professional Development Programs: Systematic Review. - Fakhar, Hamza; Lamrabet, Mohammed; Echantoufi, Noureddine; El khattabi, Khalid; Ajana, Lotfi (International Journal of Advanced Computer Science and Applications, 2024)
—In recent years, the Artificial Intelligence (AI) field has witnessed rapid growth, affecting diverse sectors, including education. In this systematic review of literature, we aimed to analyze studies concerning the integration of AI in the continuous professional development (CPD) of teachers in order to generate a global vision on its potential to enhance the quality of CPD programs in the international level, and to provide recommendations for its application in the Moroccan context. To achieve our objective, we conducted a research that involves a review of international indexed databases (Scopus, Web of Science, Eric) published between 2019 and 2023 using PICO framework to formulate our search query and PRISMA (Preferred Reporting Items for Systematic Reviews and MetaAnalyses) framework to select 25 relevant studies based on include and exclude criteria like publishing year, type of documents, publishing mode, subject area, language, and other criteria. The results reveal that AI integration has a positive impact on CPD programs by offering beneficial intelligent tools that can tailor adaptive training programs to meet teachers’ specific needs, preferences, and proficiency levels. Furthermore, our findings identify the importance of integrating AI as a core topic within CPD programs to enhance teachers’ AI literacy, enabling them to effectively navigate and utilize AI-based tools in their educational environment. This is important for preparing teachers to engage with the technological advances shaping the educational system. In conclusion, our systematic review emphasizes the significance of AI integration in CPD programs and offers tailored recommendations for its implementation in the Moroccan educational context. By adopting these recommendations, Morocco will pave the way for a dynamic CPD framework that meets the evolving needs of educators and students alike.
<标题>: 面向教师在线持续专业发展项目的新型人工智能框架：系统综述。 <AI Smry>: The findings identify the importance of integrating AI as a core topic within CPD programs to enhance teachers’ AI literacy, enabling them to effectively navigate and utilize AI-based tools in their educational environment. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: Educational Area: Continuous Professional Development; Evidence: Improve a process; Location: Africa; Research Design: Systematic Review; Research Method: Qualitative; Source: Open Source

Artificial Intelligence in Education in Cape Verde: Potential and Challenges - Gomes, António P. M.; Gonçalves, Bruno F. (Advances in Tourism, Technology and Systems, 2024)
This research aims to explore the potentialities and challenges of AI in education in Cape Verde, in order to understand how AI can be used in the teaching–learning context. To carry out the research, a qualitative methodology was adopted supported by a literature review. With support from the documents found, the benefits and potentialities were analyzed, reflecting on the ethical implications of the application of AI in education, with emphasis on privacy, data security and ethics. We discussed the findings related to the potential and challenges of applying AI and how it can play a key role in analyzing data on learners’ performance and preferences, providing immediate feedback, personalized recommendations and complementary activities. Research suggests that AI in education has the potential to improve the equity, quality and reach of education, however, it is necessary to face challenges, such as ensuring the ethical use of AI, accessibility for all students, data protection and privacy. In Cape Verde, AI can be a powerful tool to mitigate socio-economic and geographical challenges, providing a more balanced and equal education across all islands. However, the challenges of access to high-speed internet and the quality of services provided by utilities can be real challenges to overcome.
Series Title: Smart Innovation, Systems and Technologies <标题>: 佛得角教育中的人工智能：潜力与挑战 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z
Keywords: Educational Area: Assessment Tools & Processes; Educational Area: Student Learning; Location: Africa; Research Design: Non-systematic Review; Research Method: Qualitative; Source: Closed Source

Uncertainty in Language Models: Assessment through Rank-Calibration - Huang, Xinmeng; Li, Shuo; Yu, Mengxin; Sesia, Matteo; Hassani, Hamed; Lee, Insup; Bastani, Osbert; Dobriban, Edgar (Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024)
<标题>: 语言模型中的不确定性：通过排名校准进行评估 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

Detecting ChatGPT-generated essays in a large-scale writing assessment: Is there a bias against non-native English speakers? - Jiang, Yang; Hao, Jiangang; Fauss, Michael; Li, Chen (Computers & Education, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

ChatGPT's ability or prompt quality: what determines the success of generating multiple-choice questions - Kıyak, Yavuz Selim (Academic Pathology, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

ChatGPT for generating multiple-choice questions: Evidence on the use of artificial intelligence in automatic item generation for a rational pharmacotherapy exam - Kıyak, Yavuz Selim; Coşkun, Özlem; Budakoğlu, Işıl İrem and Uluoğlu, Canan (European Journal of Clinical Pharmacology, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions - Laupichler, Matthias Carl; Rother, Johanna Flora; Grunwald Kadow, Ilona C.; Ahmadi, Seifollah; Raupach, Tobias (Academic Medicine, 2024)
Abstract Problem Creating medical exam questions is time consuming, but well-written questions can be used for test-enhanced learning, which has been shown to have a positive effect on student learning. The automated generation of high-quality questions using large language models (LLMs), such as ChatGPT, would therefore be desirable. However, there are no current studies that compare students’ performance on LLM-generated questions to questions developed by humans. Approach The authors compared student performance on questions generated by ChatGPT (LLM questions) with questions created by medical educators (human questions). Two sets of 25 multiple-choice questions (MCQs) were created, each with 5 answer options, 1 of which was correct. The first set of questions was written by an experienced medical educator, and the second set was created by ChatGPT 3.5 after the authors identified learning objectives and extracted some specifications from the human questions. Students answered all questions in random order in a formative paper-and-pencil test that was offered leading up to the final summative neurophysiology exam (summer 2023). For each question, students also indicated whether they thought it had been written by a human or ChatGPT. Outcomes The final data set consisted of 161 participants and 46 MCQs (25 human and 21 LLM questions). There was no statistically significant difference in item difficulty between the 2 question sets, but discriminatory power was statistically significantly higher in human than LLM questions (mean = .36, standard deviation [SD] = .09 vs mean = .24, SD = .14; P = .001). On average, students identified 57% of question sources (human or LLM) correctly. Next Steps Future research should replicate the study procedure in other contexts (e.g., other medical subjects, semesters, countries, and languages). In addition, the question of whether LLMs are suitable for generating different question types, such as key feature questions, should be investigated.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation - Li, Yu; Zhang, Shenyu; Wu, Rui; Huang, Xiutian; Chen, Yongrui; Xu, Wenhao; Qi, Guilin; Min, Dehai (arXiv, 2024)
Recent advancements in generative Large Language Models(LLMs) have been remarkable, however, the quality of the text generated by these models often reveals persistent issues. Evaluating the quality of text generated by these models, especially in open-ended text, has consistently presented a significant challenge. Addressing this, recent work has explored the possibility of using LLMs as evaluators. While using a single LLM as an evaluation agent shows potential, it is filled with significant uncertainty and instability. To address these issues, we propose the MATEval: A "Multi-Agent Text Evaluation framework" where all agents are played by LLMs like GPT-4. The MATEval framework emulates human collaborative discussion methods, integrating multiple agents' interactions to evaluate open-ended text. Our framework incorporates self-reflection and Chain-of-Thought (CoT) strategies, along with feedback mechanisms, enhancing the depth and breadth of the evaluation process and guiding discussions towards consensus, while the framework generates comprehensive evaluation reports, including error localization, error types and scoring. Experimental results show that our framework outperforms existing open-ended text evaluation methods and achieves the highest correlation with human evaluation, which confirms the effectiveness and advancement of our framework in addressing the uncertainties and instabilities in evaluating LLMs-generated text. Furthermore, our framework significantly improves the efficiency of text evaluation and model iteration in industrial scenarios.
arXiv:2403.19305 [cs] <标题>: MATEval：一个用于推动开放式文本评估的多智能体讨论框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Leveraging Large Language Models for NLG Evaluation: A Survey - Li, Zhen; Xu, Xiaohan; Shen, Tao; Xu, Can; Gu, Jia-Chen; Tao, Chongyang ([object Object], 2024)
In the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. This survey aims to provide a thorough overview of leveraging LLMs for NLG evaluation, a burgeoning area that lacks a systematic analysis. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. Our detailed exploration includes critically assessing various LLM-based methodologies, as well as comparing their strengths and limitations in evaluating NLG outputs. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this survey seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.
Version Number: 1 <AI Smry>: A coherent taxonomy for organizing existing LLM-based evaluation metrics is proposed, offering a structured framework to understand and compare these methods and seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques. <标题>: 利用大型语言模型进行自然语言生成评估：综述 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z
Keywords: /unread; Computation and Language (cs.CL); FOS: Computer and information sciences

Leveraging Large Language Models for NLG Evaluation: A Survey - Li, Zhen; Xu, Xiaohan; Shen, Tao; Xu, Can; Gu, Jia-Chen; Tao, Chongyang ([object Object], 2024)
In the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. This survey aims to provide a thorough overview of leveraging LLMs for NLG evaluation, a burgeoning area that lacks a systematic analysis. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. Our detailed exploration includes critically assessing various LLM-based methodologies, as well as comparing their strengths and limitations in evaluating NLG outputs. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this survey seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.
Version Number: 1 <标题>: 利用大型语言模型进行自然语言生成评价：综述 <AI Smry>: A coherent taxonomy for organizing existing LLM-based evaluation metrics is proposed, offering a structured framework to understand and compare these methods and seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques. Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z
Keywords: Computation and Language (cs.CL); FOS: Computer and information sciences

Generative AI and Its Educational Implications - Łodzikowski, Kacper; Foltz, Peter W.; Behrens, John T. (arXiv, 2024)
We discuss the implications of generative AI on education across four critical sections: the historical development of AI in education, its contemporary applications in learning, societal repercussions, and strategic recommendations for researchers. We propose ways in which generative AI can transform the educational landscape, primarily via its ability to conduct assessment of complex cognitive performances and create personalized content. We also address the challenges of effective educational tool deployment, data bias, design transparency, and accurate output verification. Acknowledging the societal impact, we emphasize the need for updating curricula, redefining communicative trust, and adjusting to transformed social norms. We end by outlining the ways in which educational stakeholders can actively engage with generative AI, develop fluency with its capacities and limitations, and apply these insights to steer educational practices in a rapidly advancing digital landscape.
Version Number: 1 Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z
Keywords: Computers and Society (cs.CY); FOS: Computer and information sciences

Evaluating the Quality of AI-Generated Items for a Certification Exam - Mead, Alan; Zhou, Chenxuan (Journal of Applied Testing Technology, 2024)
OpenAI’s GPT-3 model can write multiple-choice exam items. This paper reviewed the literature on automatic item generation and then described the recent history of OpenAI GPT models and their operation, and then described a methodology for generating items using these models. This study then critically evaluated GPT-3 at the task of writing multiple-choice exam items for a hypothetical psychometrics exam. We also compared two versions of the GPT-3 model (text-davinci-002 and text-davinci-003) on 90 items generated by GPT-3. The vast majority (71% and 90%) of items were judged as useful, but the typical item will require revision to address problems with the stem, key or distractors. The most common error was a violation of the principles of multiple-choice items (e.g., having two correct responses).
<标题>: 评估人工智能生成的认证考试题目的质量 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

An Automatic Question Usability Evaluation Toolkit - Moore, Steven; Costello, Eamon; Nguyen, Huy A.; Stamper, John (Artificial Intelligence in Education, 2024)
Series Title: Lecture Notes in Computer Science <标题>: 自动化问题可用性评估工具包 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions - Ngo, Alexander; Gupta, Saumya; Perrine, Oliver; Reddy, Rithik; Ershadi, Sherry; Remick, Daniel (Academic Pathology, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems - Saad-Falcon, Jon; Khattab, Omar; Potts, Christopher; Zaharia, Matei (arXiv, 2024)
Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating RAG systems along the dimensions of context relevance, answer faithfulness, and answer relevance. By creating its own synthetic training data, ARES finetunes lightweight LM judges to assess the quality of individual RAG components. To mitigate potential prediction errors, ARES utilizes a small set of human-annotated datapoints for prediction-powered inference (PPI). Across eight different knowledge-intensive tasks in KILT, SuperGLUE, and AIS, ARES accurately evaluates RAG systems while using only a few hundred human annotations during evaluation. Furthermore, ARES judges remain effective across domain shifts, proving accurate even after changing the type of queries and/or documents used in the evaluated RAG systems. We make our code and datasets publicly available on Github.
arXiv:2311.09476 [cs] <标题>: ARES：面向检索增强生成系统的自动化评估框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Information Retrieval

Adapting to AI: how to understand, prepare for, and innovate in a changing landscape - Swaak, Taylor (The Chronicle of Higher Education, 2024)
<标题>: 适应人工智能：如何理解、准备并在变化的环境中创新 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Can artificial Intelligence Technology Promote the Improvement of Student Learning Outcomes?——Meta Analysis Based on 50 Experimental and Quasi Experimental Studies - Wang, Lijuan; Zhao, Miaomiao (Proceedings of the 3rd International Conference on Educational Innovation and Multimedia Technology, EIMT 2024, March 29–31, 2024, Wuhan, China, 2024)
<标题>: 人工智能技术能促进学生学习成果的提升吗？——基于50项实验与准实验研究的元分析 <AI Smry>: After conducting a meta-analysis of 50 experimental studies related to AI teaching technology, it was found that it can significantly improve students' learning outcomes. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: Educational Area: Student Learning; Evidence: Student Learning Outcomes; Location: Asia; Research Design: Non-systematic Review; Research Method: Qualitative; Source: Open Source

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations - Wang, Peiyi; Li, Lei; Shao, Zhihong; Xu, R. X.; Dai, Damai; Li, Yifei; Chen, Deli; Wu, Y.; Sui, Zhifang (arXiv, 2024)
In this paper, we present an innovative process-oriented math process reward model called \textbf{Math-Shepherd}, which assigns a reward score to each step of math problem solutions. The training of Math-Shepherd is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of Math-Shepherd in two scenarios: 1) \textit{Verification}: Math-Shepherd is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) \textit{Reinforcement Learning}: Math-Shepherd is employed to reinforce LLMs with step-by-step Proximal Policy Optimization (PPO). With Math-Shepherd, a series of open-source LLMs demonstrates exceptional performance. For instance, the step-by-step PPO with Math-Shepherd significantly improves the accuracy of Mistral-7B (77.9\%$\to$84.1\% on GSM8K and 28.6\%$\to$33.0\% on MATH). The accuracy can be further enhanced to 89.1\% and 43.5\% on GSM8K and MATH with the verification of Math-Shepherd, respectively. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.
arXiv:2312.08935 [cs] <标题>: 数学牧羊人：在无需人工标注的情况下逐步验证和强化大型语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.601Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes - Wang, Rose E.; Zhang, Qingyang; Robinson, Carly; Loeb, Susanna; Demszky, Dorottya (arXiv, 2024)
Scaling high-quality tutoring remains a major challenge in education. Due to growing demand, many platforms employ novice tutors who, unlike experienced educators, struggle to address student mistakes and thus fail to seize prime learning opportunities. Our work explores the potential of large language models (LLMs) to close the novice-expert knowledge gap in remediating math mistakes. We contribute Bridge, a method that uses cognitive task analysis to translate an expert's latent thought process into a decision-making model for remediation. This involves an expert identifying (A) the student's error, (B) a remediation strategy, and (C) their intention before generating a response. We construct a dataset of 700 real tutoring conversations, annotated by experts with their decisions. We evaluate state-of-the-art LLMs on our dataset and find that the expert's decision-making model is critical for LLMs to close the gap: responses from GPT4 with expert decisions (e.g., "simplify the problem") are +76% more preferred than without. Additionally, context-sensitive decisions are critical to closing pedagogical gaps: random decisions decrease GPT4's response quality by -97% than expert decisions. Our work shows the potential of embedding expert thought processes in LLM generations to enhance their capability to bridge novice-expert knowledge gaps. Our dataset and code can be found at: \url{https://github.com/rosewang2008/bridge}.
arXiv:2310.10648 [cs] <标题>: 通过决策模型弥合新手与专家的差距：一个关于纠正数学错误的案例研究 <AI Smry>: Bridge, a method that uses cognitive task analysis to translate an expert’s latent thought process into a decision-making model for remediation, is contributed, showing the potential of embedding expert thought processes in LLM generations to enhance their capability to bridge novice-expert knowledge gaps. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.338Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

The Social life of AI in Education - Williamson, Ben (International Journal of Artificial Intelligence in Education, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Elementary English learners’ engagement with automated feedback - Wilson, Joshua; Palermo, Corey; Wibowo, Arianto (Learning and Instruction, 2024)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Hallucination is Inevitable: An Innate Limitation of Large Language Models - Xu, Ziwei; Jain, Sanjay; Kankanhalli, Mohan (2024)
Hallucination has been widely recognized to be a significant drawback for large language models (LLMs). There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all of the computable functions and will therefore always hallucinate. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.
<AI Smry>: This paper formalizes the problem and shows that it is impossible to eliminate hallucination in LLMs, and defines a formal world where hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. <标题>: 幻觉是不可避免的：大型语言模型的天生局限性 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences; Machine Learning (cs.LG)

One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering - Yang, Han; Li, Mingchen; Zhou, Huixue; Xiao, Yongkang; Fang, Qian; Zhang, Rui (2023)
To enhance the accuracy and reliability of diverse medical question-answering (QA) tasks and investigate efficient approaches deploying the Large Language Models (LLM) technologies, We developed a novel ensemble learning pipeline by utilizing state-of-the-art LLMs, focusing on improving performance on diverse medical QA datasets.Materials and MethodsOur study employs three medical QA datasets: PubMedQA, MedQA-USMLE, and MedMCQA, each presenting unique challenges in biomedical question-answering. The proposed LLM-Synergy framework, focusing exclusively on zero-shot cases using LLMs, incorporates two primary ensemble methods. The first is a Boosting-based weighted majority vote ensemble, where decision-making is expedited and refined by assigning variable weights to different LLMs through a boosting algorithm. The second method is Cluster-based Dynamic Model Selection, which dynamically selects the most suitable LLM votes for each query, based on the characteristics of question contexts, using a clustering approach.ResultsThe Majority Weighted Vote and Dynamic Model Selection methods demonstrate superior performance compared to individual LLMs across three medical QA datasets. Specifically, the accuracies are 35.84%, 96.21%, and 37.26% for MedMCQA, PubMedQA, and MedQA-USMLE, respectively, with the Majority Weighted Vote. Correspondingly, the Dynamic Model Selection yields slightly higher accuracies of 38.01%, 96.36%, and 38.13%.ConclusionThe LLM-Synergy framework with two ensemble methods, represents a significant advancement in leveraging LLMs for medical QA tasks and provides an innovative way of efficiently utilizing the development with LLM Technologies, customing for both existing and potentially future challenge tasks in biomedical and health informatics research.
<标题>: 一个大型语言模型不足以应对：利用集成学习的力量进行医学问答 <AI Smry>: The proposed LLM-Synergy framework with two ensemble methods, represents a significant advancement in leveraging LLMs for medical QA tasks and provides an innovative way of efficiently utilizing the development with LLM Technologies, customing for both existing and potentially future challenge tasks in biomedical and health informatics research. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.338Z

NLP-Based Management of Large Multiple-Choice Test Item Repositories - Albano, Valentina; Firmani, Donatella; Laura, Luigi; Mathew, Jerin George; Paoletti, Anna Lucia; Torrente, Irene (Journal of Learning Analytics, 2023)
Multiple-choice questions (MCQs) are widely used in educational assessments and professional certification exams. Managing large repositories of MCQs, however, poses several challenges due to the high volume of questions and the need to maintain their quality and relevance over time. One of these challenges is the presence of questions that duplicate concepts but are formulated differently. Such questions can indeed elude syntactic controls but provide no added value to the repository. In this paper, we focus on this specific challenge and propose a workflow for the discovery and management of potential duplicate questions in large MCQ repositories. Overall, the workflow comprises three main steps: MCQ preprocessing, similarity computation, and finally a graph-based exploration and analysis of the obtained similarity values. For the preprocessing phase, we consider three main strategies: (i) removing the list of candidate answers from each question, (ii) augmenting each question with the correct answer, or (iii) augmenting each question with all candidate answers. Then, we use deep learning–based natural language processing (NLP) techniques, based on the Transformers architecture, to compute similarities between MCQs based on semantics. Finally, we propose a new approach to graph exploration based on graph communities to analyze the similarities and relationships between MCQs in the graph. We illustrate the approach with a case study of the Competenze Digitali program, a large-scale assessment project by the Italian government.
<标题>: 基于自然语言处理的大型多项选择题题库管理 <AI Smry>: This paper proposes a workflow for the discovery and management of potential duplicate questions in large MCQ repositories, and uses deep learning–based natural language processing (NLP) techniques, based on the Transformers architecture, to compute similarities between MCQs based on semantics. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Machine Learning Systems (2023)
<标题>: 机器学习系统 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Meet LLM360: The First Fully Open-Source and Transparent Large Language Models (LLMs) - Hassan, Sana (2023)
Open-source Large Language Models (LLMs) such as LLaMA, Falcon, and Mistral offer a range of choices for AI professionals and scholars. Yet, the majority of these LLMs have only made available select components like the end-model weights or inference scripts, with technical documents often narrowing their focus to broader design aspects and basic metrics. This approach restricts advances in the field by reducing clarity in the training methodologies of LLMs, leading to repeated efforts by teams to uncover numerous aspects of the training procedure. A team of researchers from Petuum, MBZUAI, USC, CMU, UIUC, and UCSD introduced LLM360 to support
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Can AI Be Too Good to Use? - Fell, Andy (2023)
Much of the discussion around implementing artificial intelligence systems focuses on whether an AI application is “trustworthy”: Does it produce useful, reliable results, free of bias, while ensuring data privacy? But a new paper published Dec. 7 in Frontiers in Artificial Intelligence poses a different question: What if an AI is just too good?
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Navigating the generative AI era: Introducing the AI assessment scale for ethical GenAI assessment - Perkins, Mike; Furze, Leon; Roe, Jasper; MacVaugh, Jason (arXiv.org, 2023)
Recent developments in Generative Artificial Intelligence (GenAI) have created a paradigm shift in multiple areas of society, and the use of these technologies is likely to become a defining feature of education in coming decades. GenAI offers transformative pedagogical opportunities, while simultaneously posing ethical and academic challenges. Against this backdrop, we outline a practical, simple, and sufficiently comprehensive tool to allow for the integration of GenAI tools into educational assessment: the AI Assessment Scale (AIAS). The AIAS empowers educators to select the appropriate level of GenAI usage in assessments based on the learning outcomes they seek to address. The AIAS offers greater clarity and transparency for students and educators, provides a fair and equitable policy tool for institutions to work with, and offers a nuanced approach which embraces the opportunities of GenAI while recognising that there are instances where such tools may not be pedagogically appropriate or necessary. By adopting a practical, flexible approach that can be implemented quickly, the AIAS can form a much-needed starting point to address the current uncertainty and anxiety regarding GenAI in education. As a secondary objective, we engage with the current literature and advocate for a refocused discourse on GenAI tools in education, one which foregrounds how technologies can help support and enhance teaching and learning, which contrasts with the current focus on GenAI as a facilitator of academic misconduct.
<标题>: 驾驭生成式人工智能时代：推出用于伦理生成式人工智能评估的AI评估量表 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Classification of Human- and AI-Generated Texts for English, French, German, and Spanish - Schaaff, Kristina; Schlippe, Tim; Mindner, Lorenz (arXiv, 2023)
In this paper we analyze features to classify human- and AI-generated text for English, French, German and Spanish and compare them across languages. We investigate two scenarios: (1) The detection of text generated by AI from scratch, and (2) the detection of text rephrased by AI. For training and testing the classifiers in this multilingual setting, we created a new text corpus covering 10 topics for each language. For the detection of AI-generated text, the combination of all proposed features performs best, indicating that our features are portable to other related languages: The F1-scores are close with 99% for Spanish, 98% for English, 97% for German and 95% for French. For the detection of AI-rephrased text, the systems with all features outperform systems with other features in many cases, but using only document features performs best for German (72%) and Spanish (86%) and only text vector features leads to best results for English (78%).
arXiv:2312.04882 [cs] <标题>: 英、法、德、西班牙语中人类与人工智能生成文本的分类 Read_Status: New Read_Status_Date: 2026-01-26T11:32:28.375Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Can AI Provide Useful Holistic Essay Scoring? - Tate, Tamara; Steiss, Jacob; Bailey, Drew; Graham, Steve; Ritchie, Daniel; Tseng, Waverly; Moon, Youngsun; Uci, Mark Warschauer (OSF, 2023)
Researchers have sought for decades to automate holistic essay scoring. Over the years, these programs have improved significantly. However, accuracy requires significant amounts of training on human-scored texts—reducing the expediency and usefulness of such programs for routine uses by teachers across the nation on non-standardized prompts. This study analyzes the output of multiple versions of ChatGPT scoring of secondary student essays from three extant corpora and compares it to quality human ratings. We find that the current iteration of ChatGPT scoring is not statistically significantly different from human scoring, but exact agreement with humans is still difficult. Consistency and agreement within one point, however, is achievable and may be sufficient for low-stakes, formative assessment purposes.
<标题>: 人工智能能提供有用的整体性作文评分吗？ Read_Status: New Read_Status_Date: 2026-01-26T11:32:25.304Z
Keywords: AI: ChatGPT; Educational Area: Assessment Tools & Processes; Evidence: Improve a process; Location: Americas; Research Design: Other Research Design; Research Method: Quantitative; Source: Open Source

What Do We Mean by GenAI? A Systematic Mapping of The Evolution, Trends, and Techniques Involved in Generative AI. - García Peñalvo, Francisco José; Vázquez Ingelmo, Andrea (International Journal of Interactive Multimedia and Artificial Intelligence, 2023)
Artificial Intelligence has become a focal point of interest across various sectors due to its ability to generate creative and realistic outputs. A specific subset, generative artificial intelligence, has seen significant growth, particularly in late 2022. Tools like ChatGPT, Dall-E, or Midjourney have democratized access to Large Language Models, enabling the creation of human-like content. However, the concept 'Generative Artificial Intelligence lacks a universally accepted definition, leading to potential misunderstandings. While a model that produces any output can be technically seen as generative, the Artificial Intelligent research community often reserves the term for complex models that generate high-quality, human-like material. This paper presents a literature mapping of AI-driven content generation, analyzing 631 solutions published over the last five years to better understand and characterize the Generative Artificial Intelligence landscape. Our findings suggest a dichotomy in the understanding and application of the term "Generative AI". While the broader public often interprets "Generative AI" as AI-driven creation of tangible content, the AI research community mainly discusses generative implementations with an emphasis on the models in use, without explicitly categorizing their work under the term "Generative AI".
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design - Regenwetter, Lyle; Srivastava, Akash; Gutfreund, Dan; Ahmed, Faez (Computer-Aided Design, 2023)
<标题>: 超越统计相似性：重新思考工程设计中深度生成模型的指标 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design - Regenwetter, Lyle; Srivastava, Akash; Gutfreund, Dan; Ahmed, Faez (Computer-Aided Design, 2023)
<标题>: 超越统计相似性：重新思考工程设计中深度生成模型的指标 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Human mobility and language: towards new multilingual approaches with AI - Santorelli, Marion; Catullo, Domenico (Economic growth in the conditions of globalization, Vol. III, 2023)
This study investigates the relationships between language and human mobility in terms of investment, accessibility and inclusion and how human-computer interactions, AI (Artificial Intelligence) speech translators might overcome language barrier in a multilingual perspective. After a brief analysis of population dynamics, demographic change and migration based on European Union publications, the aim of this paper is to highlight the strong nexus between language and mobility and how it plays a key role in citizenship, educational policies, employment and social services. The phenomenon of linguistic identity, together with the power relations of ELF (English as Lingua Franca), is observed by presenting poststructuralist perspectives on SLA (Second Language Acquisition). It emerges that the power relations can be equal and unequal and can influence both positively and negatively users’ identities, shape people interactions and, thus, place them in particular communities or statuses. This ‘communicative imperialism’ can be inferred by the English Proficiency Index that places European countries in a high position, while countries in Eastern Europe, the Middle East, Africa, Latin America, and some Asian countries in the lowest ones. Non-native speakers have differing levels of command of the language, meaning that, for them, crucial details and nuances, as well as cultural references, might often be lost. This study explores the real-time multilingual interpretation, with the new emerging technologies, as a means to reduce language discrimination, information loss, and increase the return on investment (ROI), enabling everyone to join in a conversation in their own language from anywhere around the world.
<标题>: 人类流动性与语言：借助人工智能迈向新的多语言方法 <AI Smry>: The real-time multilingual interpretation, with the new emerging technologies, is explored as a means to reduce language discrimination, information loss, and increase the return on investment (ROI), enabling everyone to join in a conversation in their own language from anywhere around the world. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.295Z

Math Education with Large Language Models: Peril or Promise? - Kumar, Harsh; Rothschild, David M.; Goldstein, Daniel G.; Hofman, Jake M. (2023)
The widespread availability of large language models (LLMs) has provoked both fear and excitement in the domain of education.On one hand, there is the concern that students will offload their coursework to LLMs, limiting what they themselves learn.On the other hand, there is the hope that LLMs might serve as scalable, personalized tutors.Here we conduct a large, pre-registered experiment involving 1200 participants to investigate how exposure to LLM-based explanations affect learning.In the experiment's learning phase, we gave participants practice problems and manipulated two key factors in a between-participants design: first, whether they were required to attempt a problem before or after seeing the correct answer, and second, whether participants were shown only the answer or were also exposed to an LLM-generated explanation of the answer.Subsequently, all participants were tested on new test questions to assess how well they had learned the underlying concepts.Overall we found that LLM-based explanations positively impacted learning relative to seeing only correct answers.The benefits were largest for those who attempted problems on their own first before consulting LLM explanations, but surprisingly this trend held even for those participants who were exposed to LLM explanations before attempting to solve practice problems on their own.An accompanying qualitative analysis revealed that these boosts in performance were indeed due to participants adopting the strategies they were shown, and that exposure to LLM explanations increased the amount people felt they learned and decreased the perceived difficulty of the test problems.
Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: /unread; AI; education; generative AI; human-AI interaction; large language models; llms; math; tutoring

AI and the Future of Skills, Volume 2: Methods for Evaluating AI Capabilities - OECD (OECD, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

ChaTA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs - Hicke, Yann; Agarwal, Anmol; Ma, Qianou; Denny, Paul (arXiv, 2023)
Responding to the thousands of student questions on online QA platforms each semester has a considerable human cost, particularly in computing courses with rapidly growing enrollments. To address the challenges of scalable and intelligent question-answering (QA), we introduce an innovative solution that leverages open-source Large Language Models (LLMs) from the LLaMA-2 family to ensure data privacy. Our approach combines augmentation techniques such as retrieval augmented generation (RAG), supervised fine-tuning (SFT), and learning from human preferences data using Direct Preference Optimization (DPO). Through extensive experimentation on a Piazza dataset from an introductory CS course, comprising 10,000 QA pairs and 1,500 pairs of preference data, we demonstrate a significant 30% improvement in the quality of answers, with RAG being a particularly impactful addition. Our contributions include the development of a novel architecture for educational QA, extensive evaluations of LLM performance utilizing both human assessments and LLM-based metrics, and insights into the challenges and future directions of educational data processing. This work paves the way for the development of CHATA, an intelligent QA assistant customizable for courses with an online QA platform
arXiv:2311.02775 [cs] <标题>: ChaTA：利用开源大型语言模型构建智能问答教学助理 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

ChaTA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs - Hicke, Yann; Agarwal, Anmol; Ma, Qianou; Denny, Paul (arXiv, 2023)
Responding to the thousands of student questions on online QA platforms each semester has a considerable human cost, particularly in computing courses with rapidly growing enrollments. To address the challenges of scalable and intelligent question-answering (QA), we introduce an innovative solution that leverages open-source Large Language Models (LLMs) from the LLaMA-2 family to ensure data privacy. Our approach combines augmentation techniques such as retrieval augmented generation (RAG), supervised fine-tuning (SFT), and learning from human preferences data using Direct Preference Optimization (DPO). Through extensive experimentation on a Piazza dataset from an introductory CS course, comprising 10,000 QA pairs and 1,500 pairs of preference data, we demonstrate a significant 30% improvement in the quality of answers, with RAG being a particularly impactful addition. Our contributions include the development of a novel architecture for educational QA, extensive evaluations of LLM performance utilizing both human assessments and LLM-based metrics, and insights into the challenges and future directions of educational data processing. This work paves the way for the development of CHATA, an intelligent QA assistant customizable for courses with an online QA platform
arXiv:2311.02775 [cs] <标题>: ChaTA：利用开源大型语言模型构建智能问答教学助理 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Review on Neural Question Generation for Education Purposes - Al Faraby, Said; Adiwijaya, Adiwijaya; Romadhony, Ade (International Journal of Artificial Intelligence in Education, 2023)
Questioning plays a vital role in education, directing knowledge construction and assessing students’ understanding. However, creating high-level questions requires significant creativity and effort. Automatic question generation is expected to facilitate the generation of not only fluent and relevant but also educationally valuable questions. While rule-based methods are intuitive for short inputs, they struggle with longer and more complex inputs. Neural question generation (NQG) has shown better results in this regard. This review summarizes the advancements in NQG between 2016 and early 2022. The focus is on the development of NQG for educational purposes, including challenges and research opportunities. We found that although NQG can generate fluent and relevant factoid-type questions, few studies focus on education. Specifically, there is limited literature using context in the form of multi-paragraphs, which due to the input limitation of the current deep learning techniques, require key content identification. The desirable key content should be important to specific topics or learning objectives and be able to generate certain types of questions. A further research opportunity is controllable NQG systems, which can be customized by taking into account factors like difficulty level, desired answer type, and other individualized needs. Equally important, the results of our review also suggest that it is necessary to create datasets specific to the question generation tasks with annotations that support better learning for neural-based methods.
<标题>: 关于教育用途的神经问题生成的综述 <AI Smry>: It is found that although NQG can generate fluent and relevant factoid-type questions, few studies focus on education, and it is necessary to create datasets specific to the question generation tasks with annotations that support better learning for neural-based methods. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Towards social generative AI for education: theory, practices and ethics (2023)
<标题>: 面向教育的社会生成式人工智能：理论、实践与伦理 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.518Z
Keywords: /unread

Using State-of-the-Art Speech Models to Evaluate Oral Reading Fluency in Ghana - Henkel, Owen; Horne-Robinson, Hannah; Hills, Libby; Roberts, Bill; McGrane, Joshua (arXiv, 2023)
This paper reports on a set of three recent experiments utilizing large-scale speech models to evaluate the oral reading fluency (ORF) of students in Ghana. While ORF is a well-established measure of foundational literacy, assessing it typically requires one-on-one sessions between a student and a trained evaluator, a process that is time-consuming and costly. Automating the evaluation of ORF could support better literacy instruction, particularly in education contexts where formative assessment is uncommon due to large class sizes and limited resources. To our knowledge, this research is among the first to examine the use of the most recent versions of large-scale speech models (Whisper V2 wav2vec2.0) for ORF assessment in the Global South. We find that Whisper V2 produces transcriptions of Ghanaian students reading aloud with a Word Error Rate of 13.5. This is close to the model's average WER on adult speech (12.8) and would have been considered state-of-the-art for children's speech transcription only a few years ago. We also find that when these transcriptions are used to produce fully automated ORF scores, they closely align with scores generated by expert human graders, with a correlation coefficient of 0.96. Importantly, these results were achieved on a representative dataset (i.e., students with regional accents, recordings taken in actual classrooms), using a free and publicly available speech model out of the box (i.e., no fine-tuning). This suggests that using large-scale speech models to assess ORF may be feasible to implement and scale in lower-resource, linguistically diverse educational contexts.
arXiv:2310.17606 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Enhancing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Approach - Nakamoto, Ryosuke; Flanagan, Brendan; Yamauchi, Taisei; Dai, Yiling; Takami, Kyosuke; Ogata, Hiroaki (Computers, 2023)
In the realm of mathematics education, self-explanation stands as a crucial learning mechanism, allowing learners to articulate their comprehension of intricate mathematical concepts and strategies. As digital learning platforms grow in prominence, there are mounting opportunities to collect and utilize mathematical self-explanations. However, these opportunities are met with challenges in automated evaluation. Automatic scoring of mathematical self-explanations is crucial for preprocessing tasks, including the categorization of learner responses, identification of common misconceptions, and the creation of tailored feedback and model solutions. Nevertheless, this task is hindered by the dearth of ample sample sets. Our research introduces a semi-supervised technique using the large language model (LLM), specifically its Japanese variant, to enrich datasets for the automated scoring of mathematical self-explanations. We rigorously evaluated the quality of self-explanations across five datasets, ranging from human-evaluated originals to ones devoid of original content. Our results show that combining LLM-based explanations with mathematical material significantly improves the model’s accuracy. Interestingly, there is an optimal limit to how many synthetic self-explanation data can benefit the system. Exceeding this limit does not further improve outcomes. This study thus highlights the need for careful consideration when integrating synthetic data into solutions, especially within the mathematics discipline.
<标题>: 利用大语言模型生成的数据集增强数学自我解释质量的自动评分：一种半监督方法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z

Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents - Chae, Hyungjoo; Song, Yongho; Ong, Kai Tzu-iunn; Kwon, Taeyoon; Kim, Minjin; Yu, Youngjae; Lee, Dongha; Kang, Dongyeop; Yeo, Jinyoung (arXiv, 2023)
Human-like chatbots necessitate the use of commonsense reasoning in order to effectively comprehend and respond to implicit information present within conversations. Achieving such coherence and informativeness in responses, however, is a non-trivial task. Even for large language models (LLMs), the task of identifying and aggregating key evidence within a single hop presents a substantial challenge. This complexity arises because such evidence is scattered across multiple turns in a conversation, thus necessitating integration over multiple hops. Hence, our focus is to facilitate such multi-hop reasoning over a dialogue context, namely dialogue chain-of-thought (CoT) reasoning. To this end, we propose a knowledge distillation framework that leverages LLMs as unreliable teachers and selectively distills consistent and helpful rationales via alignment filters. We further present DOCTOR, a DialOgue Chain-of-ThOught Reasoner that provides reliable CoT rationales for response generation. We conduct extensive experiments to show that enhancing dialogue agents with high-quality rationales from DOCTOR significantly improves the quality of their responses.
arXiv:2310.09343 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Chainpoll: A high efficacy method for LLM hallucination detection - Friel, Robert; Sanyal, Atindriyo (arXiv, 2023)
Large language models (LLMs) have experienced notable advancements in generating coherent and contextually relevant responses. However, hallucinations - incorrect or unfounded claims - are still prevalent, prompting the creation of automated metrics to detect these in LLM outputs. Our contributions include: introducing ChainPoll, an innovative hallucination detection method that excels compared to its counterparts, and unveiling RealHall, a refined collection of benchmark datasets to assess hallucination detection metrics from recent studies. While creating RealHall, we assessed tasks and datasets from previous hallucination detection studies and observed that many are not suitable for the potent LLMs currently in use. Overcoming this, we opted for four datasets challenging for modern LLMs and pertinent to real-world scenarios. Using RealHall, we conducted a comprehensive comparison of ChainPoll with numerous hallucination metrics from recent studies. Our findings indicate that ChainPoll outperforms in all RealHall benchmarks, achieving an overall AUROC of 0.781. This surpasses the next best theoretical method by 11% and exceeds industry standards by over 23%. Additionally, ChainPoll is cost-effective and offers greater transparency than other metrics. We introduce two novel metrics to assess LLM hallucinations: Adherence and Correctness. Adherence is relevant to Retrieval Augmented Generation workflows, evaluating an LLM's analytical capabilities within given documents and contexts. In contrast, Correctness identifies logical and reasoning errors.
arXiv:2310.18344 [cs] <标题>: Chainpoll：一种高效的LLM幻觉检测方法 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory - Xiao, Ziang; Zhang, Susu; Lai, Vivian; Liao, Q. Vera (arXiv, 2023)
We address a fundamental challenge in Natural Language Generation (NLG) model evaluation – the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.
tex.ids= xiaoEvaluatingEvaluationMetrics2023a, xiaoEvaluatingEvaluationMetrics2023b arXiv: 2305.14889 [cs] <标题>: 评估评价指标：使用测量理论分析自然语言生成评价指标的框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:25.304Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory - Xiao, Ziang; Zhang, Susu; Lai, Vivian; Liao, Q. Vera (arXiv, 2023)
We address a fundamental challenge in Natural Language Generation (NLG) model evaluation – the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.
arXiv:2305.14889 [cs] <标题>: 评估评价指标：使用测量理论分析自然语言生成评价指标的框架 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Stanford University and Research Institutions Release Model Transparency Index - Nowacki, Igor (TS2 SPACE, 2023)
Stanford University and Research Institutions Release Model Transparency Index TS2 SPACE
<标题>: 斯坦福大学及研究机构发布模型透明度指数 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Stanford University and Research Institutions Release Model Transparency Index - Nowacki, Igor (TS2 SPACE, 2023)
Stanford University and Research Institutions Release Model Transparency Index TS2 SPACE
<标题>: 斯坦福大学及研究机构发布模型透明度指数 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z

The Internal State of an LLM Knows When It's Lying - Azaria, Amos; Mitchell, Tom (arXiv, 2023)
While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone. In this paper, we provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements. This includes both statements provided to the LLM, and statements that the LLM itself generates. Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement. Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71\% to 83\% accuracy labeling which sentences are true versus false, depending on the LLM base model. Furthermore, we explore the relationship between our classifier's performance and approaches based on the probability assigned to the sentence by the LLM. We show that while LLM-assigned sentence probability is related to sentence truthfulness, this probability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing a more reliable approach to detecting truthfulness, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios.
arXiv:2304.13734 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Advantages and pitfalls in utilizing artificial intelligence for crafting medical examinations: a medical education pilot study with GPT-4 - E, Klang; S, Portugez; R, Gross; R, Kassif Lerner; A, Brenner; M, Gilboa; T, Ortal; S, Ron; V, Robinzon; H, Meiri; G, Segal (BMC Medical Education, 2023)
Abstract Background The task of writing multiple choice question examinations for medical students is complex, timely and requires significant efforts from clinical staff and faculty. Applying artificial intelligence algorithms in this field of medical education may be advisable. Methods During March to April 2023, we utilized GPT-4, an OpenAI application, to write a 210 multi choice questions-MCQs examination based on an existing exam template and thoroughly investigated the output by specialist physicians who were blinded to the source of the questions. Algorithm mistakes and inaccuracies, as identified by specialists were classified as stemming from age, gender or geographical insensitivities. Results After inputting a detailed prompt, GPT-4 produced the test rapidly and effectively. Only 1 question (0.5%) was defined as false; 15% of questions necessitated revisions. Errors in the AI-generated questions included: the use of outdated or inaccurate terminology, age-sensitive inaccuracies, gender-sensitive inaccuracies, and geographically sensitive inaccuracies. Questions that were disqualified due to flawed methodology basis included elimination-based questions and questions that did not include elements of integrating knowledge with clinical reasoning. Conclusion GPT-4 can be used as an adjunctive tool in creating multi-choice question medical examinations yet rigorous inspection by specialist physicians remains pivotal.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Using large language models in psychology - Demszky, Dorottya; Yang, Diyi; Yeager, David S.; Bryan, Christopher J.; Clapper, Margarett; Chandhok, Susannah; Eichstaedt, Johannes C.; Hecht, Cameron; Jamieson, Jeremy; Johnson, Meghann; Jones, Michaela; Krettek-Cobb, Danielle; et al. (Nature Reviews Psychology, 2023)
<标题>: 在心理学中使用大型语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement - Sarkar, Surjodeep; Gaur, Manas; Chen, Lujie Karen; Garg, Muskan; Srivastava, Biplav (Frontiers in Artificial Intelligence, 2023)
Virtual Mental Health Assistants (VMHAs) continuously evolve to support the overloaded global healthcare system, which receives approximately 60 million primary care visits and 6 million emergency room visits annually. These systems, developed by clinical psychologists, psychiatrists, and AI researchers, are designed to aid in Cognitive Behavioral Therapy (CBT). The main focus of VMHAs is to provide relevant information to mental health professionals (MHPs) and engage in meaningful conversations to support individuals with mental health conditions. However, certain gaps prevent VMHAs from fully delivering on their promise during active communications. One of the gaps is their inability to explain their decisions to patients and MHPs, making conversations less trustworthy. Additionally, VMHAs can be vulnerable in providing unsafe responses to patient queries, further undermining their reliability. In this review, we assess the current state of VMHAs on the grounds of user-level explainability and safety, a set of desired properties for the broader adoption of VMHAs. This includes the examination of ChatGPT, a conversation agent developed on AI-driven models: GPT3.5 and GPT-4, that has been proposed for use in providing mental health services. By harnessing the collaborative and impactful contributions of AI, natural language processing, and the mental health professionals (MHPs) community, the review identifies opportunities for technological progress in VMHAs to ensure their capabilities include explainable and safe behaviors. It also emphasizes the importance of measures to guarantee that these advancements align with the promise of fostering trustworthy conversations.
<AI Smry>: The review identifies opportunities for technological progress in VMHAs to ensure their capabilities include explainable and safe behaviors and emphasizes the importance of measures to guarantee that these advancements align with the promise of fostering trustworthy conversations. <标题>: 对用于心理健康的会话代理的可解释性和安全性进行综述，以寻找改进的途径 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

eXplainable AI with GPT4 for story analysis and generation: A novel framework for diachronic sentiment analysis - Chun, Jon; Elkins, Katherine (International Journal of Digital Humanities, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

eXplainable AI with GPT4 for story analysis and generation: A novel framework for diachronic sentiment analysis - Chun, Jon; Elkins, Katherine (International Journal of Digital Humanities, 2023)
<标题>: 使用 GPT-4 的可解释人工智能进行故事分析与生成：一种新的历时情感分析框架 <AI Smry>: A novel XAI greybox ensemble for diachronic sentiment analysis generalizable to any AI classification data points within a structured time series under human-in-the-loop supervision (HITL). Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

A Human-AI Integrated Rating System for Improving Second Language Writing:The Case of Japanese Learners of English for General Academic Purposes - Spring, Ryan (2023)
Abstract In order to solve the problem of teachers not assigning and evaluating student writing but not completely trusting AI raters, I created and tested a rating scheme in which an AI model would rate students’ language use based on understandable criteria and humans would quickly check the AI responses while rating content and structure. Teachers tried the scheme and improvements were made based on new data and newly available research. An online practice tool was also created for students so that they could understand how the AI would rate their language use and practice accordingly. The AI rating models were improved over the course of three semesters based on student data and the ratings of external professional raters. As a result, an increasing number of teachers used the rating scheme, the number of students that practiced writing and were evaluated increased university-wide, and reasonable levels of fairness assessment were maintained.
<标题>: 一种人机整合评分系统以提高第二语言写作：以日语学习者的通用学术英语为例 Read_Status: New Read_Status_Date: 2026-01-26T11:32:30.842Z

SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning - Miao, Ning; Teh, Yee Whye; Rainforth, Tom (arXiv, 2023)
The recent progress in large language models (LLMs), especially the invention of chain-of-thought prompting, has made it possible to automatically answer questions by stepwise reasoning. However, when faced with more complicated problems that require non-linear thinking, even the strongest LLMs make mistakes. To address this, we explore whether LLMs are able to recognize errors in their own step-by-step reasoning, without resorting to external resources. To this end, we propose SelfCheck, a general-purpose zero-shot verification schema for recognizing such errors. We then use the results of these checks to improve question-answering performance by conducting weighted voting on multiple solutions to the question. We test SelfCheck on three datasets (GSM8K, MathQA, and MATH) and find that it successfully recognizes errors and, in turn, increases final answer accuracies.
arXiv:2308.00436 [cs] <标题>: 自我检查：使用大型语言模型零样本验证其自身的逐步推理 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning - Miao, Ning; Teh, Yee Whye; Rainforth, Tom (arXiv, 2023)
The recent progress in large language models (LLMs), especially the invention of chain-of-thought prompting, has made it possible to automatically answer questions by stepwise reasoning. However, when faced with more complicated problems that require non-linear thinking, even the strongest LLMs make mistakes. To address this, we explore whether LLMs are able to recognize errors in their own step-by-step reasoning, without resorting to external resources. To this end, we propose SelfCheck, a general-purpose zero-shot verification schema for recognizing such errors. We then use the results of these checks to improve question-answering performance by conducting weighted voting on multiple solutions to the question. We test SelfCheck on three datasets (GSM8K, MathQA, and MATH) and find that it successfully recognizes errors and, in turn, increases final answer accuracies.
arXiv:2308.00436 [cs] <标题>: 自我检查：使用大型语言模型零样本验证其自身的逐步推理 Read_Status: New Read_Status_Date: 2026-01-26T11:32:31.459Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Are LLMs Useful in the Poorest Schools? theTeacherAI in Sierra Leone - Choi, Jun Ho; Garrod, Oliver; Atherton, Paul; Joyce-Gibbons, Andrew; Mason-Sesay, Miriam; Björkegren, Daniel (arXiv, 2023)
Education systems in developing countries have few resources to serve large, poor populations. How might generative AI integrate into classrooms? This paper introduces an AI chatbot designed to assist teachers in Sierra Leone with professional development to improve their instruction. We describe initial findings from early implementation across 122 schools and 193 teachers, and analyze its use with qualitative observations and by analyzing queries. Teachers use the system for lesson planning, classroom management, and subject matter. A subset of teachers use the system intensively. We draw conclusions from these findings about how generative AI systems can be integrated into school systems in low income countries.
tex.ids= choiAreLLMsUseful2023a, choiAreLLMsUseful2023b arXiv: 2310.02982 [cs] issue: arXiv:2310.02982 <标题>: 大型语言模型在最贫困的学校有用吗？塞拉利昂的 TeacherAI <AI Smry>: An AI chatbot designed to assist teachers in Sierra Leone with professional development to improve their instruction is introduced, and initial findings from early implementation across 122 schools and 193 teachers are described. Read_Status: New Read_Status_Date: 2026-01-26T11:32:27.022Z
Keywords: /unread; Computer Science - Computers and Society

Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference - Levonian, Zachary; Li, Chenglu; Zhu, Wangda; Gade, Anoushka; Henkel, Owen; Postle, Millie-Ellen; Xing, Wanli (arXiv, 2023)
For middle-school math students, interactive question-answering (QA) with tutors is an effective way to learn. The flexibility and emergent capabilities of generative large language models (LLMs) has led to a surge of interest in automating portions of the tutoring process - including interactive QA to support conceptual discussion of mathematical concepts. However, LLM responses to math questions can be incorrect or mismatched to the educational context - such as being misaligned with a school's curriculum. One potential solution is retrieval-augmented generation (RAG), which involves incorporating a vetted external knowledge source in the LLM prompt to increase response quality. In this paper, we designed prompts that retrieve and use content from a high-quality open-source math textbook to generate responses to real student questions. We evaluate the efficacy of this RAG system for middle-school algebra and geometry QA by administering a multi-condition survey, finding that humans prefer responses generated using RAG, but not when responses are too grounded in the textbook content. We argue that while RAG is able to improve response quality, designers of math QA systems must consider trade-offs between generating responses preferred by students and responses closely matched to specific educational resources.
arXiv:2310.03184 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language; Computer Science - Human-Computer Interaction

A Survey on Recent Approaches to Question Difficulty Estimation from Text - Benedetto, Luca; Cremonesi, Paolo; Caines, Andrew; Buttery, Paula; Cappelli, Andrea; Giussani, Andrea; Turrin, Roberto (ACM Computing Surveys, 2023)
Question Difficulty Estimation from Text (QDET) is the application of Natural Language Processing techniques to the estimation of a value, either numerical or categorical, which represents the difficulty of questions in educational settings. We give an introduction to the field, build a taxonomy based on question characteristics, and present the various approaches that have been proposed in recent years, outlining opportunities for further research. This survey provides an introduction for researchers and practitioners into the domain of question difficulty estimation from text and acts as a point of reference about recent research in this topic to date.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

Investigating the Efficacy of Large Language Models in Reflective Assessment Methods through Chain of Thoughts Prompting - Masikisiki, Baphumelele; Marivate, Vukosi; Hlope, Yvette (arXiv, 2023)
Large Language Models, such as Generative Pre-trained Transformer 3 (aka. GPT-3), have been developed to understand language through the analysis of extensive text data, allowing them to identify patterns and connections between words. While LLMs have demonstrated impressive performance across various text-related tasks, they encounter challenges in tasks associated with reasoning. To address this challenge, Chain of Thought(CoT) prompting method has been proposed as a means to enhance LLMs' proficiency in complex reasoning tasks like solving math word problems and answering questions based on logical argumentative reasoning. The primary aim of this research is to assess how well four language models can grade reflective essays of third-year medical students. The assessment will specifically target the evaluation of critical thinking skills using CoT prompting. The research will provide the following contributions; to introduce and educate on the process of instructing models to evaluate reflective essays from a dataset they have not been previously trained on; to illustrate the use of CoT prompting as an instructional approach for training large models to carry out particular tasks. Our results suggest that among all the models, Llama-7b performs the least effectively, displaying the highest mean squared error. Conversely, ChatGPT emerges as the superior model, boasting a higher Cohen kappa score value of 0.53. Lastly, it's important to note that the selected models do prioritise user privacy by allowing users to delete their own conducted conversations.
arXiv:2310.00272 [cs] <标题>: 通过链式思维提示法研究大型语言模型在反思性评估方法中的有效性 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

The Promise of Automated Writing Evaluation for English Learners - Palermo, Corey; Wilson, Joshua (2023)
<标题>: 自动化写作评估对英语学习者的前景 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.078Z

OpenAI Cookbook (2023)
Open-source examples and guides for building with the OpenAI API.
<标题>: OpenAI 烹饪书 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.531Z
Keywords: /unread

Generative AI Text Classification using Ensemble LLM Approaches - Abburi, Harika; Suesserman, Michael; Pudota, Nirmala; Veeramani, Balaji; Bowen, Edward; Bhattacharya, Sanmitra (arXiv, 2023)
Large Language Models (LLMs) have shown impressive performance across a variety of Artificial Intelligence (AI) and natural language processing tasks, such as content creation, report generation, etc. However, unregulated malign application of these models can create undesirable consequences such as generation of fake news, plagiarism, etc. As a result, accurate detection of AI-generated language can be crucial in responsible usage of LLMs. In this work, we explore 1) whether a certain body of text is AI generated or written by human, and 2) attribution of a specific language model in generating a body of text. Texts in both English and Spanish are considered. The datasets used in this study are provided as part of the Automated Text Identification (AuTexTification) shared task. For each of the research objectives stated above, we propose an ensemble neural model that generates probabilities from different pre-trained LLMs which are used as features to a Traditional Machine Learning (TML) classifier following it. For the first task of distinguishing between AI and human generated text, our model ranked in fifth and thirteenth place (with macro $F1$ scores of 0.733 and 0.649) for English and Spanish texts, respectively. For the second task on model attribution, our model ranked in first place with macro $F1$ scores of 0.625 and 0.653 for English and Spanish texts, respectively.
arXiv:2309.07755 [cs] <标题>: 使用集成大语言模型方法的生成式人工智能文本分类 Read_Status: New Read_Status_Date: 2026-01-26T11:32:28.375Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning - Zadouri, Ted; Üstün, Ahmet; Ahmadian, Arash; Ermiş, Beyza; Locatelli, Acyr; Hooker, Sara (arXiv, 2023)
The Mixture of Experts (MoE) is a widely known neural architecture where an ensemble of specialized sub-models optimizes overall performance with a constant computational cost. However, conventional MoEs pose challenges at scale due to the need to store all experts in memory. In this paper, we push MoE to the limit. We propose extremely parameter-efficient MoE by uniquely combining MoE architecture with lightweight experts.Our MoE architecture outperforms standard parameter-efficient fine-tuning (PEFT) methods and is on par with full fine-tuning by only updating the lightweight experts – less than 1% of an 11B parameters model. Furthermore, our method generalizes to unseen tasks as it does not depend on any prior task knowledge. Our research underscores the versatility of the mixture of experts architecture, showcasing its ability to deliver robust performance even when subjected to rigorous parameter constraints. Our code used in all the experiments is publicly available here: https://github.com/for-ai/parameter-efficient-moe.
arXiv:2309.05444 [cs] <标题>: 将专家混合推向极限：用于指令调优的极度参数高效 MoE Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.303Z
Keywords: /unread; Computer Science - Computation and Language; Computer Science - Machine Learning

Comparing the Quality of Human and ChatGPT Feedback on Students’ Writing - Steiss, Jacob; Tate, Tamara; Graham, Steve; Cruz, Jazmin; Hebert, Michael; Wang, Jiali; Moon, Youngsun; Tseng, Waverly; Uci, Mark Warschauer (EdArXiv, 2023)
Offering students formative feedback on drafts of their writing is an effective way to facilitate writing development. This study examined the ability of generative AI (i.e., ChatGPT) to provide formative feedback on students’ compositions. We compared the quality of human and AI feedback by scoring the feedback each provided on secondary student essays (n=200) on five measures of feedback quality: the degree to which feedback (a) was criteria-based, (b) provided clear directions for improvement, (c) was accurate, (d) prioritized essential features, and (e) used a supportive tone. We examined whether ChatGPT and human evaluators provided feedback that differed in quality for native English speakers and English learners and for compositions that differed in overall quality. Results showed that human raters were better at providing high-quality feedback to students in all categories other than criteria-based. Considering the ease of generating feedback through ChatGPT and its overall quality, practical differences between humans and ChatGPT were not substantial. Feedback did not vary by language status for humans or AI, but AI and humans showed differences in feedback based on essay quality. Implications for generative AI as an educational tool are discussed.
<标题>: 比较人类与 ChatGPT 对学生写作反馈的质量 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; AI; AI Literacy; Artificial Intelligence; Education; Language and Literacy Education; automated writing evaluation; digital writing; feedback; secondary school; writing

Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text - Elkhatat, Ahmed M.; Elsaid, Khaled; Almeer, Saeed (International Journal for Educational Integrity, 2023)
The proliferation of artificial intelligence (AI)-generated content, particularly from models like ChatGPT, presents potential challenges to academic integrity and raises concerns about plagiarism. This study investigates the capabilities of various AI content detection tools in discerning human and AI-authored content. Fifteen paragraphs each from ChatGPT Models 3.5 and 4 on the topic of cooling towers in the engineering process and five human-witten control responses were generated for evaluation. AI content detection tools developed by OpenAI, Writer, Copyleaks, GPTZero, and CrossPlag were used to evaluate these paragraphs. Findings reveal that the AI detection tools were more accurate in identifying content generated by GPT 3.5 than GPT 4. However, when applied to human-written control responses, the tools exhibited inconsistencies, producing false positives and uncertain classifications. This study underscores the need for further development and refinement of AI content detection tools as AI-generated content becomes more sophisticated and harder to distinguish from human-written text.
<标题>: 评估人工智能内容检测工具在区分人类与人工智能生成文本中的有效性 <AI Smry>: Investigation of the capabilities of various AI content detection tools in discerning human and AI-authored content reveals that the AI detection tools were more accurate in identifying content generated by GPT 3.5 than GPT 4.5. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom) - Cheung, Billy Ho Hung; Lau, Gary Kui Kai; Wong, Gordon Tin Chun; Lee, Elaine Yuen Phin; Kulkarni, Dhananjay; Seow, Choon Sheong; Wong, Ruby; Co, Michael Tiong-Hong (PLOS ONE, 2023)
Large language models, in particular ChatGPT, have showcased remarkable language processing capabilities. Given the substantial workload of university medical staff, this study aims to assess the quality of multiple-choice questions (MCQs) produced by ChatGPT for use in graduate medical examinations, compared to questions written by university professoriate staffs based on standard medical textbooks.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.078Z

FacTool (2023)
<标题>: 面具工具 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

PsyArXiv Preprints | Reclaiming AI as a theoretical tool for cognitive science (2023)
<标题>: PsyArXiv 预印本 | 将人工智能重新作为认知科学的理论工具 Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.602Z
Keywords: /unread

Exploring the Potential and Limitations of Chat Generative Pre-trained Transformer (ChatGPT) in Generating Board-Style Dermatology Questions: A Qualitative Analysis - Ayub, Ibraheim; Hamann, Dathan; Hamann, Carsten R; Davis, Matthew J (Cureus, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate - Chan, Chi-Min; Chen, Weize; Su, Yusheng; Yu, Jianxuan; Xue, Wei; Zhang, Shanghang; Fu, Jie; Liu, Zhiyuan (arXiv, 2023)
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at https://github.com/chanchimin/ChatEval.
arXiv:2308.07201 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: /unread; Computer Science - Computation and Language; Reviewed

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate - Chan, Chi-Min; Chen, Weize; Su, Yusheng; Yu, Jianxuan; Xue, Wei; Zhang, Shanghang; Fu, Jie; Liu, Zhiyuan (arXiv, 2023)
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at https://github.com/chanchimin/ChatEval.
arXiv:2308.07201 [cs] <标题>: ChatEval：通过多智能体辩论实现更优的基于大型语言模型的评估器 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Computation and Language

The Future of AI in Education: 13 Things We Can Do to Minimize the Damage - Hamilton, Arran; Wiliam, Dylan; Hattie, John (2023)
We may already be in the era of ‘peak humanity’, a time where we have the greatest levels of education, reasoning, rationality, and creativity – spread out amongst the greatest number of us. A brilliant result of the massification of universal basic education and the power of the university. But with the rapid advancement of Artificial Intelligence (AI) that can already replicate and even exceed many of our reasoning capabilities – there may soon be less incentive for us to learn and grow. The grave risk is that we then become de-educated and de-coupled from the driving seat to the future. In all the hype about AI, we need to properly assess these risks to collectively decide whether the AI upsides are worth it and whether we should ‘stick or twist’. This paper aims to catalyse the debate and reduce the probability that we sleepwalk to a destination that we don’t want and can’t reverse back out of. We also make 13 clear recommendations about how AI developments could be regulated - to slow things down a little and give time for informed choices about the best future for humanity. Those potential long-term futures include: (1) AI Curtailment; (2) Fake Work; (3) Transhumanism; and (4) Universal Basic Income – each with very different implications for the future of education.
Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z

White House launches AI Cyber Challenge to test how top AI models protect software (2023)
Today the Biden-Harris Administration launched a two-year competition that uses AI to protect the U.S.'s most critical software.
<标题>: 白宫启动人工智能网络挑战赛，测试顶级人工智能模型如何保护软件 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

Stop Judging AI Using Human Exams - Roschelle, Jeremy (2023)
Computer scientists should inform the public that human tests are not a valid way to judge the quality of an AI model, nor a good way to compare AI models to human experts. Computer scientists should stop using human tests to measure AI.
<标题>: 不要用人类考试来评判人工智能 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Exploring the potential of using an AI language model for automated essay scoring - Mizumoto, Atsushi; Eguchi, Masaki (Research Methods in Applied Linguistics, 2023)
<标题>: 探索使用人工智能语言模型进行自动化作文评分的潜力 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z

The Power of Generative AI: A Review of Requirements, Models, Input–Output Formats, Evaluation Metrics, and Challenges - Bandi, Ajay; Adapa, Pydi Venkata Satya Ramesh; Kuchi, Yudu Eswar Vinay Pratap Kumar (Future Internet, 2023)
Generative artificial intelligence (AI) has emerged as a powerful technology with numerous applications in various domains. There is a need to identify the requirements and evaluation metrics for generative AI models designed for specific tasks. The purpose of the research aims to investigate the fundamental aspects of generative AI systems, including their requirements, models, input–output formats, and evaluation metrics. The study addresses key research questions and presents comprehensive insights to guide researchers, developers, and practitioners in the field. Firstly, the requirements necessary for implementing generative AI systems are examined and categorized into three distinct categories: hardware, software, and user experience. Furthermore, the study explores the different types of generative AI models described in the literature by presenting a taxonomy based on architectural characteristics, such as variational autoencoders (VAEs), generative adversarial networks (GANs), diffusion models, transformers, language models, normalizing flow models, and hybrid models. A comprehensive classification of input and output formats used in generative AI systems is also provided. Moreover, the research proposes a classification system based on output types and discusses commonly used evaluation metrics in generative AI. The findings contribute to advancements in the field, enabling researchers, developers, and practitioners to effectively implement and evaluate generative AI models for various applications. The significance of the research lies in understanding that generative AI system requirements are crucial for effective planning, design, and optimal performance. A taxonomy of models aids in selecting suitable options and driving advancements. Classifying input–output formats enables leveraging diverse formats for customized systems, while evaluation metrics establish standardized methods to assess model quality and performance.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z
Keywords: /unread

Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators - Denny, Paul; Leinonen, Juho; Prather, James; Luxton-Reilly, Andrew; Amarouche, Thezyrie; Becker, Brett A.; Reeves, Brent N. (arXiv, 2023)
With their remarkable ability to generate code, large language models (LLMs) are a transformative technology for computing education practice. They have created an urgent need for educators to rethink pedagogical approaches and teaching strategies for newly emerging skill sets. Traditional approaches to learning programming have focused on frequent and repeated practice at writing code. The ease with which code can now be generated has resulted in a shift in focus towards reading, understanding and evaluating LLM-generated code. In parallel with this shift, a new essential skill is emerging – the ability to construct good prompts for code-generating models. This paper introduces a novel pedagogical concept known as a `Prompt Problem', designed to help students learn how to craft effective prompts for LLMs. A Prompt Problem challenges a student to create a natural language prompt that leads an LLM to produce the correct code for a specific problem. To support the delivery of Prompt Problems at scale, in this paper we also present a novel tool called Promptly which hosts a repository of Prompt Problems and automates the evaluation of prompt-generated code. We report empirical findings from a field study in which Promptly was deployed in a first-year Python programming course (n=54). We explore student interactions with the tool and their perceptions of the Prompt Problem concept. We found that Promptly was largely well-received by students for its ability to engage their computational thinking skills and expose them to new programming constructs. We also discuss avenues for future work, including variations on the design of Prompt Problems and the need to study their integration into the curriculum and teaching practice.
arXiv:2307.16364 [cs] <标题>: 及时：使用提示问题教导学习者如何有效利用 AI 代码生成器 Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.004Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Human-Computer Interaction

Navigating A World of Generative AI: Suggestions for Educators - Cao, Lydia; Dede, Chris (Next Level Lab, Harvard Graduate School of Education, 2023)
Understanding the nature of generative AI is crucial for educators to navigate the evolving landscape of teaching and learning. In a new report from the Next Level Lab, Lydia Cao and Chris Dede reflect on the role of generative AI in learning and how this pushes us to reconceptualize our visions of effective education. Though...Continue Reading Navigating A World of Generative AI: Suggestions for Educators
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Co-designing teacher support technology for problem-based learning in middle school science - Hutchins, Nicole M.; Biswas, Gautam (British Journal of Educational Technology, 2023)
This paper provides an experience report on a co‐design approach with teachers to co‐create learning analytics‐based technology to support problem‐based learning in middle school science classrooms. We have mapped out a workflow for such applications and developed design narratives to investigate the implementation, modifications and temporal roles of the participants in the design process. Our results provide precedent knowledge on co‐designing with experienced and novice teachers and co‐constructing actionable insight that can help teachers engage more effectively with their students' learning and problem‐solving processes during classroom PBL implementations.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/bjet.13363 <标题>: 面向中学科学问题导向学习的教师支持技术共同设计 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: AI: Analytics; Educational Area: Data Quality & EMIS; Evidence: Self-reported Student Behaviour/ Attitude; Location: Americas; Research Design: Qualitative Exploratory; Research Method: Qualitative; Source: Closed Source

Llama 2: Open Foundation and Fine-Tuned Chat Models - Touvron, Hugo; Martin, Louis; Stone, Kevin; Albert, Peter; Almahairi, Amjad; Babaei, Yasmine; Bashlykov, Nikolay; Batra, Soumya; Bhargava, Prajjwal; Bhosale, Shruti; Bikel, Dan; Blecher, Lukas; et al. (2023)
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.
tex.ids= touvronLlama2Open2023 arXiv: 2307.09288 [cs] <标题>: Llama 2：开放基础模型与精调聊天模型 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Model Cards (2023)
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
<标题>: 模型卡 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.518Z
Keywords: /unread

Together AI launches full stack for developers to build with open-source AI (2023)
Simple, powerful, cost effective cloud services to train, fine-tune, and run the world’s leading open-source AI models
<标题>: Together AI 推出完整开发栈，供开发者使用开源 AI 构建应用 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.078Z
Keywords: /unread

GPT detectors are biased against non-native English writers - Liang, Weixin; Yuksekgonul, Mert; Mao, Yining; Wu, Eric; Zou, James (arXiv, 2023)
The rapid adoption of generative language models has brought about substantial advancements in digital communication, while simultaneously raising concerns regarding the potential misuse of AI-generated content. Although numerous detection methods have been proposed to differentiate between AI and human-generated content, the fairness and robustness of these detectors remain underexplored. In this study, we evaluate the performance of several widely-used GPT detectors using writing samples from native and non-native English writers. Our findings reveal that these detectors consistently misclassify non-native English writing samples as AI-generated, whereas native writing samples are accurately identified. Furthermore, we demonstrate that simple prompting strategies can not only mitigate this bias but also effectively bypass GPT detectors, suggesting that GPT detectors may unintentionally penalize writers with constrained linguistic expressions. Our results call for a broader conversation about the ethical implications of deploying ChatGPT content detectors and caution against their use in evaluative or educational settings, particularly when they may inadvertently penalize or exclude non-native English speakers from the global discourse. The published version of this study can be accessed at: www.cell.com/patterns/fulltext/S2666-3899(23)00130-7
arXiv:2304.02819 [cs] <AI Smry>: GPT detectors frequently misclassify non-native English writing as AI generated, raising concerns about fairness and robustness, and addressing the biases in these detectors is crucial to prevent the marginalization of non- natives in evaluative and educational settings. Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Human-Computer Interaction; Computer Science - Machine Learning

GPT detectors are biased against non-native English writers - Liang, Weixin; Yuksekgonul, Mert; Mao, Yining; Wu, Eric; Zou, James (arXiv, 2023)
The rapid adoption of generative language models has brought about substantial advancements in digital communication, while simultaneously raising concerns regarding the potential misuse of AI-generated content. Although numerous detection methods have been proposed to differentiate between AI and human-generated content, the fairness and robustness of these detectors remain underexplored. In this study, we evaluate the performance of several widely-used GPT detectors using writing samples from native and non-native English writers. Our findings reveal that these detectors consistently misclassify non-native English writing samples as AI-generated, whereas native writing samples are accurately identified. Furthermore, we demonstrate that simple prompting strategies can not only mitigate this bias but also effectively bypass GPT detectors, suggesting that GPT detectors may unintentionally penalize writers with constrained linguistic expressions. Our results call for a broader conversation about the ethical implications of deploying ChatGPT content detectors and caution against their use in evaluative or educational settings, particularly when they may inadvertently penalize or exclude non-native English speakers from the global discourse. The published version of this study can be accessed at: www.cell.com/patterns/fulltext/S2666-3899(23)00130-7
arXiv:2304.02819 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Human-Computer Interaction; Computer Science - Machine Learning

Using LLM (Large Language Model) to Improve Efficiency in Literature Review for Undergraduate Research - Antu, Shouvik Ahmed; Chen, Haiyan; Richards, Cindy K. (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z
Keywords: /unread

A Case Study using Large Language Models to Generate Metadata for Math Questions - Bainbridge, Katie; Walkington, Candace; Ibrahim, Armon; Zhong, Iris; Mallick, Debshila Basu; Washington, Julianna; Baraniuk, Rich (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 使用大型语言模型为数学题生成元数据的案例研究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

The Unseen A+ Student: Evaluating the Performance and Detectability of Large Language Models in High School Assignments - Boháček, Matyáš (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

On the Application of Large Language Models for Language Teaching and Assessment Technology - Caines, Andrew; Benedetto, Luca; Taslimipoor, Shiva; Davis, Christopher; Gao, Yuan; Andersen, Øistein; Yuan, Zheng; Elliott, Mark; Moore, Russell; Bryant, Christopher; Rei, Marek; Yannakoudakis, Helen; et al. (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
tex.ids= cainesApplicationLargeLanguage2023a <标题>: 大型语言模型在语言教学与评估技术中的应用 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Exploring Pre-Service Teachers’ Perceptions of Large Language Models-Generated Hints in Online Mathematics Learning - Gattupalli, Sai; Lee, Will; Allessio, Danielle; Crabtree, Danielle; Arroyo, Ivon; Woolf, Beverly (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Language Modeling for Plan Generation in Game-Base Learning Environments - Goslen, Alex; Kim, Yeo Jin; Rowe, Jonathan; Lester, James (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Detecting ChatGPT-Generated Code in a CS1 Course - Hoq, Muntasir; Shi, Yang; Leinonen, Juho; Babalola, Damilola; Lynch, Collin; Akram, Bita (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

An LLM-Powered Adaptive Practicing System - Kabir, Md Rayhan; Lin, Fuhua (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Leveraging LLMs for Adaptive Testing and Learning in Taiwan Adaptive Learning Platform (TALP) - Kuo, Bor-Chen; Chang, Frederic T. Y.; Bai, Zong-En (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 在台湾自适应学习平台（TALP）中利用大型语言模型进行自适应测试和学习 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.078Z
Keywords: /unread

Prototyping the use of Large Language Models (LLMs) for Adult Learning Content Creation at Scale - Leiker, Daniel; Finnigan, Sara; Gyllen, Ashley Ricker; Cukurova, Mutlu (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming - Ma, Qianou Christina; Wu, Sherry Tongshuang; Koedinger, Ken (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 人工智能是更好的编程伙伴吗？人类-人类结对编程 vs. 人类-人工智能结对编程 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Preface Summary: There were 38 papers submitted for peer-review to this workshop. Out of these, 17 papers were accepted for this volume. - Moore, Steven; Stamper, John; Tong, Richard; Cao, Chen; Liu, Zitao; Hu, Xiangen; Lu, Yu; Liang, Joleen; Khosravi, Hassan; Denny, Paul; Singh, Anjali; Brooks, Chris (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 前言摘要：本次研讨会共收到38篇论文提交进行同行评审。其中，有17篇论文被本卷收录。 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Rewriting Math Word Problems with Large Language Models - Norberg, Kole; Almoubayyed, Husni; Fancsali, Stephen E.; Ley, Logan De; Weldon, Kyle; Murphy, April; Ritter, Steve (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 使用大型语言模型改写数学应用题 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Generative Large Language Models for Dialog-Based Tutoring: An Early Consideration of Opportunities and Concerns - Nye, Benjamin D.; Mee, Dillon; Core, Mark G. (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z
Keywords: /unread

Generating Multiple Choice Questions from a Textbook: LLMs Match Human Performance on Most Metrics - Olney, Andrew M. (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 从教科书生成选择题：大语言模型在大多数指标上达到人类水平 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Deduction under Perturbed Evidence: Probing Student Simulation (Knowledge Tracing) Capabilities of Large Language Models - Sonkar, Shashank; Baraniuk, Richard G. (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 在扰动证据下的推理：探讨大型语言模型的学生模拟（知识追踪）能力 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectives - Sridhar, Pragnya; Doyle, Aidan; Agarwal, Arav; Bogart, Christopher; Savelka, Jaromir; Sakr, Majd (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 在课程设计中利用大语言模型：使用 GPT-4 支持学习目标的撰写 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: /unread

Contextualizing Problems to Student Interests at Scale in Intelligent Tutoring System Using Large Language Models - Yadav, Gautam; Tseng, Ying-Jui; Ni, Xiaolin (Proceedings of the Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation 2023, 2023)
<标题>: 在智能辅导系统中使用大型语言模型将问题与学生兴趣进行大规模情境化 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z
Keywords: /unread

Auto-scoring Student Responses with Images in Mathematics - Baral, Sami; Botelho, Anthony; Abhishek Santhanam; Gurung, Ashish; Cheng, Li; Heffernan, Neil ([object Object], 2023)
Teachers often rely on the use of a range of open-ended problems to assess students' understanding of mathematical concepts. Beyond traditional conceptions of student open-ended work, commonly in the form of textual short-answer or essay responses, the use of figures, tables, number lines, graphs, and pictographs are other examples of open-ended work common in mathematics. While recent developments in areas of natural language processing and machine learning have led to automated methods to score student open-ended work, these methods have largely been limited to textual answers. Several computer-based learning systems allow students to take pictures of hand-written work and include such images within their answers to open-ended questions. With that, however, there are few-to-no existing solutions that support the auto-scoring of student hand-written or drawn answers to questions. In this work, we build upon an existing method for auto-scoring textual student answers and explore the use of OpenAI/CLIP, a deep learning embedding method designed to represent both images and text, as well as Optical Character Recognition (OCR) to improve model performance. We evaluate the performance of our method on a dataset of student open-responses that contains both text- and image-based responses, and find a reduction of model error in the presence of images when controlling for other answer-level features.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - AIED (2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - AIED (2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z
Keywords: /unread

Accused: How students respond to allegations of using ChatGPT on assessments - Gorichanaz, Tim (Learning: Research and Practice, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

GPT detectors are biased against non-native English writers - Liang, Weixin; Yuksekgonul, Mert; Mao, Yining; Wu, Eric; Zou, James (Patterns, 2023)
<标题>: GPT 检测器对非英语母语的写作者存在偏见 <AI Smry>: GPT detectors frequently misclassify non-native English writing as AI generated, raising concerns about fairness and robustness, and addressing the biases in these detectors is crucial to prevent the marginalization of non- natives in evaluative and educational settings. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z

GPT-3-driven pedagogical agents for training children's curious question-asking skills - Abdelghani, Rania; Wang, Yen-Hsiang; Yuan, Xingdi; Wang, Tong; Lucas, Pauline; Sauzéon, Hélène; Oudeyer, Pierre-Yves (International Journal of Artificial Intelligence in Education, 2023)
In order to train children's ability to ask curiosity-driven questions, previous research has explored designing specific exercises relying on providing semantic and linguistic cues to help formulate such questions. But despite showing pedagogical efficiency, this method is still limited as it relies on generating the said cues by hand, which can be a very costly process. In this context, we propose to leverage advances in the natural language processing field (NLP) and investigate the efficiency of using a large language model (LLM) for automating the production of the pedagogical content of a curious question-asking (QA) training. We study generating the said content using the "prompt-based" method that consists of explaining the task to the LLM in natural text. We evaluate the output using human experts annotations and comparisons with hand-generated content. Results suggested indeed the relevance and usefulness of this content. We also conduct a field study in primary school (75 children aged 9-10), where we evaluate children's QA performance when having this training. We compare 3 types of content : 1) hand-generated content that proposes "closed" cues leading to predefined questions; 2) GPT-3-generated content that proposes the same type of cues; 3) GPT-3-generated content that proposes "open" cues leading to several possible questions. We see a similar QA performance between the two "closed" trainings (showing the scalability of the approach using GPT-3), and a better one for participants with the "open" training. These results suggest the efficiency of using LLMs to support children in generating more curious questions, using a natural language prompting approach that affords usability by teachers and other users not specialists of AI techniques. Furthermore, results also show that open-ended content may be more suitable for training curious question-asking skills.
arXiv:2211.14228 [cs] <标题>: 基于GPT-3的教学代理用于培养儿童的好奇提问能力 <AI Smry>: The efficiency of using large language model (LLM) to support children in generating more curious questions is suggested, using a natural language prompting approach that affords usability by teachers and other users not specialists of AI techniques. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: Computer Science - Computation and Language; Computer Science - Human-Computer Interaction

EdArXiv Preprints | From nCoder to ChatGPT: From Automated Coding to Refining Human Coding (2023)
<标题>: EdArXiv 预印本 | 从 nCoder 到 ChatGPT：从自动编码到优化人工编码 Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.602Z
Keywords: /unread

Inside the AI Factory - Dzieza, Josh (2023)
How many humans does it take to make tech seem human? Millions.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Artificial Intelligence: assessing performance claims (2023)
If the brains behind Artificial Intelligence claim their creations can perform in tests like humans then surely those results should be assessed as if they were produced by humans. AQA's Head of Research and Development, Dr Cesare Aloisi, says this is vital to maintain trust in AI but fears that is not what is happening
<标题>: 人工智能：评估性能声明 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

ChatGPT (2023)
A conversational AI system that listens, learns, and challenges
<标题>: 聊天生成预训练变换器 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

No, GPT4 can’t ace MIT - Chowdhuri, Raunak; Deshmukh, Neil; Doplow, David (Notion, 2023)
A new tool that blends your everyday work apps into one. It's the all-in-one workspace for you and your team
<标题>: 不，GPT-4 不能轻松通过麻省理工学院 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread

WizardCoder: Empowering Code Large Language Models with Evol-Instruct - Luo, Ziyang; Xu, Can; Zhao, Pu; Sun, Qingfeng; Geng, Xiubo; Hu, Wenxiang; Tao, Chongyang; Ma, Jing; Lin, Qingwei; Jiang, Daxin (arXiv, 2023)
Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated exceptional performance in code-related tasks. However, most existing models are solely pre-trained on extensive raw code data without instruction fine-tuning. In this paper, we introduce WizardCoder, which empowers Code LLMs with complex instruction fine-tuning, by adapting the Evol-Instruct method to the domain of code. Through comprehensive experiments on four prominent code generation benchmarks, namely HumanEval, HumanEval+, MBPP, and DS-1000, we unveil the exceptional capabilities of our model. It surpasses all other open-source Code LLMs by a substantial margin. Moreover, our model even outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on HumanEval and HumanEval+. Our code, model weights, and data are public at https://github.com/nlpxucan/WizardLM
arXiv:2306.08568 [cs] <标题>: WizardCoder：通过 Evol-Instruct 赋能大型代码语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

The “it” in AI models is the dataset. – Non_Interactive – Software & ML (2023)
<标题>: AI 模型中的“它”是数据集。– 非交互式 – 软件与机器学习 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

WizardLM: Empowering Large Language Models to Follow Complex Instructions - Xu, Can; Sun, Qingfeng; Zheng, Kai; Geng, Xiubo; Zhao, Pu; Feng, Jiazhan; Tao, Chongyang; Jiang, Daxin (arXiv, 2023)
Training large language models (LLMs) with open-domain instruction following data brings colossal success. However, manually creating such instruction data is very time-consuming and labor-intensive. Moreover, humans may struggle to produce high-complexity instructions. In this paper, we show an avenue for creating large amounts of instruction data with varying levels of complexity using LLM instead of humans. Starting with an initial set of instructions, we use our proposed Evol-Instruct to rewrite them step by step into more complex instructions. Then, we mix all generated instruction data to fine-tune LLaMA. We call the resulting model WizardLM. Human evaluations on a complexity-balanced test bed and Vicuna's testset show that instructions from Evol-Instruct are superior to human-created ones. By analyzing the human evaluation results of the high complexity part, we demonstrate that outputs from our WizardLM are preferred to outputs from OpenAI ChatGPT. In GPT-4 automatic evaluation, WizardLM achieves more than 90\% capacity of ChatGPT on 17 out of 29 skills. Even though WizardLM still lags behind ChatGPT in some aspects, our findings suggest that fine-tuning with AI-evolved instructions is a promising direction for enhancing LLMs. Our code and data are public at https://github.com/nlpxucan/WizardLM
arXiv:2304.12244 [cs] <AI Smry>: The findings suggest that fine-tuning with AI-evolved instructions is a promising direction for enhancing LLMs, and it is demonstrated that outputs from the authors' WizardLM are preferred to outputs from OpenAI ChatGPT. <标题>: WizardLM：赋能大型语言模型以遵循复杂指令 Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.602Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Unbias me! Mitigating Algorithmic Bias for Less-studied Demographic Groups in the Context of Language Learning Technology - Rzepka, Nathalie; Fernsel, Linda; Müller, Hans-Georg; Simbeck, Katharina; Pinkwart, Neils (EdArXiv, 2023)
Algorithms and machine learning models are being used more frequently in educational settings, but there are concerns that they may discriminate against certain groups. While there is some research on algorithmic fairness, there are two main issues with the current research. Firstly, it often focuses on gender and race and ignores other groups. Secondly, studies often find algorithmic bias in educational models but don't explore ways to reduce it. This study evaluates three drop-out prediction models used in an online learning platform to teach German spelling skills. The aim is to assess the fairness of the models for (in part) less-studied demographic groups, including first spoken language, home literacy environment, parental education background, and gender. To evaluate the models, four fairness metrics are used: predictive parity, equalized odds, predictive equality, and ABROCA. The study also examines ways to reduce algorithmic bias by analyzing the models at each stage of the machine learning process. The results show that all three models had biases that affected the fairness of all four demographic groups to varying degrees. However, the study found that most biases could be mitigated during the process. The methods used to mitigate bias differed by demographic group, and some methods improved fairness for one group but worsened it for others. Therefore, the study concludes that reducing algorithmic bias for lessstudied demographic groups is possible, but finding the right method for each algorithm and demographic group is crucial.
<标题>: 消除偏见！在语言学习技术背景下缓解针对研究较少的人口群体的算法偏见 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread

Can Automated Feedback Improve Teachers’ Uptake of Student Ideas? Evidence From a Randomized Controlled Trial In a Large-Scale Online Course - Demszky, Dorottya; Liu, Jing; Hill, Heather C.; Jurafsky, Dan; Piech, Chris (Annenberg Institute at Brown University, 2023)
Providing consistent, individualized feedback to teachers is essential for improving instruction but can be prohibitively resource-intensive in most educational contexts. We develop M-Powering Teachers, an automated tool based on natural language processing to give teachers feedback on their uptake of student contributions, a high-leverage dialogic teaching practice that makes students feel heard. We conduct a randomized controlled trial in an online computer science course (n=1,136 instructors), to evaluate the effectiveness of our tool.
Publication Title: EdWorkingPapers.com Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction - Wang, Rose; Demszky, Dorottya (EdArXiv, 2023)
Coaching, which involves classroom observation and expert feedback, is a widespread and fundamental part of teacher training. However, the majority of teachers do not have access to consistent, high quality coaching due to limited resources and access to expertise. We explore whether generative AI could become a cost-effective complement to expert feedback by serving as an automated teacher coach. In doing so, we propose three teacher coaching tasks for generative AI: (A) scoring transcript segments based on classroom observation instruments, (B) identifying highlights and missed opportunities for good instructional strategies, and (C) providing actionable suggestions for eliciting more student reasoning. We recruit expert math teachers to evaluate the zero-shot performance of ChatGPT on each of these tasks for elementary math classroom transcripts. Our results reveal that ChatGPT generates responses that are relevant to improving instruction, but they are often not novel or insightful. For example, 82% of the model's suggestions point to places in the transcript where the teacher is already implementing that suggestion. Our work highlights the challenges of producing insightful, novel and truthful feedback for teachers while paving the way for future research to address these obstacles and improve the capacity of generative AI to coach teachers.
<标题>: ChatGPT 是一个好的教师指导工具吗？衡量零样本表现以评估课堂教学并提供可操作的见解 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.511Z
Keywords: /unread

Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction - Wang, Rose; Demszky, Dorottya (EdArXiv, 2023)
Coaching, which involves classroom observation and expert feedback, is a widespread and fundamental part of teacher training. However, the majority of teachers do not have access to consistent, high quality coaching due to limited resources and access to expertise. We explore whether generative AI could become a cost-effective complement to expert feedback by serving as an automated teacher coach. In doing so, we propose three teacher coaching tasks for generative AI: (A) scoring transcript segments based on classroom observation instruments, (B) identifying highlights and missed opportunities for good instructional strategies, and (C) providing actionable suggestions for eliciting more student reasoning. We recruit expert math teachers to evaluate the zero-shot performance of ChatGPT on each of these tasks for elementary math classroom transcripts. Our results reveal that ChatGPT generates responses that are relevant to improving instruction, but they are often not novel or insightful. For example, 82% of the model's suggestions point to places in the transcript where the teacher is already implementing that suggestion. Our work highlights the challenges of producing insightful, novel and truthful feedback for teachers while paving the way for future research to address these obstacles and improve the capacity of generative AI to coach teachers.
<标题>: ChatGPT 是好老师教练吗？评估零样本性能以对课堂教学进行评分并提供可操作的见解 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction - Wang, Rose; Demszky, Dorottya (OSF, 2023)
Coaching, which involves classroom observation and expert feedback, is a widespread and fundamental part of teacher training. However, the majority of teachers do not have access to consistent, high quality coaching due to limited resources and access to expertise. We explore whether generative AI could become a cost-effective complement to expert feedback by serving as an automated teacher coach. In doing so, we propose three teacher coaching tasks for generative AI: (A) scoring transcript segments based on classroom observation instruments, (B) identifying highlights and missed opportunities for good instructional strategies, and (C) providing actionable suggestions for eliciting more student reasoning. We recruit expert math teachers to evaluate the zero-shot performance of ChatGPT on each of these tasks for elementary math classroom transcripts. Our results reveal that ChatGPT generates responses that are relevant to improving instruction, but they are often not novel or insightful. For example, 82% of the model's suggestions point to places in the transcript where the teacher is already implementing that suggestion. Our work highlights the challenges of producing insightful, novel and truthful feedback for teachers while paving the way for future research to address these obstacles and improve the capacity of generative AI to coach teachers.
<标题>: ChatGPT 是好老师教练吗？衡量零样本性能以评分并提供课堂教学的可操作见解 Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.602Z
Keywords: /unread

Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction - Wang, Rose; Demszky, Dorottya (EdArXiv, 2023)
Coaching, which involves classroom observation and expert feedback, is a widespread and fundamental part of teacher training. However, the majority of teachers do not have access to consistent, high quality coaching due to limited resources and access to expertise. We explore whether generative AI could become a cost-effective complement to expert feedback by serving as an automated teacher coach. In doing so, we propose three teacher coaching tasks for generative AI: (A) scoring transcript segments based on classroom observation instruments, (B) identifying highlights and missed opportunities for good instructional strategies, and (C) providing actionable suggestions for eliciting more student reasoning. We recruit expert math teachers to evaluate the zero-shot performance of ChatGPT on each of these tasks for elementary math classroom transcripts. Our results reveal that ChatGPT generates responses that are relevant to improving instruction, but they are often not novel or insightful. For example, 82% of the model's suggestions point to places in the transcript where the teacher is already implementing that suggestion. Our work highlights the challenges of producing insightful, novel and truthful feedback for teachers while paving the way for future research to address these obstacles and improve the capacity of generative AI to coach teachers.
<标题>: ChatGPT 是好老师教练吗？评估零样本性能以对课堂教学进行评分并提供可操作的见解 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions - Zhang, Mengxue; Heffernan, Neil; Lan, Andrew (arXiv, 2023)
Automated scoring of student responses to open-ended questions, including short-answer questions, has great potential to scale to a large number of responses. Recent approaches for automated scoring rely on supervised learning, i.e., training classifiers or fine-tuning language models on a small number of responses with human-provided score labels. However, since scoring is a subjective process, these human scores are noisy and can be highly variable, depending on the scorer. In this paper, we investigate a collection of models that account for the individual preferences and tendencies of each human scorer in the automated scoring task. We apply these models to a short-answer math response dataset where each response is scored (often differently) by multiple different human scorers. We conduct quantitative experiments to show that our scorer models lead to improved automated scoring accuracy. We also conduct quantitative experiments and case studies to analyze the individual preferences and tendencies of scorers. We found that scorers can be grouped into several obvious clusters, with each cluster having distinct features, and analyzed them in detail.
arXiv:2306.00791 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions - Zhang, Mengxue; Heffernan, Neil; Lan, Andrew (arXiv, 2023)
Automated scoring of student responses to open-ended questions, including short-answer questions, has great potential to scale to a large number of responses. Recent approaches for automated scoring rely on supervised learning, i.e., training classifiers or fine-tuning language models on a small number of responses with human-provided score labels. However, since scoring is a subjective process, these human scores are noisy and can be highly variable, depending on the scorer. In this paper, we investigate a collection of models that account for the individual preferences and tendencies of each human scorer in the automated scoring task. We apply these models to a short-answer math response dataset where each response is scored (often differently) by multiple different human scorers. We conduct quantitative experiments to show that our scorer models lead to improved automated scoring accuracy. We also conduct quantitative experiments and case studies to analyze the individual preferences and tendencies of scorers. We found that scorers can be grouped into several obvious clusters, with each cluster having distinct features, and analyzed them in detail.
arXiv:2306.00791 [cs] <标题>: 短答数学题评分者偏好建模与分析 Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.807Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Artificial intelligence and the future of evaluation education: Possibilities and prototypes - Tilton, Zach; LaVelle, John M.; Ford, Tian; Montenegro, Maria (New Directions for Evaluation, 2023)
Advancements in Artificial Intelligence (AI) signal a paradigmatic shift with the potential for transforming many various aspects of society, including evaluation education, with implications for subsequent evaluation practice. This article explores the potential implications of AI for evaluator and evaluation education. Specifically, the article discusses key issues in evaluation education including equitable language access to evaluation education, navigating program, social science, and evaluation theory, understanding evaluation theorists and their philosophies, and case studies and simulations. The paper then considers how chatbots might address these issues, and documents efforts to prototype chatbots for three use cases in evaluation education, including a guidance counselor, teaching assistant, and mentor chatbot for young and emerging evaluations or anyone who wants to use it. The paper concludes with ruminations on additional research and activities on evaluation education topics such as how to best integrate evaluation literacy training into existing programs, making strategic linkages for practitioners, and evaluation educators.
<AI Smry>: Key issues in evaluation education including equitable language access to evaluation education, navigating program, social science, and evaluation theory, understanding evaluation theorists and their philosophies, and case studies and simulations are discussed. <标题>: 人工智能与评估教育的未来：可能性与原型 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: /unread

Artificial intelligence and the future of evaluation education: Possibilities and prototypes - Tilton, Zach; LaVelle, John M.; Ford, Tian; Montenegro, Maria (New Directions for Evaluation, 2023)
Advancements in Artificial Intelligence (AI) signal a paradigmatic shift with the potential for transforming many various aspects of society, including evaluation education, with implications for subsequent evaluation practice. This article explores the potential implications of AI for evaluator and evaluation education. Specifically, the article discusses key issues in evaluation education including equitable language access to evaluation education, navigating program, social science, and evaluation theory, understanding evaluation theorists and their philosophies, and case studies and simulations. The paper then considers how chatbots might address these issues, and documents efforts to prototype chatbots for three use cases in evaluation education, including a guidance counselor, teaching assistant, and mentor chatbot for young and emerging evaluations or anyone who wants to use it. The paper concludes with ruminations on additional research and activities on evaluation education topics such as how to best integrate evaluation literacy training into existing programs, making strategic linkages for practitioners, and evaluation educators.
<标题>: 人工智能与评估教育的未来：可能性与原型 <AI Smry>: Key issues in evaluation education including equitable language access to evaluation education, navigating program, social science, and evaluation theory, understanding evaluation theorists and their philosophies, and case studies and simulations are discussed. Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

EU-U.S. Terminology and Taxonomy for Artificial Intelligence (2023)
Following the AI Roadmap suggestions for concrete activities aimed at aligning EU and U.S. risk-based approaches, a group of experts engaged to prepare an initial draft AI terminologies and taxonomies. A total number of 65 terms were identified with reference to key documents from the EU and the U.S.
<标题>: 欧盟-美国人工智能术语和分类法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread

Academic performance prediction in a gender-imbalanced environment - Sapiezynski, Piotr; Kassarnig, Valentin; Wilson, Christo (2023)
Institution: Boise State University <标题>: 在性别不平衡环境中的学业表现预测 <AI Smry>: This paper investigates the low academic performance predictors of female and male participants of the Copenhagen Networks Study and proposes an own method that outperforms the alternatives on the dataset in question. Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z

Automated speech scoring of dialogue response by Japanese learners of English as a foreign language - Hayashi, Yuko; Kondo, Yusuke; Ishii, Yutaka (Innovation in Language Learning and Teaching, 2023)
<标题>: 对日语学习者英语作为外语的对话反应进行自动语音评分 <AI Smry>: A new reliable scoring system for learner speech using an automated DCT with multiple turns is presented, which offers useful insight into ways in which the system could be used in a low-stakes environment, including foreign language classroom settings. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Awesome-LLM - Hannibal046 (2023)
Awesome-LLM: a curated list of Large Language Model
Programmers: _:n9877 original-date: 2023-02-17T14:12:21Z Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Psychometric Considerations When Using Deep Learning for Automated Scoring - Lottridge, Susan; Ormerod, Chris; Jafari, Amir (Advancing Natural Language Processing in Educational Assessment, 2023)
Book Authors: _:n9533 <标题>: 使用深度学习进行自动评分时的心理测量学考量 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

Psychometric Considerations When Using Deep Learning for Automated Scoring - Lottridge, Susan; Ormerod, Chris; Jafari, Amir (Advancing Natural Language Processing in Educational Assessment, 2023)
Book Authors: _:n13369 <标题>: 使用深度学习进行自动评分时的心理测量学考量 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

A.I. poses existential risk of people being 'harmed or killed,' ex-Google CEO Eric Schmidt says - Kharpal, Arjun (2023)
Artificial intelligence has been thrust into the center of conversations among policymakers grappling with what the tech should look like in the future.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

QLoRA: Efficient Finetuning of Quantized LLMs - Dettmers, Tim; Pagnoni, Artidoro; Holtzman, Ari; Zettlemoyer, Luke (arXiv, 2023)
We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.
arXiv:2305.14314 [cs] <标题>: QLoRA：量化大型语言模型的高效微调 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Machine Learning

Exploring User Perspectives on ChatGPT: Applications, Perceptions, and Implications for AI-Integrated Education - Mogavi, Reza Hadi; Deng, Chao; Kim, Justin Juho; Zhou, Pengyuan; Kwon, Young D.; Metwally, Ahmed Hosny Saleh; Tlili, Ahmed; Bassanelli, Simone; Bucchiarone, Antonio; Gujar, Sujit; Nacke, Lennart E.; Hui, Pan (2023)
Understanding user perspectives on Artificial Intelligence (AI) in education is essential for creating pedagogically effective and ethically responsible AI-integrated learning environments. In this paper, we conduct an extensive qualitative content analysis of four major social media platforms (Twitter, Reddit, YouTube, and LinkedIn) to explore the user experience (UX) and perspectives of early adopters toward ChatGPT-an AI Chatbot technology-in various education sectors. We investigate the primary applications of ChatGPT in education (RQ1) and the various perceptions of the technology (RQ2). Our findings indicate that ChatGPT is most popularly used in the contexts of higher education (24.18%), K-12 education (22.09%), and practical-skills learning (15.28%). On social media platforms, the most frequently discussed topics about ChatGPT are productivity, efficiency, and ethics. While early adopters generally lean toward seeing ChatGPT as a revolutionary technology with the potential to boost students' self-efficacy and motivation to learn, others express concern that overreliance on the AI system may promote superficial learning habits and erode students' social and critical thinking skills. Our study contributes to the broader discourse on Human-AI Interaction and offers recommendations based on crowd-sourced knowledge for educators and learners interested in incorporating ChatGPT into their educational settings. Furthermore, we propose a research agenda for future studies that sets the foundation for continued investigation into the application of ChatGPT in education.
arXiv:2305.13114 [cs] <AI Smry>: The results of the research show that ChatGPT is most commonly used in the domains of higher education, K-12 education, and practical skills training, and the topics most frequently associated with ChatG PT are productivity, efficiency, and ethics. <标题>: 探索用户对 ChatGPT 的看法：应用、认知及对 AI 融合教育的影响 Read_Status: New Read_Status_Date: 2026-01-26T11:32:23.432Z
Keywords: /unread; Computer Science - Computers and Society; Computer Science - Human-Computer Interaction

Are Emergent Abilities of Large Language Models a Mirage? - Schaeffer, Rylan; Miranda, Brando; Koyejo, Sanmi (arXiv, 2023)
Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance. We present our alternative explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities; (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show to choose metrics to produce never-before-seen seemingly emergent abilities in multiple vision tasks across diverse deep networks. Via all three analyses, we provide evidence that alleged emergent abilities evaporate with different metrics or with better statistics, and may not be a fundamental property of scaling AI models.
arXiv:2304.15004 <AI Smry>: Evidence is provided that alleged emergent abilities evaporate with different metrics or with better statistics, and may not be a fundamental property of scaling AI models. <标题>: 大型语言模型的新兴能力是幻觉吗？ Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.004Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Machine Learning

Large, creative AI models will transform lives and labour markets (The Economist, 2023)
<标题>: 大型创意人工智能模型将改变生活和劳动市场 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Reflexion: Language Agents with Verbal Reinforcement Learning - Shinn, Noah; Cassano, Federico; Labash, Beck; Gopinath, Ashwin; Narasimhan, Karthik; Yao, Shunyu (arXiv, 2023)
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.
arXiv:2303.11366 [cs] Issue: arXiv:2303.11366 <标题>: 反思：具有语言强化学习的语言代理 Read_Status: New Read_Status_Date: 2026-01-26T11:32:27.022Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

How Well Do Large Language Models Support Clinician Information Needs? (Stanford HAI, 2023)
Stanford experts examine the safety and accuracy of GPT-4 in serving curbside consultation needs of doctors.
<标题>: 大型语言模型在多大程度上支持临床医生的信息需求？ Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.518Z
Keywords: /unread

PaLM 2 Technical Report - Anil, Rohan; Dai, Andrew M.; Firat, Orhan; Johnson, Melvin; Lepikhin, Dmitry; Passos, Alexandre; Shakeri, Siamak; Taropa, Emanuel; Bailey, Paige; Chen, Zhifeng; Chu, Eric; Clark, Jonathan H.; et al. (arXiv, 2023)
We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish between pre-trained models (of various sizes), fine-tuned variants of these models, and the user-facing products that use these models. In particular, user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report.
arXiv:2305.10403 [cs] <AI Smry>: PaLM 2 is a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; technical

Automatic item generation: foundations and machine learning-based approaches for assessments - Circi, Ruhan; Hicks, Juanita; Sikali, Emmanuel (Frontiers in Education, 2023)
This mini review summarizes the current state of knowledge about automatic item generation in the context of educational assessment and discusses key points in the item generation pipeline. Assessment is critical in all learning systems and digitalized assessments have shown significant growth over the last decade. This leads to an urgent need to generate more items in a fast and efficient manner. Continuous improvements in computational power and advancements in methodological approaches, specifically in the field of natural language processing, provide new opportunities as well as new challenges in automatic generation of items for educational assessment. This mini review asserts the need for more work across a wide variety of areas for the scaled implementation of AIG.
<标题>: 自动题目生成：评估的基础与基于机器学习的方法 <AI Smry>: This mini review asserts the need for more work across a wide variety of areas for the scaled implementation of AIG and discusses key points in the item generation pipeline. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Can ChatGPT and Bard generate aligned assessment items? A reliability analysis against human performance - Khademi Khademi University of Massachusetts, Am, Abdolvahab (Journal of Applied Learning & Teaching, 2023)
ChatGPT and Bard are AI chatbots based on Large Language Models (LLM) that are slated to promise different applications in diverse areas. In education, these AI technologies have been tested for applications in assessment and teaching. In assessment, AI has long been used in automated essay scoring and automated item generation. One psychometric property that these tools must have to assist or replace humans in assessment is high reliability in terms of agreement between AI scores and human raters. In this paper, we measure the reliability of OpenAI ChatGP and Google Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts. Intraclass correlation (ICC) as a performance metric showed that the inter-reliability of both the OpenAI ChatGPT and the Google Bard were low against the gold standard of human ratings.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z

Can ChatGPT and Bard Generate Aligned Assessment Items? A Reliability Analysis against Human Performance - Khademi, Abdolvahab (Journal of Applied Learning & Teaching, 2023)
ChatGPT and Bard are AI chatbots based on Large Language Models (LLM) that are slated to promise different applications in diverse areas. In education, these AI technologies have been tested for applications in assessment and teaching. In assessment, AI has long been used in automated essay scoring and automated item generation. One psychometric property that these tools must have to assist or replace humans in assessment is high reliability in terms of agreement between AI scores and human raters. In this paper, we measure the reliability of OpenAI ChatGP and Google Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts. Intraclass correlation (ICC) as a performance metric showed that the inter-reliability of both the OpenAI ChatGPT and the Google Bard were low against the gold standard of human ratings.
arXiv:2304.05372 [cs] <标题>: ChatGPT 和 Bard 能生成一致的评估项目吗？与人类表现的可靠性分析 <AI Smry>: Measure the reliability of OpenAI ChatGP and Google Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts showed that the inter-reliability of both the Open AI ChatGPT and the Google Bard were low against the gold standard of human ratings. Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

The Writing Synth Hypothesis - March 24th, sbs on; 2023 (2023)
<标题>: 写作合成假说 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread

Assessment redesign for generative AI: A taxonomy of options and their viability (2023)
with Sarah Howard and Jaclyn Broadbent Since the seemingly sudden emergence of ChatGPT at the end of 2022, there has been significant debate surrounding the impact of text-based generative AI in education. Many jurisdictions initially attempted to ban access to these tools, citing concerns that stud
<标题>: 面向生成式人工智能的评估重设计：选项分类及其可行性 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.518Z
Keywords: /unread

Study Finds ChatGPT Outperforms Physicians in High-Quality, Empathetic Answers to Patient Questions (2023)
A new study published in JAMA Internal Medicine compared written responses from physicians with those from ChatGPT to real-world health questions. A panel of licensed healthcare professionals preferred ChatGPT’s responses 79% of the time.
<标题>: 研究发现，ChatGPT在向患者提供高质量、富有同理心的回答方面表现优于医生 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z
Keywords: /unread

How AI Could Save (Not Destroy) Education | Sal Khan | TED - TED (2023)
Sal Khan, the founder and CEO of Khan Academy, thinks artificial intelligence could spark the greatest positive transformation education has ever seen. He shares the opportunities he sees for students and educators to collaborate with AI tools – including the potential of a personal AI tutor for every student and an AI teaching assistant for every teacher – and demos some exciting new features for their educational chatbot, Khanmigo. If you love watching TED Talks like this one, become a TED Member to support our mission of spreading ideas: https://ted.com/membership Follow TED! Twitter: https://twitter.com/TEDTalks Instagram: https://www.instagram.com/ted Facebook: https://facebook.com/TED LinkedIn: https://www.linkedin.com/company/ted-... TikTok: https://www.tiktok.com/@tedtoks The TED Talks channel features talks, performances and original series from the world's leading thinkers and doers. Subscribe to our channel for videos on Technology, Entertainment and Design — plus science, business, global issues, the arts and more. Visit https://TED.com to get our entire library of TED Talks, transcripts, translations, personalized talk recommendations and more. Watch more: https://go.ted.com/salkhan • How AI Could Save... TED's videos may be used for non-commercial purposes under a Creative Commons License, Attribution–Non Commercial–No Derivatives (or the CC BY – NC – ND 4.0 International) and in accordance with our TED Talks Usage Policy: https://www.ted.com/about/our-organiz.... For more information on using TED for commercial purposes (e.g. employee learning, in a film or online course), please submit a Media Request at https://media-requests.ted.com #TED #TEDTalks #education #AI #khanacademy
Directors: _:n8985 <标题>: 人工智能如何拯救（而非毁掉）教育 | 萨尔·可汗 | TED Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: /unread

Artificial Intelligence and the Future of Teaching and Learning - U.S. Department of Education, Office of Educational Technology (Department of Education, 2023)
<标题>: 人工智能与未来的教学与学习 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

MathGPT is incoming; how it fits into undergraduate math education - The Arizona State Press (2023)
Arizona State University's independent student-run news organization covering Tempe, Phoenix, Mesa and Glendale.
<标题>: MathGPT 即将到来；它在本科数学教育中的定位 - 亚利桑那州立新闻 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Responsible Use of Generative AI | Deloitte US (2023)
<标题>: 生成式人工智能的负责任使用 | 德勤美国 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.518Z
Keywords: /unread

Yann LeCun on LinkedIn: A survey of LLMs with a practical guide and evolutionary tree. Number of… | 21 comments (2023)
A survey of LLMs with a practical guide and evolutionary tree. Number of LLMs from Meta = 7 Number of open source LLMs from Meta = 7 The architecture… | 21 comments on LinkedIn
<标题>: Yann LeCun 在 LinkedIn 上：一份关于大型语言模型（LLM）的调查，包括实用指南和进化树。评论数… | 21 条评论 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

A New Approach To Mitigating AI’s Negative Impact (Stanford HAI, 2023)
Stanford launches an Ethics and Society Review Board that asks researchers to take an early look at the impact of their work.
<标题>: 减轻人工智能负面影响的新方法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

The Stanford MOOCPosts Data Set - Agrawal, Akshay; Paepcke, Andreas (Stanford, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z
Keywords: /unread

Office Overachievers Won't Be Happy About ChatGPT, Study Finds (Gizmodo, 2023)
Customer support agents given access to a generative AI chatbot were 14% more productive, but those gains were much higher for lower-performing workers.
<标题>: 研究发现，办公室高绩效者对ChatGPT不会感到高兴 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z
Keywords: /unread

Artificial intelligence in higher education: the state of the field - Crompton, Helen; Burke, Diane (International Journal of Educational Technology in Higher Education, 2023)
This systematic review provides unique findings with an up-to-date examination of artificial intelligence (AI) in higher education (HE) from 2016 to 2022. Using PRISMA principles and protocol, 138 articles were identified for a full examination. Using a priori, and grounded coding, the data from the 138 articles were extracted, analyzed, and coded. The findings of this study show that in 2021 and 2022, publications rose nearly two to three times the number of previous years. With this rapid rise in the number of AIEd HE publications, new trends have emerged. The findings show that research was conducted in six of the seven continents of the world. The trend has shifted from the US to China leading in the number of publications. Another new trend is in the researcher affiliation as prior studies showed a lack of researchers from departments of education. This has now changed to be the most dominant department. Undergraduate students were the most studied students at 72%. Similar to the findings of other studies, language learning was the most common subject domain. This included writing, reading, and vocabulary acquisition. In examination of who the AIEd was intended for 72% of the studies focused on students, 17% instructors, and 11% managers. In answering the overarching question of how AIEd was used in HE, grounded coding was used. Five usage codes emerged from the data: (1) Assessment/Evaluation, (2) Predicting, (3) AI Assistant, (4) Intelligent Tutoring System (ITS), and (5) Managing Student Learning. This systematic review revealed gaps in the literature to be used as a springboard for future researchers, including new tools, such as Chat GPT.
<AI Smry>: G gaps in the literature were revealed to be used as a springboard for future researchers, including new tools, such as Chat GPT, which was revealed in the switch from US to China in the most studies published. Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread; AI; Artificial Intelligence; Higher education; Systematic review

Meet RedPajama: An AI Project to Create Fully Open-Source Large Language Models Beginning with the Release of a 1.2 Trillion Token Dataset - Singh, Niharika (MarkTechPost, 2023)
The most advanced foundation models for AI are only partially open-source and are only available through commercial APIs. This restricts their use and limits research and customization. However, a project called RedPajama now aims to create leading, fully open-source models. The first step of this project, reproducing the LLaMA training dataset, has been completed. Open-source models have made significant progress recently, and AI is experiencing a moment similar to the Linux movement. Stable Diffusion demonstrated that open-source models could compete with commercial offerings and encourage creativity through community participation. A similar movement has now emerged around large language models, with
<标题>: 认识 RedPajama：一个旨在创建完全开源大语言模型的人工智能项目，从发布 1.2 万亿词元数据集开始 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z
Keywords: /unread

5 Things Every School District and Educator Should Know About ChatGPT - TalkingPoints - Dooley, Heather (TalkingPoints, 2023)
In schools and conferences across the country, everyone seems to be talking about AI (Artificial Intelligence) and ChatGPT. Teachers are using ChatGPT to create lesson plans. Students are using ChatGPT to do their homework. And bloggers are using ChatGPT to draft content (although we promise not this one!). We understand the opportunities these resources can
<标题>: 每个学区和教育工作者都应该了解的关于 ChatGPT 的 5 件事 - TalkingPoints Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z
Keywords: /unread

Automated Scoring of Speaking and Writing: Starting to Hit its Stride - Jones, Daniel Marc; Cheng, Liying; Tweedie, Gregory (Canadian Journal of Learning and Technology, 2023)
This article reviews recent literature (2011–present) on the automated scoring (AS) of writing and speaking. Its purpose is to first survey the current research on automated scoring of language, then highlight how automated scoring impacts the present and future of assessment, teaching, and learning. The article begins by outlining the general background of AS issues in language assessment and testing. It then positions AS research with respect to technological advancements. Section two details the literature review search process and criteria for article inclusion. In section three, the three main themes emerging from the review are presented: automated scoring design considerations, the role of humans and artificial intelligence, and the accuracy of automated scoring with different groups. Two tables show how specific articles contributed to each of the themes. Following this, each of the three themes is presented in further detail, with a sequential focus on writing, speaking, and a short summary. Section four addresses AS implementation with respect to current assessment, teaching, and learning issues. Section five considers future research possibilities related to both the research and current uses of AS, with implications for the Canadian context in terms of the next steps for automated scoring.
<标题>: 口语和写作的自动评分：开始进入状态 <AI Smry>: The purpose is to first survey the current research on automated scoring of language, then highlight how automated scoring impacts the present and future of assessment, teaching, and learning, with a sequential focus on writing, speaking, and a short summary. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT - Dai, Wei; Lin, Jionghao; Jin, Flora; Li, Tongguang; Tsai, Yi-Shan; Gasevic, Dragan; Chen, Guanliang (EdArXiv, 2023)
Educational feedback has been widely acknowledged as an effective approach to improving student learning. However, scaling effective practices can be laborious and costly, which motivated researchers to work on automated feedback systems (AFS). Inspired by the recent advancements in the pre-trained language models (e.g., ChatGPT), we posit that such models might advance the existing knowledge of textual feedback generation in AFS because of their capability to offer natural-sounding and detailed responses. Therefore, we aimed to investigate the feasibility of using ChatGPT to provide students with feedback to help them learn better. Specifically, we first examined the readability of ChatGPT-generated feedback. Then, we measured the agreement between ChatGPT and the instructor when assessing students' assignments according to the marking rubric. Finally, we used a well-known theoretical feedback framework to further investigate the effectiveness of the feedback generated by ChatGPT. Our results show that i) ChatGPT is capable of generating more detailed feedback that fluently and coherently summarizes students' performance than human instructors; ii) ChatGPT achieved high agreement with the instructor when assessing the topic of students' assignments; and iii) ChatGPT could provide feedback on the process of students completing the task, which benefits students developing learning skills.
<标题>: 大型语言模型能为学生提供反馈吗？——以ChatGPT为例的案例研究 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Automated Feedback; Education; Educational Assessment; Educational Methods; Evaluation; Feedback Effectiveness; Feedback Generation; Higher Education; Large Language Model; and Research

MxML (Exploring the paradigmatic relationship between measurement and machine learning in the history, current time, and future): Current state-of-the-field - Zheng, Yi; Nydick, Steven; Huang, Sijia; Zhang, Susu (2023)
The recent surge of machine learning (ML) has impacted many disciplines, including educational and psychological measurement (hereafter shortened as measurement, “M”). The measurement literature has seen a rapid growth in studies that explore using ML methods to solve measurement problems. However, there exist gaps between the typical paradigm of ML and fundamental principles of measurement. The MxML project was created to explore how the measurement community might potentially redefine the psychometrics discipline in the imminent future of big data and machine learning, so as to harness the power of machine learning to serve our (redefined and updated) mission. This paper describes the first study of the MxML project, in which we summarize the state of the field of applications, extensions, and discussions about ML methods in measurement contexts with a systematic review of the recent 10 years of literature (2013 - 2022). Specifically, we provide a snapshot of the literature in terms of (1) areas of measurement, (2) types of article, (3) ML methods discussed, and (4) gaps addressed between measurement goals and ML methods.
<标题>: MxML（探索历史、当前及未来测量与机器学习之间的典范关系）：该领域的现状 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: /unread

Toxicity in ChatGPT: Analyzing Persona-assigned Language Models - Deshpande, Ameet; Murahari, Vishvak; Rajpurohit, Tanmay; Kalyan, Ashwin; Narasimhan, Karthik (arXiv, 2023)
Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students or patients engaging with chatbots, the safety of these systems is of prime importance. Therefore, a clear understanding of the capabilities and limitations of LLMs is necessary. To this end, we systematically evaluate toxicity in over half a million generations of ChatGPT, a popular dialogue-based LLM. We find that setting the system parameter of ChatGPT by assigning it a persona, say that of the boxer Muhammad Ali, significantly increases the toxicity of generations. Depending on the persona assigned to ChatGPT, its toxicity can increase up to 6x, with outputs engaging in incorrect stereotypes, harmful dialogue, and hurtful opinions. This may be potentially defamatory to the persona and harmful to an unsuspecting user. Furthermore, we find concerning patterns where specific entities (e.g., certain races) are targeted more than others (3x more) irrespective of the assigned persona, that reflect inherent discriminatory biases in the model. We hope that our findings inspire the broader AI community to rethink the efficacy of current safety guardrails and develop better techniques that lead to robust, safe, and trustworthy AI systems.
arXiv:2304.05335 [cs] <AI Smry>: This work systematically evaluates toxicity in over half a million generations of ChatGPT, a popular dialogue-based LLM, and finds concerning patterns where specific entities are targeted more than others irrespective of the assigned persona, that reflect inherent discriminatory biases in the model. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Automated Reading Passage Generation with OpenAI's Large Language Model - Bezirhan, Ummugul; Davier, Matthias von (arXiv, 2023)
The widespread usage of computer-based assessments and individualized learning platforms has resulted in an increased demand for the rapid production of high-quality items. Automated item generation (AIG), the process of using item models to generate new items with the help of computer technology, was proposed to reduce reliance on human subject experts at each step of the process. AIG has been used in test development for some time. Still, the use of machine learning algorithms has introduced the potential to improve the efficiency and effectiveness of the process greatly. The approach presented in this paper utilizes OpenAI's latest transformer-based language model, GPT-3, to generate reading passages. Existing reading passages were used in carefully engineered prompts to ensure the AI-generated text has similar content and structure to a fourth-grade reading passage. For each prompt, we generated multiple passages, the final passage was selected according to the Lexile score agreement with the original passage. In the final round, the selected passage went through a simple revision by a human editor to ensure the text was free of any grammatical and factual errors. All AI-generated passages, along with original passages were evaluated by human judges according to their coherence, appropriateness to fourth graders, and readability.
arXiv:2304.04616 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Grounded in reality: artificial intelligence in medical education - Krive, Jacob; Isola, Miriam; Chang, Linda; Patel, Tushar; Anderson, Max; Sreedhar, Radhika (JAMIA Open, 2023)
Abstract Background In a recent survey, medical students expressed eagerness to acquire competencies in the use of artificial intelligence (AI) in medicine. It is time that undergraduate medical education takes the lead in helping students develop these competencies. We propose a solution that integrates competency-driven AI instruction in medical school curriculum. Methods We applied constructivist and backwards design principles to design online learning assignments simulating the real-world work done in the healthcare industry. Our innovative approach assumed no technical background for students, yet addressed the need for training clinicians to be ready to practice in the new digital patient care environment. This modular 4-week AI course was implemented in 2019, integrating AI with evidence-based medicine, pathology, pharmacology, tele-monitoring, quality improvement, value-based care, and patient safety. Results This educational innovation was tested in 2 cohorts of fourth year medical students who demonstrated an improvement in knowledge with an average quiz score of 97% and in skills with an average application assignment score of 89%. Weekly reflections revealed how students learned to transition from theory to practice of AI and how these concepts might apply to their upcoming residency training programs and future medical practice. Conclusions We present an innovative product that achieves the objective of competency-based education of students regarding the role of AI in medicine. This course can be integrated in the preclinical years with a focus on foundational knowledge, vocabulary, and concepts, and in clinical years with a focus on application of core knowledge to real-world scenarios.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Human scoring versus automated scoring for english learners in a statewide evidence-based writing assessment - Vo, Yen; Rickels, Heather; Welch, Catherine; Dunbar, Stephen (Assessing Writing, 2023)
<标题>: 全州基于证据的写作评估中针对英语学习者的人类评分与自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z

Whose Opinions Do Language Models Reflect? - Santurkar, Shibani; Durmus, Esin; Ladhak, Faisal; Lee, Cinoo; Liang, Percy; Hashimoto, Tatsunori (arXiv, 2023)
Language models (LMs) are increasingly being used in open-ended contexts, where the opinions reflected by LMs in response to subjective queries can have a profound impact, both on user satisfaction, as well as shaping the views of society at large. In this work, we put forth a quantitative framework to investigate the opinions reflected by LMs – by leveraging high-quality public opinion polls and their associated human responses. Using this framework, we create OpinionsQA, a new dataset for evaluating the alignment of LM opinions with those of 60 US demographic groups over topics ranging from abortion to automation. Across topics, we find substantial misalignment between the views reflected by current LMs and those of US demographic groups: on par with the Democrat-Republican divide on climate change. Notably, this misalignment persists even after explicitly steering the LMs towards particular demographic groups. Our analysis not only confirms prior observations about the left-leaning tendencies of some human feedback-tuned LMs, but also surfaces groups whose opinions are poorly reflected by current LMs (e.g., 65+ and widowed individuals). Our code and data are available at https://github.com/tatsu-lab/opinions_qa.
arXiv:2303.17548 [cs] <AI Smry>: This work creates OpinionsQA, a new dataset for evaluating the alignment of LM opinions with those of 60 US demographic groups over topics ranging from abortion to automation, and finds substantial misalignment. <标题>: 语言模型反映了谁的观点？ Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.602Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Computers and Society; Computer Science - Machine Learning

FATE (Fairness and Transparency) Hits the Mainstream: UK's 5 Principles for AI Regulation - Harris, Robert (Feedzai, 2023)
The UK released a whitepaper detailing a pro-innovation approach to AI regulation. Here are the potential impacts for banks
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Conversational agents in language learning - Xiao, Feiwen; Zhao, Priscilla; Sha, Hanyue; Yang, Dandan; Warschauer, Mark (Journal of China Computer-Assisted Language Learning, 2023)
Due to advances in technology, conversational agents are emerging as intelligent spoken dialogue systems that simulate natural conversation with human beings. A growing body of literature has investigated the potential of conversational agents in enhancing language learning across multiple contexts. In this paper, a broad scoping review examining the current literature on conversational agents and language learning was conducted. This review mapped APA PsycINFO, ERIC and ProQuest Dissertations & Theses databases, which yielded 23 papers for further analysis. Our examination of these papers suggests that there are three main ways in which conversational agents are used for language learning. This review discusses these three approaches and points to directions that require further research to fully exploit the potential of conversational agents in language learning.
<AI Smry>: There are three main ways in which conversational agents are used for language learning, and this review discusses these three approaches and points to directions that require further research to fully exploit the potential of Conversational agents in language learning. <标题>: 语言学习中的对话式智能体 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread; CALL; conversational agents; language learning; natural language processing (NLP); technology

GPT-4 Technical Report - OpenAI (arXiv, 2023)
We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.
arXiv:2303.08774 [cs] <标题>: GPT-4 技术报告 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models - Eloundou, Tyna; Manning, Sam; Mishkin, Pamela; Rock, Daniel (arXiv, 2023)
We investigate the potential implications of Generative Pre-trained Transformer (GPT) models and related technologies on the U.S. labor market. Using a new rubric, we assess occupations based on their correspondence with GPT capabilities, incorporating both human expertise and classifications from GPT-4. Our findings indicate that approximately 80% of the U.S. workforce could have at least 10% of their work tasks affected by the introduction of GPTs, while around 19% of workers may see at least 50% of their tasks impacted. The influence spans all wage levels, with higher-income jobs potentially facing greater exposure. Notably, the impact is not limited to industries with higher recent productivity growth. We conclude that Generative Pre-trained Transformers exhibit characteristics of general-purpose technologies (GPTs), suggesting that these models could have notable economic, social, and policy implications.
arXiv:2303.10130 [cs, econ, q-fin] <标题>: GPT就是GPT：大语言模型对劳动力市场潜在影响的早期观察 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computers and Society; Economics - General Economics

Towards Automated Analysis of Rhetorical Categories in Students Essay Writings using Bloom’s Taxonomy - Iqbal, Sehrish; Rakovic, Mladen; Chen, Guanliang; Li, Tongguang; Ferreira Mello, Rafael; Fan, Yizhou; Fiorentino, Giuseppe; Radi Aljohani, Naif; Gasevic, Dragan (LAK23: 13th International Learning Analytics and Knowledge Conference, 2023)
<标题>: 利用布鲁姆分类法对学生论文写作中的修辞类别进行自动化分析 <AI Smry>: A statistical difference between the associations of rhetorical categories in low-achieving, medium-achiever and high-achievers groups implies that rhetorical categories can be predictive of writing performance. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Predicting Item Response Theory Parameters Using Question Statements Texts - Marinho, Wemerson; Clua, Esteban Walter; Martí, Luis; Marinho, Karla (LAK23: 13th International Learning Analytics and Knowledge Conference, 2023)
<标题>: 使用问题陈述文本预测项目反应理论参数 <AI Smry>: The application of new Neural Language Models pre-trained on a massive corpus of texts to predict Item Response Theory parameters in multiple choice questions for the Brazilian National High School Exam (ENEM) is investigated and a novel optimization target for regression is proposed: Item Characteristic Curve. Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

Empowering Teacher Learning with AI: Automated Evaluation of Teacher Attention to Student Ideas during Argumentation-focused Discussion - Nazaretsky, Tanya; Mikeska, Jamie N.; Beigman Klebanov, Beata (LAK23: 13th International Learning Analytics and Knowledge Conference, 2023)
<标题>: 用人工智能赋能教师学习：在以论证为重点的讨论中自动评估教师对学生观点的关注度 <AI Smry>: The results are encouraging: Not only can the models score the transcript as well as humans can, but they can also provide justifications for the scores comparable to those provided by human raters. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z

Survey of Hallucination in Natural Language Generation - Ji, Ziwei; Lee, Nayeon; Frieske, Rita; Yu, Tiezheng; Su, Dan; Xu, Yan; Ishii, Etsuko; Bang, Yejin; Chen, Delong; Chan, Ho Shu; Dai, Wenliang; Madotto, Andrea; et al. (ACM Computing Surveys, 2023)
Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as ve summarization, dialogue generation, and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before.In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions, and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely ve summarization, dialogue generation, generative question answering, data-to-text generation, and machine translation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.
arXiv:2202.03629 [cs] <标题>: 自然语言生成中的幻觉调查 <AI Smry>: A broad overview of the research progress and challenges in the hallucination problem in NLG is provided, including task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, and machine translation. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: A.1; Computer Science - Computation and Language

A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity - Bang, Yejin; Cahyawijaya, Samuel; Lee, Nayeon; Dai, Wenliang; Su, Dan; Wilie, Bryan; Lovenia, Holy; Ji, Ziwei; Yu, Tiezheng; Chung, Willy; Do, Quyet V.; Xu, Yan; et al. (2023)
This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn "prompt engineering" fashion. We also release codebase for evaluation set extraction.
arXiv:2302.04023 [cs] Issue: arXiv:2302.04023 <标题>: 对 ChatGPT 在推理、幻觉和交互性方面的多任务、多语言、多模态评估 Read_Status: New Read_Status_Date: 2026-01-26T11:32:27.022Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

How ChatGPT Works: The Models Behind The Bot - Ruby, Molly (Medium, 2023)
A brief introduction to the intuition and methodology behind the chat bot you can’t stop hearing about.
<标题>: ChatGPT 如何工作：支持该机器人的模型 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread

A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms - Coston, Amanda; Kawakami, Anna; Zhu, Haiyi; Holstein, Ken; Heidari, Hoda (arXiv, 2023)
Recent research increasingly brings to question the appropriateness of using predictive tools in complex, real-world tasks. While a growing body of work has explored ways to improve value alignment in these tools, comparatively less work has centered concerns around the fundamental justifiability of using these tools. This work seeks to center validity considerations in deliberations around whether and how to build data-driven algorithms in high-stakes domains. Toward this end, we translate key concepts from validity theory to predictive algorithms. We apply the lens of validity to re-examine common challenges in problem formulation and data issues that jeopardize the justifiability of using predictive algorithms and connect these challenges to the social science discourse around validity. Our interdisciplinary exposition clarifies how these concepts apply to algorithmic decision making contexts. We demonstrate how these validity considerations could distill into a series of high-level questions intended to promote and document reflections on the legitimacy of the predictive task and the suitability of the data.
arXiv:2206.14983 [cs] <标题>: 从有效性角度评估数据驱动决策算法的合理使用 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Computer Science - Computers and Society; Computer Science - Human-Computer Interaction; Computer Science - Machine Learning

Learning gain differences between ChatGPT and human tutor generated algebra hints - Pardos, Zachary A.; Bhandari, Shreya (arXiv, 2023)
Large Language Models (LLMs), such as ChatGPT, are quickly advancing AI to the frontiers of practical consumer use and leading industries to re-evaluate how they allocate resources for content production. Authoring of open educational resources and hint content within adaptive tutoring systems is labor intensive. Should LLMs like ChatGPT produce educational content on par with human-authored content, the implications would be significant for further scaling of computer tutoring system approaches. In this paper, we conduct the first learning gain evaluation of ChatGPT by comparing the efficacy of its hints with hints authored by human tutors with 77 participants across two algebra topic areas, Elementary Algebra and Intermediate Algebra. We find that 70% of hints produced by ChatGPT passed our manual quality checks and that both human and ChatGPT conditions produced positive learning gains. However, gains were only statistically significant for human tutor created hints. Learning gains from human-created hints were substantially and statistically significantly higher than ChatGPT hints in both topic areas, though ChatGPT participants in the Intermediate Algebra experiment were near ceiling and not even with the control at pre-test. We discuss the limitations of our study and suggest several future directions for the field. Problem and hint content used in the experiment is provided for replicability.
arXiv:2302.06871 [cs] <AI Smry>: This paper conducts the first learning gain evaluation of ChatGPT by comparing the efficacy of its hints with hints authored by human tutors with 77 participants across two algebra topic areas, Elementary Algebra and Intermediate Algebra. <标题>: ChatGPT 与人类导师生成的代数提示之间的学习收益差异 Read_Status: New Read_Status_Date: 2026-01-26T11:32:20.376Z
Keywords: /unread; Computer Science - Computation and Language; Computer Science - Computers and Society; Computer Science - Human-Computer Interaction

AI in the Foreign Language Classroom: A Pedagogical Overview of Automated Writing Assistance Tools - Alharbi, Wael (Education Research International, 2023)
Recent technological advances in artificial intelligence (AI) have paved the way for improved and in many cases the creation of entirely new and innovative, electronic writing tools. These writing support systems assist during and after the writing process making them indispensable to many writers in general and to students in particular who can get human-like sentence completion suggestions and text generation. Although the wide adoption of these tools by students has been faced with a steady growth of scientific publications in the field, the results of these studies are often contradictory and their validity may be questioned. To gain a deeper understanding of the validity of AI-powered writing assistance tools, we conducted a systematic review of the recent empirical AI-powered writing assistance studies. The purpose of this review is twofold. First, we wanted to explore the recent scholarly publications that evaluated the use of AI-powered writing assistance tools in the classroom in terms of their types, uses, limits, and potential for improving students’ writing skills. Second, the review also sought to explore the perceptions of educators and researchers about learners’ use of AI-powered writing tools and review their recommendations on how to best ingrate these tools into the contemporary and future classroom. Using the Scopus research database, a total of 104 peer-reviewed papers were identified and analyzed. The findings indicate that students are increasingly using a variety of AI-powered writing assistance tools for improving their writing. The tools they are using can be categorized into four main groups: (1) automated writing evaluation tools, (2) tools that provide automated writing corrective feedback, (3) AI-powered machine translators, and (4) GPT-3 automatic text generators. The analysis also highlighted the scholars’ recommendations regarding dealing with learners’ use of AI-powered writing assistance tools and grouped the recommendations into two groups for researchers and educators.
<标题>: 外语课堂中的人工智能：自动写作辅助工具的教学概述 <AI Smry>: A systematic review of the recent empirical AI-powered writing assistance studies to gain a deeper understanding of the validity of these tools and their recommendations on how to best ingrate these tools into the contemporary and future classroom. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z

A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms - Coston, Amanda; Kawakami, Anna; Zhu, Haiyi; Holstein, Ken; Heidari, Hoda (arXiv, 2023)
Recent research increasingly brings to question the appropriateness of using predictive tools in complex, real-world tasks. While a growing body of work has explored ways to improve value alignment in these tools, comparatively less work has centered concerns around the fundamental justifiability of using these tools. This work seeks to center validity considerations in deliberations around whether and how to build data-driven algorithms in high-stakes domains. Toward this end, we translate key concepts from validity theory to predictive algorithms. We apply the lens of validity to re-examine common challenges in problem formulation and data issues that jeopardize the justifiability of using predictive algorithms and connect these challenges to the social science discourse around validity. Our interdisciplinary exposition clarifies how these concepts apply to algorithmic decision making contexts. We demonstrate how these validity considerations could distill into a series of high-level questions intended to promote and document reflections on the legitimacy of the predictive task and the suitability of the data.
arXiv:2206.14983 [cs] <标题>: 从有效性角度评估数据驱动决策算法的合理使用 <AI Smry>: This work seeks to center validity considerations in deliberations around whether and how to build data-driven algorithms in high-stakes domains by applying the lens of validity to re-examine common challenges in problem formulation and data issues that jeopardize the justifiability of using predictive algorithms. Read_Status: New Read_Status_Date: 2026-01-26T11:32:31.459Z
Keywords: Computer Science - Computers and Society; Computer Science - Human-Computer Interaction; Computer Science - Machine Learning

ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? - Rudolph, Jurgen; Tan, Samson; Tan, Shannon (Journal of Applied Learning & Teaching, 2023)
<标题>: ChatGPT：胡说八道的机器，还是高等教育传统评估的终结者？ Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Educational Research and AI-Generated Writing: Confronting the Coming Tsunami - Tate, Tamara; Doroudi, Shayan; Ritchie, Daniel; Xu, Ying; Uci, Mark Warschauer (EdArXiv, 2023)
The public release and surprising capacity of ChatGPT has brought AI-enabled text generation into the forefront for educators and academics. ChatGPT and similar text generation tools raise numerous questions for educational practitioners, policymakers, and researchers. We begin by first describing what large language models are and how they function, and then situate them in the history of technology’s complex interrelationship with literacy, cognition, and education. Finally, we discuss implications for the field.
<标题>: 教育研究与人工智能生成写作：迎接即将到来的浪潮 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Artificial Intelligence; Education; Language and Literacy Education; education; literacy; ml; technology

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models - Wei, Jason; Wang, Xuezhi; Schuurmans, Dale; Bosma, Maarten; Ichter, Brian; Xia, Fei; Chi, Ed; Le, Quoc; Zhou, Denny (arXiv, 2023)
We explore how generating a chain of thought – a series of intermediate reasoning steps – significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.
arXiv:2201.11903 [cs] <标题>: 连锁思维提示在大型语言模型中引发推理 Read_Status: New Read_Status_Date: 2026-01-26T11:32:25.304Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models - Wei, Jason; Wang, Xuezhi; Schuurmans, Dale; Bosma, Maarten; Ichter, Brian; Xia, Fei; Chi, Ed; Le, Quoc; Zhou, Denny (arXiv, 2023)
We explore how generating a chain of thought – a series of intermediate reasoning steps – significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.
arXiv:2201.11903 [cs] <标题>: 链式思维提示在大型语言模型中引发推理 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Integrating generative AI in knowledge building - Chen, Bodong; Zhu, Xinran; Díaz del Castillo H., Fernando (Computers and Education: Artificial Intelligence, 2023)
Generative artificial intelligence (GenAI) is penetrating in various social sectors, motivating a strong need for teaching AI literacy in younger generations. While substantial efforts have been made to teach AI literacy and to use AI to facilitate learning, few studies have provided empirical accounts of students' nuanced processes of using GenAI for learning. In this study, we engaged a group of high school students in leveraging ChatGPT to support their knowledge building efforts. Following the teacher's pedagogical design, students used ChatGPT for a range of distinct purposes. Student interviews showed detailed processes of using ChatGPT for knowledge building and students' emerging AI literacy in multiple dimensions. This study offers practical implications for the integration of GenAI in K-12 education and urges educators to create spaces and scaffolds for students to mindfully engage with GenAI in the classroom.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: /unread; ChatGPT; Generative AI; Human-AI partnership; Knowledge building

English grammar multiple-choice question generation using Text-to-Text Transfer Transformer - Chomphooyod, Peerawat; Suchato, Atiwong; Tuaycharoen, Nuengwong; Punyabukkana, Proadpran (Computers and Education: Artificial Intelligence, 2023)
English grammar multiple-choice questions (MCQs) can be automatically generated to reduce preparation time. Previous studies have focused on semiautomated methods based on the transformation of human-made sentences/articles into MCQs, owing to which the number of generated questions is dependent on the size of a given text corpus. This study proposes an artificial intelligence-assisted MCQ generation system that increases the number of generable questions using controllable text generation techniques. In this system, the questions for MCQs are generated using a text generation model trained using the Text-to-Text Transfer Transformer (T5) architecture, a powerful deep learning model for performing text generation tasks, with a keyword and a part-of-speech (POS) template as the input for content and grammar topic control. For the text-to-MCQ transformation process, answer-to-MCQ and distractor-selection strategies are proposed for 10 grammar topics using rule-based algorithms. The quality of the generated MCQs is evaluated by human experts. The acceptance rate of the questions generated using the proposed system is 86%. The controllability of content and grammar topics are 96.86% and 98.57%, respectively. The findings of this study show that the T5 model achieves good performance in terms of controlling the POS structure in a keyword-to-text generation task. Moreover, the good acceptance rate indicates that artificial intelligence has the potential to help teachers speed up the process of selecting examination questions. We also discuss extending the proposed system to other grammar topics and the limitations of the proposed system.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: /unread; Artificial intelligence–based learning; Automatic question generation; Education technology; Language learning; Multiple-choice question

Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation - Abdelnabi, Sahar; Gomaa, Amr; Sivaprasad, Sarath; Schönherr, Lea; Fritz, Mario (arXiv, 2023)
There is an growing interest in using Large Language Models (LLMs) in multi-agent systems to tackle interactive real-world tasks that require effective collaboration and assessing complex situations. Yet, we still have a limited understanding of LLMs' communication and decision-making abilities in multi-agent setups. The fundamental task of negotiation spans many key features of communication, such as cooperation, competition, and manipulation potentials. Thus, we propose using scorable negotiation to evaluate LLMs. We create a testbed of complex multi-agent, multi-issue, and semantically rich negotiation games. To reach an agreement, agents must have strong arithmetic, inference, exploration, and planning capabilities while integrating them in a dynamic and multi-turn setup. We propose multiple metrics to rigorously quantify agents' performance and alignment with the assigned role. We provide procedures to create new games and increase games' difficulty to have an evolving benchmark. Importantly, we evaluate critical safety aspects such as the interaction dynamics between agents influenced by greedy and adversarial players. Our benchmark is highly challenging; GPT-3.5 and small models mostly fail, and GPT-4 and SoTA large models (e.g., Llama-3 70b) still underperform.
Version Number: 2 <标题>: 合作、竞争与恶意：大型语言模型利益相关者的互动谈判 <AI Smry>: This work proposes using scorable negotiation games as a new evaluation framework for LLMs, and shows that these games can help evaluate other critical aspects, such as the interaction dynamics between agents in the presence of greedy and adversarial players. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computation and Language (cs.CL); Computers and Society (cs.CY); FOS: Computer and information sciences; Machine Learning (cs.LG)

From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting - Adams, Griffin; Fabbri, Alexander; Ladhak, Faisal; Lehman, Eric; Elhadad, Noémie (arXiv, 2023)
Selecting the ``right'' amount of information to include in a summary is a difficult task. A good summary should be detailed and entity-centric without being overly dense and hard to follow. To better understand this tradeoff, we solicit increasingly dense GPT-4 summaries with what we refer to as a ``Chain of Density'' (CoD) prompt. Specifically, GPT-4 generates an initial entity-sparse summary before iteratively incorporating missing salient entities without increasing the length. Summaries generated by CoD are more abstractive, exhibit more fusion, and have less of a lead bias than GPT-4 summaries generated by a vanilla prompt. We conduct a human preference study on 100 CNN DailyMail articles and find that that humans prefer GPT-4 summaries that are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Qualitative analysis supports the notion that there exists a tradeoff between informativeness and readability. 500 annotated CoD summaries, as well as an extra 5,000 unannotated summaries, are freely available on HuggingFace (https://huggingface.co/datasets/griffin/chain_of_density).
arXiv:2309.04269 [cs] <标题>: 从稀疏到密集：使用密度链提示的 GPT-4 摘要 <AI Smry>: Qualitative analysis supports the notion that there exists a tradeoff between informativeness and readability and that humans prefer GPT-4 summaries that are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Computation and Language

WHO WROTE THIS ESSAY? DETECTING AI-GENERATED WRITING IN SECOND LANGUAGE EDUCATION IN HIGHER EDUCATION - Alexander, Katarzyna; Savvidou, Christine; Alexander, Chris (Teaching English With Technology, 2023)
<标题>: 谁写了这篇文章？在高等教育中检测第二语言教育的 AI 生成写作 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z

The Internal State of an LLM Knows When It's Lying - Azaria, Amos; Mitchell, Tom (arXiv, 2023)
While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone. In this paper, we provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements. This includes both statements provided to the LLM, and statements that the LLM itself generates. Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement. Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71\% to 83\% accuracy labeling which sentences are true versus false, depending on the LLM base model. Furthermore, we explore the relationship between our classifier's performance and approaches based on the probability assigned to the sentence by the LLM. We show that while LLM-assigned sentence probability is related to sentence truthfulness, this probability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing a more reliable approach to detecting truthfulness, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios.
arXiv:2304.13734 [cs] <标题>: 大型语言模型的内部状态知道它什么时候在撒谎 <AI Smry>: Evidence that the LLM's internal state can be used to reveal the truthfulness of statements is provided, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios. Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Automating Question Generation From Educational Text - Bhowmick, Ayan Kumar; Jagmohan, Ashish; Vempaty, Aditya; Dey, Prasenjit; Hall, Leigh; Hartman, Jeremy; Kokku, Ravi; Maheshwari, Hema (Artificial Intelligence XL, 2023)
Series Title: Lecture Notes in Computer Science Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

A Meta Systematic Review of Artificial Intelligence in Higher Education: A call for increased ethics, collaboration, and rigour - Bond, Melissa; Khosravi, Hassan; De Laat, Maarten; Bergdahl, Nina; Negrea, Violeta; Oxley, Emily; Pham, Phuong; Chong, Sin Wang; Siemens, George (Unpublished, 2023)
Version Number: 1 Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Responsible AI Standards - Duolingo English Test - Burstein, Jill (2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z
Keywords: /unread

Advancing Intelligent Textbooks with Automatically Generated Practice: A Large-Scale Analysis of Student Data - Campenhout, Rachel Van; Clark, Michelle; Jerome, Bill; Dittel, Jeffrey S.; Johnson, Benny G. (iTextbooks@AIED, 2023)
<标题>: 通过自动生成练习推动智能教材发展：学生数据的大规模分析 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Artificial Intelligence in Education Technologies: New Development and Innovative Practices: Proceedings of 2022 3rd International Conference on Artificial Intelligence in Education Technology - Cheng, Eric C. K.; Wang, Tianchong; Schlippe, Tim; Beligiannis, Grigorios N. (Springer Nature Singapore, 2023)
<标题>: 教育技术中的人工智能：新发展与创新实践：2022年第3届国际教育技术人工智能大会论文集 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: /unread

Artificial Intelligence in Education Technologies: New Development and Innovative Practices: Proceedings of 2022 3rd International Conference on Artificial Intelligence in Education Technology - Cheng, Eric C. K.; Wang, Tianchong; Schlippe, Tim; Beligiannis, Grigorios N. (Springer Nature Singapore, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

English grammar multiple-choice question generation using Text-to-Text Transfer Transformer - Chomphooyod, Peerawat; Suchato, Atiwong; Tuaycharoen, Nuengwong; Punyabukkana, Proadpran (Computers and Education: Artificial Intelligence, 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT - Dai, Wei; Lin, Jionghao; Jin, Hua; Li, Tongguang; Tsai, Yi-Shan; Gašević, Dragan; Chen, Guanliang (2023 IEEE International Conference on Advanced Learning Technologies (ICALT), 2023)
Educational feedback has been widely acknowledged as an effective approach to improving student learning. However, scaling effective practices can be laborious and costly, which motivated researchers to work on automated feedback systems (AFS). Inspired by the recent advancements in the pre-trained language models (e.g., ChatGPT), we posit that such models might advance the existing knowledge of textual feedback generation in AFS because of their capability to offer natural-sounding and detailed responses. Therefore, we aimed to investigate the feasibility of using ChatGPT to provide students with feedback to help them learn better. Our results show that i) ChatGPT is capable of generating more detailed feedback that fluently and coherently summarizes students’ performance than human instructors; ii) ChatGPT achieved high agreement with the instructor when assessing the topic of students’ assignments; and iii) ChatGPT could provide feedback on the process of students completing the task, which might benefit students developing learning skills.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z

How Useful are Educational Questions Generated by Large Language Models? - Elkins, Sabina; Kochmar, Ekaterina; Cheung, Jackie C. K.; Serban, Iulian (arXiv, 2023)
Controllable text generation (CTG) by large language models has a huge potential to transform education for teachers and students alike. Specifically, high quality and diverse question generation can dramatically reduce the load on teachers and improve the quality of their educational content. Recent work in this domain has made progress with generation, but fails to show that real teachers judge the generated questions as sufficiently useful for the classroom setting; or if instead the questions have errors and/or pedagogically unhelpful content. We conduct a human evaluation with teachers to assess the quality and usefulness of outputs from combining CTG and question taxonomies (Bloom's and a difficulty taxonomy). The results demonstrate that the questions generated are high quality and sufficiently useful, showing their promise for widespread use in the classroom setting.
arXiv:2304.06638 [cs] Issue: arXiv:2304.06638 <标题>: 由大型语言模型生成的教育问题有多大用处？ Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Computers and Society; Computer Science - Machine Learning

GPTScore: Evaluate as You Desire - Fu, Jinlan; Ng, See-Kiong; Jiang, Zhengbao; Liu, Pengfei (arXiv, 2023)
Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models. Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently. This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., zero-shot instruction) of generative pre-trained models to score generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., FLAN-T5-small) to 175B (e.g., GPT3). Experimental results on four text generation tasks, 22 evaluation aspects, and corresponding 37 datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions. This nature helps us overcome several long-standing challenges in text evaluation–how to achieve customized, multi-faceted evaluation without the need for annotated samples. We make our code publicly available at https://github.com/jinlanfu/GPTScore.
Version Number: 2 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread; Computation and Language (cs.CL); FOS: Computer and information sciences; Reviewed

GPTScore: Evaluate as You Desire - Fu, Jinlan; Ng, See-Kiong; Jiang, Zhengbao; Liu, Pengfei (arXiv, 2023)
Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models. Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently. This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., zero-shot instruction) of generative pre-trained models to score generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., FLAN-T5-small) to 175B (e.g., GPT3). Experimental results on four text generation tasks, 22 evaluation aspects, and corresponding 37 datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions. This nature helps us overcome several long-standing challenges in text evaluation–how to achieve customized, multi-faceted evaluation without the need for annotated samples. We make our code publicly available at https://github.com/jinlanfu/GPTScore.
Version Number: 2 <标题>: GPT评分：按您的意愿进行评估 <AI Smry>: A novel evaluation framework, GPTScore, which utilizes the emergent abilities of generative pre-trained models to score generated texts simply by natural language instructions to overcome several long-standing challenges in text evaluation. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: Computation and Language (cs.CL); FOS: Computer and information sciences

Bias and Fairness in Large Language Models: A Survey - Gallegos, Isabel O.; Rossi, Ryan A.; Barrow, Joe; Tanjim, Md Mehrab; Kim, Sungchul; Dernoncourt, Franck; Yu, Tong; Zhang, Ruiyi; Ahmed, Nesreen K. (arXiv, 2023)
Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases. In this paper, we present a comprehensive survey of bias evaluation and mitigation techniques for LLMs. We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. We then unify the literature by proposing three intuitive taxonomies, two for bias evaluation, namely metrics and datasets, and one for mitigation. Our first taxonomy of metrics for bias evaluation disambiguates the relationship between metrics and evaluation datasets, and organizes metrics by the different levels at which they operate in a model: embeddings, probabilities, and generated text. Our second taxonomy of datasets for bias evaluation categorizes datasets by their structure as counterfactual inputs or prompts, and identifies the targeted harms and social groups; we also release a consolidation of publicly-available datasets for improved access. Our third taxonomy of techniques for bias mitigation classifies methods by their intervention during pre-processing, in-training, intra-processing, and post-processing, with granular subcategories that elucidate research trends. Finally, we identify open problems and challenges for future work. Synthesizing a wide range of recent research, we aim to provide a clear guide of the existing literature that empowers researchers and practitioners to better understand and prevent the propagation of bias in LLMs.
Version Number: 2 <AI Smry>: A comprehensive survey of bias evaluation and mitigation techniques for LLMs, consolidating, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: /unread; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); FOS: Computer and information sciences; Machine Learning (cs.LG); fairness

Bias and Fairness in Large Language Models: A Survey - Gallegos, Isabel O.; Rossi, Ryan A.; Barrow, Joe; Tanjim, Md Mehrab; Kim, Sungchul; Dernoncourt, Franck; Yu, Tong; Zhang, Ruiyi; Ahmed, Nesreen K. (arXiv, 2023)
Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases. In this paper, we present a comprehensive survey of bias evaluation and mitigation techniques for LLMs. We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. We then unify the literature by proposing three intuitive taxonomies, two for bias evaluation, namely metrics and datasets, and one for mitigation. Our first taxonomy of metrics for bias evaluation disambiguates the relationship between metrics and evaluation datasets, and organizes metrics by the different levels at which they operate in a model: embeddings, probabilities, and generated text. Our second taxonomy of datasets for bias evaluation categorizes datasets by their structure as counterfactual inputs or prompts, and identifies the targeted harms and social groups; we also release a consolidation of publicly-available datasets for improved access. Our third taxonomy of techniques for bias mitigation classifies methods by their intervention during pre-processing, in-training, intra-processing, and post-processing, with granular subcategories that elucidate research trends. Finally, we identify open problems and challenges for future work. Synthesizing a wide range of recent research, we aim to provide a clear guide of the existing literature that empowers researchers and practitioners to better understand and prevent the propagation of bias in LLMs.
Version Number: 2 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); FOS: Computer and information sciences; Machine Learning (cs.LG)

Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning - Han, Lifeng; Erofeev, Gleb; Sorokina, Irina; Gladkoff, Serge; Nenadic, Goran (arXiv, 2023)
Massively multilingual pre-trained language models (MMPLMs) are developed in recent years demonstrating superpowers and the pre-knowledge they acquire for downstream tasks. This work investigates whether MMPLMs can be applied to clinical domain machine translation (MT) towards entirely unseen languages via transfer learning. We carry out an experimental investigation using Meta-AI's MMPLMs ``wmt21-dense-24-wide-en-X and X-en (WMT21fb)'' which were pre-trained on 7 language pairs and 14 translation directions including English to Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese, and the opposite direction. We fine-tune these MMPLMs towards English-\textit{Spanish} language pair which \textit{did not exist at all} in their original pre-trained corpora both implicitly and explicitly. We prepare carefully aligned \textit{clinical} domain data for this fine-tuning, which is different from their original mixed domain knowledge. Our experimental result shows that the fine-tuning is very successful using just 250k well-aligned in-domain EN-ES segments for three sub-task translation testings: clinical cases, clinical terms, and ontology concepts. It achieves very close evaluation scores to another MMPLM NLLB from Meta-AI, which included Spanish as a high-resource setting in the pre-training. To the best of our knowledge, this is the first work on using MMPLMs towards \textit{clinical domain transfer-learning NMT} successfully for totally unseen languages during pre-training.
arXiv:2210.06068 [cs] <标题>: 通过迁移学习研究面向临床领域的大型多语言预训练机器翻译模型 <AI Smry>: This is the first work on using MMPLMs towards clinical domain transfer-learning NMT successfully for totally unseen languages during pre-training and achieves very close evaluation scores to another MMPLM NLLB from Meta-AI. Read_Status: New Read_Status_Date: 2026-01-26T11:32:18.434Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Fine-tuning ChatGPT for Automatic Scoring - Latif, Ehsan; Zhai, Xiaoming (arXiv, 2023)
This study highlights the potential of fine-tuned ChatGPT (GPT-3.5) for automatically scoring student written constructed responses using example assessment tasks in science education. Recent studies on OpenAI's generative model GPT-3.5 proved its superiority in predicting the natural language with high accuracy and human-like responses. GPT-3.5 has been trained over enormous online language materials such as journals and Wikipedia; therefore, more than direct usage of pre-trained GPT-3.5 is required for automatic scoring as students utilize a different language than trained material. These imply that a domain-specific model, fine-tuned over data for specific tasks, can enhance model performance. In this study, we fine-tuned GPT-3.5 on six assessment tasks with a diverse dataset of middle-school and high-school student responses and expert scoring. The six tasks comprise two multi-label and four multi-class assessment tasks. We compare the performance of fine-tuned GPT-3.5 with the fine-tuned state-of-the-art Google's generated language model, BERT. The results show that in-domain training corpora constructed from science questions and responses for BERT achieved average accuracy = 0.838, SD = 0.069. GPT-3.5 shows a remarkable average increase (9.1%) in automatic scoring accuracy (mean = 9.15, SD = 0.042) for the six tasks, p =0.001 < 0.05. Specifically, for multi-label tasks (item 1 with 5 labels; item 2 with 10 labels), GPT-3.5 achieved significantly higher scoring accuracy than BERT across all the labels, with the second item achieving a 7.1% increase. The average scoring increase for the four multi-class items for GPT-3.5 was 10.6% compared to BERT. Our study confirmed the effectiveness of fine-tuned GPT-3.5 for automatic scoring of student responses on domain-specific data in education with high accuracy. We have released fine-tuned models for public use and community engagement.
Version Number: 1 <标题>: 微调 ChatGPT 以实现自动评分 <AI Smry>: This study highlights the potential of fine-tuned ChatGPT (GPT-3.5) for automatically scoring student written constructed responses using example assessment tasks in science education with high accuracy and confirms the effectiveness of GPT- 3.5 for automatic scoring of student responses on domain-specific data in education withhigh accuracy. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences

Applying Large Language Models and Chain-of-Thought for Automatic Scoring - Lee, Gyeong-Geon; Latif, Ehsan; Wu, Xuansheng; Liu, Ninghao; Zhai, Xiaoming ([object Object], 2023)
This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1,650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics. Results indicated that few-shot (acc = .67) outperformed zero-shot learning (acc = .60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = .60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT -3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches.
Version Number: 2 <AI Smry>: A more balanced accuracy is found across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences

Applying Large Language Models and Chain-of-Thought for Automatic Scoring - Lee, Gyeong-Geon; Latif, Ehsan; Wu, Xuansheng; Liu, Ninghao; Zhai, Xiaoming ([object Object], 2023)
This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1,650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics. Results indicated that few-shot (acc = .67) outperformed zero-shot learning (acc = .60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = .60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT -3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches.
Version Number: 2 <标题>: 将大型语言模型与链式思维应用于自动评分 <AI Smry>: A more balanced accuracy is found across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences

Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference - Levonian, Zachary; Li, Chenglu; Zhu, Wangda; Gade, Anoushka; Henkel, Owen; Postle, Millie-Ellen; Xing, Wanli (arXiv, 2023)
For middle-school math students, interactive question-answering (QA) with tutors is an effective way to learn. The flexibility and emergent capabilities of generative large language models (LLMs) has led to a surge of interest in automating portions of the tutoring process - including interactive QA to support conceptual discussion of mathematical concepts. However, LLM responses to math questions can be incorrect or mismatched to the educational context - such as being misaligned with a school's curriculum. One potential solution is retrieval-augmented generation (RAG), which involves incorporating a vetted external knowledge source in the LLM prompt to increase response quality. In this paper, we designed prompts that retrieve and use content from a high-quality open-source math textbook to generate responses to real student questions. We evaluate the efficacy of this RAG system for middle-school algebra and geometry QA by administering a multi-condition survey, finding that humans prefer responses generated using RAG, but not when responses are too grounded in the textbook content. We argue that while RAG is able to improve response quality, designers of math QA systems must consider trade-offs between generating responses preferred by students and responses closely matched to specific educational resources.
Version Number: 2 <标题>: 检索增强生成以提高数学问答：扎根性与人类偏好的权衡 <AI Smry>: It is argued that while RAG is able to improve response quality, designers of math QA systems must consider trade-offs between generating responses preferred by students and responses closely matched to specific educational resources. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: Computation and Language (cs.CL); FOS: Computer and information sciences; Human-Computer Interaction (cs.HC)

Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference - Levonian, Zachary; Li, Chenglu; Zhu, Wangda; Gade, Anoushka; Henkel, Owen; Postle, Millie-Ellen; Xing, Wanli (arXiv, 2023)
For middle-school math students, interactive question-answering (QA) with tutors is an effective way to learn. The flexibility and emergent capabilities of generative large language models (LLMs) has led to a surge of interest in automating portions of the tutoring process - including interactive QA to support conceptual discussion of mathematical concepts. However, LLM responses to math questions can be incorrect or mismatched to the educational context - such as being misaligned with a school's curriculum. One potential solution is retrieval-augmented generation (RAG), which involves incorporating a vetted external knowledge source in the LLM prompt to increase response quality. In this paper, we designed prompts that retrieve and use content from a high-quality open-source math textbook to generate responses to real student questions. We evaluate the efficacy of this RAG system for middle-school algebra and geometry QA by administering a multi-condition survey, finding that humans prefer responses generated using RAG, but not when responses are too grounded in the textbook content. We argue that while RAG is able to improve response quality, designers of math QA systems must consider trade-offs between generating responses preferred by students and responses closely matched to specific educational resources.
Version Number: 2 <标题>: 检索增强生成以提高数学问答：扎根性与人类偏好的权衡 <AI Smry>: It is argued that while RAG is able to improve response quality, designers of math QA systems must consider trade-offs between generating responses preferred by students and responses closely matched to specific educational resources. Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z
Keywords: Computation and Language (cs.CL); FOS: Computer and information sciences; Human-Computer Interaction (cs.HC)

A Framework for Responsible Development of Automated Student Feedback with Generative AI - Lindsay, Euan D; Johri, Aditya; Bjerva, Johannes ([object Object], 2023)
Providing rich feedback to students is essential for supporting student learning. Recent advances in generative AI, particularly within large language modelling (LLM), provide the opportunity to deliver repeatable, scalable and instant automatically generated feedback to students, making abundant a previously scarce and expensive learning resource. Such an approach is feasible from a technical perspective due to these recent advances in Artificial Intelligence (AI) and Natural Language Processing (NLP); while the potential upside is a strong motivator, doing so introduces a range of potential ethical issues that must be considered as we apply these technologies. The attractiveness of AI systems is that they can effectively automate the most mundane tasks; but this risks introducing a "tyranny of the majority", where the needs of minorities in the long tail are overlooked because they are difficult to automate. Developing machine learning models that can generate valuable and authentic feedback requires the input of human domain experts. The choices we make in capturing this expertise – whose, which, when, and how – will have significant consequences for the nature of the resulting feedback. How we maintain our models will affect how that feedback remains relevant given temporal changes in context, theory, and prior learning profiles of student cohorts. These questions are important from an ethical perspective; but they are also important from an operational perspective. Unless they can be answered, our AI generated systems will lack the trust necessary for them to be useful features in the contemporary learning environment. This article will outline the frontiers of automated feedback, identify the ethical issues involved in the provision of automated feedback and present a framework to assist academics to develop such systems responsibly.
Version Number: 1 <标题>: 一个面向负责任开发的生成式人工智能自动学生反馈框架 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z
Keywords: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); FOS: Computer and information sciences

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment - Liu, Yang; Iter, Dan; Xu, Yichong; Wang, Shuohang; Xu, Ruochen; Zhu, Chenguang (arXiv, 2023)
The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs. We experiment with two generation tasks, text summarization and dialogue generation. We show that G-Eval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin. We also propose preliminary analysis on the behavior of LLM-based evaluators, and highlight the potential issue of LLM-based evaluators having a bias towards the LLM-generated texts. The code is at https://github.com/nlpyang/geval
arXiv:2303.16634 [cs] <标题>: G-Eval：使用 GPT-4 进行更好人类对齐的自然语言生成评估 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Deciphering Stereotypes in Pre-Trained Language Models - Ma, Weicheng; Scheible, Henry; Wang, Brian; Veeramachaneni, Goutham; Chowdhary, Pratim; Sun, Alan; Koulogeorge, Andrew; Wang, Lili; Yang, Diyi; Vosoughi, Soroush (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023)
Warning: This paper contains content that is stereotypical and may be upsetting. This paper addresses the issue of demographic stereotypes present in Transformer-based pre-trained language models (PLMs) and aims to deepen our understanding of how these biases are encoded in these models. To accomplish this, we introduce an easy-to-use framework for examining the stereotype-encoding behavior of PLMs through a combination of model probing and textual analyses. Our findings reveal that a small subset of attention heads within PLMs are primarily responsible for encoding stereotypes and that stereotypes toward specific minority groups can be identified using attention maps on these attention heads. Leveraging these insights, we propose an attention-head pruning method as a viable approach for debiasing PLMs, without compromising their language modeling capabilities or adversely affecting their performance on downstream tasks.
<标题>: 解读预训练语言模型中的刻板印象 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Sources of Hallucination by Large Language Models on Inference Tasks - McKenna, Nick; Li, Tianyi; Cheng, Liang; Hosseini, Mohammad Javad; Johnson, Mark; Steedman, Mark (arXiv, 2023)
Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization. We present a series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled experiments. We establish two biases originating from pretraining which predict much of their behavior, and show that these are major sources of hallucination in generative LLMs. First, memorization at the level of sentences: we show that, regardless of the premise, models falsely label NLI test samples as entailing when the hypothesis is attested in training data, and that entities are used as ``indices'' to access the memorized data. Second, statistical patterns of usage learned at the level of corpora: we further show a similar effect when the premise predicate is less frequent than that of the hypothesis in the training data, a bias following from previous studies. We demonstrate that LLMs perform significantly worse on NLI test samples which do not conform to these biases than those which do, and we offer these as valuable controls for future LLM evaluation.
arXiv:2305.14552 [cs] <标题>: 大型语言模型在推理任务中的幻觉来源 Read_Status: New Read_Status_Date: 2026-01-26T11:32:28.375Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Exploring Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning - McNichols, Hunter; Feng, Wanyong; Lee, Jaewook; Scarlatos, Alexander; Smith, Digory; Woodhead, Simon; Lan, Andrew (arXiv, 2023)
Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and are a reliable format in both assessments and practices. An important aspect of MCQs is the distractors, i.e., incorrect options that are designed to target specific misconceptions or insufficient knowledge among students. To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers, which has limited scalability. In this work, we explore the task of automated distractor and corresponding feedback message generation in math MCQs using large language models. We establish a formulation of these two tasks and propose a simple, in-context learning-based solution. Moreover, we explore using two non-standard metrics to evaluate the quality of the generated distractors and feedback messages. We conduct extensive experiments on these tasks using a real-world MCQ dataset that contains student response information. Our findings suggest that there is a lot of room for improvement in automated distractor and feedback generation. We also outline several directions for future work
Version Number: 1 <标题>: 通过情境学习探索数学选择题的自动干扰项和反馈生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: Computation and Language (cs.CL); FOS: Computer and information sciences

Guidance for generative AI in education and research - Miao, Fengchun; Holmes, Wayne (UNESCO, 2023)
<标题>: 教育和科研中生成式人工智能的指导 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.151Z
Keywords: /unread

Using AI to Implement Effective Teaching Strategies in Classrooms: Five Strategies, Including Prompts - Mollick, Ethan R.; Mollick, Lilach (2023)
This paper provides guidance for using AI to quickly and easily implement evidence-based teaching strategies that instructors can integrate into their teaching. We discuss five teaching strategies that have proven value but are hard to implement in practice due to time and effort constraints. We show how AI can help instructors create material that supports these strategies and improve student learning. The strategies include providing multiple examples and explanations; uncovering and addressing student misconceptions; frequent low-stakes testing; assessing student learning; and distributed practice. The paper provides guidelines for how AI can support each strategy, and discusses both the promises and perils of this approach, arguing that AI may act as a “force multiplier” for instructors if implemented cautiously and thoughtfully in service of evidence-based teaching practices.
<AI Smry>: The paper provides guidelines for how AI can support each strategy, and discusses both the promises and perils of this approach, arguing that AI may act as a “force multiplier” for instructors if implemented cautiously and thoughtfully in service of evidence-based teaching practices. <标题>: 在课堂上使用人工智能实施有效教学策略：五种策略，包括提示 Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.807Z
Keywords: /unread; AI; ChatGPT; GPT4; Learning

Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods - Moore, Steven; Nguyen, Huy A.; Chen, Tianying; Stamper, John (Responsive and Sustainable Educational Futures, 2023)
Series Title: Lecture Notes in Computer Science Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

GenQ: Automated Question Generation to Support Caregivers While Reading Stories with Children - Narayanan, Arun Balajiee Lekshmi; Gomez, Ligia E.; Fernandez, Martha Michelle Soto; Nguyen, Tri; Blais, Chris; Restrepo, M. Adelaida; Glenberg, Art (arXiv, 2023)
When caregivers ask open–ended questions to motivate dialogue with children, it facilitates the child's reading comprehension skills.Although there is scope for use of technological tools, referred here as "intelligent tutoring systems", to scaffold this process, it is currently unclear whether existing intelligent systems that generate human–language like questions is beneficial. Additionally, training data used in the development of these automated question generation systems is typically sourced without attention to demographics, but people with different cultural backgrounds may ask different questions. As a part of a broader project to design an intelligent reading support app for Latinx children, we crowdsourced questions from Latinx caregivers and noncaregivers as well as caregivers and noncaregivers from other demographics. We examine variations in question–asking within this dataset mediated by individual, cultural, and contextual factors. We then design a system that automatically extracts templates from this data to generate open–ended questions that are representative of those asked by Latinx caregivers.
Version Number: 3 <标题>: GenQ：自动化问题生成，支持在与孩子共读故事时照顾者 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences; Human-Computer Interaction (cs.HC)

Enhancing Programming eTextbooks with ChatGPT Generated Counterfactual-Thinking-Inspired Questions - Narayanan, Arun Balajiee Lekshmi; Hendrawan, Rully Agus; V, Venktesh (arXiv, 2023)
Digital textbooks have become an integral part of everyday learning tasks. In this work, we consider the use of digital textbooks for programming classes. Generally, students struggle with utilizing textbooks on programming to the maximum, with a possible reason being that the example programs provided as illustration of concepts in these textbooks don't offer sufficient interactivity for students, and thereby not sufficiently motivating to explore or understand these programming examples better. In our work, we explore the idea of enhancing the navigability of intelligent textbooks with the use of ``counterfactual'' questions, to make students think critically about these programs and enhance possible program comprehension. Inspired from previous works on nudging students on counter factual thinking, we present the possibility to enhance digital textbooks with questions generated using GPT.
Version Number: 2 <标题>: 利用 ChatGPT 生成的反事实思维启发问题增强编程电子教材 <AI Smry>: This work explores the idea of enhancing the navigability of intelligent textbooks with the use of ``counterfactual'' questions, to make students think critically about these programs and enhance possible program comprehension. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: Computation and Language (cs.CL); FOS: Computer and information sciences; Human-Computer Interaction (cs.HC)

Ethical principles for artificial intelligence in education - Nguyen, Andy; Ngo, Ha Ngan; Hong, Yvonne; Dang, Belle; Nguyen, Bich-Phuong Thi (Education and Information Technologies, 2023)
Abstract The advancement of artificial intelligence in education (AIED) has the potential to transform the educational landscape and influence the role of all involved stakeholders. In recent years, the applications of AIED have been gradually adopted to progress our understanding of students’ learning and enhance learning performance and experience. However, the adoption of AIED has led to increasing ethical risks and concerns regarding several aspects such as personal data and learner autonomy. Despite the recent announcement of guidelines for ethical and trustworthy AIED, the debate revolves around the key principles underpinning ethical AIED. This paper aims to explore whether there is a global consensus on ethical AIED by mapping and analyzing international organizations’ current policies and guidelines. In this paper, we first introduce the opportunities offered by AI in education and potential ethical issues. Then, thematic analysis was conducted to conceptualize and establish a set of ethical principles by examining and synthesizing relevant ethical policies and guidelines for AIED. We discuss each principle and associated implications for relevant educational stakeholders, including students, teachers, technology developers, policymakers, and institutional decision-makers. The proposed set of ethical principles is expected to serve as a framework to inform and guide educational stakeholders in the development and deployment of ethical and trustworthy AIED as well as catalyze future development of related impact studies in the field.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Navigating the generative AI era: Introducing the AI assessment scale for ethical GenAI assessment - Perkins, Mike; Furze, Leon; Roe, Jasper; MacVaugh, Jason (arXiv, 2023)
Recent developments in Generative Artificial Intelligence (GenAI) have created a paradigm shift in multiple areas of society, and the use of these technologies is likely to become a defining feature of education in coming decades. GenAI offers transformative pedagogical opportunities, while simultaneously posing ethical and academic challenges. Against this backdrop, we outline a practical, simple, and sufficiently comprehensive tool to allow for the integration of GenAI tools into educational assessment: the AI Assessment Scale (AIAS). The AIAS empowers educators to select the appropriate level of GenAI usage in assessments based on the learning outcomes they seek to address. The AIAS offers greater clarity and transparency for students and educators, provides a fair and equitable policy tool for institutions to work with, and offers a nuanced approach which embraces the opportunities of GenAI while recognising that there are instances where such tools may not be pedagogically appropriate or necessary. By adopting a practical, flexible approach that can be implemented quickly, the AIAS can form a much-needed starting point to address the current uncertainty and anxiety regarding GenAI in education. As a secondary objective, we engage with the current literature and advocate for a refocused discourse on GenAI tools in education, one which foregrounds how technologies can help support and enhance teaching and learning, which contrasts with the current focus on GenAI as a facilitator of academic misconduct.
Version Number: 1 <标题>: 驾驭生成式人工智能时代：推出用于伦理生成式人工智能评估的AI评估量表 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: Artificial Intelligence (cs.AI); FOS: Computer and information sciences; K.4

Sabiá: Portuguese Large Language Models - Pires, Ramon; Abonizio, Hugo; Almeida, Thales Sales; Nogueira, Rodrigo (2023)
As the capabilities of language models continue to advance, it is conceivable that "one-size-fits-all" model will remain as the main paradigm. For instance, given the vast number of languages worldwide, many of which are low-resource, the prevalent practice is to pretrain a single model on multiple languages. In this paper, we add to the growing body of evidence that challenges this practice, demonstrating that monolingual pretraining on the target language significantly improves models already extensively trained on diverse corpora. More specifically, we further pretrain GPT-J and LLaMA models on Portuguese texts using 3% or less of their original pretraining budget. Few-shot evaluations on Poeta, a suite of 14 Portuguese datasets, reveal that our models outperform English-centric and multilingual counterparts by a significant margin. Our best model, Sabiá-65B, performs on par with GPT-3.5-turbo. By evaluating on datasets originally conceived in the target language as well as translated ones, we study the contributions of language-specific pretraining in terms of 1) capturing linguistic nuances and structures inherent to the target language, and 2) enriching the model's knowledge about a domain or culture. Our results indicate that the majority of the benefits stem from the domain-specific knowledge acquired through monolingual pretraining.
<标题>: Sabiá：葡萄牙语大型语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:32:23.432Z
Keywords: /unread; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences; LLM; fine-tune

Sabiá: Portuguese Large Language Models - Pires, Ramon; Abonizio, Hugo; Almeida, Thales Sales; Nogueira, Rodrigo (2023)
As the capabilities of language models continue to advance, it is conceivable that "one-size-fits-all" model will remain as the main paradigm. For instance, given the vast number of languages worldwide, many of which are low-resource, the prevalent practice is to pretrain a single model on multiple languages. In this paper, we add to the growing body of evidence that challenges this practice, demonstrating that monolingual pretraining on the target language significantly improves models already extensively trained on diverse corpora. More specifically, we further pretrain GPT-J and LLaMA models on Portuguese texts using 3% or less of their original pretraining budget. Few-shot evaluations on Poeta, a suite of 14 Portuguese datasets, reveal that our models outperform English-centric and multilingual counterparts by a significant margin. Our best model, Sabiá-65B, performs on par with GPT-3.5-turbo. By evaluating on datasets originally conceived in the target language as well as translated ones, we study the contributions of language-specific pretraining in terms of 1) capturing linguistic nuances and structures inherent to the target language, and 2) enriching the model's knowledge about a domain or culture. Our results indicate that the majority of the benefits stem from the domain-specific knowledge acquired through monolingual pretraining.
<标题>: Sabiá：葡萄牙语大型语言模型 Read_Status: New Read_Status_Date: 2026-01-26T11:32:15.784Z
Keywords: /unread; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences; LLM; fine-tune

ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope - Ray, Partha Pratim (Internet of Things and Cyber-Physical Systems, 2023)
In recent years, artificial intelligence (AI) and machine learning have been transforming the landscape of scientific research. Out of which, the chatbot technology has experienced tremendous advancements in recent years, especially with ChatGPT emerging as a notable AI language model. This comprehensive review delves into the background, applications, key challenges, and future directions of ChatGPT. We begin by exploring its origins, development, and underlying technology, before examining its wide-ranging applications across industries such as customer service, healthcare, and education. We also highlight the critical challenges that ChatGPT faces, including ethical concerns, data biases, and safety issues, while discussing potential mitigation strategies. Finally, we envision the future of ChatGPT by exploring areas of further research and development, focusing on its integration with other technologies, improved human-AI interaction, and addressing the digital divide. This review offers valuable insights for researchers, developers, and stakeholders interested in the ever-evolving landscape of AI-driven conversational agents. This study explores the various ways ChatGPT has been revolutionizing scientific research, spanning from data processing and hypothesis generation to collaboration and public outreach. Furthermore, the paper examines the potential challenges and ethical concerns surrounding the use of ChatGPT in research, while highlighting the importance of striking a balance between AI-assisted innovation and human expertise. The paper presents several ethical issues in existing computing domain and how ChatGPT can invoke challenges to such notion. This work also includes some biases and limitations of ChatGPT. It is worth to note that despite of several controversies and ethical concerns, ChatGPT has attracted remarkable attentions from academia, research, and industries in a very short span of time.
<标题>: ChatGPT：关于背景、应用、主要挑战、偏见、伦理、局限性及未来发展方向的综合评述 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: /unread; ChatGPT; Context understanding; Conversational AI; GPT-3.5; Generative AI; Language model; Natural language processing

CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles - Sonkar, Shashank; Liu, Naiming; Mallick, Debshila; Baraniuk, Richard (Findings of the Association for Computational Linguistics: EMNLP 2023, 2023)
<AI Smry>: A design framework for building advanced Intelligent Tutoring Systems (ITS) powered by high-performance Large Language Models (LLMs) using the CLASS framework, which equips ITS with essential problem-solving strategies, enabling it to provide tutor-like, step-by-step guidance to students. <标题>: 课程：一种基于学习科学原理构建智能辅导系统的设计框架 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them - Suzgun, Mirac; Scales, Nathan; Schärli, Nathanael; Gehrmann, Sebastian; Tay, Yi; Chung, Hyung Won; Chowdhery, Aakanksha; Le, Quoc V.; Chi, Ed H.; Zhou, Denny; Wei, Jason (arXiv, 2023)
BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models. Language models have already made good progress on this benchmark, with the best model in the BIG-Bench paper outperforming average reported human-rater results on 65% of the BIG-Bench tasks via few-shot prompting. But on what tasks do language models fall short of average human-rater performance, and are those tasks actually unsolvable by current language models? In this work, we focus on a suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH). These are the task for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average human-rater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.
arXiv:2210.09261 [cs] <标题>: 具有挑战性的 BIG-Bench 任务以及思维链是否能解决它们 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language

Towards Generalizable Detection of Urgency of Discussion Forum Posts - Švábenský, Valdemar; Baker, Ryan S.; Zambrano, Andrés; Zou, Yishan; Slater, Stefan (Proceedings of the 16th International Conference on Educational Data Mining, 2023)
<标题>: 面向讨论论坛帖子紧急性的一般化检测 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: /unread

Approximating Online Human Evaluation of Social Chatbots with Prompting - Svikhnushina, Ekaterina; Pu, Pearl (arXiv, 2023)
As conversational models become increasingly available to the general public, users are engaging with this technology in social interactions. Such unprecedented interaction experiences may pose considerable social and psychological risks to the users unless the technology is properly controlled. This highlights the need for scalable and robust evaluation metrics for conversational chatbots. Existing evaluation metrics aim to automate offline user evaluation and approximate human judgment of pre-curated dialogs. However, they are limited in their ability to capture subjective perceptions of users who actually interact with the bots and might not generalize to real-world settings. To address this limitation, we propose an approach to approximate online human evaluation leveraging large language models (LLMs) from the GPT family. We introduce a new Dialog system Evaluation framework based on Prompting (DEP), which enables a fully automatic evaluation pipeline that replicates live user studies and achieves an impressive correlation with human judgment (up to Pearson r=0.95 on a system level). The DEP approach involves collecting synthetic chat logs of evaluated bots with an LLM in the other-play setting, where the LLM is carefully conditioned to follow a specific scenario. We further explore different prompting approaches to produce evaluation scores with the same LLM. The best performing prompts, which contain few-shot demonstrations and instructions, show outstanding performance on the tested dataset and demonstrate the ability to generalize to other dialog corpora.
arXiv:2304.05253 [cs] <标题>: 使用提示语近似在线社交聊天机器人的人工评估 <AI Smry>: A new Dialog system Evaluation framework based on Prompting is introduced, which enables a fully automatic evaluation pipeline that replicates live user studies and achieves an impressive correlation with human judgment (up to Pearson r=0.95 on a system level). Read_Status: New Read_Status_Date: 2026-01-26T11:32:21.058Z
Keywords: /unread; Computer Science - Computation and Language; Computer Science - Human-Computer Interaction

Auditing and Mitigating Cultural Bias in LLMs - Tao, Yan; Viberg, Olga; Baker, Ryan S.; Kizilcec, Rene F. (arXiv, 2023)
Culture fundamentally shapes people's reasoning, behavior, and communication. Generative artificial intelligence (AI) technologies may cause a shift towards a dominant culture. As people increasingly use AI to expedite and even automate various professional and personal tasks, cultural values embedded in AI models may bias authentic expression. We audit large language models for cultural bias, comparing their responses to nationally representative survey data, and evaluate country-specific prompting as a mitigation strategy. We find that GPT-4, 3.5 and 3 exhibit cultural values resembling English-speaking and Protestant European countries. Our mitigation strategy reduces cultural bias in recent models but not for all countries/territories. To avoid cultural bias in generative AI, especially in high-stakes contexts, we suggest using culture matching and ongoing cultural audits.
Version Number: 1 Read_Status: New Read_Status_Date: 2026-01-26T11:33:01.187Z
Keywords: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences

Is ChatGPT a Good NLG Evaluator? A Preliminary Study - Wang, Jiaan; Liang, Yunlong; Meng, Fandong; Sun, Zengkui; Shi, Haoxiang; Li, Zhixu; Xu, Jinan; Qu, Jianfeng; Zhou, Jie (arXiv, 2023)
Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regard ChatGPT as a human evaluator and give task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruction to prompt ChatGPT to evaluate the generated results of NLG models. We conduct experiments on five NLG meta-evaluation datasets (including summarization, story generation and data-to-text tasks). Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases. In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets. For the meta-evaluation datasets which are created greatly depending on the reference and thus are biased, the ChatGPT evaluator might lose its effectiveness. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.
Version Number: 3 <AI Smry>: Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases and it is hoped this preliminary study could prompt the emergence of a general-purposed reliable NLG metric. <标题>: ChatGPT 是一个好的自然语言生成评估器吗？初步研究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences

CRASE+® for ACT Writing Technical Report - Wood, Scott (2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Rating Short L2 Essays on the CEFR Scale with GPT-4 - Yancey, Kevin P.; Laflair, Geoffrey; Verardi, Anthony; Burstein, Jill (Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), 2023)
Essay scoring is a critical task used to evaluate second-language (L2) writing proficiency on high-stakes language assessments. While automated scoring approaches are mature and have been around for decades, human scoring is still considered the gold standard, despite its high costs and well-known issues such as human rater fatigue and bias. The recent introduction of large language models (LLMs) brings new opportunities for automated scoring. In this paper, we evaluate how well GPT-3.5 and GPT-4 can rate short essay responses written by L2 English learners on a high-stakes language assessment, computing inter-rater agreement with human ratings. Results show that when calibration examples are provided, GPT-4 can perform almost as well as modern Automatic Writing Evaluation (AWE) methods, but agreement with human ratings can vary depending on the test-taker's first language (L1).
<AI Smry>: Results show that when calibration examples are provided, GPT-4 can perform almost as well as modern Automatic Writing Evaluation (AWE) methods, but agreement with human ratings can vary depending on the test-taker’s first language (L1). <标题>: 使用 GPT-4 按 CEFR 量表评估短篇 L2 论文 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency - Zelikman, Eric; Ma, Wanjing Anya; Tran, Jasmine E.; Yang, Diyi; Yeatman, Jason D.; Haber, Nick (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023)
<标题>: 使用语言模型模拟为中小学生生成和评估测试：以句子阅读效率为例的案例研究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code - Zhou, Shuyan; Alon, Uri; Agarwal, Sumit; Neubig, Graham (Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023)
<标题>: CodeBERTScore：使用预训练代码模型评估代码生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.511Z
Keywords: /unread

Learning by Analogy: Diverse Questions Generation in Math Word Problem - Zhou, Zihao; Ning, Maizhen; Wang, Qiufeng; Yao, Jie; Wang, Wei; Huang, Xiaowei; Huang, Kaizhu (Findings of the Association for Computational Linguistics: ACL 2023, 2023)
<标题>: 类比学习：数学应用题的多样化问题生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

Using Demographic Data as Predictor Variables: a Questionable Choice - Baker, Ryan Shaun; Esbenshade, Lief; Vitale, Jon; Karumbaiah, Shamya (EdArXiv, 2022)
Predictive analytics methods in education are seeing widespread use and are producing increasingly accurate predictions of students’ outcomes. With the increased use of predictive analytics comes increasing concern about fairness for specific subgroups of the population. One approach that has been proposed to increase fairness is using demographic variables directly in models, as predictors. In this paper we explore issues of fairness in the use of demographic variables as predictors of long-term student outcomes, studying the arguments for and against this practice in the contexts where this literature has been published. We analyze arguments for the inclusion of demographic variables, specifically claims that this approach improves model performance and charges that excluding such variables amounts to a form of ‘color-blind’ racism. We also consider arguments against including demographic variables as predictors, including reduced actionability of predictions, risk of reinforcing bias, and limits of categorization. We then discuss how contextual factors of predictive models should influence case-specific decisions for the inclusion or exclusion of demographic variables and discuss the role of proxy variables. We conclude that, on balance, there are greater benefits to fairness if demographic variables are used to validate fairness rather than as predictors within models.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.824Z
Keywords: /unread; fairness

Less is More: Parameter-Free Text Classification with Gzip - Jiang, Zhiying; Yang, Matthew Y. R.; Tsirlin, Mikhail; Tang, Raphael; Lin, Jimmy (arXiv, 2022)
Deep neural networks (DNNs) are often used for text classification tasks as they usually achieve high levels of accuracy. However, DNNs can be computationally intensive with billions of parameters and large amounts of labeled data, which can make them expensive to use, to optimize and to transfer to out-of-distribution (OOD) cases in practice. In this paper, we propose a non-parametric alternative to DNNs that's easy, light-weight and universal in text classification: a combination of a simple compressor like gzip with a $k$-nearest-neighbor classifier. Without any training, pre-training or fine-tuning, our method achieves results that are competitive with non-pretrained deep learning methods on six in-distributed datasets. It even outperforms BERT on all five OOD datasets, including four low-resource languages. Our method also performs particularly well in few-shot settings where labeled data are too scarce for DNNs to achieve a satisfying accuracy.
arXiv:2212.09410 [cs] <标题>: 少即是多：使用 Gzip 的无参数文本分类 Read_Status: New Read_Status_Date: 2026-01-26T11:32:30.842Z
Keywords: Computer Science - Computation and Language

A Graph Convolutional Network Feature Learning Framework for Interpretable Geometry Problem Solving - Guo, Fucheng; Jian, Pengpeng (2022 International Conference on Intelligent Education and Intelligent Research (IEIR), 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

MinMax fairness: from Rawlsian Theory of Justice to solution for algorithmic bias - Barsotti, Flavia; Koçer, Rüya Gökhan (AI & SOCIETY, 2022)
This paper presents an intuitive explanation about why and how Rawlsian Theory of Justice (Rawls in A theory of justice, Harvard University Press, Harvard, 1971) provides the foundations to a solution for algorithmic bias. The contribution of the paper is to discuss and show why Rawlsian ideas in their original form (e.g. the veil of ignorance, original position, and allowing inequalities that serve the worst-off) are relevant to operationalize fairness for algorithmic decision making. The paper also explains how this leads to a specific MinMaxfairness solution, which addresses the basic challenges of algorithmic justice. We combine substantive elements of Rawlsian perspective with an intuitive explanation in order to provide accessible and practical insights. The goal is to propose and motivate why and how the MinMaxfairness solution derived from Rawlsian principles overcomes some of the current challenges for algorithmic bias and highlight the benefits provided when compared to other approaches. The paper presents and discusses the solution by building a bridge between the qualitative theoretical aspects and the quantitative technical approach.
<标题>: 极大极小公平性：从罗尔斯的正义理论到算法偏差的解决方案 <AI Smry>: The goal is to propose and motivate why and how the MinMaxfairness solution derived from Rawlsian principles overcomes some of the current challenges for algorithmic bias and highlight the benefits provided when compared to other approaches. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z

AI-assisted automated scoring of picture-cued writing tasks for language assessment - Zhao, Ruibin; Zhuang, Yipeng; Zou, Di; Xie, Qin; Yu, Philip L. H. (Education and Information Technologies, 2022)
<标题>: 用于语言评估的图画提示写作任务的AI辅助自动评分 <AI Smry>: A novel method for automatically scoring student responses to picture-cued writing tasks by incorporating the emerging cross-modal matching technology and some NLP algorithms is proposed, which could reduce the subjective elements inherent in human grading and save teachers’ time from the mundane task of grading to other valuable endeavors such as designing teaching plans based on AI-generated diagnosis of student progress. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Rubric development for AI-enabled scoring of three-dimensional constructed-response assessment aligned to NGSS learning progression - Kaldaras, Leonora; Yoshida, Nicholas R.; Haudek, Kevin C. (Frontiers in Education, 2022)
The Framework for K-12 Science Education (the Framework) and the Next- Generation Science Standards (NGSS) define three dimensions of science: disciplinary core ideas, scientific and engineering practices, and crosscutting concepts and emphasize the integration of the three dimensions (3D) to reflect deep science understanding. The Framework also emphasizes the importance of using learning progressions (LPs) as roadmaps to guide assessment development. These assessments capable of measuring the integration of NGSS dimensions should probe the ability to explain phenomena and solve problems. This calls for the development of constructed response (CR) or open-ended assessments despite being expensive to score. Artificial intelligence (AI) technology such as machine learning (ML)-based approaches have been utilized to score and provide feedback on open-ended NGSS assessments aligned to LPs. ML approaches can use classifications resulting from holistic and analytic coding schemes for scoring short CR assessments. Analytic rubrics have been shown to be easier to evaluate for the validity of ML-based scores with respect to LP levels. However, a possible drawback of using analytic rubrics for NGSS-aligned CR assessments is the potential for oversimplification of integrated ideas. Here we describe how to deconstruct a 3D holistic rubric for CR assessments probing the levels of an NGSS-aligned LP for high school physical sciences.MethodsWe deconstruct this rubric into seven analytic categories to preserve the 3D nature of the rubric and its result scores and provide subsequent combinations of categories to LP levels.ResultsThe resulting analytic rubric had excellent human- human inter-rater reliability across seven categories (Cohen’s kappa range 0.82–0.97). We found overall scores of responses using the combination of analytic rubric very closely agreed with scores assigned using a holistic rubric (99% agreement), suggesting the 3D natures of the rubric and scores were maintained. We found differing levels of agreement between ML models using analytic rubric scores and human-assigned scores. ML models for categories with a low number of positive cases displayed the lowest level of agreement.DiscussionWe discuss these differences in bin performance and discuss the implications and further applications for this rubric deconstruction approach.
<标题>: 为三维构建性回答评估开发与 NGSS 学习进展对齐的 AI 驱动评分量表 <AI Smry>: This paper deconstructs a 3D holistic rubric for CR assessments probing the levels of an NGSS-aligned LP for high school physical sciences into seven analytic categories to preserve the 3D nature of the rubric and its result scores and provide subsequent combinations of categories to LP levels. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z

A scoping review on the use of speech-to-text technology for adolescents with learning difficulties in secondary education - Matre, Marianne Engen; Cameron, David Lansing (Disability and Rehabilitation: Assistive Technology, 2022)
To identify and describe the aims, methodological approaches, and major findings of studies on the use of STT among secondary pupils (age 12–18) with learning difficulties published from January 2000 to April 2022. This scoping review includes empirical studies published in peer-reviewed journals and grey literature between January 2000 and April 2022. Searches were conducted in April 2022 in three databases: ERIC, PsycINFO and Scopus. In addition, related reviews were manually screened for relevant papers. Eight peer-revied studies and five publications of grey literature were found to meet the inclusion criteria; two studies employed experimental designs, four employed quasi-experimental designs and seven employed explorative designs. Six studies described STT as an assistive technology (a compensatory aid for poor writing performance); two assessed STT as an instructional technology to determine whether it improves overall writing and related skills (e.g., reading). Results suggest that STT may increase pupils’ abilities to produce texts with fewer errors, provide help with spelling and improve reading comprehension and word recognition. To date, there is a paucity of high-quality research on the use of STT among adolescents with LD. The scoping review shows that very little research has been conducted on the use of STT for adolescents with learning difficulties in secondary education. Findings from the studies identified five areas of interest: writing related skills, text assessment, writing processes, accuracy of the technology, and participants’ experiences. Findings indicate that writing performance among students with learning difficulties improves when using STT. Parents, teachers, and pupils report positive experiences with the technology, particularly for students with severe reading and writing difficulties.IMPLICATIONS FOR REHABILITATIONThere is a great need for more robust research on the use of speech-to-text technology (STT) in educational settings, especially on its effect on writing skillsStudies describe STT as either an assistive (a compensatory aid for poor writing performance) or instructional technology (aiming to improve learning in general). It is important that practitioners are aware of the different aims and possible consequences of introducing STT to learners with writing difficulties.STT provides both opportunities and challenges for writers with learning difficulties in secondary education. Findings indicate that writing performance among students with learning difficulties improves when using STT, yet inaccuracy of the technology was presented as one of the main challenges.Parents, teachers, and pupils report positive experiences with the technology, particularly for students with severe reading and writing difficulties. There is a great need for more robust research on the use of speech-to-text technology (STT) in educational settings, especially on its effect on writing skills Studies describe STT as either an assistive (a compensatory aid for poor writing performance) or instructional technology (aiming to improve learning in general). It is important that practitioners are aware of the different aims and possible consequences of introducing STT to learners with writing difficulties. STT provides both opportunities and challenges for writers with learning difficulties in secondary education. Findings indicate that writing performance among students with learning difficulties improves when using STT, yet inaccuracy of the technology was presented as one of the main challenges. Parents, teachers, and pupils report positive experiences with the technology, particularly for students with severe reading and writing difficulties.
_eprint: https://doi.org/10.1080/17483107.2022.2149865 <标题>: 关于在中学教育中对有学习困难的青少年使用语音转文字技术的范围综述 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: AI: Speech Recognition; Educational Area: Inclusion; Evidence: Student Learning Outcomes; Location: Europe; Research Design: Non-systematic Review; Research Method: Qualitative; Source: Open Source

Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model - Luccioni, Alexandra Sasha; Viguier, Sylvain; Ligozat, Anne-Laure (arXiv, 2022)
Progress in machine learning (ML) comes with a cost to the environment, given that training ML models requires significant computational resources, energy and materials. In the present article, we aim to quantify the carbon footprint of BLOOM, a 176-billion parameter language model, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes of~\carboneq~if we consider only the dynamic power consumption, and 50.5 tonnes if we account for all processes ranging from equipment manufacturing to energy-based operational consumption. We also study the energy requirements and carbon emissions of its deployment for inference via an API endpoint receiving user queries in real-time. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of ML models and future research directions that can contribute towards improving carbon emissions reporting.
arXiv:2211.02001 [cs] <标题>: 估算BLOOM（一种拥有1760亿参数的语言模型）的碳足迹 Read_Status: New Read_Status_Date: 2026-01-26T11:32:29.807Z
Keywords: /unread; Computer Science - Machine Learning; risks

Validity of Chatbot Use for Mental Health Assessment: Experimental Study - Schick, Anita; Feine, Jasper; Morana, Stefan; Maedche, Alexander; Reininghaus, Ulrich (JMIR mHealth and uHealth, 2022)
Mental disorders in adolescence and young adulthood are major public health concerns. Digital tools such as text-based conversational agents (ie, chatbots) are a promising technology for facilitating mental health assessment. However, the human-like interaction style of chatbots may induce potential biases, such as socially desirable responding (SDR), and may require further effort to complete assessments.
<标题>: 聊天机器人在心理健康评估中的使用有效性：实验研究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Best Practices for Constructed‐Response Scoring - McCaffrey, Daniel F.; Casabianca, Jodi M.; Ricker‐Pedley, Kathryn L.; Lawless, René R.; Wendler, Cathy (ETS Research Report Series, 2022)
This document describes a set of best practices for developing, implementing, and maintaining the critical process of scoring constructed‐response tasks. These practices address both the use of human raters and automated scoring systems as part of the scoring process and cover the scoring of written, spoken, performance, or multimodal responses. Best Practices for Constructed‐Response Scoring is designed not to act as an independent guide, but rather to be used in conjunction with other ETS publications, including the Guidelines for Constructed‐Response and Other Performance Assessments, ETS Standards for Quality and Fairness, ETS Guidelines for Fair Tests and Communications, and the ETS Guidelines for Fairness.
<标题>: 构建式回答评分的最佳实践 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

An Evaluation of Automatic Item Generation: A Case Study of Weak Theory Approach - Fu, Yanyan; Choe, Edison M.; Lim, Hwanggyu; Choi, Jaehwa (Educational Measurement: Issues and Practice, 2022)
This case study applied the weak theory of Automatic Item Generation (AIG) to generate isomorphic item instances (i.e., unique but psychometrically equivalent items) for a large‐scale assessment. Three representative instances were selected from each item template (i.e., model) and pilot‐tested. In addition, a new analytical framework, differential child item functioning (DCIF) analysis, based on the existing differential item functioning statistics, was applied to evaluate the psychometric equivalency of item instances within each template. The results showed that, out of 23 templates, nine successfully generated isomorphic instances, five required minor revisions to make them isomorphic, and the remaining templates required major modifications. The results and insights obtained from the AIG template development procedure may help item writers and psychometricians effectively develop and manage the templates that generate isomorphic instances.
<标题>: 自动题目生成评估：弱理论方法的案例研究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z

The Intertwined Histories of Artificial Intelligence and Education - Doroudi, Shayan (International Journal of Artificial Intelligence in Education, 2022)
In this paper, I argue that the fields of artificial intelligence (AI) and education have been deeply intertwined since the early days of AI. Specifically, I show that many of the early pioneers of AI were cognitive scientists who also made pioneering and impactful contributions to the field of education. These researchers saw AI as a tool for thinking about human learning and used their understanding of how people learn to further AI. Furthermore, I trace two distinct approaches to thinking about cognition and learning that pervade the early histories of AI and education. Despite their differences, researchers from both strands were united in their quest to simultaneously understand and improve human and machine cognition. Today, this perspective is neither prevalent in AI nor the learning sciences. I conclude with some thoughts on how the artificial intelligence in education and learning sciences communities might reinvigorate this lost perspective.
<标题>: 人工智能与教育的交织历史 <AI Smry>: It is argued that the fields of artificial intelligence (AI) and education have been deeply intertwined since the early days of AI and that many of the early pioneers of AI were cognitive scientists who also made pioneering and impactful contributions to the field of education. Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation - Chhun, Cyril; Colombo, Pierre; Clavel, Chloé; Suchanek, Fabian M. (arXiv, 2022)
Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no analysis of how well automatic criteria correlate with them. In this paper, we propose to re-evaluate ASG evaluation. We introduce a set of 6 orthogonal and comprehensive human criteria, carefully motivated by the social sciences literature. We also present HANNA, an annotated dataset of 1,056 stories produced by 10 different ASG systems. HANNA allows us to quantitatively evaluate the correlations of 72 automatic metrics with human criteria. Our analysis highlights the weaknesses of current metrics for ASG and allows us to formulate practical recommendations for ASG evaluation.
arXiv:2208.11646 [cs] <标题>: 关于人工标准与自动指标：故事生成评估的基准 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Computation and Language

Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation - Chhun, Cyril; Colombo, Pierre; Clavel, Chloé; Suchanek, Fabian M. (arXiv, 2022)
Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no analysis of how well automatic criteria correlate with them. In this paper, we propose to re-evaluate ASG evaluation. We introduce a set of 6 orthogonal and comprehensive human criteria, carefully motivated by the social sciences literature. We also present HANNA, an annotated dataset of 1,056 stories produced by 10 different ASG systems. HANNA allows us to quantitatively evaluate the correlations of 72 automatic metrics with human criteria. Our analysis highlights the weaknesses of current metrics for ASG and allows us to formulate practical recommendations for ASG evaluation.
arXiv:2208.11646 [cs] <标题>: 关于人工标准与自动指标：故事生成评估的基准 Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language

Semi‐automatic coding of open‐ended text responses in large‐scale assessments - Andersen, Nico; Zehner, Fabian; Goldhammer, Frank (Journal of Computer Assisted Learning, 2022)
In the context of large‐scale educational assessments, the effort required to code open‐ended text responses is considerably more expensive and time‐consuming than the evaluation of multiple‐choice responses because it requires trained personnel and long manual coding sessions.AimOur semi‐supervised coding method eco (exploring coding assistant) dynamically supports human raters by automatically coding a subset of the responses.MethodWe map normalized response texts into a semantic space and cluster response vectors based on their semantic similarity. Assuming that similar codes represent semantically similar responses, we propagate codes to responses in optimally homogeneous clusters. Cluster homogeneity is assessed by strategically querying informative responses and presenting them to a human rater. Following each manual coding, the method estimates the code distribution respecting a certainty interval and assumes a homogeneous distribution if certainty exceeds a predefined threshold. If a cluster is determined to certainly comprise homogeneous responses, all remaining responses are coded accordingly automatically. We evaluated the method in a simulation using different data sets.ResultsWith an average miscoding of about 3%, the method reduced the manual coding effort by an average of about 52%.ConclusionCombining the advantages of automatic and manual coding produces considerable coding accuracy and reduces the required manual effort.
<标题>: 半自动 编码 开放式 文本回答在 大规模 评估中的使用 <AI Smry>: A semi-supervised coding method that dynamically supports human raters by automatically coding a subset of the responses by combining the advantages of automatic and manual coding produces considerable coding accuracy and reduces the required manual effort. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z

Comparing Measures From Computer-Administered and Examiner-Administered Narrative Retells in Spanish: A Pilot Study - Heilmann, John; Finneran, Denise; Moyle, Maura (American Journal of Speech-Language Pathology, 2022)
Narrative language sample analysis (LSA) is a recommended best practice for the assessment of monolingual and bilingual children. With business-as-usual narrative LSA, examiners are actively involved in all aspects of the elicitation. Software advancements have shown multiple benefits of computer-administered language assessments, some of which may be beneficial for narrative assessments, particularly for bilingual children. The goal of this pilot study was to test the feasibility of computer-administered narrative retells in bilingual children.
<标题>: 比较西班牙语计算机管理与考官管理叙述复述的测量：一项初步研究 <AI Smry>: This study provides preliminary evidence on the use of a computer-administered narrative procedure and motivates further research on the method to confirm its validity and to document its effectiveness within clinical practice. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z

Using Chatbots as AI Conversational Partners in Language Learning - Belda-Medina, Jose; Calvo-Ferrer, José Ramón (Applied Sciences, 2022)
Recent advances in Artificial Intelligence (AI) and machine learning have paved the way for the increasing adoption of chatbots in language learning. Research published to date has mostly focused on chatbot accuracy and chatbot–human communication from students’ or in-service teachers’ perspectives. This study aims to examine the knowledge, level of satisfaction and perceptions concerning the integration of conversational AI in language learning among future educators. In this mixed method research based on convenience sampling, 176 undergraduates from two educational settings, Spain (n = 115) and Poland (n = 61), interacted autonomously with three conversational agents (Replika, Kuki, Wysa) over a four-week period. A learning module about Artificial Intelligence and language learning was specifically designed for this research, including an ad hoc model named the Chatbot–Human Interaction Satisfaction Model (CHISM), which was used by teacher candidates to evaluate different linguistic and technological features of the three conversational agents. Quantitative and qualitative data were gathered through a pre-post-survey based on the CHISM and the TAM2 (technology acceptance) models and a template analysis (TA), and analyzed through IBM SPSS 22 and QDA Miner software. The analysis yielded positive results regarding perceptions concerning the integration of conversational agents in language learning, particularly in relation to perceived ease of use (PeU) and attitudes (AT), but the scores for behavioral intention (BI) were more moderate. The findings also unveiled some gender-related differences regarding participants’ satisfaction with chatbot design and topics of interaction.
<标题>: 使用聊天机器人作为语言学习中的人工智能对话伙伴 <AI Smry>: Positive results are yielded regarding perceptions concerning the integration of conversational agents in language learning, particularly in relation to perceived ease of use (PeU) and attitudes (AT), but the scores for behavioral intention (BI) were more moderate. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Using HeuristicsMiner to Analyze Problem-Solving Processes: Exemplary Use Case of a Productive-Failure Study - Hartmann, Christian; Rummel, Nikol; Bannert, Maria (Journal of Learning Analytics, 2022)
This paper presents a fine-grained process analysis of 22 students in a classroom-based learning setting. The students engaged (and failed) in problem-solving attempts prior to instruction (i.e., the Productive-Failure approach). We used the HeuristicsMiner algorithm to analyze the data of a quasi-experimental study. The applied algorithm allowed us to investigate temporally structured think-aloud data, to outline productive and unproductive problem-solving strategies. Our analyses and findings demonstrated that HeuristicsMiner enables researchers to effectively mine problem-solving processes and sequences, even for smaller sample sizes, which cannot be done with traditional code-and-count strategies. The limitations of the algorithm, as well as further implications for educational research and practice, are also discussed.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

The Machines Take Over: A Comparison of Various Supervised Learning Approaches for Automated Scoring of Divergent Thinking Tasks - Buczak, Philip; Huang, He; Forthmann, Boris; Doebler, Philipp (The Journal of Creative Behavior, 2022)
Traditionally, researchers employ human raters for scoring responses to creative thinking tasks. Apart from the associated costs this approach entails two potential risks. First, human raters can be subjective in their scoring behavior (inter‐rater‐variance). Second, individual raters are prone to inconsistent scoring patterns (intra‐rater‐variance). In light of these issues, we present an approach for automated scoring of Divergent Thinking (DT) Tasks. We implemented a pipeline aiming to generate accurate rating predictions for DT responses using text mining and machine learning methods. Based on two existing data sets from two different laboratories, we constructed several prediction models incorporating features representing meta information of the response or features engineered from the response’s word embeddings that were obtained using pre‐trained GloVe and Word2Vec word vector spaces. Out of these features, word embeddings and features derived from them proved to be particularly effective. Overall, longer responses tended to achieve higher ratings as well as responses that were semantically distant from the stimulus object. In our comparison of three state‐of‐the‐art machine learning algorithms, Random Forest and XGBoost tended to slightly outperform the Support Vector Regression.
<标题>: 机器接管：多种监督学习方法在发散性思维任务自动评分中的比较 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Automatic grading for Arabic short answer questions using optimized deep learning model - Abdul Salam, Mustafa; El-Fatah, Mohamed Abd; Hassan, Naglaa Fathy (PLOS ONE, 2022)
Auto-grading of short answer questions is considered a challenging problem in the processing of natural language. It requires a system to comprehend the free text answers to automatically assign a grade for a student answer compared to one or more model answers. This paper suggests an optimized deep learning model for grading short-answer questions automatically by using various sizes of datasets collected in the Science subject for students in seventh grade in Egypt. The proposed system is a hybrid approach that optimizes a deep learning technique called LSTM (Long Short Term Memory) with a recent optimization algorithm called a Grey Wolf Optimizer (GWO). The GWO is employed to optimize the LSTM by selecting the best dropout and recurrent dropout rates of LSTM hyperparameters rather than manual choice. Using GWO makes the LSTM model more generalized and can also avoid the problem of overfitting in forecasting the students’ scores to improve the learning process and save instructors’ time and effort. The model’s performance is measured in terms of the Root Mean Squared Error (RMSE), the Pearson correlation coefficient, and R-Square. According to the simulation results, the hybrid GWO with the LSTM model ensured the best performance and outperformed the classical LSTM model and other compared models such that it had the highest Pearson correlation coefficient value, the lowest RMSE value, and the best R square value in all experiments, but higher training time than the traditional deep learning model.
<标题>: 使用优化的深度学习模型对阿拉伯语简答题进行自动评分 <AI Smry>: An optimized deep learning model for grading short-answer questions automatically by using various sizes of datasets collected in the Science subject for students in seventh grade in Egypt is suggested. Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level - Drori, Iddo; Zhang, Sarah; Shuttleworth, Reece; Tang, Leonard; Lu, Albert; Ke, Elizabeth; Liu, Kevin; Chen, Linda; Tran, Sunny; Cheng, Newman; Wang, Roman; Singh, Nikhil; et al. (Proceedings of the National Academy of Sciences, 2022)
We demonstrate that a neural network pretrained on text and fine-tuned on code solves mathematics course problems, explains solutions, and generates questions at a human level. We automatically synthesize programs using few-shot learning and OpenAI’s Codex transformer and execute them to solve course problems at 81% automatic accuracy. We curate a dataset of questions from Massachusetts Institute of Technology (MIT)’s largest mathematics courses (Single Variable and Multivariable Calculus, Differential Equations, Introduction to Probability and Statistics, Linear Algebra, and Mathematics for Computer Science) and Columbia University’s Computational Linear Algebra. We solve questions from a MATH dataset (on Prealgebra, Algebra, Counting and Probability, Intermediate Algebra, Number Theory, and Precalculus), the latest benchmark of advanced mathematics problems designed to assess mathematical reasoning. We randomly sample questions and generate solutions with multiple modalities, including numbers, equations, and plots. The latest GPT-3 language model pretrained on text automatically solves only 18.8% of these university questions using zero-shot learning and 30.8% using few-shot learning and the most recent chain of thought prompting. In contrast, program synthesis with few-shot learning using Codex fine-tuned on code generates programs that automatically solve 81% of these questions. Our approach improves the previous state-of-the-art automatic solution accuracy on the benchmark topics from 8.8 to 81.1%. We perform a survey to evaluate the quality and difficulty of generated questions. This work automatically solves university-level mathematics course questions at a human level and explains and generates university-level mathematics course questions at scale, a milestone for higher education.
<标题>: 神经网络通过程序综合和少量示例学习，以人类水平解决、解释和生成大学数学问题 <AI Smry>: This work solves university-level mathematics courses and improves upon state-of-the-art, increasing automatic accuracy on randomly sampled questions on a benchmark by order of magnitude. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

The interactive reading task: Transformer-based automatic item generation - Attali, Yigal; Runge, Andrew; LaFlair, Geoffrey T.; Yancey, Kevin; Goodwin, Sarah; Park, Yena; von Davier, Alina A. (Frontiers in Artificial Intelligence, 2022)
Automatic item generation (AIG) has the potential to greatly expand the number of items for educational assessments, while simultaneously allowing for a more construct-driven approach to item development. However, the traditional item modeling approach in AIG is limited in scope to content areas that are relatively easy to model (such as math problems), and depends on highly skilled content experts to create each model. In this paper we describe the interactive reading task, a transformer-based deep language modeling approach for creating reading comprehension assessments. This approach allows a fully automated process for the creation of source passages together with a wide range of comprehension questions about the passages. The format of the questions allows automatic scoring of responses with high fidelity (e.g., selected response questions). We present the results of a large-scale pilot of the interactive reading task, with hundreds of passages and thousands of questions. These passages were administered as part of the practice test of the Duolingo English Test. Human review of the materials and psychometric analyses of test taker results demonstrate the feasibility of this approach for automatic creation of complex educational assessments.
<标题>: 互动式阅读任务：基于Transformer的自动题目生成 <AI Smry>: The interactive reading task is described, a transformer-based deep language modeling approach for creating reading comprehension assessments that allows a fully automated process for the creation of source passages together with a wide range of comprehension questions about the passages. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z

The interactive reading task: Transformer-based automatic item generation - Attali, Yigal; Runge, Andrew; LaFlair, Geoffrey T.; Yancey, Kevin; Goodwin, Sarah; Park, Yena; Von Davier, Alina A. (Frontiers in Artificial Intelligence, 2022)
Automatic item generation (AIG) has the potential to greatly expand the number of items for educational assessments, while simultaneously allowing for a more construct-driven approach to item development. However, the traditional item modeling approach in AIG is limited in scope to content areas that are relatively easy to model (such as math problems), and depends on highly skilled content experts to create each model. In this paper we describe the interactive reading task, a transformer-based deep language modeling approach for creating reading comprehension assessments. This approach allows a fully automated process for the creation of source passages together with a wide range of comprehension questions about the passages. The format of the questions allows automatic scoring of responses with high fidelity (e.g., selected response questions). We present the results of a large-scale pilot of the interactive reading task, with hundreds of passages and thousands of questions. These passages were administered as part of the practice test of the Duolingo English Test. Human review of the materials and psychometric analyses of test taker results demonstrate the feasibility of this approach for automatic creation of complex educational assessments.
<标题>: 互动式阅读任务：基于Transformer的自动题目生成 <AI Smry>: The interactive reading task is described, a transformer-based deep language modeling approach for creating reading comprehension assessments that allows a fully automated process for the creation of source passages together with a wide range of comprehension questions about the passages. Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

On the Opportunities and Risks of Foundation Models - Bommasani, Rishi; Hudson, Drew A.; Adeli, Ehsan; Altman, Russ; Arora, Simran; von Arx, Sydney; Bernstein, Michael S.; Bohg, Jeannette; Bosselut, Antoine; Brunskill, Emma; Brynjolfsson, Erik; Buch, Shyamal; et al. (arXiv, 2022)
AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities (e.g., language, vision, robotics, reasoning, human interaction) and technical principles(e.g., model architectures, training procedures, data, systems, security, evaluation, theory) to their applications (e.g., law, healthcare, education) and societal impact (e.g., inequity, misuse, economic and environmental impact, legal and ethical considerations). Though foundation models are based on standard deep learning and transfer learning, their scale results in new emergent capabilities,and their effectiveness across so many tasks incentivizes homogenization. Homogenization provides powerful leverage but demands caution, as the defects of the foundation model are inherited by all the adapted models downstream. Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration commensurate with their fundamentally sociotechnical nature.
arXiv:2108.07258 [cs] <标题>: 关于基础模型的机遇与风险 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computers and Society; Computer Science - Machine Learning

InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation - Colombo, Pierre Jean A.; Clavel, Chloé; Piantanida, Pablo (Proceedings of the AAAI Conference on Artificial Intelligence, 2022)
Assessing the quality of natural language generation (NLG) systems through human annotation is very expensive. Additionally, human annotation campaigns are time-consuming and include non-reusable human labour. In practice, researchers rely on automatic metrics as a proxy of quality. In the last decade, many string-based metrics (e.g., BLEU or ROUGE) have been introduced. However, such metrics usually rely on exact matches and thus, do not robustly handle synonyms. In this paper, we introduce InfoLM a family of untrained metrics that can be viewed as a string-based metric that addresses the aforementioned flaws thanks to a pre-trained masked language model. This family of metrics also makes use of information measures allowing the possibility to adapt InfoLM to different evaluation criteria. Using direct assessment, we demonstrate that InfoLM achieves statistically significant improvement and two figure correlation gains in many configurations compared to existing metrics on both summarization and data2text generation tasks.
<AI Smry>: This paper introduces InfoLM a family of untrained metrics that can be viewed as a string-based metric that addresses the aforementioned flaws thanks to a pre-trained masked language model and makes use of information measures allowing the possibility to adapt InfoLM to different evaluation criteria. Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Fully Adaptive Framework: Neural Computerized Adaptive Testing for Online Education - Zhuang, Yan; Liu, Qi; Huang, Zhenya; Li, Zhi; Shen, Shuanghong; Ma, Haiping (Proceedings of the AAAI Conference on Artificial Intelligence, 2022)
Computerized Adaptive Testing (CAT) refers to an efficient and personalized test mode in online education, aiming to accurately measure student proficiency level on the required subject/domain. The key component of CAT is the "adaptive" question selection algorithm, which automatically selects the best suited question for student based on his/her current estimated proficiency, reducing test length. Existing algorithms rely on some manually designed and pre-fixed informativeness/uncertainty metrics of question for selections, which is labor-intensive and not sufficient for capturing complex relations between students and questions. In this paper, we propose a fully adaptive framework named Neural Computerized Adaptive Testing (NCAT), which formally redefines CAT as a reinforcement learning problem and directly learns selection algorithm from real-world data. Specifically, a bilevel optimization is defined and simplified under CAT's application scenarios to make the algorithm learnable. Furthermore, to address the CAT task effectively, we tackle it as an equivalent reinforcement learning problem and propose an attentive neural policy to model complex non-linear interactions. Extensive experiments on real-world datasets demonstrate the effectiveness and robustness of NCAT compared with several state-of-the-art methods.
<标题>: 完全自适应框架：在线教育的神经计算机自适应测试 <AI Smry>: A fully adaptive framework named Neural Computerized Adaptive Testing (NCAT) is proposed, which formally redefines CAT as a reinforcement learning problem and directly learns selection algorithm from real-world data. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z

The use of semantic similarity tools in automated content scoring of fact-based essays written by EFL learners - Wang, Qiao (Education and Information Technologies, 2022)
This study searched for open-source semantic similarity tools and evaluated their effectiveness in automated content scoring of fact-based essays written by English-as-a-Foreign-Language (EFL) learners. Fifty writing samples under a fact-based writing task from an academic English course in a Japanese university were collected and a gold standard was produced by a native expert. A shortlist of carefully selected tools, including InferSent, spaCy, DKPro, ADW, SEMILAR and Latent Semantic Analysis, generated semantic similarity scores between student writing samples and the expert sample. Three teachers who were lecturers of the course manually graded the student samples on content. To ensure validity of human grades, samples with discrepant agreement were excluded and an inter-rater reliability test was conducted on remaining samples with quadratic weighted kappa. After the grades of the remaining samples were proven valid, a Pearson correlation analysis between semantic similarity scores and human grades was conducted and results showed that InferSent was the most effective tool in predicting the human grades. The study further pointed to the limitations of the six tools and suggested three alternatives to traditional methods in turning semantic similarity scores into reporting grades on content.
<标题>: 在自动评分以英语为外语的学习者所写事实性文章时使用语义相似性工具 <AI Smry>: This study searched for open-source semantic similarity tools and evaluated their effectiveness in automated content scoring of fact-based essays written by English-as-a-Foreign-Language (EFL) learners and found InferSent was the most effective tool in predicting the human grades. Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.518Z

Validity Arguments Meet Artificial Intelligence in Innovative Educational Assessment: A Discussion and Look Forward - Dorsey, David W.; Michaels, Hillary R. (Journal of Educational Measurement, 2022)
In this concluding article of the special issue, we provide an overall discussion and point to future emerging trends in AI that might shape our approach to validity and building validity arguments.
<标题>: 有效性论证在创新教育评估中遇到人工智能：讨论与展望 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance - Raji, Inioluwa Deborah; Xu, Peggy; Honigsberg, Colleen; Ho, Daniel E. (arXiv, 2022)
Much attention has focused on algorithmic audits and impact assessments to hold developers and users of algorithmic systems accountable. But existing algorithmic accountability policy approaches have neglected the lessons from non-algorithmic domains: notably, the importance of interventions that allow for the effective participation of third parties. Our paper synthesizes lessons from other fields on how to craft effective systems of external oversight for algorithmic deployments. First, we discuss the challenges of third party oversight in the current AI landscape. Second, we survey audit systems across domains - e.g., financial, environmental, and health regulation - and show that the institutional design of such audits are far from monolithic. Finally, we survey the evidence base around these design components and spell out the implications for algorithmic auditing. We conclude that the turn toward audits alone is unlikely to achieve actual algorithmic accountability, and sustained focus on institutional design will be required for meaningful third party involvement.
arXiv:2206.04737 [cs] <标题>: 外部监督：为人工智能治理设计第三方审计生态系统 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.951Z
Keywords: /unread; Computer Science - Computers and Society

Retrieval-Augmented Reinforcement Learning - Goyal, Anirudh; Friesen, Abram L.; Banino, Andrea; Weber, Theophane; Ke, Nan Rosemary; Badia, Adria Puigdomenech; Guez, Arthur; Mirza, Mehdi; Humphreys, Peter C.; Konyushkova, Ksenia; Sifre, Laurent; Valko, Michal; et al. (arXiv, 2022)
Most deep reinforcement learning (RL) algorithms distill experience into parametric behavior policies or value functions via gradient updates. While effective, this approach has several disadvantages: (1) it is computationally expensive, (2) it can take many updates to integrate experiences into the parametric model, (3) experiences that are not fully integrated do not appropriately influence the agent's behavior, and (4) behavior is limited by the capacity of the model. In this paper we explore an alternative paradigm in which we train a network to map a dataset of past experiences to optimal behavior. Specifically, we augment an RL agent with a retrieval process (parameterized as a neural network) that has direct access to a dataset of experiences. This dataset can come from the agent's past experiences, expert demonstrations, or any other relevant source. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context, to help the agent achieve its goal faster and more efficiently. he proposed method facilitates learning agents that at test-time can condition their behavior on the entire dataset and not only the current state, or current trajectory. We integrate our method into two different RL agents: an offline DQN agent and an online R2D2 agent. In offline multi-task problems, we show that the retrieval-augmented DQN agent avoids task interference and learns faster than the baseline DQN agent. On Atari, we show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores. We run extensive ablations to measure the contributions of the components of our proposed method.
arXiv:2202.08417 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:45.601Z
Keywords: /unread; Computer Science - Machine Learning

The U.S. can improve its AI governance strategy by addressing online biases - Nicol Turner Lee; Samantha Lai (Brookings, 2022)
Stakeholders in artificial intelligence must trace back to the roots of the problems, which lie in the lack of diversity in design teams and data that continues to carry on trauma and discrimination of the past, Nicol Turner Lee and Samantha Lai write.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Anchoring Validity Evidence for Automated Essay Scoring - Shermis, Mark D. (Journal of Educational Measurement, 2022)
One of the challenges of discussing validity arguments for machine scoring of essays centers on the absence of a commonly held definition and theory of good writing. At best, the algorithms attempt to measure select attributes of writing and calibrate them against human ratings with the goal of accurate prediction of scores for new essays. Sometimes these attributes are based on the fundamentals of writing (e.g., fluency), but quite often they are based on locally developed rubrics that may be confounded with specific content coverage expectations. This lack of transparency makes it difficult to provide systematic evidence that machine scoring is assessing writing, but slices or correlates of writing performance.
<标题>: 自动作文评分的锚定有效性证据 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Now Is the Time to Build a National Data Ecosystem for Materials Science and Chemistry Research Data - Campo, Eva M.; Shankar, Sadasivan; Szalay, Alexander S.; Hanisch, Robert J. (ACS Omega, 2022)
<标题>: 现在是建立国家材料科学和化学研究数据生态系统的时机 Read_Status: New Read_Status_Date: 2026-01-26T11:33:01.187Z

An error-analysis study from an EFL writing context: Human and Automated Essay Scoring Approaches - Almusharraf, Norah; Alotaibi, Hind (Technology, Knowledge and Learning, 2022)
<标题>: 来自EFL写作环境的错误分析研究：人工与自动化作文评分方法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.151Z

A linear model approach to bias detection - Justice, Derek (2022)
<标题>: 用于偏差检测的线性模型方法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: DONE

A Theoretical Assessment Ecosystem for a Digital-First Assessment—The Duolingo English Test - Burstein, Jill; LaFlair, Geoffrey T.; Kunnan, Antony John; von Davier, Alina A. (2022)
The Duolingo English Test is a groundbreaking, digital-first, computer-adaptive English language proficiency test intended to support stakeholder admissions decisions at English-medium institutions. The test measures four key constructs for university English language proficiency: Speaking, Writing, Reading, and Listening (SWRL), and is aligned with the Common European Framework of Reference for Languages (CEFR) proficiency levels and descriptors. As a digital-first assessment, the test leverages “human-in-the-loop AI” from end to end for test security, automated item generation, and scoring of test-taker responses. This paper presents a novel theoretical assessment ecosystem for the Duolingo English Test. It is a theoretical representation of language assessment design, measurement, and test security processes, and the test-taker experience factors that contribute to the test validity argument and test impact. The test validity argument is constructed with a digitally-informed chain of inferences that addresses digital affordances applied to the test. The ecosystem is composed of an integrated set of complex frameworks: (1) the Language Assessment Design Framework, (2) the Expanded Evidence-Centered Design Framework, (3) the Computational Psychometrics Framework, and (4) the Test Security Framework. Test-taker experience (TTX) is a test priority throughout the test-taking pipeline, such as low cost, anytime/anywhere, and shorter testing time. The test’s expected impact is aligned with Duolingo’s social mission to lower barriers to education access and offer a secure and delightful test experience, and to provide a valid, fair and reliable test score. The ecosystem leverages principles from assessment theory, computational psychometrics, design, data science, language assessment theory, NLP/AI and machine learning, and test security.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Generating and Evaluating Collective Concept Maps - Brennan, Riordan; Perouli, Debbie (LAK22: 12th International Learning Analytics and Knowledge Conference, 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

PeerBERT: Automated Characterization of Peer Review Comments across Courses - Dood, Amber; Winograd, Blair; Finkenstaedt-Quinn, Solaire; Gere, Anne; Shultz, Ginger (LAK22: 12th International Learning Analytics and Knowledge Conference, 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Hybrid Human-AI Curriculum Development for Personalised Informal Learning Environments - Tavakoli, Mohammadreza; Faraji, Abdolali; Molavi, Mohammadreza; T. Mol, Stefan; Kismihók, Gábor (LAK22: 12th International Learning Analytics and Knowledge Conference, 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Training language models to follow instructions with human feedback - Ouyang, Long; Wu, Jeff; Jiang, Xu; Almeida, Diogo; Wainwright, Carroll L.; Mishkin, Pamela; Zhang, Chong; Agarwal, Sandhini; Slama, Katarina; Ray, Alex; Schulman, John; Hilton, Jacob; et al. (arXiv, 2022)
Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.
arXiv:2203.02155 [cs] <标题>: 通过人类反馈训练语言模型以遵循指令 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Short-answer scoring with ensembles of pretrained language models - Ormerod, Christopher (arXiv, 2022)
We investigate the effectiveness of ensembles of pretrained transformer-based language models on short answer questions using the Kaggle Automated Short Answer Scoring dataset. We fine-tune a collection of popular small, base, and large pretrained transformer-based language models, and train one feature-base model on the dataset with the aim of testing ensembles of these models. We used an early stopping mechanism and hyperparameter optimization in training. We observe that generally that the larger models perform slightly better, however, they still fall short of state-of-the-art results one their own. Once we consider ensembles of models, there are ensembles of a number of large networks that do produce state-of-the-art results, however, these ensembles are too large to realistically be put in a production environment.
arXiv:2202.11558 [cs] <标题>: 使用预训练语言模型集群进行简答题评分 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Computation and Language; Computer Science - Machine Learning

Leveraging Natural Language Processing for Quality Assurance of a Situational Judgement Test - Bulut, Okan; MacIntosh, Alexander; Walsh, Cole (Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium, 2022)
Series Title: Lecture Notes in Computer Science <标题>: 利用自然语言处理进行情境判断测试的质量保证 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

The Promises and Challenges of Artificial Intelligence for Teachers: a Systematic Review of Research - Celik, Ismail; Dindar, Muhterem; Muukkonen, Hanni; Järvelä, Sanna (TechTrends, 2022)
Abstract This study provides an overview of research on teachers’ use of artificial intelligence (AI) applications and machine learning methods to analyze teachers’ data. Our analysis showed that AI offers teachers several opportunities for improved planning (e.g., by defining students’ needs and familiarizing teachers with such needs), implementation (e.g., through immediate feedback and teacher intervention), and assessment (e.g., through automated essay scoring) of their teaching. We also found that teachers have various roles in the development of AI technology. These roles include acting as models for training AI algorithms and participating in AI development by checking the accuracy of AI automated assessment systems. Our findings further underlined several challenges in AI implementation in teaching practice, which provide guidelines for developing the field.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Validity Arguments Meet Artificial Intelligence in Innovative Educational Assessment - Dorsey, David W.; Michaels, Hillary R. (Journal of Educational Measurement, 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.078Z

Validity Arguments Meet Artificial Intelligence in Innovative Educational Assessment - Dorsey, David W.; Michaels, Hillary R. (Journal of Educational Measurement, 2022)
Abstract We have dramatically advanced our ability to create rich, complex, and effective assessments across a range of uses through technology advancement. Artificial Intelligence (AI) enabled assessments represent one such area of advancement—one that has captured our collective interest and imagination. Scientists and practitioners within the domains of organizational and workforce assessment have increasingly used AI in assessment, and its use is now becoming more common in education. While these types of solutions offer their users the promise of efficiency, effectiveness, and a “wow factor,” users need to maintain high standards for validity and fairness in high stakes settings. Due to the complexity of some AI methods and tools, this requirement for adherence to standards may challenge our traditional approaches to building validity and fairness arguments. In this edition, we review what these challenges may look like as validity arguments meet AI in educational assessment domains. We specifically explore how AI impacts Evidence‐Centered Design (ECD) and development from assessment concept and coding to scoring and reporting. We also present information on ways to ensure that bias is not built into these systems. Lastly, we discuss future horizons, many that are almost here, for maximizing what AI offers while minimizing negative effects on test takers and programs.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Automated Scoring for Reading Comprehension via In-context BERT Tuning - Fernandez, Nigel; Ghosh, Aritra; Liu, Naiming; Wang, Zichao; Choffin, Benoît; Baraniuk, Richard; Lan, Andrew (arXiv, 2022)
Automated scoring of open-ended student responses has the potential to significantly reduce human grader effort. Recent advances in automated scoring often leverage textual representations based on pre-trained language models such as BERT and GPT as input to scoring models. Most existing approaches train a separate model for each item/question, which is suitable for scenarios such as essay scoring where items can be quite different from one another. However, these approaches have two limitations: 1) they fail to leverage item linkage for scenarios such as reading comprehension where multiple items may share a reading passage; 2) they are not scalable since storing one model per item becomes difficult when models have a large number of parameters. In this paper, we report our (grand prize-winning) solution to the National Assessment of Education Progress (NAEP) automated scoring challenge for reading comprehension. Our approach, in-context BERT fine-tuning, produces a single shared scoring model for all items with a carefully-designed input structure to provide contextual information on each item. We demonstrate the effectiveness of our approach via local evaluations using the training dataset provided by the challenge. We also discuss the biases, common error types, and limitations of our approach.
arXiv:2205.09864 [cs] <标题>: 通过上下文BERT调优进行阅读理解的自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:32:31.459Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computers and Society; Computer Science - Machine Learning

Validity Arguments for AI‐Based Automated Scores: Essay Scoring as an Illustration - Ferrara, Steve; Qunbar, Saed (Journal of Educational Measurement, 2022)
Abstract In this article, we argue that automated scoring engines should be transparent and construct relevant—that is, as much as is currently feasible. Many current automated scoring engines cannot achieve high degrees of scoring accuracy without allowing in some features that may not be easily explained and understood and may not be obviously and directly relevant to the target assessment construct. We address the current limitations on evidence and validity arguments for scores from automated scoring engines from the points of view of the Standards for Educational and Psychological Testing (i.e., construct relevance, construct representation, and fairness) and emerging principles in Artificial Intelligence (e.g., explainable AI, an examinee's right to explanations, and principled AI). We illustrate these concepts and arguments for automated essay scores.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

A Multilevel Multinomial Logit Approach to Bias Detection - He, Yong; Jing, Shumin; Lu, Y (2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z
Keywords: ⛔ No DOI found

Toward Argument‐Based Fairness with an Application to AI‐Enhanced Educational Assessments - Huggins‐Manley, A. Corinne; Booth, Brandon M.; D'Mello, Sidney K. (Journal of Educational Measurement, 2022)
Abstract The field of educational measurement places validity and fairness as central concepts of assessment quality. Prior research has proposed embedding fairness arguments within argument‐based validity processes, particularly when fairness is conceived as comparability in assessment properties across groups. However, we argue that a more flexible approach to fairness arguments that occurs outside of and complementary to validity arguments is required to address many of the views on fairness that a set of assessment stakeholders may hold. Accordingly, we focus this manuscript on two contributions: (a) introducing the argument‐based fairness approach to complement argument‐based validity for both traditional and artificial intelligence (AI)‐enhanced assessments and (b) applying it in an illustrative AI assessment of perceived hireability in automated video interviews used to prescreen job candidates. We conclude with recommendations for further advancing argument‐based fairness approaches.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Psychometric Methods to Evaluate Measurement and Algorithmic Bias in Automated Scoring - Johnson, Matthew S.; Liu, Xiang; McCaffrey, Daniel F. (Journal of Educational Measurement, 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Examining Bias in Automated Scoring of Reading Comprehension Items - Lottridge, Susan; Young, Mackenzie (2022)
The use of automated scoring (AS) of constructed responses has become increasingly common in k - 12 formative, interim, and summative assessment programs. AS has been shown to perform well in essay writing, reading comprehension, and mathematics. However, less is known about how automated scoring engines perform for key subgroups such as gender, race/ethnicity, English proficiency status, disability status, and economic status. Bias evaluations have focused primarily on mean score differences that may occur between the engine scores and the human scores within a subgroup and most published work has been on international examinees using ETS' e-rater system. Gregg et al. (2021) recently examined bias for limited English proficient examinees in United States k-12 examinees on writing and mathematics items, examining engine performance for the presence of bias and examining what elements of the engine processing steps may contribute to bias. This study builds upon the Gregg study by extending the examination to a broader set of subgroups, including gender (Male/Female), economic status (Title I/non-Title I), special education status (SPED/non-SPED) and race/ethnicity (Asian, Black, Hispanic, White). The study examined the performance of the engine on 24 reading comprehension items in grades 3-8 and 11. Bia swas examined on subgroups using the full set of data and when matched using propensity score matching on ability and subgroup covariates. In addition to the usual agreement metrics of quadratic weighted kappa (QWK), exact agreement, and standardized mean difference (SMD), the study also used agreement matrices to examine results for possible bias in patterns of rubric score applications both for human raters and for the automated scoring engine. These matrices show the level of agreement between the second human rater against the first human rater, which can be compared to the level of agreement between the engine and the first human rater. The value of such analyses is that they help to illustrate where in the rubric scale any errors or bias may be appearing and they also help to better understand the pattern of score distributions across the scale, The results of the study indicated that, across methods, the engine showed little evidence of bias for most subgroups on the full sample but showed more evidence of bias when groups were matched. Additionally, the engine showed some bias, as indicated by lower agreements when conditioned on the human score, compared to other subgroups or a matched counterpart at some, but not all, points in the rubric scale. Future work will examine matched responses where the engine does appear to behave differently at locations of the rubric scale (e.g., score points 2 for Black, White, non-Title I and SPED examinees). Such a review may help to identify whether there are patterns in responses that may be causing issues in how the engine is modeling the responses. This review may also uncover any trends in human scoring
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z
Keywords: ⛔ No DOI found

Mapping Between Hidden States and Features to Validate Automated Essay Scoring Using DeBERTa Models - Ormerod, Christopher (Psychological Test and Assessment Modeling, 2022)
We introduce a regression-based framework to explore the dependence that global features have on score predictions from pretrained transformer-based language models used for Automated Essay Scoring (AES). We demonstrate that neural networks use approximations of rubric-relevant global features to determine a score prediction. By considering linear models on the hidden states, we can approximate global features and measure their importance to score predictions. This study uses DeBERTa models trained on overall scores and trait-level scores to demonstrate this framework with a specific focus on convention errors, which are errors in the use of language, encompassing spelling, grammar, and punctuation errors. This introduces a new form of explainability and provides evidence of validity for Language Model based AES.
<标题>: 使用 DeBERTa 模型验证自动作文评分的隐藏状态与特征映射 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z

Artificial Intelligence in Education: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part I - Rodrigo, Maria Mercedes; Matsuda, Noburu; Cristea, Alexandra I.; Dimitrova, Vania (Springer International Publishing, 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems - Sato, Shiki; Kishinami, Yosuke; Sugiyama, Hiroaki; Akama, Reina; Tokuhisa, Ryoko; Suzuki, Jun (Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Student Research Workshop, 2022)
Automation of dialogue system evaluation is a driving force for the efficient development of dialogue systems. This paper introduces the bipartite-play method, a dialogue collection method for automating dialogue system evaluation. It addresses the limitations of existing dialogue collection methods: (i) inability to compare with systems that are not publicly available, and (ii) vulnerability to cheating by intentionally selecting systems to be compared. Experimental results show that the automatic evaluation using the bipartite-play method mitigates these two drawbacks and correlates as strongly with human subjectivity as existing methods.
<标题>: 面向对话系统实用自动评估的双向对话集 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread

Assessment in the age of artificial intelligence - Swiecki, Zachari; Khosravi, Hassan; Chen, Guanliang; Martinez-Maldonado, Roberto; Lodge, Jason M.; Milligan, Sandra; Selwyn, Neil; Gašević, Dragan (Computers and Education: Artificial Intelligence, 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Automatic scoring of short answers using justification cues estimated by BERT - Takano, Shunya; Ichikawa, Osamu (Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 2022)
<标题>: 使用 BERT 估计的理由提示进行简答自动评分 <AI Smry>: The proposed method reduces the training data from the 800 data required in the past to about 400 data, and still achieves scoring accuracy comparable to that of humans on the RIKEN dataset. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z

Automated scoring of the autobiographical interview with natural language processing - van Genugten, Ruben; Schacter, Daniel L (PsyArXiv, 2022)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Toward Efficient Automated Feature Engineering - Wang, Kafeng; Wang, Pengyang; xu, Chengzhong (arXiv, 2022)
Automated Feature Engineering (AFE) refers to automatically generate and select optimal feature sets for downstream tasks, which has achieved great success in real-world applications. Current AFE methods mainly focus on improving the effectiveness of the produced features, but ignoring the low-efficiency issue for large-scale deployment. Therefore, in this work, we propose a generic framework to improve the efficiency of AFE. Specifically, we construct the AFE pipeline based on reinforcement learning setting, where each feature is assigned an agent to perform feature transformation \com{and} selection, and the evaluation score of the produced features in downstream tasks serve as the reward to update the policy. We improve the efficiency of AFE in two perspectives. On the one hand, we develop a Feature Pre-Evaluation (FPE) Model to reduce the sample size and feature size that are two main factors on undermining the efficiency of feature evaluation. On the other hand, we devise a two-stage policy training strategy by running FPE on the pre-evaluation task as the initialization of the policy to avoid training policy from scratch. We conduct comprehensive experiments on 36 datasets in terms of both classification and regression tasks. The results show $2.9\%$ higher performance in average and 2x higher computational efficiency comparing to state-of-the-art AFE methods.
Version Number: 1 <标题>: 迈向高效的自动化特征工程 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: FOS: Computer and information sciences; Machine Learning (cs.LG)

Towards Human-Like Educational Question Generation with Large Language Models - Wang, Zichao; Valdez, Jakob; Basu Mallick, Debshila; Baraniuk, Richard G. (Artificial Intelligence in Education, 2022)
Series Title: Lecture Notes in Computer Science Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Towards Human-Like Educational Question Generation with Large Language Models - Wang, Zichao; Valdez, Jakob; Basu Mallick, Debshila; Baraniuk, Richard G. (Artificial Intelligence in Education, 2022)
Series Title: Lecture Notes in Computer Science <标题>: 利用大型语言模型实现类人教育题目生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Towards Human-Like Educational Question Generation with Large Language Models - Wang, Zichao; Valdez, Jakob; Basu Mallick, Debshila; Baraniuk, Richard G. (Artificial Intelligence in Education, 2022)
Series Title: Lecture Notes in Computer Science <标题>: 利用大型语言模型实现类人教育题目生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.499Z

Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension - Xu, Ying; Wang, Dakuo; Yu, Mo; Ritchie, Daniel; Yao, Bingsheng; Wu, Tongshuang; Zhang, Zheng; Li, Toby Jia-Jun; Bradford, Nora; Sun, Branda; Hoang, Tran Bao; Sang, Yisi; et al. (arXiv, 2022)
Question answering (QA) is a fundamental means to facilitate assessment and training of narrative comprehension skills for both machines and young children, yet there is scarcity of high-quality QA datasets carefully designed to serve this purpose. In particular, existing datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. Drawing on the reading education research, we introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. Generated by educational experts based on an evidence-based theoretical framework, FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. Our dataset is valuable in two folds: First, we ran existing QA models on our dataset and confirmed that this annotation helps assess models' fine-grained learning skills. Second, the dataset supports question generation (QG) task in the education domain. Through benchmarking with QG models, we show that the QG model trained on FairytaleQA is capable of asking high-quality and more diverse questions.
Version Number: 1 <标题>: 奇妙的问题及其来源：FairytaleQA —— 一个用于叙事理解的真实数据集 <AI Smry>: Through benchmarking with QG models, it is shown that the QG model trained on FairytaleQA is capable of asking high-quality and more diverse questions, and the dataset supports question generation (QG) task in the education domain. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z
Keywords: Computation and Language (cs.CL); FOS: Computer and information sciences

Towards a Unified Multi-Dimensional Evaluator for Text Generation - Zhong, Ming; Liu, Yang; Yin, Da; Mao, Yuning; Jiao, Yizhu; Liu, Pengfei; Zhu, Chenguang; Ji, Heng; Han, Jiawei (arXiv, 2022)
Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Language Generation (NLG), i.e., evaluating the generated text from multiple explainable dimensions, such as coherence and fluency. However, automatic evaluation in NLG is still dominated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional evaluator UniEval for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions. Furthermore, thanks to the unified Boolean QA format, we are able to introduce an intermediate learning phase that enables UniEval to incorporate external knowledge from multiple related tasks and gain further improvement. Experiments on three typical NLG tasks show that UniEval correlates substantially better with human judgments than existing metrics. Specifically, compared to the top-performing unified evaluators, UniEval achieves a 23% higher correlation on text summarization, and over 43% on dialogue response generation. Also, UniEval demonstrates a strong zero-shot learning ability for unseen evaluation dimensions and tasks. Source code, data and all pre-trained evaluators are available on our GitHub repository (https://github.com/maszhongming/UniEval).
arXiv:2210.07197 [cs] <标题>: 迈向统一的多维文本生成评估器 Read_Status: New Read_Status_Date: 2026-01-26T11:32:27.022Z
Keywords: /unread; Computer Science - Computation and Language

Towards a Unified Multi-Dimensional Evaluator for Text Generation - Zhong, Ming; Liu, Yang; Yin, Da; Mao, Yuning; Jiao, Yizhu; Liu, Pengfei; Zhu, Chenguang; Ji, Heng; Han, Jiawei (arXiv, 2022)
Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Language Generation (NLG), i.e., evaluating the generated text from multiple explainable dimensions, such as coherence and fluency. However, automatic evaluation in NLG is still dominated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional evaluator UniEval for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions. Furthermore, thanks to the unified Boolean QA format, we are able to introduce an intermediate learning phase that enables UniEval to incorporate external knowledge from multiple related tasks and gain further improvement. Experiments on three typical NLG tasks show that UniEval correlates substantially better with human judgments than existing metrics. Specifically, compared to the top-performing unified evaluators, UniEval achieves a 23% higher correlation on text summarization, and over 43% on dialogue response generation. Also, UniEval demonstrates a strong zero-shot learning ability for unseen evaluation dimensions and tasks. Source code, data and all pre-trained evaluators are available on our GitHub repository (https://github.com/maszhongming/UniEval).
arXiv:2210.07197 [cs] <标题>: 迈向统一的多维文本生成评估器 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: Computer Science - Computation and Language

Learning Automated Essay Scoring Models Using Item-Response-Theory-Based Scores to Decrease Effects of Rater Biases - Uto, Masaki; Okano, Masashi (IEEE Transactions on Learning Technologies, 2021)
<标题>: 使用基于项目反应理论分数的自动作文评分模型学习以减少评分者偏差的影响 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

A Method to Construct Guidelines for Spanish Comments Annotation for Sentiment Analysis - Urpay-Camasi, John; Garcia-Calderon, Jorge; Shiguihara, Pedro (2021 IEEE Sciences and Humanities International Research Conference (SHIRCON), 2021)
<标题>: 一种用于西班牙语评论情感分析标注指南的构建方法 <AI Smry>: This work proposes a method to construct guidelines to annotators reach a consensus in the entire annotation process of spanish comments from social networks, where the concordance analysis from the annotators was 84% and the highest F1-Score with Multilayer Perceptron was 78.63%. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Evaluation Metrics for Health Chatbots: A Delphi Study - Denecke, Kerstin; Abd-Alrazaq, Alaa; Househ, Mowafa; Warren, Jim (Methods of Information in Medicine, 2021)
Abstract Background In recent years, an increasing number of health chatbots has been published in app stores and described in research literature. Given the sensitive data they are processing and the care settings for which they are developed, evaluation is essential to avoid harm to users. However, evaluations of those systems are reported inconsistently and without using a standardized set of evaluation metrics. Missing standards in health chatbot evaluation prevent comparisons of systems, and this may hamper acceptability since their reliability is unclear. Objectives The objective of this paper is to make an important step toward developing a health-specific chatbot evaluation framework by finding consensus on relevant metrics. Methods We used an adapted Delphi study design to verify and select potential metrics that we retrieved initially from a scoping review. We invited researchers, health professionals, and health informaticians to score each metric for inclusion in the final evaluation framework, over three survey rounds. We distinguished metrics scored relevant with high, moderate, and low consensus. The initial set of metrics comprised 26 metrics (categorized as global metrics, metrics related to response generation, response understanding and aesthetics). Results Twenty-eight experts joined the first round and 22 (75%) persisted to the third round. Twenty-four metrics achieved high consensus and three metrics achieved moderate consensus. The core set for our framework comprises mainly global metrics (e.g., ease of use, security content accuracy), metrics related to response generation (e.g., appropriateness of responses), and related to response understanding. Metrics on aesthetics (font type and size, color) are less well agreed upon—only moderate or low consensus was achieved for those metrics. Conclusion The results indicate that experts largely agree on metrics and that the consensus set is broad. This implies that health chatbot evaluation must be multifaceted to ensure acceptability.
<AI Smry>: The results indicate that experts largely agree on metrics and that the consensus set is broad, which implies that health chatbot evaluation must be multifaceted to ensure acceptability. Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: /unread

FairER: Entity Resolution With Fairness Constraints - Efthymiou, Vasilis; Stefanidis, Kostas; Pitoura, Evaggelia; Christophides, Vassilis (Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021)
<AI Smry>: FairER, a greedy algorithm for solving the data bias in entity resolution (ER) tasks aiming to discover and unify records/descriptions from different data sources that refer to the same real-world entity is introduced. Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Integrating Deep Learning into An Automated Feedback Generation System for Automated Essay Scoring - Lu, Chang; Cutumisu, Maria (2021)
<标题>: 将深度学习整合到自动化作文评分的自动反馈生成系统中 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z

L2 Learner Cognitive Psychological Factors About Artificial Intelligence Writing Corrective Feedback - Wu, LiQin; Wu, Yong; Zhang, XiangYang (English Language Teaching, 2021)
Although the study of artificial intelligence (AI) used in language teaching and learning is increasingly prevailing, research on language two (L2) learner cognitive psychological factors about AI writing corrective feedback (WCF) is scarce. This paper explores L2 learner cognitive psychology of pigai, an AI evaluating system for English writings in China, from perspectives of perception, noticing, uptake, initiative, retention and emotion. It investigates the consistency between learner cognitive psychology about AI WCF and the expected one and probes into the correlation of learner cognitive psychological factors about AI WCF, aiming at bridging the gap between the research of AI WCF and that of L2 learner cognitive psychology. After a 5-point Likert anonymous questionnaire survey of 1952 undergraduate L2 learners in Anhui University of Finance and Economics (AUFE), the statistical data of Pearson correlation coefficient indicate that learner perception, noticing, uptake, initiative, retention and emotion are positively related in the context of AI WCF, which conforms to the early research of learner cognitive psychology about WCF. But one sample t-test reveals that learner cognitive psychology of AI WCF only occasionally or sometimes consists with the expected one. The subsequent random interviews with 15 respondents suggest that pigai WCF is beneficial to L2 writing, yet there is still much room for it to improve to be deeply integrated with human WCF.
<标题>: 关于人工智能写作纠正反馈的二语学习者认知心理因素 <AI Smry>: It is suggested that pigai WCF is beneficial to L2 writing, yet there is still much room for it to improve to be deeply integrated with human WCF, which conforms to the early research of learner cognitive psychology about WCF. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

An automated essay scoring systems: a systematic literature review - Ramesh, Dadi; Sanampudi, Suresh Kumar (Artificial Intelligence Review, 2021)
<标题>: 自动化作文评分系统：一项系统的文献综述 <AI Smry>: It is observed that the essay evaluation is not done based on the relevance of the content and coherence, which means that there is no proper evaluation system for grading essays and short answers. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Beyond Agreement: Exploring Rater Effects in Large-Scale Mixed Format Assessments - Wind, Stefanie A.; Guo, Wenjing (Educational Assessment, 2021)
<标题>: 超越一致性：在大规模混合形式评估中探索评分者效应 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

shinyReCoR: A Shiny Application for Automatically Coding Text Responses Using R - Andersen, Nico; Zehner, Fabian (Psych, 2021)
In this paper, we introduce shinyReCoR: a new app that utilizes a cluster-based method for automatically coding open-ended text responses. Reliable coding of text responses from educational or psychological assessments requires substantial organizational and human effort. The coding of natural language in responses to tests depends on the texts’ complexity, corresponding coding guides, and the guides’ quality. Manual coding is thus not only expensive but also error-prone. With shinyReCoR, we provide a more efficient alternative. The use of natural language processing makes texts utilizable for statistical methods. shinyReCoR is a Shiny app deployed as an R-package that allows users with varying technical affinity to create automatic response classifiers through a graphical user interface based on annotated data. The present paper describes the underlying methodology, including machine learning, as well as peculiarities of the processing of language in the assessment context. The app guides users through the workflow with steps like text corpus compilation, semantic space building, preprocessing of the text data, and clustering. Users can adjust each step according to their needs. Finally, users are provided with an automatic response classifier, which can be evaluated and tested within the process.
<标题>: shinyReCoR：一个使用 R 自动编码文本回应的 Shiny 应用程序 <AI Smry>: The present paper describes the underlying methodology, including machine learning, as well as peculiarities of the processing of language in the assessment context, that utilizes a cluster-based method for automatically coding open-ended text responses. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Understanding Potential Sources of Harm throughout the Machine Learning Life Cycle - Suresh, Harini; Guttag, John (MIT Case Studies in Social and Ethical Responsibilities of Computing, 2021)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Automatic Speech Recognition (ASR) Systems Applied to Pronunciation Assessment of L2 Spanish for Japanese Speakers - Tejedor-García, Cristian; Cardeñoso-Payo, Valentín; Escudero-Mancebo, David (Applied Sciences, 2021)
General-purpose automatic speech recognition (ASR) systems have improved in quality and are being used for pronunciation assessment. However, the assessment of isolated short utterances, such as words in minimal pairs for segmental approaches, remains an important challenge, even more so for non-native speakers. In this work, we compare the performance of our own tailored ASR system (kASR) with the one of Google ASR (gASR) for the assessment of Spanish minimal pair words produced by 33 native Japanese speakers in a computer-assisted pronunciation training (CAPT) scenario. Participants in a pre/post-test training experiment spanning four weeks were split into three groups: experimental, in-classroom, and placebo. The experimental group used the CAPT tool described in the paper, which we specially designed for autonomous pronunciation training. A statistically significant improvement for the experimental and in-classroom groups was revealed, and moderate correlation values between gASR and kASR results were obtained, in addition to strong correlations between the post-test scores of both ASR systems and the CAPT application scores found at the final stages of application use. These results suggest that both ASR alternatives are valid for assessing minimal pairs in CAPT tools, in the current configuration. Discussion on possible ways to improve our system and possibilities for future research are included.
<标题>: 自动语音识别（ASR）系统在日本母语者西班牙语第二语言发音评估中的应用 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Complementary strengths? Evaluation of a hybrid human-machine scoring approach for a test of oral academic English - Davis, Larry; Papageorgiou, Spiros (Assessment in Education: Principles, Policy & Practice, 2021)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Complementary strengths? Evaluation of a hybrid human-machine scoring approach for a test of oral academic English - Davis, Larry; Papageorgiou, Spiros (Assessment in Education: Principles, Policy & Practice, 2021)
<标题>: 互补优势？针对学术英语口语测试的混合人机评分方法评估 <AI Smry>: An empirical study that explored the possibility of combining human and machine scores using responses from the speaking section of the TOEFL iBT® test found that composite scores calculated solely from human analytic scores showed the highest reliability and reliability steadily decreased as more machine scores replaced human scores. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z

Assessing L2 English speaking using automated scoring technology: examining automarker reliability - Xu, Jing; Jones, Edmund; Laxton, Victoria; Galaczi, Evelina (Assessment in Education: Principles, Policy & Practice, 2021)
<标题>: 使用自动评分技术评估二语英语口语：考察自动评分器的可靠性 <AI Smry>: A study that investigated the reliability of an automarker using candidate responses produced in an online oral English test found that the automarkser was slightly more lenient than examiner fair average scores, particularly for low-proficiency speakers. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.531Z

Determinants of grader agreement: an analysis of multiple short answer corpora - Padó, Ulrike; Padó, Sebastian (Language Resources and Evaluation, 2021)
The ’short answer’ question format is a widely used tool in educational assessment, in which students write one to three sentences in response to an open question. The answers are subsequently rated by expert graders. The agreement between these graders is crucial for reliable analysis, both in terms of educational strategies and in terms of developing automatic models for short answer grading (SAG), an active research topic in NLP. This makes it important to understand the properties that influence grader agreement (such as question difficulty, answer length, and answer correctness). However, the twin challenges towards such an understanding are the wide range of SAG corpora in use (which differ along a number of dimensions) and the hierarchical structure of potentially relevant properties (which can be located at the corpus, answer, or question levels). This article uses generalized mixed effects models to analyze the effect of various such properties on grader agreement in six major SAG corpora for two main assessment tasks (language and content assessment). Overall, we find broad agreement among corpora, with a number of properties behaving similarly across corpora (e.g., shorter answers and correct answers are easier to grade). Some properties show more corpus-specific behavior (e.g., the question difficulty level), and some corpora are more in line with general tendencies than others. In sum, we obtain a nuanced picture of how the major short answer grading corpora are similar and dissimilar from which we derive suggestions for corpus development and analysis.
<标题>: 评分员一致性的决定因素：多篇简答语料库分析 <AI Smry>: A nuanced picture is obtained of how the major short answer grading corpora are similar and dissimilar from which suggestions for corpus development and analysis are derived. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z

Designing Effective Interview Chatbots: Automatic Chatbot Profiling and Design Suggestion Generation for Chatbot Debugging - Han, Xu; Zhou, Michelle; Turner, Matthew J.; Yeh, Tom (Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021)
<AI Smry>: iChatProfile is an assistive chatbot design tool that can automatically generate a profile of an interview chatbot with quantified performance metrics and offer design suggestions for improving the chatbot based on such metrics. Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Engineering an Intelligent Essay Scoring and Feedback System: An Experience Report - Chadda, Akriti; Song, Kelly; Chandrasekar, Raman; Gorton, Ian (arXiv, 2021)
Artificial Intelligence (AI) / Machine Learning (ML)-based systems are widely sought-after commercial solutions that can automate and augment core business services. Intelligent systems can improve the quality of services offered and support scalability through automation. In this paper we describe our experience in engineering an exploratory system for assessing the quality of essays supplied by customers of a specialized recruitment support service. The problem domain is challenging because the open-ended customer-supplied source text has considerable scope for ambiguity and error, making models for analysis hard to build. There is also a need to incorporate specialized business domain knowledge into the intelligent processing systems. To address these challenges, we experimented with and exploited a number of cloud-based machine learning models and composed them into an application-specific processing pipeline. This design allows for modification of the underlying algorithms as more data and improved techniques become available. We describe our design, and the main challenges we faced, namely keeping a check on the quality control of the models, testing the software and deploying the computationally expensive ML models on the cloud.
arXiv:2103.13590 [cs] <标题>: 智能作文评分与反馈系统的工程设计：经验报告 <AI Smry>: This paper describes the experience in engineering an exploratory system for assessing the quality of essays supplied by customers of a specialized recruitment support service, and describes the design, and the main challenges it faced. Read_Status: New Read_Status_Date: 2026-01-26T11:32:28.375Z
Keywords: Computer Science - Machine Learning; Computer Science - Software Engineering

Advanced Methods in Automatic Item Generation - Gierl, Mark J.; Lai, Hollis; Tanygin, Vasily (Routledge, 2021)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Defining and Demystifying Automated Decision Systems - Richardson, Rashida (Maryland Law Review, 2021)
Government agencies are increasingly using automated decision systems to aid or supplant human decision-making and policy enforcement in various sensitive social domains. They determine who will have their food subsidies terminated, how much healthcare benefits a person is entitled to, and who is likely to be a victim of a crime. Yet, existing legislative and regulatory definitions fail to adequately describe or clarify how these technologies are used in practice and their impact on society. This failure to adequately describe and define “automated decision systems” leads to such systems evading scrutiny that policymakers are increasingly recognizing is warranted and potentially impedes avenues for legal redress. Such oversights can have concrete consequences for individuals and communities, such as increased law enforcement harassment, deportation, denial of housing or employment opportunities, and death.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

The role and challenges of education for responsible AI - Dignum, Virginia (London Review of Education, 2021)
Artificial intelligence (AI) is impacting education in many different ways. From virtual assistants for personalized education, to student or teacher tracking systems, the potential benefits of AI for education often come with a discussion of its impact on privacy and well-being. At the same time, the social transformation brought about by AI requires reform of traditional education systems. This article discusses what a responsible, trustworthy vision for AI is and how this relates to and affects education.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Enhancing prediction of student success: Automated machine learning approach - Zeineddine, Hassan; Braendle, Udo; Farah, Assaad (Computers & Electrical Engineering, 2021)
<标题>: 提升学生成功预测：自动化机器学习方法 <AI Smry>: This paper proposes the use of Automated Machine Learning to enhance the accuracy of predicting student performance using data available prior to the start of the academic program. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z

Improving Automated Scoring of Student Open Responses in Mathematics - Baral, Sami; Botelho, Anthony F.; Erickson, John A.; Benachamardi, Priyanka; Heffernan, Neil T. (Educational Data Mining, 2021)
<标题>: 改进数学中学生开放性回答的自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

How Do Educationally At-Risk Men and Women Differ in Their Essay-Writing Processes? - Bennett, Randy Elliot; Zhang, Mo; Sinharay, Sandip (Chinese/English Journal of Educational Measurement and Evaluation, 2021)
This study examined differences in the composition processes used by educationally at-risk males and females who wrote essays as part of a high-school equivalency examination. Over 30,000 individuals were assessed, each taking one of 12 forms of the examination’s language arts writing subtest in 23 US states. Writing processes were inferred using features extracted from keystroke logs and aggregated into seven composite indicators. Results showed that females earned higher essay and total language arts writing composite scores than did males, but only by trivial amounts. More pertinent was that, after controlling for language arts writing composite score, age, and essay prompt, all seven process indicators showed nontrivial, statistically significant differences, the most notable being for indicators related to fluency and different aspects of editing. The study’s findings are consistent in important ways with those from other investigations of school-age students and adults, and with results from both online and paper-based writing tasks. Implications are offered for conducting similar research for individuals composing in character-based languages like Chinese.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

A Machine Learning Prediction of Automatic Text Based Assessment for Open and Distance Learning: A Review - Blessing, Guembe; Azeta, Ambrose; Misra, Sanjay; Chigozie, Felix; Ahuja, Ravin (Innovations in Bio-Inspired Computing and Applications, 2021)
Series Title: Advances in Intelligent Systems and Computing Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

On the Opportunities and Risks of Foundation Models - Bommasani, Rishi; Hudson, Drew A.; Adeli, Ehsan; Altman, Russ; Arora, Simran; von Arx, Sydney; Bernstein, Michael S.; Bohg, Jeannette; Bosselut, Antoine; Brunskill, Emma; Brynjolfsson, Erik; Buch, Shyamal; et al. (arXiv, 2021)
AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities (e.g., language, vision, robotics, reasoning, human interaction) and technical principles(e.g., model architectures, training procedures, data, systems, security, evaluation, theory) to their applications (e.g., law, healthcare, education) and societal impact (e.g., inequity, misuse, economic and environmental impact, legal and ethical considerations). Though foundation models are based on standard deep learning and transfer learning, their scale results in new emergent capabilities,and their effectiveness across so many tasks incentivizes homogenization. Homogenization provides powerful leverage but demands caution, as the defects of the foundation model are inherited by all the adapted models downstream. Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration commensurate with their fundamentally sociotechnical nature.
Version Number: 3 <标题>: 关于基础模型的机遇与风险 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: Artificial Intelligence (cs.AI); Computers and Society (cs.CY); FOS: Computer and information sciences; Machine Learning (cs.LG)

All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text - Clark, Elizabeth; August, Tal; Serrano, Sofia; Haduong, Nikita; Gururangan, Suchin; Smith, Noah A. (arXiv, 2021)
Human evaluations are typically considered the gold standard in natural language generation, but as models' fluency improves, how well can evaluators detect and judge machine-generated text? We run a study assessing non-experts' ability to distinguish between human- and machine-authored text (GPT2 and GPT3) in three domains (stories, news articles, and recipes). We find that, without training, evaluators distinguished between GPT3- and human-authored text at random chance level. We explore three approaches for quickly training evaluators to better identify GPT3-authored text (detailed instructions, annotated examples, and paired examples) and find that while evaluators' accuracy improved up to 55%, it did not significantly improve across the three domains. Given the inconsistent results across text domains and the often contradictory reasons evaluators gave for their judgments, we examine the role untrained human evaluations play in NLG evaluation and provide recommendations to NLG researchers for improving human evaluations of text generated from state-of-the-art models.
arXiv:2107.00061 [cs] <AI Smry>: The role untrained human evaluations play in NLG evaluation is examined and three approaches for quickly training evaluators to better identify GPT3-authored text are explored and it is found that while evaluation accuracy improved up to 55%, it did not significantly improve across the three domains. <标题>: 并非所有“人性”皆为黄金：评估生成文本的人类评价 Read_Status: New Read_Status_Date: 2026-01-26T11:32:33.304Z
Keywords: /unread; Computer Science - Computation and Language

All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text - Clark, Elizabeth; August, Tal; Serrano, Sofia; Haduong, Nikita; Gururangan, Suchin; Smith, Noah A. (arXiv, 2021)
Human evaluations are typically considered the gold standard in natural language generation, but as models' fluency improves, how well can evaluators detect and judge machine-generated text? We run a study assessing non-experts' ability to distinguish between human- and machine-authored text (GPT2 and GPT3) in three domains (stories, news articles, and recipes). We find that, without training, evaluators distinguished between GPT3- and human-authored text at random chance level. We explore three approaches for quickly training evaluators to better identify GPT3-authored text (detailed instructions, annotated examples, and paired examples) and find that while evaluators' accuracy improved up to 55%, it did not significantly improve across the three domains. Given the inconsistent results across text domains and the often contradictory reasons evaluators gave for their judgments, we examine the role untrained human evaluations play in NLG evaluation and provide recommendations to NLG researchers for improving human evaluations of text generated from state-of-the-art models.
arXiv:2107.00061 [cs] <标题>: 并非所有“人性”皆为黄金：评估生成文本的人类评价 <AI Smry>: The role untrained human evaluations play in NLG evaluation is examined and three approaches for quickly training evaluators to better identify GPT3-authored text are explored and it is found that while evaluation accuracy improved up to 55%, it did not significantly improve across the three domains. Read_Status: New Read_Status_Date: 2026-01-26T11:32:19.981Z
Keywords: Computer Science - Computation and Language

Generate: A NLG system for educational content creation - Khan, Saad; Hamer, Jesse; Almeida, Tiago (Proceedings of The 14th International Conference on Educational Data Mining, 2021)
We present Generate, a AI-human hybrid system to help education content creators interactively generate assessment content in an efficient and scalable manner. Our system integrates advanced natural language generation (NLG) approaches with subject matter expertise of assessment developers to efficiently generate a large number of highly customized and valid assessment items. We utilize the powerful Transformer architecture which is capable of leveraging substantive pretraining on several generic text corpora in order to produce sophisticated, context-dependent text as the basis for item creation. We present early results from experimental studies demonstrating the efficiency of our approach.
<标题>: 生成：用于教育内容创作的自然语言生成系统 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Automated Essay Scoring and the Deep Learning Black Box: How Are Rubric Scores Determined? - Kumar, Vivekanandan S.; Boulanger, David (International Journal of Artificial Intelligence in Education, 2021)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Evaluating Dialogue Systems - McTear, Michael (Conversational AI, 2021)
Book Authors: _:n9473 Series Title: Synthesis Lectures on Human Language Technologies <标题>: 评估对话系统 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

CIDEr-R: Robust Consensus-based Image Description Evaluation - Santos, Gabriel Oliveira dos; Colombini, Esther Luna; Avila, Sandra (arXiv, 2021)
This paper shows that CIDEr-D, a traditional evaluation metric for image description, does not work properly on datasets where the number of words in the sentence is significantly greater than those in the MS COCO Captions dataset. We also show that CIDEr-D has performance hampered by the lack of multiple reference sentences and high variance of sentence length. To bypass this problem, we introduce CIDEr-R, which improves CIDEr-D, making it more flexible in dealing with datasets with high sentence length variance. We demonstrate that CIDEr-R is more accurate and closer to human judgment than CIDEr-D; CIDEr-R is more robust regarding the number of available references. Our results reveal that using Self-Critical Sequence Training to optimize CIDEr-R generates descriptive captions. In contrast, when CIDEr-D is optimized, the generated captions' length tends to be similar to the reference length. However, the models also repeat several times the same word to increase the sentence length.
arXiv:2109.13701 [cs] <AI Smry>: The results reveal that using Self-Critical Sequence Training to optimize CIDEr-R generates descriptive captions, and when C IDEr-D is optimized, the generated captions’ length tends to be similar to the reference length, but the models also repeat several times the same word to increase the sentence length. <标题>: CIDEr-R：基于共识的鲁棒图像描述评估 Read_Status: New Read_Status_Date: 2026-01-26T11:32:36.950Z
Keywords: /unread; Computer Science - Computation and Language; Computer Science - Computer Vision and Pattern Recognition

BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text - University of Wolverhampton, UK; Saadany, Hadeel; Orăsan, Constantin; University of Surrey, UK (Proceedings of the Translation and Interpreting Technology Online Conference TRITON 2021, 2021)
<AI Smry>: The ability of automatic quality metrics to detect critical machine translation errors which can cause serious misunderstanding of the affect message is assessed and the need for the fine-tuning of automatic metrics to make them more robust in detecting sentiment critical errors is demonstrated. <标题>: BLEU、METEOR、BERTScore：评估情感导向文本中关键翻译错误的度量指标表现 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

A review of deep-neural automated essay scoring models - Uto, Masaki (Behaviormetrika, 2021)
Abstract Automated essay scoring (AES) is the task of automatically assigning scores to essays as an alternative to grading by humans. Although traditional AES models typically rely on manually designed features, deep neural network (DNN)-based AES models that obviate the need for feature engineering have recently attracted increased attention. Various DNN-AES models with different characteristics have been proposed over the past few years. To our knowledge, however, no study has provided a comprehensive review of DNN-AES models while introducing each model in detail. Therefore, this review presents a comprehensive survey of DNN-AES models, describing the main idea and detailed architecture of each model. We classify the AES task into four types and introduce existing DNN-AES models according to this classification.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Automated Scoring of Chinese Grades 7–9 Students’ Competence in Interpreting and Arguing from Evidence - Wang, Cong; Liu, Xiufeng; Wang, Lei; Sun, Ying; Zhang, Hongyan (Journal of Science Education and Technology, 2021)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Data Augmentation by Rubrics for Short Answer Grading - Wang, Tianqi; Funayama, Hiroaki; Ouchi, Hiroki; Inui, Kentaro (Journal of Natural Language Processing, 2021)
<标题>: 通过评分量表进行简答题评分的数据增强 <AI Smry>: This paper proposes a semi-supervised method to train a neural SAG model that extracts keyphrases that are highly related to answers scores from rubrics, and demonstrates that both performance of grading and justiﬁcation identi-cation is improved by integrating attention semi- supervised training, especially in a low-resource setting. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Math Word Problem Generation with Mathematical Consistency and Problem Context Constraints - Wang, Zichao; Lan, Andrew; Baraniuk, Richard (Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021)
<标题>: 具有数学一致性和问题情境约束的数学应用题生成 <AI Smry>: A novel MWP generation approach is developed that leverages i) pre-trained language models and a context keyword selection model to improve the language quality of generated MWPs and ii) an equation consistency constraint for math equations to improved the mathematical validity of the generatedMWPs. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Effects of Algorithmic Transparency in Bayesian Knowledge Tracing on Trust and Perceived Accuracy - Williamson, Kim Christopher; Kizilcec, René F. (Educational Data Mining, 2021)
<标题>: 贝叶斯知识追踪中算法透明度对信任与感知准确性的影响 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z

ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback - Wu, Mike; Goodman, Noah; Piech, Chris; Finn, Chelsea (arXiv, 2021)
High-quality computer science education is limited by the difficulty of providing instructor feedback to students at scale. While this feedback could in principle be automated, supervised approaches to predicting the correct feedback are bottlenecked by the intractability of annotating large quantities of student code. In this paper, we instead frame the problem of providing feedback as few-shot classification, where a meta-learner adapts to give feedback to student code on a new programming question from just a few examples annotated by instructors. Because data for meta-training is limited, we propose a number of amendments to the typical few-shot learning framework, including task augmentation to create synthetic tasks, and additional side information to build stronger priors about each task. These additions are combined with a transformer architecture to embed discrete sequences (e.g. code) to a prototypical representation of a feedback class label. On a suite of few-shot natural language processing tasks, we match or outperform state-of-the-art performance. Then, on a collection of student solutions to exam questions from an introductory university course, we show that our approach reaches an average precision of 88% on unseen questions, surpassing the 82% precision of teaching assistants. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university. This is, to the best of our knowledge, the first successful deployment of a machine learning based feedback to open-ended student code.
Version Number: 2 <标题>: ProtoTransformer：一种用于提供学生反馈的元学习方法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: Computers and Society (cs.CY); FOS: Computer and information sciences; Machine Learning (cs.LG)

A Systematic Review of Fairness in Artificial Intelligence Algorithms - Xivuri, Khensani; Twinomurinzi, Hossana (Responsible AI and Analytics for an Ethical and Inclusive Digitized Society, 2021)
Series Title: Lecture Notes in Computer Science <标题>: 人工智能算法公平性的系统综述 <AI Smry>: There is a lack of formalised AI terminology and definitions which subsequently results in contrasting views of AI algorithmic fairness, and the public services sector is the target of most research, particularly criminal justice and immigration, followed by the health sector. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Bot-Adversarial Dialogue for Safe Conversational Agents - Xu, Jing; Ju, Da; Li, Margaret; Boureau, Y-Lan; Weston, Jason; Dinan, Emily (Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021)
<AI Smry>: This work introduces a new human-and-model-in-the-loop framework for evaluating the toxicity of generative models, and proposes two novel methods for safe conversational agents by either training on data from the framework in a two-stage system, or ”baking-in” safety to the generative model itself. <标题>: 用于安全对话代理的对抗性机器人对话 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Using AI Technology to Support Speaking Skill Development for the Teaching of Chinese as a Foreign Language: - Yingsoon, Goh Ying (Multidisciplinary Functions of Blockchain Technology in AI and IoT Applications, 2021)
The teaching of Chinese as a foreign language can be supported by using AI technology. Traditionally, the non-native learners can only interact with the instructors and depend on them solely for speaking practices. However, with the advancement of AI technology, the learners can use AI technology for interactive speaking skill development. In this study, the learners were instructed to download an application at https://m.wandoujia.com/apps/6790950. The process on the preparation of this AI technology in supporting their speaking skill development can be accessed at https://sites.google.com/site/gohyi141/assignments/projectinteractivespeakingapp. By using this AI technology, the findings showed a tremendous affirmative responses pertaining to the use of this AI technology. Hence, AI technology should be encouraged in active utilization for the teaching of Chinese as a foreign language in particular and for all language speaking skill development in general.
<标题>: 利用人工智能技术支持对外汉语教学中口语技能的发展 <AI Smry>: The findings showed a tremendous affirmative responses pertaining to the use of this AI technology for the teaching of Chinese as a foreign language in particular and for all language speaking skill development in general. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

BARTScore: Evaluating Generated Text as Text Generation - Yuan, Weizhe; Neubig, Graham; Liu, Pengfei (arXiv, 2021)
A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
Version Number: 2 <标题>: BART评分：将生成的文本作为文本生成进行评估 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: /unread; Computation and Language (cs.CL); FOS: Computer and information sciences

BARTScore: Evaluating Generated Text as Text Generation - Yuan, Weizhe; Neubig, Graham; Liu, Pengfei (arXiv, 2021)
A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
Version Number: 2 <标题>: BART评分：将生成的文本作为文本生成进行评估 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.293Z
Keywords: Computation and Language (cs.CL); FOS: Computer and information sciences

A Primer in BERTology: What We Know About How BERT Works - Rogers, Anna; Kovaleva, Olga; Rumshisky, Anna (Transactions of the Association for Computational Linguistics, 2020)
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
<标题>: BERT入门：我们对BERT工作原理的了解 <AI Smry>: This paper is the first survey of over 150 studies of the popular BERT model, reviewing the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Classification accuracy and efficiency of writing screening using automated essay scoring - Wilson, Joshua; Rodrigues, Jessica (Journal of School Psychology, 2020)
<标题>: 使用自动作文评分进行写作筛查的分类准确性和效率 <AI Smry>: Results indicated that the AES-scored multi-prompt screener and screeners with fewer prompts yield acceptable classification accuracy, are efficient, and are more accurate than screeners scored for word count. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Automated Essay Scoring and the Deep Learning Black Box: How Are Rubric Scores Determined? - Kumar, Vivekanandan S.; Boulanger, David (International Journal of Artificial Intelligence in Education, 2020)
<标题>: 自动化作文评分与深度学习黑箱：评分标准是如何确定的？ <AI Smry>: Predicting rubric scores is essential to automated essay scoring, because it reveals the reasoning behind AIED-based AES systems, and this study seeks to expand the horizons of AES to make the technique practical, explainable, and trustable. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Automated Scoring of Chinese Grades 7–9 Students’ Competence in Interpreting and Arguing from Evidence - Wang, Cong; Liu, Xiufeng; Wang, Lei; Sun, Ying; Zhang, Hongyan (Journal of Science Education and Technology, 2020)
<标题>: 中文七至九年级学生证据解释与论证能力的自动评分 <AI Smry>: It is suggested that automated scoring of Chinese writings produced a similar level of accuracy compared with that of English writings reported in literature, although there are specific considerations, e.g., training data set size, scoring rubric, and text lengths, to be considered using automatedscoring of student written responses in Chinese. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Measuring the Expressive Language and Vocabulary of Latino English Learners Using Hand Transcribed Speech Data and Automated Scoring - Makoto Sano; Doris Luft Baker; Marlen Collazo; Nancy Le; Akihito Kamata (International Journal of Intelligent Technologies and Applied Statistics, 2020)
Purpose: Explore how different automated scoring (AS) models score reliably the expressive language and vocabulary knowledge in depth of young second grade Latino English learners. Design/methodology/approach: Analyze a total of 13,471 English utterances from 217 Latino English learners with random forest, end-to-end memory networks, long short-term memory, and other AS models. Findings: Random forest outperformed the other AS models as measured by the mean of quadratic weighted kappa (QWK = 0.70) followed by the end-to-end memory networks-long short-term memory (QWK = 0.69) across all tasks and data points. The QWK between humans was 0.90, while the human-machine agreement of three AS models and humans ranged from 0.66 to 0.70. Practical implications: Examine closely misclassifications between human and machine scoring to better understand the specific words and structures the systems were not capturing. Originality/value: Discuss findings in the context of developing efficient and reliable ways to analyze the natural speech of young English learners. This information could guide the vocabulary and language proficiency instruction in the early grades.
<标题>: 使用手工转录语音数据和自动评分测量拉美裔英语学习者的表达性语言和词汇 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z

Automated Scoring of Teachers’ Pedagogical Content Knowledge – A Comparison Between Human and Machine Scoring - Wahlen, Andreas; Kuhn, Christiane; Zlatkin-Troitschanskaia, Olga; Gold, Christian; Zesch, Torsten; Horbach, Andrea (Frontiers in Education, 2020)
<标题>: 教师教学内容知识的自动评分——人工评分与机器评分的比较 <AI Smry>: The results of the single-sector variance analysis show a significant influence of the answers for each homogeneous subgroup on the automated scoring results, suggesting a convergent validity of the scoring results. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z

More efficient processes for creating automated essay scoring frameworks: A demonstration of two algorithms - Shin, Jinnie; Gierl, Mark J. (Language Testing, 2020)
Automated essay scoring (AES) has emerged as a secondary or as a sole marker for many high-stakes educational assessments, in native and non-native testing, owing to remarkable advances in feature engineering using natural language processing, machine learning, and deep-neural algorithms. The purpose of this study is to compare the effectiveness and the performance of two AES frameworks, each based on machine learning with deep language features, or complex language features, and deep neural algorithms. More specifically, support vector machines (SVMs) in conjunction with Coh-Metrix features were used for a traditional AES model development, and the convolutional neural networks (CNNs) approach was used for more contemporary deep-neural model development. Then, the strengths and weaknesses of the traditional and contemporary models under different circumstances (e.g., types of the rubric, length of the essay, and the essay type) were tested. The results were evaluated using the quadratic weighted kappa (QWK) score and compared with the agreement between the human raters. The results indicated that the CNNs model performs better, meaning that it produced more comparable results to the human raters than the Coh-Metrix + SVMs model. Moreover, the CNNs model also achieved state-of-the-art performance in most of the essay sets with a high average QWK score.
<标题>: 用于创建自动化作文评分框架的更高效流程：两种算法的演示 <AI Smry>: The CNNs model performs better, meaning that it produced more comparable results to the human raters than the Coh-Metrix + SVMs model, and achieved state-of-the-art performance in most of the essay sets with a high average QWK score. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Predictive Modeling of Rater Behavior: Implications for Quality Assurance in Essay Scoring - Bejar, Isaac I.; Li, Chen; McCaffrey, Daniel (Applied Measurement in Education, 2020)
<标题>: 评分者行为的预测模型：对论文评分质量保障的影响 <AI Smry>: Predictive rater models merit further investigation as a means of quality controlling human scoring because they capture stable individual differences among raters. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Evaluations of Automated Scoring Systems in Practice - Rotou, Ourania; Rupp, André A. (ETS Research Report Series, 2020)
This research report provides a description of the processes of evaluating the “deployability” of automated scoring (AS) systems from the perspective of large‐scale educational assessments in operational settings. It discusses a comprehensive psychometric evaluation that entails analyses that take into consideration the specific purpose of AS, the test design, the quality of human scores, the data collection design needed to train and evaluate the AS model, and the application of statistics and evaluation criteria. Finally, it notes that an effective evaluation of an AS system requires professional judgment coupled with statistical and psychometric knowledge and understanding of the risk assessment and business metrics.
<标题>: 自动评分系统在实践中的评估 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

BERTScore: Evaluating Text Generation with BERT - Zhang, Tianyi; Kishore, Varsha; Wu, Felix; Weinberger, Kilian Q.; Artzi, Yoav (arXiv, 2020)
We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.
arXiv:1904.09675 [cs] <标题>: BERT评分：使用BERT评估文本生成 Read_Status: New Read_Status_Date: 2026-01-26T11:32:25.304Z
Keywords: /unread; Computer Science - Computation and Language

Bias in data‐driven artificial intelligence systems—An introductory survey - Ntoutsi, Eirini; Fafalios, Pavlos; Gadiraju, Ujwal; Iosifidis, Vasileios; Nejdl, Wolfgang; Vidal, Maria‐Esther; Ruggieri, Salvatore; Turini, Franco; Papadopoulos, Symeon; Krasanakis, Emmanouil; Kompatsiaris, Ioannis; Kinder‐Kurlanda, Katharina; et al. (WIREs Data Mining and Knowledge Discovery, 2020)
Artificial Intelligence (AI)‐based systems are widely employed nowadays to make decisions that have far‐reaching impact on individuals and society. Their decisions might affect everyone, everywhere, and anytime, entailing concerns about potential human rights issues. Therefore, it is necessary to move beyond traditional AI algorithms optimized for predictive performance and embed ethical and legal principles in their design, training, and deployment to ensure social good while still benefiting from the huge potential of the AI technology. The goal of this survey is to provide a broad multidisciplinary overview of the area of bias in AI systems, focusing on technical challenges and solutions as well as to suggest new research directions towards approaches well‐grounded in a legal frame. In this survey, we focus on data‐driven AI, as a large part of AI is powered nowadays by (big) data and powerful machine learning algorithms. If otherwise not specified, we use the general term bias to describe problems related to the gathering or processing of data that might result in prejudiced decisions on the bases of demographic features such as race, sex, and so forth.
<标题>: 数据驱动的人工智能系统中的偏见——入门调查 <AI Smry>: A broad multidisciplinary overview of the area of bias in AI systems is provided, focusing on technical challenges and solutions as well as to suggest new research directions towards approaches well‐grounded in a legal frame. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Artificial Intelligence (AI): A Review of its Uses in Language Teaching and Learning - Ali, Zuraina (IOP Conference Series: Materials Science and Engineering, 2020)
The uses of Artificial Intelligence (AI) seems to be relevant in many fields nowadays due to its ability in providing a simulation of human intelligence processes that are handled by machines; in particular computer systems. This paper concerns with reviewing the uses of AI in language teaching and learning. In particular, it reviews the research on the uses of AI in its application in the learning and teaching of language. Qualitative research method; specifically content analysis, is employed as the technique to review the articles that are obtained from relevant databases. Findings from the study reveal that there are four (4) themes emerge in the uses of AI in relation to teaching and learning a language. The uses of AI for pedagogy, therefore, prove that its uses eases language teaching and learning.
<标题>: 人工智能（AI）：在语言教学与学习中的应用综述 <AI Smry>: The uses of AI for pedagogy, therefore, prove that its uses eases language teaching and learning. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

How technology affects instruction for English learners - Altavilla, Jennifer (Phi Delta Kappan, 2020)
In response to federal policy and the COVID-19 pandemic, schools and districts are using technology to support students designated as English learners (ELs). However, school leaders and teachers have little guidance about how to implement technology effectively to foster these students’ language development and content instruction. To address this need, Jennifer Altavilla raises three concerns specific to technology use with ELs: (1) Technology accessibility and use are equally important, (2) Technologies have baked-in biases, (3) Technology should promote authentic social interaction.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization - Durmus, Esin; He, He; Diab, Mona (Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020)
<AI Smry>: An automatic question answering (QA) based metric for faithfulness, FEQA, is proposed, which leverages recent advances in reading comprehension and has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries. Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Should You Fine-Tune BERT for Automated Essay Scoring? - Mayfield, Elijah; Black, Alan W (Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, 2020)
<标题>: 你应该为自动作文评分微调BERT吗？ <AI Smry>: It is found that fine-tuning BERT produces similar performance to classical models at significant additional cost, and a review of promising areas for research on student essays where the unique characteristics of Transformers may provide benefits over classical methods to justify the costs. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

On Faithfulness and Factuality in Abstractive Summarization - Maynez, Joshua; Narayan, Shashi; Bohnet, Bernd; McDonald, Ryan (arXiv, 2020)
It is well known that the standard likelihood training and approximate decoding objectives in neural text generation models lead to less human-like responses for open-ended tasks such as language modeling and story generation. In this paper we have analyzed limitations of these models for abstractive document summarization and found that these models are highly prone to hallucinate content that is unfaithful to the input document. We conducted a large scale human evaluation of several neural abstractive summarization systems to better understand the types of hallucinations they produce. Our human annotators found substantial amounts of hallucinated content in all model generated summaries. However, our analysis does show that pretrained models are better summarizers not only in terms of raw metrics, i.e., ROUGE, but also in generating faithful and factual summaries as evaluated by humans. Furthermore, we show that textual entailment measures better correlate with faithfulness than standard metrics, potentially leading the way to automatic evaluation metrics as well as training and decoding criteria.
arXiv:2005.00661 [cs] <标题>: 关于抽象摘要中的忠实性与事实性 <AI Smry>: It is found that neural abstractive summarization models are highly prone to hallucinate content that is unfaithful to the input document and textual entailment measures better correlate with faithfulness than standard metrics, potentially leading the way to automatic evaluation metrics as well as training and decoding criteria. Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Computation and Language

USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation - Mehri, Shikib; Eskenazi, Maxine (Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020)
<AI Smry>: USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog and is shown to strongly correlate with human judgment on both Topical-Chat and PersonaChat. <标题>: USR：一种用于对话生成的无监督且无需参考的评估指标 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z
Keywords: /unread

USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation - Mehri, Shikib; Eskenazi, Maxine (Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020)
<标题>: USR：一种用于对话生成的无监督且无需参考的评估指标 <AI Smry>: USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog and is shown to strongly correlate with human judgment on both Topical-Chat and PersonaChat. Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking - Rashkin, Hannah; Celikyilmaz, Asli; Choi, Yejin; Gao, Jianfeng (Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020)
<标题>: 情节机器：基于大纲的生成与动态情节状态跟踪 <AI Smry>: PlotMachines is presented, a neural narrative model that learns to transform an outline into a coherent story by tracking the dynamic plot states, and is enriched with high-level discourse structure so that the model can learn different styles of writing corresponding to different parts of the narrative. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

BLEURT: Learning Robust Metrics for Text Generation - Sellam, Thibault; Das, Dipanjan; Parikh, Ankur (Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020)
<AI Smry>: BLEURT, a learned evaluation metric for English based on BERT, can model human judgment with a few thousand possibly biased training examples and yields superior results even when the training data is scarce and out-of-distribution. <标题>: BLEURT：面向文本生成的鲁棒度量学习 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread

Algorithm inspection for chatbot performance evaluation - Vijayaraghavan, V; Cooper, Jack Brian; others (Procedia Computer Science, 2020)
<AI Smry>: This paper aims to explore the prominent types of chatbot testing methods with detailed emphasis on algorithm testing techniques, and involves the use of techniques such as cross-validation, grammar and parsing, verification and validation and statistical parsing. <标题>: 聊天机器人性能评估的算法检查 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread

Human Scoring versus Automated Scoring for English Learners in a Statewide Writing Assessment - Vo, Yen; Welch, Catherine; Dunbar, Stephen (2020)
Presenters: _:n11930 Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

The Impact of Training Data Quality on Automated Content Scoring Performance - Yao, Lili; Cahill, A.; McCaffrey, Daniel F. (Association for the Advancement of Artificial Intelligence, 2020)
<标题>: 训练数据质量对自动内容评分性能的影响 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Evaluating China’s Automated Essay Scoring System iWrite - Qian, Leyi; Zhao, Yali; Cheng, Yan (Journal of Educational Computing Research, 2019)
Automated writing scoring can not only provide holistic scores but also instant and corrective feedback on L2 learners’ writing quality. It has been increasing in use throughout China and internationally. Given the advantages, the past several years has witnessed the emergence and growth of writing evaluation products in China. To the best of our knowledge, no previous studies have touched upon the validity of China’s automated essay scoring systems. By drawing on the four major categories of argument for validity framework proposed by Kane—scoring, generalization, extrapolation, and implication, this article aims to evaluate the performance of one of the China’s automated essay scoring systems—iWrite against human scores. The results show that iWrite fails to be a valid tool to assess L2 writings and predict human scores. Therefore, iWrite currently should be restricted to nonconsequential uses and cannot be employed as an alternative to or a substitute for human raters.
<标题>: 评估中国的自动作文评分系统 iWrite <AI Smry>: iWrite fails to be a valid tool to assess L2 writings and predict human scores, and should be restricted to nonconsequential uses and cannot be employed as an alternative to or a substitute for human raters. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Question Generation by Transformers - Kriangchaivech, Kettip; Wangperawong, Artit (arXiv, 2019)
A machine learning model was developed to automatically generate questions from Wikipedia passages using transformers, an attention-based model eschewing the paradigm of existing recurrent neural networks (RNNs). The model was trained on the inverted Stanford Question Answering Dataset (SQuAD), which is a reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles. After training, the question generation model is able to generate simple questions relevant to unseen passages and answers containing an average of 8 words per question. The word error rate (WER) was used as a metric to compare the similarity between SQuAD questions and the model-generated questions. Although the high average WER suggests that the questions generated differ from the original SQuAD questions, the questions generated are mostly grammatically correct and plausible in their own right.
arXiv:1909.05017 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.339Z
Keywords: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Machine Learning

Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces? - Holmes, Samuel; Moorhead, Anne; Bond, Raymond; Zheng, Huiru; Coates, Vivien; Mctear, Michael (Proceedings of the 31st European Conference on Cognitive Ergonomics, 2019)
<AI Smry>: Usability test outcomes confirm what is already known about chatbots - that they are highly usable but conventional methods for assessing usability and user experience may not be as accurate when applied to chatbots. Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Automated language essay scoring systems: a literature review - Hussein, Mohamed Abdellatif; Hassan, Hesham; Nassef, Mohammad (PeerJ Computer Science, 2019)
Writing composition is a significant factor for measuring test-takers’ ability in any language exam. However, the assessment (scoring) of these writing compositions or essays is a very challenging process in terms of reliability and time. The need for objective and quick scores has raised the need for a computer system that can automatically grade essay questions targeting specific prompts. Automated Essay Scoring (AES) systems are used to overcome the challenges of scoring writing tasks by using Natural Language Processing (NLP) and machine learning techniques. The purpose of this paper is to review the literature for the AES systems used for grading the essay questions.
<标题>: 自动化语言作文评分系统：文献综述 <AI Smry>: The purpose of this paper is to review the literature for the AES systems used for grading the essay questions, and presents a structured literature review of the available Handcrafted Features AES systems. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Using Learning Analytics to evaluate the quality of multiple-choice questions: A perspective with Classical Test Theory and Item Response Theory - Azevedo, Jose Manuel; Oliveira, Ema P.; Beites, Patrícia Damas (The International Journal of Information and Learning Technology, 2019)
The purpose of this paper is to find appropriate forms of analysis of multiple-choice questions (MCQ) to obtain an assessment method, as fair as possible, for the students. The authors intend to ascertain if it is possible to control the quality of the MCQ contained in a bank of questions, implemented in Moodle, presenting some evidence with Item Response Theory (IRT) and Classical Test Theory (CTT). The used techniques can be considered a type of Descriptive Learning Analytics since they allow the measurement, collection, analysis and reporting of data generated from students’ assessment.
<标题>: 利用学习分析评估多项选择题的质量：基于经典测验理论与项目反应理论的视角 <AI Smry>: The main contribution and originality that can be found in this research is the definition of groups of questions with similar features, regarding their difficulty and discrimination properties, which allow teachers to build tests, randomly generated with Moodle, that include questions with several difficulty levels in the tests, as it should be done. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity - Holstein, Kenneth; McLaren, Bruce M.; Aleven, Vincent (Journal of Learning Analytics, 2019)
Involving stakeholders throughout the creation of new educational technologies can help ensure their usefulness and usability in real-world contexts. However, given the complexity of learning analytics (LA) systems, it can be challenging to meaningfully involve non-technical stakeholders throughout their design and development. This article reports on the iterative co-design, development, and classroom evaluation of Konscia, a wearable, real-time awareness tool for teachers working in AI-enhanced K-12 classrooms. In the process, we argue that the co-design of LA systems requires new kinds of prototyping methods. We introduce one of our own prototyping methods, REs, to address unique challenges of co-prototyping LA tools. This work presents the first end-to-end demonstration of how non-technical stakeholders can participate throughout the whole design process for a complex LA system—from early generative phases to the selection and tuning of analytics to evaluation in real-world contexts. We conclude by providing methodological recommendations for future LA co-design efforts.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

Examining Cultural Responsiveness in Large-Scale Assessment: The Matrix of Evidence for Validity Argumentation - Solano-Flores, Guillermo (Frontiers in Education, 2019)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z

An Empirical Study on the Artificial Intelligence Writing Evaluation System in China CET - Lu, Xiaoxia (Big Data, 2019)
<标题>: 中国大学英语四、六级人工智能写作评估系统的实证研究 <AI Smry>: The author finds that using AWE does effectively help the students with their English writing and both teachers and students have positive attitude to the use of AWE in terms of immediate and clear feedback, time-saving, and arousing interests in English writing. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.529Z

A Systematic Review of Deep Learning Approaches to Educational Data Mining - Hernández-Blanco, Antonio; Herrera-Flores, Boris; Tomás, David; Navarro-Colorado, Borja (Complexity, 2019)
Educational Data Mining (EDM) is a research field that focuses on the application of data mining, machine learning, and statistical methods to detect patterns in large collections of educational data. Different machine learning techniques have been applied in this field over the years, but it has been recently that Deep Learning has gained increasing attention in the educational domain. Deep Learning is a machine learning method based on neural network architectures with multiple layers of processing units, which has been successfully applied to a broad set of problems in the areas of image recognition and natural language processing. This paper surveys the research carried out in Deep Learning techniques applied to EDM, from its origins to the present day. The main goals of this study are to identify the EDM tasks that have benefited from Deep Learning and those that are pending to be explored, to describe the main datasets used, to provide an overview of the key concepts, main architectures, and configurations of Deep Learning and its applications to EDM, and to discuss current state-of-the-art and future directions on this area of research.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Question Bank Similarity Searching System (QB3S) Using NLP and Information Retrieval Technique - Mia, Md. Raihan; Latiful Hoque, Abu Sayed Md. (2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), 2019)
<标题>: 使用自然语言处理和信息检索技术的题库相似性搜索系统（QB3S） <AI Smry>: An efficient Question Bank Similarity Searching System (QB3S) to find similar questions, handle duplicate question and rank search result of a query input based on NLP and Information Retrieval techniques is developed. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Automated summary evaluation with inbuilt rubric method: An alternative to constructed responses and multiple-choice tests assessments - Martínez-Huertas, José Á.; Jastrzebska, Olga; Olmos, Ricardo; León, José A. (Assessment & Evaluation in Higher Education, 2019)
<标题>: 带内置评分标准方法的自动化摘要评估：构造性回答和选择题测试评估的替代方案 <AI Smry>: It is concluded that new automated summary evaluation approaches such as the inbuilt rubric method can be practical in terms of reliability and efficiency, and, thus, they can offer an affordable and valuable form of knowledge assessment in different educational levels. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts - Clark, Elizabeth; Celikyilmaz, Asli; Smith, Noah A. (Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019)
<AI Smry>: This work introduces methods based on sentence mover’s similarity, and finds that sentence-based metrics correlate with human judgments significantly better than ROUGE, both on machine-generated summaries and human-authored essays. Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Linguistic features and proficiency classification in L2 Spanish and L2 Portuguese - del Rio, Iria (Proceedings of the 8th Workshop on Natural Language Processing for Computer Assisted Language Learning, 2019)
This work explores the relationship between L2 proficiency levels and certain linguistic features through experiments in automatic proficiency classification. We use L2 Spanish and L2 Portuguese data and perform monolingual and cross-lingual experiments. We also compare native and leaner Spanish texts. To the best of our knowledge, this is the first work that performs automatic proficiency classification for L2 Spanish, as well as cross-lingual proficiency classification between L2 Portuguese and L2 Spanish. Our results for L2 Spanish are similar to the state-of-the-art, while our crosslingual experiments got lower results than similar works. In general, all the experiments suggest new insights about the relationship between linguistic features and proficiency levels in L2 Portuguese and L2 Spanish.
<标题>: L2西班牙语和L2葡萄牙语的语言特征与熟练度分类 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z

The many dimensions of algorithmic fairness in educational applications - Loukina, Anastassia; Madnani, Nitin; Zechner, Klaus (Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, 2019)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Best practices for the human evaluation of automatically generated text - Van Der Lee, Chris; Gatt, Albert; Van Miltenburg, Emiel; Wubben, Sander; Krahmer, Emiel (Proceedings of the 12th International Conference on Natural Language Generation, 2019)
<AI Smry>: This paper provides an overview of how human evaluation is currently conducted, and presents a set of best practices, grounded in the literature, for Natural Language Generation systems. <标题>: 自动生成文本的人类评估最佳实践 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Best practices for the human evaluation of automatically generated text - Van Der Lee, Chris; Gatt, Albert; Van Miltenburg, Emiel; Wubben, Sander; Krahmer, Emiel (Proceedings of the 12th International Conference on Natural Language Generation, 2019)
<标题>: 自动生成文本的人类评估最佳实践 <AI Smry>: This paper provides an overview of how human evaluation is currently conducted, and presents a set of best practices, grounded in the literature, for Natural Language Generation systems. Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

Automatic distractor generation for multiple-choice English vocabulary questions - Susanti, Yuni; Tokunaga, Takenobu; Nishikawa, Hitoshi; Obari, Hiroyuki (Research and Practice in Technology Enhanced Learning, 2018)
<标题>: 英语词汇选择题的自动干扰项生成 <AI Smry>: A novel method for automatically generating distractors for multiple-choice English vocabulary questions is proposed that introduces new sources for collecting distractor candidates and utilises semantic similarity and collocation information when ranking the collected candidates. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

A Survey of Methods for Explaining Black Box Models - Guidotti, Riccardo; Monreale, Anna; Ruggieri, Salvatore; Turini, Franco; Giannotti, Fosca; Pedreschi, Dino (ACM Computing Surveys, 2018)
In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
<标题>: 黑箱模型解释方法综述 <AI Smry>: A classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system is provided to help the researcher to find the proposals more useful for his own work. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Automated Scoring: Beyond Natural Language Processing - Madnani, Nitin; Cahill, Aoife (Proceedings of the 27th International Conference on Computational Linguistics, 2018)
In this position paper, we argue that building operational automated scoring systems is a task that has disciplinary complexity above and beyond standard competitive shared tasks which usually involve applying the latest machine learning techniques to publicly available data in order to obtain the best accuracy. Automated scoring systems warrant significant cross-discipline collaboration of which natural language processing and machine learning are just two of many important components. Such systems have multiple stakeholders with different but valid perspectives that can often times be at odds with each other. Our position is that it is essential for us as NLP researchers to understand and incorporate these perspectives in our research and work towards a mutually satisfactory solution in order to build automated scoring systems that are accurate, fair, unbiased, and useful.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Designing, evaluating, and deploying automated scoring systems with validity in mind: Methodological design decisions - Rupp, André A. (Applied Measurement in Education, 2018)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z

Interaction Challenges in AI Equipped Environments Built to Teach Foreign Languages Through Dialogue and Task-Completion - Divekar, Rahul R.; Drozdal, Jaimie; Zhou, Yalun; Song, Ziyi; Allen, David; Rouhani, Robert; Zhao, Rui; Zheng, Shuyue; Balagyozyan, Lilit; Su, Hui (Proceedings of the 2018 Designing Interactive Systems Conference, 2018)
<标题>: 在配备人工智能的环境中进行外语教学的对话与任务完成中的互动挑战 <AI Smry>: A human-scale, immersive room with a virtual AI agent that aids foreign language learning through task-completion exercises using multi-modal dialogue and enhanced the AI agent, virtual environment, and the on-boarding process for new users to mitigate these challenges. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

A comparison of grammatical proficiency measures in the automated assessment of spontaneous speech - Yoon, Su-Youn; Bhat, Suma (Speech Communication, 2018)
<标题>: 自动化评估自发性语音中的语法能力测量比较 <AI Smry>: New measures that assess the level of grammatical proficiency for an automated speech proficiency scoring system and an automated scoring model which predicted the holistic oral proficiency scores led to statistically significant improvement in agreement between human and machine scores over the previous system. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z

Understanding Mean Score Differences Between the e‐rater ® Automated Scoring Engine and Humans for Demographically Based Groups in the GRE ® General Test - Ramineni, Chaitanya; Williamson, David (ETS Research Report Series, 2018)
Notable mean score differences for the e‐rater® automated scoring engine and for humans for essays from certain demographic groups were observed for the GRE® General Test in use before the major revision of 2012, called rGRE. The use of e‐rater as a check‐score model with discrepancy thresholds prevented an adverse impact on the examinee score at the item or test level. Despite this control, there remains a need to understand the root causes of these demographically based score differences and to identify potential mechanisms for avoiding future instances of discrepancy. In this study, we used a combination of statistical methods and human review to propose hypotheses about the root cause of score differences and whether such discrepancies reflect inadequacies of e‐rater, human scoring, or both. The human rating process was found to be influenced strongly by the scale structure and did not fully correspond to the e‐rater scoring mechanism. The human raters appeared to be using conditional logic and a rule‐based approach to their scoring, while e‐rater uses linear weighting of all the features. These analyses have implications for future research and operational policies for the scoring of the rGRE.
<标题>: 理解 e‐rater ® 自动评分引擎与人类在 GRE ® 通用考试中不同人口群体的平均分差异 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks - Ganesan, Kavita (arXiv, 2018)
Evaluation of summarization tasks is extremely crucial to determining the quality of machine generated summaries. Over the last decade, ROUGE has become the standard automatic evaluation measure for evaluating summarization tasks. While ROUGE has been shown to be effective in capturing n-gram overlap between system and human composed summaries, there are several limitations with the existing ROUGE measures in terms of capturing synonymous concepts and coverage of topics. Thus, often times ROUGE scores do not reflect the true quality of summaries and prevents multi-faceted evaluation of summaries (i.e. by topics, by overall content coverage and etc). In this paper, we introduce ROUGE 2.0, which has several updated measures of ROUGE: ROUGE-N+Synonyms, ROUGE-Topic, ROUGE-Topic+Synonyms, ROUGE-TopicUniq and ROUGE-TopicUniq+Synonyms; all of which are improvements over the core ROUGE measures.
arXiv:1803.01937 [cs] Read_Status: New Read_Status_Date: 2026-01-26T11:32:41.340Z
Keywords: /unread; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Information Retrieval

The Influence of Rater Effects in Training Sets on the Psychometric Quality of Automated Scoring for Writing Assessments - Wind, Stefanie A.; Wolfe, Edward W.; Engelhard, George; Foltz, Peter; Rosenstein, Mark (International Journal of Testing, 2018)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Automated Essay Scoring in the Presence of Biased Ratings - Amorim, Evelin; Cançado, Marcia; Veloso, Adriano (Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

CNN for Text-Based Multiple Choice Question Answering - Chaturvedi, Akshay; Pandit, Onkar; Garain, Utpal (Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018)
<标题>: 用于基于文本的多项选择题问答的卷积神经网络 <AI Smry>: A Convolutional Neural Network model for text-based multiple choice question answering where questions are based on a particular article and given an article and a multiple choice questions, this model assigns a score to each question-option tuple and chooses the final option accordingly. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z

Finding Similar Medical Questions from Question Answering Websites - Li, Yaliang; Yao, Liuyi; Du, Nan; Gao, Jing; Li, Qi; Meng, Chuishi; Zhang, Chenwei; Fan, Wei (arXiv, 2018)
The past few years have witnessed the flourishing of crowdsourced medical question answering (Q&A) websites. Patients who have medical information demands tend to post questions about their health conditions on these crowdsourced Q&A websites and get answers from other users. However, we observe that a large portion of new medical questions cannot be answered in time or receive only few answers from these websites. On the other hand, we notice that solved questions have great potential to solve this challenge. Motivated by these, we propose an end-to-end system that can automatically find similar questions for unsolved medical questions. By learning the vector presentation of unsolved questions and their candidate similar questions, the proposed system outputs similar questions according to the similarity between vector representations. Through the vector representation, the similar questions are found at the question level, and the diversity of medical questions expression issue can be addressed. Further, we handle two more important issues, i.e., training data generation issue and efficiency issue, associated with the LSTM training procedure and the retrieval of candidate similar questions. The effectiveness of the proposed system is validated on a large-scale real-world dataset collected from a crowdsourced maternal-infant Q&A website.
Version Number: 1 <标题>: 从问答网站中查找相似的医疗问题 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); FOS: Computer and information sciences

Towards Controllable Story Generation - Peng, Nanyun; Ghazvininejad, Marjan; May, Jonathan; Knight, Kevin (Proceedings of the First Workshop on Storytelling, 2018)
<标题>: 面向可控的故事生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.077Z

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead - Rudin, Cynthia (arXiv, 2018)
Black box machine learning models are currently being used for high stakes decision-making throughout society, causing problems throughout healthcare, criminal justice, and in other domains. People have hoped that creating methods for explaining these black box models will alleviate some of these problems, but trying to \textit{explain} black box models, rather than creating models that are \textit{interpretable} in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society. There is a way forward – it is to design models that are inherently interpretable. This manuscript clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare, and computer vision.
Version Number: 3 <标题>: 停止为高风险决策解释黑箱机器学习模型，转而使用可解释的模型 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.129Z
Keywords: FOS: Computer and information sciences; Machine Learning (cs.LG); Machine Learning (stat.ML)

Use of Automated Scoring Features to Generate Hypotheses Regarding Language-Based DIF - Shermis, Mark D.; Mao, Liyang; Mulholland, Matthew; Kieftenbeld, Vincent (International Journal of Testing, 2017)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Approaches to Automated Scoring of Speaking for K–12 English Language Proficiency Assessments - Evanini, Keelan; Hauck, Maurice Cogan; Hakuta, Kenji (ETS Research Report Series, 2017)
This report is the fifth in a series concerning English language proficiency (ELP) assessments for English learners (ELs) in kindergarten through 12th grade in the United States. The series, produced by Educational Testing Service (ETS), is intended to provide theory‐ and evidence‐based principles and recommendations for improving next‐generationELPassessment systems, policies, and practices and to stimulate discussion on better serving K–12ELstudents. The first report articulated a high‐level conceptualization of next‐generationELPassessment systems (Hauck, Wolf, & Mislevy, 2016). The second report addressed accessibility issues in the context ofELPassessments forELsandELswith disabilities (Guzman‐Orth, Laitusis, Thurlow, & Christensen, 2016). The third report focused on critical policy and research issues of summativeELPassessments that state use for accountability purposes (Wolf, Guzman‐Orth, & Hauck, 2016). The fourth report dealt with one of the major uses ofELPassessments—the initial identification and classification of ELs (Lopez, Pooler, & Linquanti, 2016). The present report discusses approaches to using automated scoring technology for the evaluation of student‐spoken responses on K–12ELPassessments. As many states have begun to use computer‐basedELPassessments, there is a growing interest in automated scoring of spoken responses to increase the efficiency of scoring. This report delineates major areas to consider in using automated speech scoring for K–12ELPassessments (i.e., assessment construct and task design, scoring and score reporting, and artificial intelligence (AI) model development and test delivery) and makes recommendations for states on how to evaluate these considerations and determine a path forward.
<标题>: 面向 K–12 英语语言能力评估的口语自动评分方法 <AI Smry>: Major areas to consider in using automated speech scoring for K–12 ELP assessments are delineated and recommendations for states on how to evaluate these considerations are made and determine a path forward are made. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.151Z

Use of Automated Scoring Features to Generate Hypotheses Regarding Language-Based DIF - Shermis, Mark D.; Mao, Liyang; Mulholland, Matthew; Kieftenbeld, Vincent (International Journal of Testing, 2017)
<标题>: 使用自动评分功能生成关于基于语言的差异项目功能（DIF）假设 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

A Simple but Tough-to-Beat Baseline for Sentence Embeddings - Arora, Sanjeev; Liang, Yingyu; Ma, Tengyu (International Conference on Learning Representations, 2017)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z
Keywords: ⛔ No DOI found

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses - Lowe, Ryan; Noseworthy, Michael; Serban, Iulian Vlad; Angelard-Gontier, Nicolas; Bengio, Yoshua; Pineau, Joelle (Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017)
<AI Smry>: An evaluation model that learns to predict human-like scores to input responses, using a new dataset of human response scores is presented and it is shown that the ADEM model’s predictions correlate significantly, and at a level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level. <标题>: 迈向自动图灵测试：学习评估对话回复 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

Building Better Open-Source Tools to Support Fairness in Automated Scoring - Madnani, Nitin; Loukina, Anastassia; von Davier, Alina; Burstein, Jill; Cahill, Aoife (Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, 2017)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Exploring Automated Essay Scoring for Nonnative English Speakers - Nigam, Amber (Proceedings of the Conference EUROPHRAS 2017 - Computational and Corpus-based Phraseology: Recent Advances and Interdisciplinary Approaches, Volume II (short papers, posters and student workshop papers), 2017)
Automated Essay Scoring (AES) has been quite popular and is being widely used. However, lack of appropriate methodology for rating nonnative English speakers' essays has meant a lopsided advancement in this field. In this paper, we report initial results of our experiments with nonnative AES that learns from manual evaluation of nonnative essays. For this purpose, we conducted an exercise in which essays written by nonnative English speakers in test environment were rated both manually and by the automated system designed for the experiment. In the process, we experimented with a few features to learn about nuances linked to nonnative evaluation. The proposed methodology of automated essay evaluation has yielded a correlation coefficient of 0.750 with the manual evaluation.
arXiv:1706.03335 [cs] <标题>: 探索面向非英语母语者的自动化作文评分 <AI Smry>: Initial results of experiments with nonnative AES that learns from manual evaluation of nonnative essays are reported, and the proposed methodology of automated essay evaluation has yielded a correlation coefficient of 0.750 with the manual evaluation. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z
Keywords: Computer Science - Computation and Language; I.2.7

Investigating neural architectures for short answer scoring - Riordan, Brian; Horbach, Andrea; Cahill, Aoife; Zesch, Torsten; Lee, Chong Min (Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, 2017)
<标题>: 研究用于简答评分的神经架构 <AI Smry>: This work investigates how several basic neural approaches similar to those used for automated essay scoring perform on short answer scoring, and shows that neural architectures can outperform a strong non-neural baseline, but performance and optimal parameter settings vary across the more diverse types of prompts typical of shortanswer scoring. Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

How Does Predicate Invention Affect Human Comprehensibility? - Schmid, Ute; Zeller, Christina; Besold, Tarek; Tamaddoni-Nezhad, Alireza; Muggleton, Stephen (Inductive Logic Programming, 2017)
Series Title: Lecture Notes in Computer Science <标题>: 谓词发明如何影响人类可理解性？ <AI Smry>: A definition of comprehensibility of hypotheses which can be estimated using human participant trials is provided and results indicate that comprehensibility is affected not only by the complexity of the presented program but also by the existence of anonymous predicate symbols. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Associated effects of automated essay evaluation software on growth in writing quality for students with and without disabilities - Wilson, Joshua (Reading and Writing, 2017)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Combining Multiple Corpora for Readability Assessment for People with Cognitive Disabilities - Yaneva, Victoria; Orasan, Constantin; Evans, Richard; Rohanian, Omid (Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, 2017)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Monitoring the performance of human and automated scores for spoken responses - Wang, Zhen; Zechner, Klaus; Sun, Yu (Language Testing, 2016)
As automated scoring systems for spoken responses are increasingly used in language assessments, testing organizations need to analyze their performance, as compared to human raters, across several dimensions, for example, on individual items or based on subgroups of test takers. In addition, there is a need in testing organizations to establish rigorous procedures for monitoring the performance of both human and automated scoring processes during operational administrations. This paper provides an overview of the automated speech scoring system SpeechRaterSM and how to use charts and evaluation statistics to monitor and evaluate automated scores and human rater scores of spoken constructed responses.
<标题>: 监控人工评分和自动评分对口语回答的表现 <AI Smry>: An overview of the automated speech scoring system SpeechRaterSM is provided and how to use charts and evaluation statistics to monitor and evaluate automated scores and human rater scores of spoken constructed responses is provided. Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Comparing Human and Automated Essay Scoring for Prospective Graduate Students With Learning Disabilities and/or ADHD - Buzick, Heather; Oliveri, Maria Elena; Attali, Yigal; Flor, Michael (Applied Measurement in Education, 2016)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Creating a Next‐Generation System of K–12 English Learner Language Proficiency Assessments - Hauck, Maurice Cogan; Wolf, Mikyung Kim; Mislevy, Robert (ETS Research Report Series, 2016)
This paper is the first in a series from Educational Testing Service (ETS) concerning English language proficiency (ELP) assessments for K–12 English learners (ELs). The goal of this paper, and the series, is to present research‐based ideas, principles, and recommendations for consideration by those who are conceptualizing, developing, and implementing ELP assessments for K–12 ELs and by all stakeholders in their education and assessment. We also hope to contribute to the active current discussion in the field on improving the instruction and assessment of EL students as well as contribute to bringing these ideas into practice. This paper articulates a high‐level vision for a next‐generation assessment system serving K–12 English learners with a brief overview of the current state of the art for K–12 ELP assessments and the context of current reforms. In addition, the paper discusses the role of digital technology in next‐generation K‐12 ELP assessment systems, noting the distinct conceptual and practical advantages that digital technology offers in assessing ELP as well as principles for evaluating the potential challenges of implementing a digitally based assessment system against these benefits. A proposed theory of action for K–12 ELP assessment systems is also discussed in this paper. The second paper in the series addressed accessibility issues in the context of ELP assessments for ELs and ELs with disabilities (Guzman‐Orth, Laitusis, Thurlow, & Christensen, 2016), the third paper addressed issues related to summative ELP assessments that emerged from the presentations and discussions at the English Language Proficiency Assessment Research working meeting (Wolf, Guzman‐Orth, & Hauck, 2016), and the fourth paper focused on a key concern within such systems—the initial identification and classification of ELs (Lopez, Pooler, & Linquanti, 2016).
<标题>: 创建下一代K–12英语学习者语言能力评估系统 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

Analyzing the Behavior of Visual Question Answering Models - Agrawal, Aishwarya; Batra, Dhruv; Parikh, Devi (arXiv, 2016)
Recently, a number of deep-learning based models have been proposed for the task of Visual Question Answering (VQA). The performance of most models is clustered around 60-70%. In this paper we propose systematic methods to analyze the behavior of these models as a first step towards recognizing their strengths and weaknesses, and identifying the most fruitful directions for progress. We analyze two models, one each from two major classes of VQA models – with-attention and without-attention and show the similarities and differences in the behavior of these models. We also analyze the winning entry of the VQA Challenge 2016. Our behavior analysis reveals that despite recent progress, today's VQA models are "myopic" (tend to fail on sufficiently novel instances), often "jump to conclusions" (converge on a predicted answer after 'listening' to just half the question), and are "stubborn" (do not change their answers across images).
Version Number: 2 <标题>: 分析视觉问答模型的行为 Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.956Z
Keywords: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); FOS: Computer and information sciences; Machine Learning (cs.LG)

Automatic Generation of Context-Based Fill-in-the-Blank Exercises Using Co-occurrence Likelihoods and Google n-grams - Hill, Jennifer; Simha, Rahul (Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, 2016)
<标题>: 使用共现可能性和谷歌 n-gram 自动生成基于上下文的填空练习 <AI Smry>: A unique application of word co-occurrence likelihoods and the Google n-grams corpus is used to select words with strong contextual links to their surrounding text, and to generate distractors that make sense only in an isolated narrow context and not in the full context of the passage. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Automated Essay Evaluation for English Language Learners:A Case Study of MY Access - Hoang, Giang Thi Linh; Kunnan, Antony John (Language Assessment Quarterly, 2016)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Human-level concept learning through probabilistic program induction - Lake, Brenden M.; Salakhutdinov, Ruslan; Tenenbaum, Joshua B. (Science, 2015)
Handwritten characters drawn by a model Not only do children learn effortlessly, they do so quickly and with a remarkable ability to use what they have learned as the raw material for creating new stuff. Lake et al. describe a computational model that learns in a similar fashion and does so better than current deep learning algorithms. The model classifies, parses, and recreates handwritten characters, and can generate new letters of the alphabet that look “right” as judged by Turing-like tests of the model's output in comparison to what real humans produce. Science , this issue p. 1332 , Combining the capacity to handle noise with probabilistic learning yields humanlike performance in a computational model. , People learning new concepts can often generalize successfully from just a single example, yet machine learning algorithms typically require tens or hundreds of examples to perform with similar accuracy. People can also use learned concepts in richer ways than conventional algorithms—for action, imagination, and explanation. We present a computational model that captures these human learning abilities for a large class of simple visual concepts: handwritten characters from the world’s alphabets. The model represents concepts as simple programs that best explain observed examples under a Bayesian criterion. On a challenging one-shot classification task, the model achieves human-level performance while outperforming recent deep learning approaches. We also present several “visual Turing tests” probing the model’s creative generalization abilities, which in many cases are indistinguishable from human behavior.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Comparing the Effectiveness of Self‐Paced and Collaborative Frame‐of‐Reference Training on Rater Accuracy in a Large‐Scale Writing Assessment - Raczynski, Kevin R.; Cohen, Allan S.; Engelhard, George; Lu, Zhenqiu (Journal of Educational Measurement, 2015)
There is a large body of research on the effectiveness of rater training methods in the industrial and organizational psychology literature. Less has been reported in the measurement literature on large‐scale writing assessments. This study compared the effectiveness of two widely used rater training methods—self‐paced and collaborative frame‐of‐reference training—in the context of a large‐scale writing assessment. Sixty‐six raters were randomly assigned to the training methods. After training, all raters scored the same 50 representative essays prescored by a group of expert raters. A series of generalized linear mixed models were then fitted to the rating data. Results suggested that the self‐paced method was equivalent in effectiveness to the more time‐intensive and expensive collaborative method. Implications for large‐scale writing assessments and suggestions for further research are discussed.
<标题>: 在大型写作评估中自定进度与协作框架参考培训对评分者准确性的有效性比较 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z

Using Technology-Enhanced Processes to Generate Test Items in Multiple Languages - Gierl, Mark J.; Lai, Hollis; Fung, Karen; Zheng, Bin (Technology and Testing, 2015)
Book Authors: _:n12231 <标题>: 使用技术增强的流程生成多语言测试题 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

CIDEr: Consensus-based Image Description Evaluation - Vedantam, Ramakrishna; Zitnick, C. Lawrence; Parikh, Devi (arXiv, 2015)
Automatically describing an image with a sentence is a long-standing challenge in computer vision and natural language processing. Due to recent progress in object detection, attribute classification, action recognition, etc., there is renewed interest in this area. However, evaluating the quality of descriptions has proven to be challenging. We propose a novel paradigm for evaluating image descriptions that uses human consensus. This paradigm consists of three main parts: a new triplet-based method of collecting human annotations to measure consensus, a new automated metric (CIDEr) that captures consensus, and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences describing each image. Our simple metric captures human judgment of consensus better than existing metrics across sentences generated by various sources. We also evaluate five state-of-the-art image description approaches using this new protocol and provide a benchmark for future comparisons. A version of CIDEr named CIDEr-D is available as a part of MS COCO evaluation server to enable systematic evaluation and benchmarking.
arXiv:1411.5726 [cs] <AI Smry>: A novel paradigm for evaluating image descriptions that uses human consensus is proposed and a new automated metric that captures human judgment of consensus better than existing metrics across sentences generated by various sources is evaluated. <标题>: CIDEr：基于共识的图像描述评估 Read_Status: New Read_Status_Date: 2026-01-26T11:32:21.425Z
Keywords: /unread; Computer Science - Computation and Language; Computer Science - Computer Vision and Pattern Recognition; Computer Science - Information Retrieval

The Impact of Training Data on Automated Short Answer Scoring Performance - Heilman, Michael; Madnani, Nitin (Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, 2015)
<标题>: 训练数据对自动短答案评分性能的影响 <AI Smry>: This paper conducts experiments using scored responses to 44 prompts from 5 diverse datasets in order to better understand how training set size and other factors relate to system performance. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

LeBLEU: N-gram-based Translation Evaluation Score for Morphologically Complex Languages - Virpioja, Sami; Grönroos, Stig-Arne (Proceedings of the Tenth Workshop on Statistical Machine Translation, 2015)
<AI Smry>: The results on WMT data sets show that fuzzy n-gram matching improves correlations to human evaluation especially for highly compounding languages. <标题>: LeBLEU：用于形态复杂语言的基于N-gram的翻译评估分数 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

The Eras and Trends of Automatic Short Answer Grading - Burrows, Steven; Gurevych, Iryna; Stein, Benno (International Journal of Artificial Intelligence in Education, 2014)
<标题>: 自动简答评分的发展时代与趋势 <AI Smry>: A comprehensive review of ASAG research and systems according to history and components concludes that an era of evaluation is the newest trend in ASAGResearch, which is paving the way for the consolidation of the field. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.151Z

Automated versus human scoring: A case study in an EFL context - Huang, S.-J (Electronic Journal of Foreign Language Teaching, 2014)
<标题>: 自动评分与人工评分：一项在英语作为外语环境中的案例研究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Automated Scoring of Constructed-Response Science Items: Prospects and Obstacles - Liu, Ou Lydia; Brew, Chris; Blackmore, John; Gerard, Libby; Madhok, Jacquie; Linn, Marcia C. (Educational Measurement: Issues and Practice, 2014)
Content‐based automated scoring has been applied in a variety of science domains. However, many prior applications involved simplified scoring rubrics without considering rubrics representing multiple levels of understanding. This study tested a concept‐based scoring tool for content‐based scoring, c‐rater™, for four science items with rubrics aiming to differentiate among multiple levels of understanding. The items showed moderate to good agreement with human scores. The findings suggest that automated scoring has the potential to score constructed‐response items with complex scoring rubrics, but in its current design cannot replace human raters. This article discusses sources of disagreement and factors that could potentially improve the accuracy of concept‐based automated scoring.
<标题>: 构建式科学题目的自动评分：前景与障碍 Read_Status: New Read_Status_Date: 2026-01-26T11:33:31.076Z

Using Automated Essay Scores as an Anchor When Equating Constructed Response Writing Tests - Almond, Russell G. (International Journal of Testing, 2013)
<标题>: 在对构造性回应写作测试进行等值处理时使用自动评分作为基准 <AI Smry>: Output from an automated scoring program designed to measure writing fluency can be used in place of a more traditional anchor to produce a viable alternative, and a mechanism for checking that the identity equating is appropriate. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z

Exploring the Assessment of Summaries: Using Latent Semantic Analysis to Grade Summaries Written by Spanish Students - León, J.A.; Olmos, R.; Escudero, I.; Jorge-Botana, G.; Perry, D. (Procedia - Social and Behavioral Sciences, 2013)
<标题>: 探索摘要评估：使用潜在语义分析对西班牙学生撰写的摘要进行评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z

An Application of Reverse Engineering to Automatic Item Generation: A Proof of Concept Using Automatically Generated Figures - Lorie, William A. (Questar Assessment, Inc., 2013)
A reverse engineering approach to automatic item generation (AIG) was applied to a figurebased publicly released test item from the Organisation for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA) mathematical literacy cognitive instrument as part of a proof of concept. The author created an item template from which three items were randomly generated from within each of six types defined by a feature deemed to be most likely to affect item difficulty, for a total of eighteen distinct items. To assess their equivalence, these items were embedded in otherwise identical test forms and administered to human intelligence task workers on the Amazon Mechanical Turk system. One level of the type-defining feature appeared to affect item difficulty systematically. The author provides a task requirement rationale for removing this level. Implications for AIG theory and practice are discussed.
<标题>: 逆向工程在自动生成题目中的应用：使用自动生成图形的概念验证 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

English language learners and automated scoring of essays: Critical considerations - Weigle, Sara Cushing (Assessing Writing, 2013)
<标题>: 英语学习者与自动化作文评分：关键考量 <AI Smry>: Concerns for using automated scoring systems to evaluate second language writing and an approach to validity based on articulating an interpretive argument are presented and discussed with reference to a recent study of the use of e-rater on the TOEFL. Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.511Z

English language learners and automated scoring of essays: Critical considerations - Weigle, Sara Cushing (Assessing Writing, 2013)
<标题>: 英语学习者与自动化作文评分：关键考量 <AI Smry>: Concerns for using automated scoring systems to evaluate second language writing and an approach to validity based on articulating an interpretive argument are presented and discussed with reference to a recent study of the use of e-rater on the TOEFL. Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Automatic item generation: theory and practice - Gierl, Mark J.; Haladyna, Thomas M. (Routledge, 2013)
"Automatic item generation (AIG) represents a relatively new and unique research area where specific cognitive and psychometric theories are applied to test construction practices for the purpose of producing test items using technology. The purpose of this book is to bring researchers and practitioners up-to-date on the growing body of research on AIG by organizing in one volume what is currently known about this research area. Part I begins with an overview of the concepts and topics necessary for understanding AIG by focusing on both its history and current applications. Part II presents two theoretical frameworks and practical applications of these frameworks in the production of item generation. Part III summarizes the psychological and substantive characteristics of generated items . Part IV concludes with a discussion of the statistical models that can be used to estimate the item characteristics of generated items, features one future application of AIG, describes the current technologies used for AIG, and also highlights the unresolved issues that must be addressed as AIG continues to mature as a research area. Comprehensive The book provides a comprehensive analysis of both the theoretical concepts that define automatic item generation and the practical considerations required to implement these concepts. Varied Applications Readers are provided with novel applications in diverse content areas (e.g., science and reading comprehension) that range across all educational levels elementary thru university"–
<标题>: 自动题目生成：理论与实践 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

‘Realness’ in Chatbots: Establishing Quantifiable Criteria - Morrissey, Kellie; Kirakowski, Jurek (Human-Computer Interaction. Interaction Modalities and Techniques, 2013)
Series Editors: _:n9363 Series Title: Lecture Notes in Computer Science <AI Smry>: The aim of this research is to generate measurable evaluation criteria acceptable to chatbot users, resulting in four subscales with strong reliability which discriminated well between the two categories of chatbots. Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

Indices of lexical diversity for Automated Essay Scoring - Cohen, Yoav; Safran, Yael (2012)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Comparison of Human and Machine Scoring of Essays: Differences by Gender, Ethnicity, and Country - Bridgeman, Brent; Trapani, Catherine; Attali, Yigal (Applied Measurement in Education, 2012)
<标题>: 人类与机器作文评分比较：按性别、族裔和国家的差异 <AI Smry>: Human and machine scores were very similar across most subgroups, but there were some notable exceptions. Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z

Rater Cognition: Implications for Validity - Bejar, Issac I. (Educational Measurement: Issues and Practice, 2012)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.206Z

Comparison of Human and Machine Scoring of Essays: Differences by Gender, Ethnicity, and Country - Bridgeman, Brent; Trapani, Catherine; Attali, Yigal (Applied Measurement in Education, 2012)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

Entity resolution: theory, practice & open challenges - Getoor, Lise; Machanavajjhala, Ashwin (Proceedings of the VLDB Endowment, 2012)
This tutorial brings together perspectives on ER from a variety of fields, including databases, machine learning, natural language processing and information retrieval, to provide, in one setting, a survey of a large body of work. We discuss both the practical aspects and theoretical underpinnings of ER. We describe existing solutions, current challenges, and open research problems.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread

A Framework for Evaluation and Use of Automated Scoring - Williamson, David M.; Xi, Xiaoming; Breyer, F. Jay (Educational Measurement: Issues and Practice, 2012)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z

Computing and Evaluating Syntactic Complexity Features for Automated Scoring of Spontaneous Non-Native Speech - Chen, Miao; Zechner, Klaus (Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Fairness Through Awareness - Dwork, Cynthia; Hardt, Moritz; Pitassi, Toniann; Reingold, Omer; Zemel, Rich (arXiv, 2011)
We study fairness in classification, where individuals are classified, e.g., admitted to a university, and the goal is to prevent discrimination against individuals based on their membership in some group, while maintaining utility for the classifier (the university). The main conceptual contribution of this paper is a framework for fair classification comprising (1) a (hypothetical) task-specific metric for determining the degree to which individuals are similar with respect to the classification task at hand; (2) an algorithm for maximizing utility subject to the fairness constraint, that similar individuals are treated similarly. We also present an adaptation of our approach to achieve the complementary goal of "fair affirmative action," which guarantees statistical parity (i.e., the demographics of the set of individuals receiving any classification are the same as the demographics of the underlying population), while treating similar individuals as similarly as possible. Finally, we discuss the relationship of fairness to privacy: when fairness implies privacy, and how tools developed in the context of differential privacy may be applied to fairness.
Version Number: 2 <标题>: 通过认知实现公平 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z
Keywords: Computational Complexity (cs.CC); Computers and Society (cs.CY); FOS: Computer and information sciences

Analyzing language samples of Spanish–English bilingual children for the automated prediction of language dominance - Solorio, T.; Sherman, M.; Liu, Y.; Bedore, L. M.; Peña, E. D.; Iglesias, A. (Natural Language Engineering, 2010)
In this work we study how features typically used in natural language processing tasks, together with measures from syntactic complexity, can be adapted to the problem of developing language profiles of bilingual children. Our experiments show that these features can provide high discriminative value for predicting language dominance from story retells in a Spanish–English bilingual population of children. Moreover, some of our proposed features are even more powerful than measures commonly used by clinical researchers and practitioners for analyzing spontaneous language samples of children. This study shows that the field of natural language processing has the potential to make significant contributions to communication disorders and related areas.
<标题>: 分析西班牙语–英语双语儿童的语言样本以自动预测语言优势 <AI Smry>: The experiments show that features typically used in natural language processing tasks together with measures from syntactic complexity can provide high discriminative value for predicting language dominance from story retells in a Spanish–English bilingual population of children. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Data Mining for Education - Baker, R.S.J.d (International Encyclopedia of Education, 2010)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z

Automated scoring and feedback systems: Where are we and where are we heading? - Xi, Xiaoming (Language Testing, 2010)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:27.957Z

An Investigation into the Validity of Some Metrics for Automatically Evaluating Natural Language Generation Systems - Reiter, Ehud; Belz, Anja (Computational Linguistics, 2009)
There is growing interest in using automatically computed corpus-based evaluation metrics to evaluate Natural Language Generation (NLG) systems, because these are often considerably cheaper than the human-based evaluations which have traditionally been used in NLG. We review previous work on NLG evaluation and on validation of automatic metrics in NLP, and then present the results of two studies of how well some metrics which are popular in other areas of NLP (notably BLEU and ROUGE) correlate with human judgments in the domain of computer-generated weather forecasts. Our results suggest that, at least in this domain, metrics may provide a useful measure of language quality, although the evidence for this is not as strong as we would ideally like to see; however, they do not provide a useful measure of content quality. We also discuss a number of caveats which must be kept in mind when interpreting this and other validation studies.
<AI Smry>: The results of two studies of how well some metrics which are popular in other areas of NLP correlate with human judgments in the domain of computer-generated weather forecasts suggest that, at least in this domain, metrics may provide a useful measure of language quality, although the evidence for this is not as strong as one would ideally like to see. <标题>: 关于某些用于自动评估自然语言生成系统的指标有效性的研究 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.513Z
Keywords: /unread

Web-based essay critiquing system and EFL students' writing: a quantitative and qualitative investigation - Lee, Cynthia; Wong, Kelvin C.K.; Cheung, William K.; Lee, Fion S.L. (Computer Assisted Language Learning, 2009)
<标题>: 基于网络的论文批改系统与英语作为外语学生的写作：一项定量与定性研究 <AI Smry>: A web-based essay critiquing system developed by the authors using latent semantic analysis (LSA), an automatic text analysis technique, to provide students with immediate feedback on content and organisation for revision whenever there is an internet connection is described. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.531Z

SpeechRater™: A Construct-Driven Approach to Scoring Spontaneous Non-Native Speech - Zechner, Klaus; Higgins, Derrick; Xi, Xiaoming (2007)
<标题>: SpeechRater™：一种基于构念的非母语自发语音评分方法 <AI Smry>: An overview of the SpeechRater system of Educational Testing Service (ETS), a fully operational automated scoring system for non-native spontaneous speech employed in a practice context, which combines a selected set of speech features to predict a speaking score using multiple regression. Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.149Z

Toward More Substantively Meaningful Automated Essay Scoring - Ben-Simon, Anat; Bennett, Randy Elliot (Journal of Technology, Learning, and Assessment, 2007)
This study evaluated a “substantively driven” method for scoring NAEP writing assessments automatically. The study used variations of an existing commercial program, e-rater®, to compare the performance of three approaches to automated essay scoring: a brute-empirical approach in which variables are selected and weighted solely according to statistical criteria, a hybrid approach in which a fixed set of variables more closely tied to the characteristics of good writing was used but the weights were still statistically determined, and a substantively driven approach in which a fixed set of variables was weighted according to the judgments of two independent committees of writing experts. The research questions concerned (a) the reproducibility of weights across writing nexperts, (b) the comparison of scores generated by the three automated approaches, and (c) the extent to which models developed for scoring one NAEP prompt generalize to other NAEP prompts of the same genre. Data came from the 2002 NAEP Writing Online study and from the main NAEP 2002 writing assessment.
<标题>: 迈向更具实质意义的自动化作文评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:43.294Z

Evaluation of machine translation and its evaluation - Turian, Joseph P.; Shen, Luke; Melamed, I. Dan (Proceedings of Machine Translation Summit IX: Papers, 2006)
Evaluation of MT evaluation measures is limited by inconsistent human judgment data. Nonetheless, machine translation can be evaluated using the well-known measures precision, recall, and their average, the F-measure. The unigram-based F-measure has significantly higher correlation with human judgments than recently proposed alternatives. More importantly, this standard measure has an intuitive graphical interpretation, which can facilitate insight into how MT systems might be improved. The relevant software is publicly available from http://nlp.cs.nyu.edu/GTM/.
<AI Smry>: The unigram-based F-measure has significantly higher correlation with human judgments than recently proposed alternatives and has an intuitive graphical interpretation, which can facilitate insight into how MT systems might be improved. <标题>: 机器翻译及其评估的评价 Read_Status: New Read_Status_Date: 2026-01-26T11:33:38.512Z
Keywords: /unread

The Significance of Recall in Automatic Metrics for MT Evaluation - Lavie, Alon; Sagae, Kenji; Jayaraman, Shyamsundar (Machine Translation: From Real Users to Research, 2004)
Series Editors: _:n9635 Series Title: Lecture Notes in Computer Science <标题>: 召回率在机器翻译评估自动指标中的重要性 <AI Smry>: This work shows that correlation with human judgments is highest when almost all of the weight is assigned to recall, and shows that stemming is significantly beneficial not just to simpler unigram precision and recall based metrics, but also to BLEU and NIST. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics - Lin, Chin-Yew; Och, Franz Josef (Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04, 2004)
<AI Smry>: Two new objective automatic evaluation methods for machine translation based on longest common subsequence between a candidate translation and a set of reference translations and relaxes strict n-gram matching to skip-bigram matching are described. <标题>: 使用最长公共子序列和跳跃二元统计进行机器翻译质量的自动评估 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z
Keywords: /unread

Automatic Item Generation via Frame Semantics: Natural Language Generation of Math Word Problems - Deane, Paul; Sheehan, Kathleen M. (2003)
<标题>: 通过框架语义的自动题目生成：数学应用题的自然语言生成 Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

Automatic evaluation of machine translation quality using n-gram co-occurrence statistics - Doddington, George (Proceedings of the second international conference on Human Language Technology Research -, 2002)
<AI Smry>: NIST commissioned NIST to develop an MT evaluation facility based on the IBM work, which is now available from NIST and serves as the primary evaluation measure for TIDES MT research. Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

Automated Essay Scoring Using Bayes' Theorem - Rudner, Lawrence M.; Liang, T. (2002)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z
Keywords: ⛔ No DOI found

BLEU: a method for automatic evaluation of machine translation - Papineni, Kishore; Roukos, Salim; Ward, Todd; Zhu, Wei-Jing (Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02, 2001)
<AI Smry>: This work proposes a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. <标题>: BLEU：一种用于机器翻译自动评估的方法 Read_Status: New Read_Status_Date: 2026-01-26T11:33:34.519Z
Keywords: /unread

Automated Essay Scoring for Nonnative English Speakers - Burstein, Jill; Chodorow, Martin (Computer Mediated Language Assessment and Evaluation in Natural Language Processing, 1999)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Automated Essay Scoring for Nonnative English Speakers - Burstein, Jill; Chodorow, Martin (Proceedings of a Symposium on Computer Mediated Language Assessment and Evaluation in Natural Language Processing - ASSESSEVALNLP '99, 1999)
The e-rater system™ is an operational automated essay scoring system, developed at Educational Testing Service (ETS). The average agreement between human readers, and between independent human readers and e-rater is approximately 92%. There is much interest in the larger writing community in examining the system's performance on nonnative speaker essays. This paper focuses on results of a study that show e-rater's performance on Test of Written English (TWE) essay responses written by nonnative English speakers whose native language is Chinese, Arabic, or Spanish. In addition, one small sample of the data is from US-born English speakers, and another is from non-US-born candidates who report that their native language is English. As expected, significant differences were found among the scores of the English groups and the nonnative speakers. While there were also differences between e-rater and the human readers for the various language groups, the average agreement rate was as high as operational agreement. At least four of the five features that are included in e-rater's current operational models (including discourse, topical, and syntactic features) also appear in the TWE models. This suggests that the features generalize well over a wide range of linguistic variation, as e-rater was not confounded by non-standard English syntactic structures or stylistic discourse structures which one might expect to be a problem for a system designed to evaluate native speaker writing.
<标题>: 针对非英语母语者的自动作文评分 <AI Smry>: Results of a study that show e-rater's performance on Test of Written English (TWE) essay responses written by nonnative English speakers whose native language is Chinese, Arabic, or Spanish suggest that the features generalize well over a wide range of linguistic variation. Read_Status: New Read_Status_Date: 2026-01-26T11:33:53.530Z

2025-benchmarks-in-the-AI-lifecycle-diagram
<标题>: 人工智能生命周期图中的2025基准

Benchmark Evaluation Resources 10April2026
<标题>: 基准评估资源 2026年4月10日

Benchmarks 101
<标题>: 基准 101

How2Bench
<标题>: 如何测试

Inducing anxiety in large language models can induce bias
<标题>: 在大型语言模型中诱发焦虑会诱发偏见 Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments - Banerjee, Satanjeev; Lavie, Alon
We describe METEOR, an automatic metric for machine translation evaluation that is based on a generalized concept of unigram matching between the machine produced translation and human-produced reference translations. Unigrams can be matched based on their surface forms, stemmed forms, and meanings; furthermore, METEOR can be easily extended to include more advanced matching strategies. Once all generalized unigram matches between the two strings have been found, METEOR computes a score for this matching using a combination of unigram-precision, unigram-recall, and a measure of fragmentation that is designed to directly capture how well-ordered the matched words in the machine translation are in relation to the reference. We evaluate METEOR by measuring the correlation between the metric scores and human judgments of translation quality. We compute the Pearson R correlation value between its scores and human quality assessments of the LDC TIDES 2003 Arabic-to-English and Chinese-to-English datasets. We perform segment-bysegment correlation, and show that METEOR gets an R correlation value of 0.347 on the Arabic data and 0.331 on the Chinese data. This is shown to be an improvement on using simply unigram precision, unigram-recall and their harmonic F1 combination. We also perform experiments to show the relative contributions of the various mapping modules
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: /unread; ⛔ No DOI found

Responsible AI Standards - Burstein, Jill
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.207Z

Automated Scoring of Nonnative Speech Using the SpeechRaterSM v. 5.0 Engine - Chen, Lei; Zechner, Klaus; Yoon, Su-Youn
<标题>: 使用 SpeechRaterSM 5.0 引擎对非母语语音进行自动评分 Read_Status: New Read_Status_Date: 2026-01-26T11:33:57.130Z

Teacher Coaching - ACL preprint - demzsky
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: /unread

The m-rater™ engine: Introduction to the automated scoring of mathematics items - Fife, J. H. (Research Memorandum No. Rm-17-02)
This report provides an introduction to the m-rater™ engine, ETS’s automated scoring engine for computer-delivered constructed-response items when the response is a number, an equation (or mathematical expression), or a graph. This introduction is intended to acquaint the reader with the types of items that m-rater can score, the requirements for authoring these items onscreen, the methods m-rater uses to score these items, and the features these items must possess to be reliably scored. M-rater can score 3 types of responses—numeric responses, equations, and graphs—and these 3 types of responses are considered separately. Each type of response has certain technical requirements that must be satisfied and certain considerations for scoring. These requirements are discussed in detail in the Numeric Response Items, Equation Items, and Graph Items sections. Also discussed are generating example items—items with many correct solutions for which the student must produce 1 correct solution. The final chapter explains how to convert scoring rubrics into concepts and scoring rules, a necessary step for building m-rater scoring models.
<标题>: m-rater™ 引擎：数学题目自动评分简介 Read_Status: New Read_Status_Date: 2026-01-26T11:33:49.150Z

Math Education With Large Language Models: Peril or Promise? - Kumar, Harsh; Rothschild, David M.; Goldstein, Daniel G.; Hofman, Jake M.
The widespread availability of large language models (LLMs) has provoked both fear and excitement in the domain of education. On one hand, there is the concern that students will offload their coursework to LLMs, limiting what they themselves learn. On the other hand, there is the hope that LLMs might serve as scalable, personalized tutors. Here we conduct a large, pre-registered experiment involving 1200 participants to investigate how exposure to LLM-based explanations affect learning. In the experiment’s learning phase, we gave participants practice problems and manipulated two key factors in a between-participants design: first, whether they were required to attempt a problem before or after seeing the correct answer, and second, whether participants were shown only the answer or were also exposed to an LLM-generated explanation of the answer. Subsequently, all participants were tested on new test questions to assess how well they had learned the underlying concepts. Overall we found that LLM-based explanations positively impacted learning relative to seeing only correct answers. The benefits were largest for those who attempted problems on their own first before consulting LLM explanations, but surprisingly this trend held even for those participants who were exposed to LLM explanations before attempting to solve practice problems on their own. An accompanying qualitative analysis revealed that these boosts in performance were indeed due to participants adopting the strategies they were shown, and that exposure to LLM explanations increased the amount people felt they learned and decreased the perceived difficulty of the test problems.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:07.208Z
Keywords: ❓ Multiple DOI

Write On with Cambi: The development of an argumentative writing feedback tool - Lottridge, Susan; Burkhardt, Amy; Ormerod, Christopher; Woolf, Sherri; Young, Mackenzie; Patel, Milan; Wang, Harry; Frost, Julius; McBeth, Kevin; Benson, Julie; Flynn, Michael; Dwyer, Kevin; et al.
Every year, millions of middle-school students write argumentative essays that are evaluated against a scoring rubric. However, the scores they receive don’t necessarily offer clear guidance on how to improve their essay or what they’ve done well. With advancements in natural language processing technology, we now have the capability to provide more detailed feedback. At this juncture, we’ve developed an artificial intelligence-supported editing tool to assist students in revising their essays. In this paper, we introduce this tool. We then delve into its underlying components, covering how the feedback aligns with the rubric and standards, the techniques employed in modeling, and the process of crafting feedback. The paper concludes by summarizing the results obtained from interviews with teachers who have used the tool.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:25.498Z
Keywords: ⛔ No DOI found

Comparing Different Approaches to Generating Mathematics Explanations Using Large Language Models. - Prihar, E; Lee, M; Hopman, M; Kalai, A; Vempala, S; Wang, A; Wickline, G; Heffernan, N (AIED 2023)
Read_Status: New Read_Status_Date: 2026-01-26T11:33:10.101Z
Keywords: ⛔ No DOI found

Impact of LLM Feedback on Learner Persistence in Programming - Zhou, Yiqiu; Pankiewicz, Maciej; Paquette, Luc; Baker, Ryan S
This study examines how Large Language Model (LLM) feedback generated for compiler errors impacts learners’ persistence in programming tasks within a system for automated assessment of programming assignments. Persistence, the ability to maintain effort in the face of challenges, is crucial for academic success but can sometimes lead to unproductive "wheel spinning" when students struggle without progress. We investigated how additional LLM feedback based on the GPT-4 model, provided for compiler errors affects learners’ persistence within a CS1 course. Specifically, we examined whether its impacts differ based on task difficulty, and if the effects persist after the feedback is removed. A randomized controlled trial involving 257 students across various programming tasks was conducted. Our findings reveal that LLM feedback improved some aspects of students’ performance and persistence, such as increased scores, a higher likelihood of solving problems, and a lower tendency to demonstrate unproductive "wheel spinning" behavior. Notably, this positive impact was also observed in challenging tasks. However, its benefits did not sustain once the feedback was removed. The results highlight both the potential and limitations of LLM feedback, pointing out the need to promote long-term skill development and learning independent of immediate AI assistance.
Read_Status: New Read_Status_Date: 2026-01-26T11:33:20.825Z
Keywords: ⛔ No DOI found