Automatic item generation may supply many items instantly and efficiently to assessment and learning environments. Yet, the evaluation of item quality persists to be a bottleneck for deploying generated items in learning and assessment settings. In this study, we investigated the utility of using large‐language models, specifically Llama 3‐8B, for evaluating automatically generated cloze items. The trained large‐language model was able to filter out majority of good and bad items accurately. Evaluating items automatically with instruction‐tuned LLMs may aid educators and test developers in understanding the quality of items created in an efficient and scalable manner. The item evaluation process with LLMs may also act as an intermediate step between item creation and field testing to reduce the cost and time associated with multiple rounds of revision.

Citation Key: gorgun2024 <标题>: 用于自动题目生成质量控制的指令调优大型语言模型：可行性研究 <AI Smry>: The utility of using large‐language models, specifically Llama 3‐8B, for evaluating automatically generated cloze items was investigated and the trained large‐language model was able to filter out majority of good and bad items accurately.