This study explores the potential of the large language model GPT-4 as an automated tool for qualitative data analysis by educational researchers, exploring which techniques are most successful for different types of constructs. Specifically, we assess three different prompt engineering strategies-Zero-shot, Few-shot, and Few-shot with contextual information-as well as the use of embeddings. We do so in the context of qualitatively coding three distinct educational datasets: Algebra I semi-personalized tutoring session transcripts, student observations in a game-based learning environment, and debugging behaviors in an introductory programming course. We evaluated each approach's performance based on its inter-rater agreement with human coders and explored how different methods vary in effectiveness depending on a construct's degree of clarity, concreteness, objectivity, granularity, and specificity. Our findings suggest that while GPT-4 can code a broad range of constructs, no single method consistently outperforms the others, and the selection of a particular method should be tailored to the specific properties of the construct and context being analyzed. We also found that the constructs that GPT-4 has the most difficulty with are the same constructs than human coders find more difficult to reach inter-rater reliability on. Notes for Practice (research paper)  GPT-4 can be used to code qualitative data for educationally-relevant constructs.  Using embeddings and examples can improve agreement with humans. Examples are more useful for constructs that are more difficult to define.  Constructs that human beings find difficult to agree on are also difficult for GPT-4.

Citation Key: zotero-12383