Large Language Models (LLMs) face documented challenges in solving mathematical problems. While substantial work has been done to quantify and improve LLMs’ abilities to solve static math problems, evaluating their performance in real-time math tutoring scenarios presents distinct challenges that remain underexplored. This paper specifically addresses the accuracy of LLMs in performing math correctly while tutoring students. It highlights the unique difficulties of this context, classifies types of interactions students may have with an LLM, presents a dataset, Conversation-Based Math Tutoring Accuracy Dataset (CoMTA Dataset), for evaluating the mathematical accuracy in tutoring scenarios, and discusses techniques to address these issues. Additionally, it evaluates the mathematical accuracy of a range of models in LLM-based tutoring.