A recent study has shed light on the limitations of artificial intelligence in educational settings. Researchers at the University of Wollongong and UNSW Sydney developed a chatbot called SmartTest to assist with teaching a university law class. However, the AI-powered chatbot struggled to provide accurate feedback, particularly when dealing with complex problem-scenario questions.
The study found that SmartTest provided inaccurate, misleading, or incorrect feedback in 40-54% of conversations when tackling complex questions. Even when the questions were simplified to short-answer format, the error rate remained significant, ranging from 6-27%. The researchers noted that the chatbot's behavior was unpredictable, offering excellent feedback at times and incorrect information at others.
The study also revealed that students preferred human feedback over AI-generated feedback, with 76% of students appreciating the immediate feedback and conversational tone of SmartTest but only 27% preferring AI feedback. The researchers concluded that while AI may be useful for low-stakes, formative learning activities, it lacks the reliability, nuance, and subject-matter depth required for many educational contexts.
The findings highlight the need for caution when integrating generative AI into educational settings, emphasizing the importance of rigorous evaluation to ensure that AI-powered tools do not compromise educational outcomes. As AI technology continues to evolve, it's essential to carefully consider its limitations and potential applications in education.
