The US Food and Drug Administration (FDA) has been using an artificial intelligence tool called Elsa to help with daily tasks, such as drafting reviews and summarizing documents. However, insiders have raised concerns about the tool's reliability, citing its tendency to "hallucinate confidently" and provide incorrect answers on research areas and drug labels.
According to reports, Elsa has been generating fake studies and misrepresenting research, which could have serious implications for public health. The tool's inability to link to third-party citations and its lack of access to crucial internal documents make it unreliable for critical tasks.
Despite these concerns, the FDA is continuing to use and expand Elsa's role in the approval process. However, employees are spending more time double-checking the AI's output, which defeats the purpose of using AI to speed up the process.
Experts warn that the technology is moving too fast, and safety protocols are lagging behind. Dr. Jonathan Chen, a Stanford University professor, notes that the FDA's use of AI tools like Elsa highlights the need for more robust testing and validation procedures.
The FDA's use of Elsa raises important questions about the role of AI in high-stakes decision-making and the potential risks to public health. As the agency continues to rely on AI tools, it's essential to prioritize transparency, accountability, and rigorous testing to ensure that these tools are safe and effective.
