In an interview featured in Healthcare IT News, clinical data expert Betsy Castillo, RN, explains how healthcare organisations engaging AI-software vendors must proceed with caution. She emphasises that while many AI vendors bring strong technical capabilities, they may lack the clinical domain expertise required to operate reliably in healthcare settings. When tools are built without deep involvement of clinicians and data stewards, the result can be mis-interpretations of documentation, added workload for staff, and ultimately weaker trust in the system.
One significant risk highlighted is poor alignment with clinical workflows and quality data standards. If AI tools mislabel or mis-abstract patient records, the errors don't remain isolated—they can cascade into dashboards, performance metrics and decision-making frameworks across the health system. Castillo observes that even “mathematically sound” outputs might be clinically irrelevant or misleading. Hospitals relying on such outputs may incur wasted effort, delayed improvement initiatives or worse, compromised patient outcomes.
For health IT leaders evaluating AI vendors, Castillo lays out a checklist of questions: Who helped design and validate the system? How does the system handle ambiguous or unstructured data common in healthcare settings? Can every AI-generated data point be traced back to a source record for auditability? She warns that vendors who claim to “eliminate” human oversight are a red flag—clinician judgement remains indispensable in healthcare, and AI should be built to amplify rather than replace that judgement.
The article concludes by advising organisations to insist on measurable outcomes, not just sales promises. Real-world success is shown by saved labour hours, improved data quality or faster turnaround times in quality reporting—preferably documented across multiple implementations. Without such evidence, health systems risk investing in tools that underperform or become shelf-ware. In short: the promise of AI in healthcare is real, but so are the risks—and partnering with the right vendor means ensuring the solution is clinically and operationally grounded.
