During the early stages of the Covid-19 pandemic, hospitals were facing a significant challenge in managing the influx of seriously ill patients. To address this issue, many hospitals turned to artificial intelligence algorithms, such as the one developed by Epic Systems, to help predict which patients were at a higher risk of deteriorating rapidly. This allowed hospitals to prioritize critical care for those who needed it the most.
However, the effectiveness of these proprietary AI algorithms has been called into question, as many health systems have implemented them without a clear understanding of how well they actually perform. In a recent study conducted by Yale’s health system, researchers evaluated the performance of six early warning scores using clinical data from seven hospitals. The results, published in JAMA Network Open, revealed that some clinical AI models may not be as reliable as previously thought.
Co-author Deborah Rhodes, who serves as the chief quality officer for Yale New Haven Health System and associate dean of quality for Yale School of Medicine, emphasized that the goal of the study was to identify the best tool available for healthcare providers. Despite the widespread use of Epic’s early warning score due to its integration with the company’s electronic health record, the study found that the tool may not be as effective as believed.
The study highlights the importance of rigorously evaluating the performance of AI algorithms in healthcare settings to ensure that they are providing accurate and reliable information. As the use of AI in healthcare continues to grow, it is essential for health systems to choose tools that have been thoroughly validated and proven to be effective.
Moving forward, more research and analysis will be needed to determine the best practices for implementing AI algorithms in healthcare settings. By conducting thorough assessments of these tools, healthcare providers can ensure that they are making informed decisions that will ultimately benefit patient care and outcomes.