ACR’s Assess-AI: A Registry for Real-World Performance Monitoring of Clinical Imaging AI
Journal of the American College of Radiology, 2026
ACR's Assess-AI is a national registry developed to monitor the real-world performance of clinical imaging AI after deployment. The registry was designed to support post-deployment monitoring of imaging AI. This is important because AI tools may not maintain the same performance in routine practice as observed during regulatory evaluation, particularly when local data, equipment, workflows, or patient populations differ.
Assess-AI collects de-identified AI outputs, radiology report text, and DICOM metadata through ACR Connect. It then uses large language model–based pipelines to extract surrogate labels from reports and compare them with AI outputs, calculating concordance and discordance. Results are presented through interactive dashboards that allow facilities to monitor trends, benchmark performance, and investigate discrepancies. Local review of discordant cases is supported through the Forensics App.
The article concludes that Assess-AI offers a scalable, privacy-preserving infrastructure for AI governance, quality improvement, and post-deployment monitoring in radiology, with future expansion planned for additional model types and modalities.
With the initiative now expanding to Europe, it is one to watch for anyone interested in the real-world performance, safety, and governance of imaging AI.
Currently listed as a Journal Pre-proof article