TL;DR
An Ontario audit of 20 AI note-taking systems revealed that many routinely produce inaccurate or fabricated medical information. The findings highlight concerns about AI reliability in healthcare documentation. The systems’ evaluation criteria may have underestimated the importance of accuracy and safety.
The Ontario Office of the Auditor General has found that 9 out of 20 AI note-taking systems used by healthcare providers routinely produce inaccurate or fabricated patient records, raising concerns about their safety and reliability.
The audit evaluated 20 AI systems approved for use in Ontario’s healthcare sector, using simulated doctor-patient recordings. It revealed that nine systems fabricated information, such as suggesting treatments or symptoms not discussed during the recordings. Twelve systems inserted incorrect drug information into patient notes, and 17 missed key mental health details discussed during consultations.
While OntarioMD, a support group for physicians, recommends manual review of AI-generated notes, none of the approved systems include mandatory attestation features to verify accuracy. The evaluation process itself was criticized for giving disproportionate weight to criteria unrelated to accuracy, such as vendor presence in Ontario, which accounted for 30 percent of the score, while accuracy contributed only 4 percent.
Why It Matters
The findings raise serious questions about the safety and effectiveness of AI tools used in medical documentation. Inaccurate or fabricated records can lead to misdiagnosis, inappropriate treatment, and patient harm. The report suggests that current evaluation standards may inadequately prioritize accuracy and safety, potentially allowing subpar systems to be approved for clinical use.
Plaud Note AI Voice Recorder, Note Taker w/Case, App Control, Transcribe & Summarize with AI, Support 112 Languages, for Meetings, Calls, Lectures, Professionals, Teams, Black, Non-Pro Version
Plaud Intelligence: Capture conversations in 112 languages and generate accurate transcripts with the Plaud App and Web. Plaud…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background
The use of AI for medical note-taking has expanded rapidly in Ontario, with more than 5,000 physicians participating in the program. The initiative aims to improve efficiency but has faced scrutiny over the quality of AI-generated documentation. Previous studies have shown that large language models often produce incorrect medical information, but this is the first formal audit highlighting systemic issues in AI scribe systems used in healthcare.
“Inaccurate weightings could result in the selection of vendors whose AI tools may produce inaccurate or biased medical records or lack adequate protection to safeguard sensitive personal health information.”
— Office of the Auditor General of Ontario
“More than 5,000 physicians in Ontario are participating in the AI Scribe program and there have been no reports of patient harms associated with the technology so far.”
— Ontario Ministry of Health spokesperson
AI for Therapists: The Practical Guide to HIPAA-Compliant AI Tools, Prompt Engineering, and Ethical Workflows for Mental Health Professionals (AI for Professionals)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What Remains Unclear
It remains unclear whether the issues identified are widespread across all AI scribe systems or limited to the evaluated sample. The impact on patient safety and clinical outcomes is still being assessed, and the Ministry has not yet announced specific corrective actions or changes to evaluation processes.
GENERATIVE AI IN THE CLINIC: Ambient Documentation Systems
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What’s Next
The Ontario Ministry of Health is expected to review the audit findings and may revise its evaluation criteria for AI systems. Further testing and monitoring are likely, along with possible updates to regulations or mandatory review procedures for AI-generated medical records.
Systematic Review and Meta-Analysis: Stepwise Approach for Medical and Biomedical Researchers
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Are AI note-taking systems currently safe to use in Ontario healthcare?
While no reports of patient harm have been confirmed, the audit raises concerns about the accuracy and safety of these AI systems, suggesting caution and the need for manual review.
What specific errors did the Ontario audit find in AI-generated medical notes?
The audit found that many systems fabricated information, inserted incorrect drug details, and missed key mental health issues discussed during patient consultations.
Will the government change how AI systems are approved for healthcare use?
The Ministry of Health is expected to review the audit results and may adjust its evaluation process, emphasizing accuracy and safety criteria more heavily.
Could these errors lead to patient harm?
Potentially, yes. Incorrect or fabricated medical records can cause misdiagnosis or inappropriate treatment, though no direct harm has been reported yet.
