The Documentation Quality Score (E7): An Explainable Composite Metric for AI-Assisted Clinical Note Generation in Ontario Home Care

1. Introduction

1.1 The Rise of AI-Assisted Clinical Documentation

The documentation burden in home and community care is substantial. Personal Support Workers (PSWs) and registered nurses conducting home visits are expected to produce structured clinical records immediately following visits that may be both time-pressured and emotionally demanding. Incomplete or delayed documentation creates downstream risks: inaccurate billing, missed clinical escalation triggers, audit failures, and erosion of trust between care teams and the clients they serve.

Large language models (LLMs) can generate clinically plausible narrative notes from structured inputs, voice transcripts, and brief natural-language descriptions of visit activities. Ontario home care, however, presents a distinct set of regulatory and workflow constraints that generic clinical AI platforms do not fully address.

1.2 The Problem with a Single Confidence Number

The standard output of an AI documentation system is a generated note accompanied by a model confidence score — typically a self-reported probability between 0 and 1. This design conflates two fundamentally different questions: How sure is the model of its own output? and How well does that output reflect what actually happened during the visit? These are not the same question, and they require different evidence to answer.

LLM self-confidence is known to be poorly calibrated. A model generating fluent, grammatically complete prose about a visit it has no evidence for will often report high confidence precisely because the generated text is internally coherent — not because it is grounded in the clinical record. In regulated healthcare settings, a note that confidently fabricates or omits visit data is a compliance liability, not an acceptable documentation artifact.

1.3 Contributions of This Paper

1. We describe the Documentation Quality Score (E7), a composite 0–100 metric with eight explainable weighted dimensions, implemented in CareScribe for Ontario home-care SOAP note generation.
2. We formally define transcript grounding as a sentence-level coverage measure linking generated SOAP text to visit source evidence, distinguishing it from citation counting.
3. We describe a grounding-capped confidence calibration approach that reduces AI overconfidence without requiring logprob access.
4. We distinguish generation quality from workflow assurance as two interpretable sub-scores.
5. We discuss design tradeoffs between heuristic and semantic grounding, and outline a roadmap toward empirical validation against supervisor approval outcomes.

2. Related Work

2.1 Clinical NLP Quality Metrics

Evaluation of clinical text generation has predominantly relied on n-gram overlap metrics (BLEU, ROUGE) adapted from machine translation. While computationally convenient, these measure surface similarity to a reference text and are poorly suited to home-care documentation, where no gold-standard reference note exists at generation time and the relevant comparison is to the visit's source evidence, not a human-written exemplar.

2.2 Grounding and Faithfulness in Generated Clinical Text

The concept of grounding — requiring that generated claims be traceable to source evidence — has been formalized in several clinical NLP contexts. Prior work showed that standard ROUGE scores correlate poorly with clinician judgments of faithfulness, and model-based factual consistency checkers such as FactCC have been adapted for clinical domains. The E7 approach is inspired by this body of work but prioritizes operational deployability: grounding checks are heuristic and reproducible without embedding models, with semantic methods identified as future work.

2.3 Human-in-the-Loop Documentation Systems

Human oversight of AI-generated clinical documentation is increasingly recognized as a governance requirement, not merely a quality preference. Regulatory frameworks (the FDA's proposed approach to AI/ML-based software as a medical device, and Health Canada's analogous guidance) require that human review be an integral part of the documentation workflow. CareScribe's rescoring model — which updates E7 as workers complete attestation, review clinical events, and receive supervisor approval — implements this requirement at the metric level.

2.4 PHIPA and Ontario Documentation Compliance

Ontario's Personal Health Information Protection Act (PHIPA) requires that personal health information be accurate, complete, and subject to appropriate custodianship. For AI-generated documentation this translates to three auditable requirements: the note must be traceable to source data, the generating system must record provenance information, and human review must be demonstrably part of the workflow before information is shared with the broader circle of care. E7 addresses all three through its audit readiness dimension and integration with the CareScribe approval pipeline.

3. Methods: The Eight-Dimension Composite Score

3.1 Score Architecture Overview

The Documentation Quality Score E7 is a composite integer between 0 and 100 computed automatically for each AI-generated SOAP document. The score is structured across two signal families: Generation / Evidence Signals (60 points), which assess how well the note reflects captured visit data, and Workflow / Compliance Signals (30 points), which assess whether the human-in-the-loop approval process has been followed. An additional 10 points derive from the calibrated AI confidence signal, which sits at the intersection of both families.

Each dimension has a defined weight, a pass/warn/fail threshold, and a human-readable rationale for any deduction. The full breakdown is stored in documents.doc_quality_breakdown alongside the aggregate documents.doc_quality_score, enabling per-check auditing without inspecting the model's internals.

3.2 Dimension Weights and Thresholds

4. Generation & Evidence Signals (60 Points)

4.1 Transcript Grounding (20 Points)

Transcript grounding is the primary evidence-quality signal in E7. Its central claim is that a high-quality AI-generated note is not merely grammatically complete — it is textually traceable to the visit source narrative. Grounding is defined at the sentence level across all four SOAP sections. A sentence is classified as grounded if any of the following holds:

1. An evidence span explicitly linked to it has its span text present in the visit transcript.
2. An evidence-backed statement is associated with it via the evidence extraction pipeline.
3. The stopword-filtered token overlap between it and the full visit transcript exceeds 40%.

The grounding ratio is the proportion of grounded sentences across the note, and the transcript grounding score scales that ratio to the dimension's 20-point weight:

Thresholds are applied as pass (≥75%), warn (≥50%), and fail (<50%). This moves beyond citation counting toward coverage of generated clinical text: a note that cites a transcript once but generates three additional ungrounded sentences is not well-grounded, even though citations exist.

The heuristic overlap threshold (40%, stopword-filtered) was chosen for operational deployability — it requires no embedding model, is deterministic and reproducible, and produces results a clinical reviewer can verify manually. Embedding-based grounding is identified as the primary future improvement.

4.2 Structured Capture Completeness (15 Points)

Home-care visits produce structured data through the PSW mobile interface: a task checklist with completion flags and wellness chip selections representing observed client states, persisted in visits.capture_metadata. Completed tasks are matched against the Objective and Plan sections; wellness chip labels are matched against the Subjective and Objective sections. The completeness ratio is:

A note that ignores these is missing the structured foundation of the visit record, regardless of how fluent its prose may be.

4.3 Capture Data Quality Penalties (10 Points)

Note quality is a function not only of what was written, but of what raw evidence was available. This dimension encodes a graded evidence hierarchy through a cumulative penalty model:

4.4 SOAP Section Completeness (15 Points)

A structurally complete SOAP note requires substantive content across all four sections — Subjective, Objective, Assessment, and Plan. Sections that are present but minimal (e.g., a single-sentence Assessment) are treated as warn-level rather than pass-level, reflecting the clinical reality that a perfunctory Assessment provides insufficient guidance for the next care provider.

4.5 Calibrated AI Confidence (10 Points)

Raw LLM self-confidence is unreliable as a quality proxy. CareScribe applies a two-stage calibration that does not require logprob access. At generation time the model's self-reported confidence is immediately capped:

At scoring time a further grounding cap is applied:

This ensures that if a note is later determined to be poorly grounded, the confidence score reflects that revision rather than preserving an inflated self-report. The effective_confidence is mapped to the 10-point scale with pass (≥0.80), warn (≥0.60), and fail (<0.60) thresholds — aligning with the principle that AI confidence should never exceed the evidentiary support for the generated content.

5. Workflow & Compliance Signals (30 Points)

The workflow signals address a different governance question from the generation signals: not how well was the note written, but has the correct human oversight process been followed before this note enters the permanent clinical record?

5.1 Clinical Event Review (10 Points)

CareScribe's evidence extraction pipeline identifies clinically significant events — falls, pain reports, medication concerns, behavioral changes — and surfaces them to the PSW as a review list before submission. This dimension measures the fraction of extracted events reviewed (acknowledged, dismissed, or escalated):

A note with unreviewed clinical events is not safe for supervisor submission.

5.2 Worker Attestation (10 Points)

Worker attestation captures the PSW's sign-off before supervisor approval. CareScribe tracks five states:

• Pending — not yet reviewed fail
• Uncertain — flagged for clarification fail
• Rejected — rejected, requires manual rewrite fail
• Edited — reviewed and modified warn
• Approved — reviewed and approved without modification pass

The dimension scores 10 for approved, 7 for edited (human correction is a positive governance signal), and 0 for pending, uncertain, or rejected. Edited notes trigger a full rescore of E7, including re-evaluation of grounding and completeness against the worker's changes.

5.3 Audit Readiness (10 Points)

Audit readiness consolidates PHIPA provenance requirements into a single check with five sub-components: (1) transcript presence, (2) transcript source type recorded, (3) minimum content length, (4) evidence-chain integrity, and (5) document workflow status consistency. Each contributes proportionally to the 10-point score. The check is immutable once a document reaches published status — the final score and breakdown are frozen in the audit log alongside a compliance.scored event.

6. Grade Bands, Aggregation & Dashboards

6.1 Grade Bands

6.2 Score Evolution: A Representative Example

6.3 Organizational-Level Aggregation

The organizational quality metric is the arithmetic mean of E7 scores across all published documents in a configurable time window, excluding documents without a computed score. This aggregate is available through the CareScribe supervisor and administrator dashboards, enabling quality trend monitoring at the team, site, and organization level. Compliance events are written to an immutable audit log at each score computation — recording document identifier, score, grade, fail count, and the identities of the associated worker and supervisor.

7. System Implementation

7.1 Pipeline Placement and Trigger Points

E7 computation runs automatically after SOAP note generation, risk extraction, and evidence extraction are complete, ensuring the grounding and capture-completeness checks have access to the full evidence output. Rescoring is triggered by four events: note edit, clinical event review completion, worker attestation status change, and supervisor approval. Each rescore produces a new doc_quality_breakdown record linked to the triggering event, preserving a complete audit trail of score evolution.

7.2 Data Model

• documents.doc_quality_score — integer 0–100, the current aggregate score.
• documents.doc_quality_breakdown — JSONB containing the full per-dimension breakdown (status, score, rationale) for each of the eight checks.
• visits.capture_metadata — JSONB storing task lists, wellness chips, transcript source flags, and completion states used by the capture dimensions.

The JSONB breakdown is designed for forward compatibility: new dimensions can be added without schema migration, and historical breakdowns remain queryable even after dimension weights change.

7.3 User Interface: The DocQualityPanel

The E7 score and breakdown are surfaced through the DocQualityPanel, an expandable component on both the Review and Documentation pages. It displays the aggregate score, grade band, and a per-dimension list showing status, point contribution, and rationale for any warn or fail. A supervisor reviewing a note scored 68 (Needs Review) can see immediately that the primary deductions are transcript grounding (warn: 61% grounded) and worker attestation (pending), rather than receiving an opaque low-confidence warning that provides no actionable guidance.

8. Discussion

8.1 Generation Quality vs. Workflow Assurance

The two-family architecture reflects a deliberate governance choice. Separating the sub-scores lets organizations diagnose failures at the right level: a persistent pattern of low generation sub-scores suggests a data-capture or model-quality problem, while low workflow sub-scores suggest a training or adoption problem with the PSW or supervisor population. Conflating these into a single number obscures the diagnostic information.

8.2 Heuristic Grounding vs. Semantic Methods

Stopword-filtered token overlap is transparent, deterministic, and auditable in a way embedding similarity is not — a clinical reviewer can reproduce the check manually. The limitation is real: a sentence that paraphrases a transcript with different vocabulary may be classified as ungrounded. Embedding-based semantic similarity using a domain-adapted clinical model is the natural upgrade path for E7 v2, at the cost of auditability.

8.3 Confidence Calibration

The grounding-capped approach is available through any LLM API and interpretable without statistical training, but it does not learn from feedback. Future work will explore fitting a Platt scaling or isotonic regression calibrator against supervisor approval/rejection outcomes, using the E7 score trajectory as the calibration target while preserving the grounding-capped floor as a lower bound.

8.4 Scope and Generalization

E7 is designed and validated for Ontario home-care PSW workflows. The PHIPA audit dimension, PSW attestation workflow, and wellness-chip capture structure are Ontario-specific; other jurisdictions should expect to adjust weights and thresholds. The two-family architecture and grounding-based calibration are not Ontario-specific and should generalize to other AI-assisted clinical documentation contexts.

9. Limitations

10. Conclusion

We have described the Documentation Quality Score (E7), a composite, explainable 0–100 metric for AI-assisted SOAP note generation in Ontario home care. E7 integrates eight weighted dimensions across two signal families into a single governance-oriented score that updates dynamically as human-in-the-loop approval actions are completed.

The central design principle is attributability: every point deduction is traceable to a named, auditable check rather than an opaque model output. A score that cannot be explained to a PHIPA auditor, a hospital quality officer, or a retirement-home medical director is not a governance tool; it is another source of unexamined AI risk. The most important next step is correlation analysis against supervisor approval outcomes on real visit data.

References

1. Sinsky C, Colligan L, Li L, et al. "Allocation of Physician Time in Ambulatory Practice." Annals of Internal Medicine, 2016. link
2. Giorgi S, Zamani M, Jo Y, et al. "Clinical Text Generation with Large Language Models." arXiv, 2024. link
3. Kadavath S, Conerly T, Askell A, et al. "Language Models (Mostly) Know What They Know." arXiv, 2022. link
4. Papineni K, Roukos S, Ward T, Zhu W-J. "BLEU: A Method for Automatic Evaluation of Machine Translation." ACL 2002. link
5. Maynez J, Narayan S, Bohnet B, McDonald R. "On Faithfulness and Factuality in Abstractive Summarization." ACL 2020. link
6. Abacha AB, Shivade C, Demner-Fushman D. "Investigation of Transfer Learning for Named Entity Recognition." BioNLP 2019. link
7. Kryscinski W, McCann B, Xiong C, Socher R. "Evaluating the Factual Consistency of Abstractive Text Summarization." EMNLP 2020. link
8. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations." Science, 2019. link
9. Health Canada. "Software as a Medical Device (SaMD): Definition and Classification." 2019. link
10. Ontario Legislative Assembly. Personal Health Information Protection Act, 2004, S.O. 2004, c. 3, Sched. A. link
11. Guo C, Pleiss G, Sun Y, Weinberger KQ. "On Calibration of Modern Neural Networks." ICML 2017. link