Background Postoperative complications (PCs) rates are crucial quality metrics in surgery, as they reflect both patient outcomes, perioperative care effectiveness and healthcare resource strain. Despite their importance, efficient, accurate, and affordable methods for tracking PCs are lacking. This study aimed to evaluate whether natural language processing (NLP) models could detect eleven PCs from surgical electronic health records (EHRs) at a level comparable to human curation.

Methods 17 486 surgical cases from 18 hospitals across two regions in Denmark, spanning six years, were included. The dataset was divided into training, validation, and test sets for NLP-model development and evaluation (50.2%/33.6%/16.2%). Model performance was compared against the current method of PC monitoring (ICD-10 codes) and manual curation, the latter serving as the gold standard.

Results The NLP-models had a ROC AUC between 0.901 to 0.999 for the test set. Sensitivity of the models when compared to manual curation ranged from 0.701 to 1.00, except for myocardial infarction (0.500). Positive Predictive Value (PPV) ranged from 0.0165 to 0.947, and Negative Predictive Value from 0.995 to 1.00. The NLP-models significantly outperformed ICD-10 coding in detecting PC, resulting in 16.3% of cases would require manual curation to reach a PPV of 1.00

Conclusion The NLP models alone were able to detect PCs at an acceptable level and performed superior to ICD-10 codes. Combining NLP based and manual curation was required to reach a PPV of 1.00. Therefore, NLP algorithms present a potential solution for comprehensive and real-time monitoring of PCs across the surgical field.

Conflicts of interest: MS, AB and AT have founded Aiomic, a company developing AI models for healthcare systems. This study incorporates methodological scientific validations of the Aiomic product and underlying intellectual property, but the specific models within the study are not used for commercial purposes.

Funding: Supported by a grant (#NNF19OC0055183) from the Novo Nordisk Foundation to MS

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The study received approval from the Institutional Review Board (IRB) for retrospective patient studies in Denmark, the Danish Patients Safety Board (Styrelsen for Patientsikkerhed, approval #31-1521-182), and the Danish Capital Region Data Safety Board (Videnscenter for Dataanmeldelser, approval #P-2020-180). As it was retrospective, utilized de-identified data, and involved no patient contact, informed consent was not required under Danish law.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Data availability statement: Due to GDPR-regulations and as this study utilizes patient sensitive data, the authors are not permitted to share the data without authorization. The dataset can be accessed with authorization from the Danish Patients Safety Board (Styrelsen for Patientsikkerhed) and the Danish Capital Region Data Safety Board (Videnscenter for Dataanmeldelser).