Automated Calculation of the Pediatric Emergency Care Applied Research Network (PECARN) IntraAbdominal Injury Rule from Unstructured Emergency Department Notes Using a Large Language Model

The Pediatric Emergency Care Applied Research Network (PECARN) intra-abdominal injury clinical prediction rule identifies children at very low risk for clinically significant intra-abdominal injury after blunt abdominal trauma. However, retrospective evaluation of the rule’s use in practice requires resource-intensive chart review. We investigated whether large language models (LLMs) can automatically calculate PECARN intra-abdominal injury risk scores from real-world unstructured emergency department (ED) notes.

We conducted a retrospective secondary analysis of children after blunt abdominal trauma presenting to 2 urban pediatric EDs. The LLM, OpenAI o1, was compared with clinician-derived standards for PECARN risk scores and predictor variable extraction from unstructured physician ED notes. Performance was assessed using standard diagnostic test characteristics.

Among 196 children with complete ED notes, the LLM calculated binary PECARN risk scores with 95.4% accuracy (95% confidence interval [CI] 92.5 to 98.3), 94.4% sensitivity (95% CI 90.6 to 98.2), 98.2% specificity (95% CI 94.6 to 100), and an F1 score of 0.97 (95% CI 0.90 to 0.98). Agreement by weighted Cohen’s kappa between multicategorical LLM-derived risk scores and clinician-derived risk scores was high, 0.96 (95% CI 0.92 to 1.00). Predictor variable extraction demonstrated strong performance across all PECARN variables, with accuracy greater than 94%.

The LLM had comparable performance to clinicians in calculating binary and multicategorical PECARN intra-abdominal injury risk scores and extracting predictor variables from unstructured ED notes. LLMs may enable automated calculation of clinical prediction rules from clinical documentation.