Abstract

Background: Artificial Intelligence (AI) has advanced tremendously over the past few years and has demonstrated great potential in supporting existing healthcare systems. Due to their rapid data processing, AI tools are increasingly being used by people to assess health concerns and make preliminary decisions about seeking health care. This study aims to analyse the diagnostic accuracy of AI tools in determining common pediatric emergencies.

Methods: 120 pediatric case reports were collected from open access, peer reviewed journals based on the inclusion criteria. The cases were rewritten in layman language and entered into three AI tools : ChatGPT, WebMD Symptom Checker and Mayo Clinic symptom checker and the top three differential diagnoses were analyzed. A standard 3-point scoring system was used to assess the diagnostic performance of each tool by two independent reviewers. Statistical analysis of the scores were done using Friedman’s test followed by post hoc pairwise comparisons with Bonferroni correction.

Results: Analysis showed that ChatGPT had the highest average mean of 0.95 ± 0.94 and was ranked the highest with a mean rank of 2.42 out of the three tools studied. Friedman’s test and post hoc analysis with Bonferroni correction confirmed the statistical significance (p < .001). There was no statistical difference between the scores for WebMD and Mayo Clinic symptom checkers (adjusted p = 1.000).

Conclusion: With increasing access and a large number of people resorting to AI as a reliable tool for diagnosis, ChatGPT seems to show potential in this regard. Although the diagnostic accuracy seems significant , it is important to consider the limitations of such tools which do not take into consideration the contextual information, social determinants of health and emotional aspects of clinical presentation.

Keywords: Artificial Intelligence; Pediatrics; Emergency; Diagnosis; Symptom Checkers; Accuracy

Introduction

Artificial Intelligence (AI) is a machine-learning process that mimics human cognitive abilities by drawing inferences from new and existing data. Over the years, the use of AI has exhibited great potential in the field of medical diagnostics, which is quite complex and requires extensive training and experience to be able to gauge the differentials from presenting signs and symptoms. AI has shown promise in the context of medical diagnosis, appropriate testing and treatment options, even the possibility of identifying management plans for rare conditions. This can be attributed to the large datasets that AI has access to and its ability to recognize patterns which might be beyond human capacity [1]. While AI tools offer an easy-to-use, rapid and accessible technology to its users, their diagnostic accuracy, especially when it comes to the pediatric age group, is a matter of concern. Inaccurate assessments and incorrect diagnoses given by them could lead to delay in patients getting critical care or unnecessary visits to the Emergency Department on the other hand. Limited studies exist in current literature that study the reliability and diagnostic accuracy of these tools in assessing pediatric emergency cases. In order to address this gap, we conducted a retrospective, cross-sectional study to evaluate and compare the diagnostic accuracy of three publicly available AI tools most used by caregivers in the United States for pediatric emergency health concerns- ChatGPT-4^o (OpenAI), WebMD Symptom Checker and Mayo Clinic Symptom Checker.

Methodology

This retrospective, cross-sectional study was conducted to evaluate the diagnostic accuracy of three publicly available Artificial Intelligence (AI) symptom checkers which are most frequently used by caregivers for pediatric health concerns in the United States. The AI tools selected were ChatGPT-4^o (OpenAI), WebMD Symptom Checker and Mayo Clinic Symptom Checker.

A total of 120 pediatric case reports were sourced from open access, peer-reviewed journals including SAGE Journals, Cureus, American Journal of Case Reports, BMC and Wiley Online Library. Cases were included based on the following criteria: (1) patients were under 18 years of age, (2) the presentation occurred in an emergency or acute care setting, (3) the clinical condition was relevant and commonly encountered. No rare or ambiguous presentations were included, (4) the report was published in or translated into English and (5) the case was published within the last 10 years.

Each case was rewritten in a standardized, caregiver friendly format. The language and tone were adapted to simulate a layperson speech, avoiding medical terminology to simulate realistic queries. The case descriptions were carefully constructed to include the presenting complaints, the duration and progression of symptoms, the impact on the child’s daily activities (e.g., being too tired to play or having a poor appetite) and pertinent negatives.

Each standardized vignette was individually entered into the three AI symptom checkers. To ensure consistency, all reviewers used the same version of each tool throughout the study. For every interaction, the top three diagnoses suggested were recorded.

Diagnostic performance was then assessed using a structured 3-point scoring system: a score of 2 was assigned if the correct diagnosis was listed first, 1 if it was included among the top three suggestions but not ranked first and 0 if it was not included in the top three at all. For each tool, two independent reviewers assessed the AI-generated outputs and assigned scores accordingly. Any disagreements in scoring were resolved through discussion to ensure inter-rater reliability.

Data Analysis

Null hypothesis – There would be no significant difference in diagnostic accuracy among the three AI tools.

Alternate hypothesis – There would be significant difference in the diagnostic accuracy among the three AI tools.

Descriptive statistics were used to calculate the mean diagnostic score for each tool – bar charts for average scores and stacked bar charts for the distribution of individual scores (0, 1, 2). To compare diagnostic performance, the Friedman test was used, followed by post hoc pairwise comparisons with Bonferroni correction.

Results

A total of 120 publicly available case reports were evaluated by ChatGPT, Mayo Clinic and WebMD to analyse the reliability of the tools. Scores obtained by each tool with respect to the standard scoring system is represented graphically (Fig. 1). Descriptive analysis showed that ChatGPT received the highest average score (Mean = 0.95, SD = 0.942), followed by WebMD (Mean = 0.24, SD = 0.622) and Mayo Clinic (Mean = 0.17, SD = 0.508) (Fig. 2).

Given the non-normal distribution or ordinal nature of the data, Friedman’s Two-Way ANOVA by ranks was applied to compare the three sources. The test revealed a statistically significant difference in scores among the sources (χ² = 80.687, df = 2, p < .001). The results are statistically significant at the 0.001 level, leading to rejection of the null hypothesis (Table 1). This indicates that there are significant differences in the distributions of scores across the three sources. The mean ranks further supported this finding, with ChatGPT ranked highest (Mean Rank = 2.42), followed by WebMD (1.82) and Mayo Clinic (1.75) (Table 2).

Post hoc pairwise comparisons with Bonferroni correction demonstrated that ChatGPT was rated significantly higher than both Mayo Clinic (adjusted p < 0.001) and WebMD (adjusted p < 0.001). However, no significant difference was found between the scores for Mayo Clinic and WebMD (adjusted p = 1.000) (Table 3, Fig. 3).