GPT-4 based AI agents - the new expert system for detection of antimicrobial resistance mechanisms?

Abstract

BackgroundEUCAST recommends a two-step process for beta-lactamases in Gram-negative bacteria. Screening with minimal inhibitory concentrations (MICs) or inhibition zone diameters for potential extended spectrum beta-lactamase (ESBL), plasmid-mediated AmpC beta-lactamase, or carbapenemase production is followed by confirmatory tests. GPT-4 and its newly released customized GPT-agent may support the initial EUCAST-screening process. We aimed to validate a customized GPT-agent to identify potential resistance mechanisms. MethodsWe used 225 Gram-negative isolates. Based on phenotypic resistances against beta-lactam antibiotics, we formed four categories: "none", "ESBL", "AmpC", or "carbapenemase". We included 862 phenotypic categories. Next, we customized a GPT-agent with EUCAST-guidelines, expert rules, and EUCAST-breakpoint table (v13.1). We compared routine diagnostic outputs (reference) to (i) EUCAST-GPT-expert, (ii) medical microbiologists, and (iii) GPT-4 without customization. We determined performance as sensitivities and specificities to flag suspect resistance mechanisms. ResultsThree human readers showed concordance in 814/862 (94.4%) phenotypic categories and used in median eight words (IQR 4-11) for reasoning. Median sensitivity and specificity for ESBL, AmpC, and carbapenemase were 98%/99.1%, 96.8%/97.1%, and 95.5%/98.5%, respectively. Three independent prompting rounds of the GPT-agent showed concordance in 706/862 (81.9%) categories but used in median 158 words (IQR 140-174) for reasoning. Median sensitivity and specificity for ESBL, AmpC, and carbapenemase prediction were 95.4%/69.23%, 96.9%/86.3%, and 100%/98.8%, respectively. In the non-customized GPT-4, 169/862 (19.6%) categories could be interpreted. Of these 137/169 (81.1%) categories agreed with routine diagnostic. The non-customized GPT-4 used in median 85 words (IQR 72-105) for reasoning. ConclusionHuman experts showed higher concordance and shorter argumentations compared to GPT-agents. Human experts showed comparable median sensitivities and higher specificities compared to GPT-agents. GPT-agents showed more unspecific flagging of ESBL and AmpC, potentially, resulting in additional testing, diagnostic delays, and higher costs. GPT-4 and GPT-agents are not IVDR/FDA-approved, but validation of LLMs is critical and datasets for benchmarking are needed.