A strategy for cost-effective large language model use at health system-scale

Abstract

Large language models (LLMs) can optimize clinical workflows; however, the economic and computational challenges of their utilization at the health system scale are underexplored. We evaluated how concatenating queries with multiple clinical notes and tasks simultaneously affects model performance under increasing computational loads. We assessed ten LLMs of different capacities and sizes utilizing real-world patient data. We conducted >300,000 experiments of various task sizes and configurations, measuring accuracy in question-answering and the ability to properly format outputs. Performance deteriorated as the number of questions and notes increased. High-capacity models, like Llama-3-70b, had low failure rates and high accuracies. GPT-4-turbo-128k was similarly resilient across task burdens, but performance deteriorated after 50 tasks at large prompt sizes. After addressing mitigable failures, these two models can concatenate up to 50 simultaneous tasks effectively, with validation on a public medical question-answering dataset. An economic analysis demonstrated up to a 17-fold cost reduction at 50 tasks using concatenation. These results identify the limits of LLMs for effective utilization and highlight avenues for cost-efficiency at the enterprise scale.

Fig. 1. Study design and overview.

Figure 1 shows the overall study. We used real-world notes from the electronic health records. We then used GPT-4–8k to create bespoke question-answer pairs of multiple types, namely fact-based, temporal, and numerical. We then tested 10 other LLMs of various sizes against different burdens of questions and notes and assessed the accuracy and proper formatting performance.

Fig. 2. Performance of formatting JSON output responses.

The overall accuracy of question answers for the clinical notes is presented across question burdens for each task size. Results are presented when including Omission Failures for Small (a), Medium (c), and Large (e) tasks. Results are also presented when excluding Omission Failures for Small (b), Medium (d), and Large (f). The shaded area reflects 95% Confidence Intervals. Models that were unable to properly format responses were not included.

Fig. 3. Accuracy of LLMs for question–answer pairs across task sizes with or without Omission Failures.

Assessment of LLMs’ ability to properly format outputs aggregated across configurations by task size. Models that had high JSON failure rates were not assessed for Omission failures. The JSON loading error failure rates are plotted with 95% confidence intervals across all experiments (a) Small, b Medium, and c Large task sizes. Omission errors are plotted similarly for (d) Small, e Medium, and f Large task sizes.

Fig. 4. Graphical representation of output accuracy for a given experiment highlighting different types of errors.

Graphical depiction of accuracy and errors for GPT-4-Turbo-128k for the Medium task (4 notes) and 15 question burden for each. Omission and JSON failures are demonstrated in addition to correct and incorrect answers.

Fig. 5. Impact of prompt size on LLM performance for 50 total tasks.

The impact of prompt size when a number of tasks is held constant at 50 for GPT-3.5-turbo-16k, GPT-4-turbo-128k, Mixtral-8x22B, and Mixtral-8x7B, with 50 iterations for each experiment. a JSON failure rate; b Omission failure rate; c overall accuracy including Omission failures as errors; and d overall accuracy excluding Omission failures as errors aggregated across configurations. As notes increased, the number of tokens increased, but the number of questions decreased (i.e., 2 notes with 25 questions each, 5 notes with 10 questions each, etc.). Shaded areas in plots (c) and d reflect 95% Confidence Intervals.