AI-based nanotoxicity data extraction and prediction of nanotoxicity

Abstract

With the growing use of nanomaterials (NMs), assessing their toxicity has become increasingly important. Among toxicity assessment methods, computational models for predicting nanotoxicity are emerging as alternatives to traditional in vitro and in vivo assays, which involve high costs and ethical concerns. As a result, the qualitative and quantitative importance of data is now widely recognized. However, collecting large, high-quality data is both time-consuming and labor-intensive. Artificial intelligence (AI)-based data extraction techniques hold significant potential for extracting and organizing information from unstructured text. However, the use of large language models (LLMs) and prompt engineering for nanotoxicity data extraction has not been widely studied. In this study, we developed an AI-based automated data extraction pipeline to facilitate efficient data collection. The automation process was implemented using Python-based LangChain. We used 216 nanotoxicity research articles as training data to refine prompts and evaluate LLM performance. Subsequently, the most suitable LLM with refined prompts was used to extract test data, from 605 research articles. As a result, data extraction performance on training data achieved F1_D.E. (F1 score for Data Extraction) ranging from 84.6 % to 87.6 % across different LLMs. Furthermore, using the extracted dataset from test set, we constructed automated machine learning (AutoML) models that achieved F1_N.P. (F1 score for Nanotoxicity Prediction) exceeding 86.1 % in predicting nanotoxicity. Additionally, we assessed the reliability and applicability of models by comparing them in terms of ground truth, size, and balance. This study highlights the potential of AI-based data extraction, representing a significant contribution to nanotoxicity research.

Keywords: Automated machine learning; Data extraction; LangChain; Large Language Models; Nanotoxicity; Prompt engineering.

Graphical abstract

Fig. 1

Overall workflow of (1) data preparation, (2) data extraction using LLMs, and (3) model development. The training data undergoes automated extraction, evaluation, and prompt engineering to refine the prompts (red line). The test data is automatically extracted using the selected LLM and prompts, followed by data processing and model development through AutoML (blue line).

Fig. 2

A flowchart illustrating the systematic querying of material information, physicochemical (PChem) properties, and toxicological (Tox) properties. The process begins with querying material information, followed by questions on PChem and Tox properties. For PChem properties, the output from the material information query (i) is combined with specific questions on PChem properties (q) and provided as input to the LLM. Missing data points, where material information or Tox properties are labeled as "None," are used for evaluating extraction performance but are excluded from the final datasets.

Fig. 3

Prompt engineering from a standard prompt (left) to a carefully designed prompt (right) for extracting material information. The refined prompt guides the LLM in adjusting the output format by providing detailed instructions and employing few-shot prompting, ensuring a consistent and structured output format.

Fig. 4

Prompt engineering from a standard prompt (left) to a carefully designed prompt (right) for extracting physicochemical (A) and toxicological (B) properties. For physicochemical properties (A), the prompt standardizes numerical values, including ranges and error margins, while for toxicological properties (B), it enforces standardized categorical formats, such as abbreviations and delimiters, to enhance accuracy and usability in downstream analysis.

Fig. 5

Evaluation of data extraction performance. (A) Comparison of time spent on data extraction per paper using manual extraction versus automated extraction by each LLM. The total time for each method is further broken down into manual extraction time, code run time and embedding time. (B) Average Precision_D.E., Recall_D.E., and F1_D.E. for each LLM, with standard deviations displayed as error bars. (C) Heatmaps showing the percentage of Precision_D.E., Recall_D.E., and F1_D.E. for specific nanotoxicity attributes across each LLM, with color gradients representing performance levels (50 %–100 %).

Fig. 6

Performance metrics of AutoML models trained on HaHa-Auto, HaHa-Manual, and Ha IIIB datasets. The box plots show (A) Accuracy_N.P., (B) F1_N.P., (C) Precision_N.P., and (D) Recall_N.P., comparing results across Vertex AI, Azure, SageMaker, and Dataiku. Each colored dot represents the best-performing model on each platform. The black hoizontal line represents the median, while the plus sign indicates the mean value. Note that the Ha IIIB dataset did not meet the required threshold of 1000 rows, necessary for model training on Vertex AI. Therefore, performance metrics for Vertex AI on Ha IIIB are not shown.

Fig. 7

The feature importance of model trained on HaHa-Auto dataset. The chart represents the mean and standard deviation of feature importance across each AutoML platform.

Fig. 8

Comparison of the applicability domain (AD) of models trained on HaHa-Auto and Ha IIIB datasets for numerical attributes. The thresholds are determined based on the 95 % confidence level of the Euclidean distance distribution. Only the training data from each dataset is visualized.