Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Aug 1.
doi: 10.1097/CIN.0000000000001351. Online ahead of print.

Assessing Artificial Intelligence to Support Grant Evaluation in Nursing Education

Regina G Russell  1 , Y Mike Wang, Marianna D LaNoue, Md Mashiur Rahman Khan, Hannah McGrew, Patricia Sengstack, Julia Phillippi
Affiliations

Assessing Artificial Intelligence to Support Grant Evaluation in Nursing Education

Regina G Russell et al. Comput Inform Nurs. .

Abstract

Artificial intelligence, including chatbots that produce outputs in response to user prompts, is poised to revolutionize nursing education and program evaluation. This article summarizes a pilot evaluation of three core features in a chatbot system: (1) defining a persona or system prompt, (2) creating specific task prompts, and (3) developing an improvement mechanism. First, an "act as" system prompt was written to generate "Future-FLO" based on professional values. Second, a task prompt was written to generate outputs specific to program evaluation. Third, a separate ImproverBot was created to provide structured assessments of chatbot outputs. Human raters, knowledgeable about grant goals, provided feedback on AI-generated outputs from FLO and non-FLO system prompts. The ImproverBot rated the same outputs as the human raters on accuracy, completeness, and usefulness. Statistical tests were used to compare ratings across AI versions and rater types. The custom prompt provided no added benefits compared with using the standard model, as rated by human raters and the ImproverBot. The ImproverBot gave significantly higher ratings than human raters. Comments indicated that outputs were error-filled and unreliable. Engaging with artificial intelligence to support the work of grant evaluation will require ongoing efforts to develop chatbots and evaluate their outputs.

Keywords: Chatbot; Ethical design; Generative artificial intelligence; Grant evaluation; Large language model; Nursing education; Program evaluation; Prompt engineering; Responsible use; Values-based development.

PubMed Disclaimer

Conflict of interest statement

In addition to the grants reported above, J.P. received unrelated workload support from the HRSA (23-120), the National Institute of Nursing Research (R21NR020857-01), and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (1P50HD106446-01). She also received travel reimbursement from the American College of Nurse-Midwivesand and royalties from Jones & Bartlett Learning. These relationships and funding are not related to the manuscript and do not represent a conflict of interest. The other authors have disclosed that they have no significant relationships with, or financial interest in, any commercial companies pertaining to this article.

References

    1. Hassmiller SB. Florence Nightingale as an applied statistician. 2021. www.ajnoffthecharts.com.
    1. Bradshaw NA. Florence Nightingale (1820–1910): an unexpected master of data. Patterns (N Y). 2020;1(2): 100036. doi:10.1016/j.patter.2020.100036. - DOI
    1. Andrews RJ. Let there be light: a tribute to Florence Nightingale. Significance. 2020;17(2): 16–17. doi:10.1111/1740-9713.01373. - DOI
    1. Hedley A. Florence Nightingale and Victorian data visualisation. Significance. 2020;17(2): 26–30. doi:10.1111/1740-9713.01376. - DOI
    1. Buchanan C, Howitt ML, Wilson R, Booth RG, Risling T, Bamford M. Predicted influences of artificial intelligence on nursing education: scoping review. JMIR Nursing. 2021;4(1): e23933. doi:10.2196/23933. - DOI

LinkOut - more resources