Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Randomized Controlled Trial
. 2025 Dec;29(12):4030-4042.
doi: 10.1007/s10461-025-04838-0. Epub 2025 Aug 19.

Comparing Implementation and Effectiveness Outcomes for Two Implementation Strategies of the Keep It Up! Digital HIV Prevention Program: A Type 3 Hybrid Effectiveness-Implementation Trial

Brian Mustanski  1   2   3 , Nanette Benbow  4   5 , Kathryn Macapagal  4   6 , Dennis Li  4   5 , Krystal Madkins  4   7 , Rana Saber  4 , Benjamin Linas  8 , J D Smith  9 , C Hendricks Brown  5 , Sarah Munroe  8 , Susheel Reddy  4   10 , Bruce R Schackman  11 , Gregory Swann  4 , Patrick Janulis  4   6 , Alithia Zamantakis  4   6 , Juan Pablo Zapata  4   6
Affiliations
Randomized Controlled Trial

Comparing Implementation and Effectiveness Outcomes for Two Implementation Strategies of the Keep It Up! Digital HIV Prevention Program: A Type 3 Hybrid Effectiveness-Implementation Trial

Brian Mustanski et al. AIDS Behav. 2025 Dec.

Abstract

This study sought to compare implementation and effectiveness outcomes for two delivery strategies of a digital HIV prevention intervention: community-based organization (CBO) versus direct-to-consumer (DTC). A type III hybrid effectiveness-implementation trial compared two implementation strategies of Keep It Up! (KIU! ) for young men who have sex with men (YMSM) (n = 2124). Data were collected at baseline and 12-weeks in 2019-2023. DTC enrolled more participants, but CBO recruited more Black (11.7% vs. 21.7%) and Latino YMSM (27.3% vs. 32.9%). CBO enrollees were at higher HIV risk based on condomless anal sex (p < .05) and rectal Gonorrhea rates (p < .01). Although pre-exposure prophylaxis (PrEP) use with adherence was 1.94 times higher at follow-up vs. baseline (p < .0001) across arms, PrEP increase was 1.60 times higher in CBO vs. DTC (p < .0001). CBO delivery averted more HIV infections per 100 people-years than DTC (1.65 vs. 0.62), but the cost of averting an HIV infection was higher in CBO than DTC ($983,029 vs. $173,313). Although CBOs were able to reach participants at higher HIV risk, the DTC strategy was estimated to cost less per infection averted and below established thresholds for cost savings. Findings must be interpreted within the context of data collection during the COVID-19 epidemic. More information about KIU! can be found at https://kiu.northwestern.edu/ .Trial registration: NCT03896776 (date of registration: 04/01/2019).

Keywords: Digital health interventions; HIV/AIDS; Men who have sex with men; Type III hybrid effectiveness implementation trial.

PubMed Disclaimer

Conflict of interest statement

Declarations. Competing interests: The authors have no competing interests to declare that are relevant to the content of this article. Ethical Approval: The trial was approved by the IRB at Northwestern University (STU00207476). All participants were informed of the aims of the study and of data protection; all participants consented to participate in our trial. Research Involving Human and /or Animal Participants: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Figures

Fig. 1
Fig. 1
Participant Flow in the Keep It Up! Type III Hybrid Effectiveness–Implementation Trial, 2020–2023
See this image and copyright information in PMC

References

    1. Centers for Disease Control and Prevention. HV Surveillance Supplemental Report: estimated HIV Incidence and Prevalence in the United States, 2018–2022. 2024. https://stacks.cdc.gov/view/cdc/156513
    1. Centers for Disease Control and Prevention. Compendium of evidence-based interventions and best practices for HIV prevention. 2024. https://www.cdc.gov/hiv/research/interventionresearch/compendium/index.html. Accessed 10 Feb 2024
    1. Mustanski B, Fisher CB. HIV rates are increasing in gay/bisexual teens: IRB barriers to research must be resolved to Bend the curve. Am J Prev Med Aug. 2016;51(2):249–52. - PMC - PubMed
    1. Smith JD, Li DH, Merle JL, Keiser B, Mustanski B, Benbow ND. Adjunctive interventions: change methods directed at recipients that support uptake and use of health innovations. Implement Sci Feb. 2024;8(1):10. - PMC - PubMed
    1. Mustanski B, Queiroz A, Merle JL, et al. A systematic review of implementation research on determinants and strategies of effective HIV interventions for men who have sex with men in the United States. Annu Rev Psychol. 2024;18:55–85. - PMC - PubMed

Publication types

Associated data