Gamification in Radiology Training Module Developed During the Society for Imaging Informatics in Medicine Annual Meeting Hackathon

Abstract

The purpose of this manuscript is to report our experience in the 2021 SIIM Virtual Hackathon, where we developed a proof-of-concept of a radiology training module with elements of gamification. In the 50 h allotted in the hackathon, we proposed an idea, connected with colleagues from five different countries, and completed an operational proof-of-concept, which was demonstrated live at the hackathon showcase, competing with eight other teams. Our prototype involved participants annotating publicly available chest radiographs of patients with tuberculosis. We showed how we could give experience points to trainees based on annotation precision compared to ground truth radiologists' annotation, ranked in a live leaderboard. We believe that gamification elements could provide an engaging solution for radiology education. Our project was awarded first place out of eight participating hackathon teams.

Keywords: Informatics; Internship and residency; Medical Education; Radiology.

Fig. 1

Hackathon timeline presented in the virtual orientation. Before the hackathon week, a live orientation was presented via Zoom (Zoom Video Communications, San Jose, CA) on May 6, 2021 [5]. At the start of the hackathon week, a live brainstorming session took place on Zoom on May 17, 2021, in which everyone had a chance to propose one or more ideas, and programmers were paired with non-programmers. On May 19, 2021, the hacking proper began. All communications moved to Slack (Slack Technologies, San Francisco, CA), a proprietary business communication platform. There were two live check-in meetings per day at 12 pm and 5 pm EST on Zoom to ask participants if they needed anything or connect participants with code heroes or domain experts. On Friday, May 21, at 2 pm, all hacking ceased (“pencils down”), and each team began preparing their presentations and demonstrations for the showcase that immediately followed. The orientation can be found on the SIIM YouTube channel as “2021 SIIM Hackathon Orientation Webinar” (https://www.youtube.com/watch?v=_C404uhjzY8)

Fig. 2

Example of a Lung-RADS set of modules (A) that was the example proposed during the brainstorming session. An individual module (B) within the set would comprise a series of imaging cases (C and D) for the trainee to annotate or choose an answer for in some way. Panel D shows a zoomed-in view of the module prototype. Every subsequent module (dungeon) within a set of modules would be more challenging than the previous. Answer correctness would be compared to the ground truth specific for that task. Each correct answer would grant the trainee a certain amount of XP, with each module having a limited maximum number of XPs. At the end of each dungeon, the trainee would amass XPs, which would move into the trainee’s total pool of XPs. Those XPs would amount to levels, and a trainee would keep increasing their individual level, just like a video game character. In addition, a resident would earn badges for accomplishing streaks and other achievements. Each correct answer would grant the trainee a certain amount of XP, with each module having a limited maximum number of XPs

Fig. 3

Example of an interface for the web page of a trainee profile showing their experience points (XPs), level, badges, and a leaderboard. Individual trainees could accumulate XPs and go up in levels (indicated in the “My level” bar). Badges could also be granted for specific achievements (exemplified in the “My badges” section of the webpage. For each module, there could be a leaderboard showing all trainees’ XPs. There could also be a general leaderboard for each residency year and the entire residency

Fig. 4

VGG Image Annotator (VIA) website interface and example of an annotated image. Users were instructed to place bounding boxes on imaging findings of tuberculosis on chest radiographs of 68 different patients. Example bounding boxes shown here #1 left upper and #2 right lower lung fields. The ground truth for our proof-of-concept was the annotations by a radiologist (L.R.F.) with over 30 years of experience

Fig. 5

Screenshot of the custom webpage we created on Streamlit to calculate the XPs and display them into a leaderboard based on a JSON file. The trainee can type their name, drag-and-drop their JSON file, and see the leaderboard updating in real-time

Fig. 6

Overview of steps including uploading publicly available radiographs into VIA for annotation (upper left), showing how residents annotate abnormalities. The upper right shows how they submit the resultant JSON file for analysis for score calculation (comparing to ground truth radiologist), where the Dice coefficient (for example) “scores” for placement of trainee in the leaderboard (lower right)