Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 May 6;14(5):e0215964.
doi: 10.1371/journal.pone.0215964. eCollection 2019.

Visualizing learner engagement, performance, and trajectories to evaluate and optimize online course design

Michael Ginda  1 , Michael C Richey  2 , Mark Cousino  2 , Katy Börner  1
Affiliations

Visualizing learner engagement, performance, and trajectories to evaluate and optimize online course design

Michael Ginda et al. PLoS One. .

Abstract

Learning analytics and visualizations make it possible to examine and communicate learners' engagement, performance, and trajectories in online courses to evaluate and optimize course design for learners. This is particularly valuable for workforce training involving employees who need to acquire new knowledge in the most effective manner. This paper introduces a set of metrics and visualizations that aim to capture key dynamical aspects of learner engagement, performance, and course trajectories. The metrics are applied to identify prototypical behavior and learning pathways through and interactions with course content, activities, and assessments. The approach is exemplified and empirically validated using more than 30 million separate logged events that capture activities of 1,608 Boeing engineers taking the MITxPro Course, "Architecture of Complex Systems," delivered in Fall 2016. Visualization results show course structure and patterns of learner interactions with course material, activities, and assessments. Tree visualizations are used to represent course hierarchical structures and explicit sequence of content modules. Learner trajectory networks represent pathways and interactions of individual learners through course modules, revealing patterns of learner engagement, content access strategies, and performance. Results provide evidence for instructors and course designers for evaluating the usage and effectiveness of course materials and intervention strategies.

PubMed Disclaimer

Conflict of interest statement

Michael Richey and Mark Cousino are employed by The Boeing Company. Michael Ginda and Katy Börner were partially supported by The Boeing Company. There are no patents, products in development or marketed products to declare. This does not alter our adherence to all the PLOS ONE policies on sharing data and materials.

Figures

Fig 1
Fig 1. Data flow diagram shows the processing steps used with edX course data.
Fig 2
Fig 2. EdX course structure provide levels of learner interaction analysis and guide base map construction.
(A) Shows the 5-level course module hierarchy used in the “Architecture of Complex Systems” course; (B) a bar graph shows difference between instructor’s predictions and the average time learners spent on the same set of modules, aggregated at the page sequence level of the course hierarchy; (C) learner path shows temporal sequence of learning modules accessed by a high performing learner.
Fig 3
Fig 3. Learners’ path overlaid on force-directed layout of used course modules.
Learner path of a learners with a high (left) and a low (right) performance scores overlaid on force-directed layout of course modules.
Fig 4
Fig 4. Percentage of learners accessing a module in course sequence.
Plotted are the 310 of 551 modules at lowest level of course structure used by learners. Vertical lines indicate the last module of a given course section.
Fig 5
Fig 5. Mean number of interaction events per learners with a given module in a course.
(A) Plots the 117 HTML modules, (B) shows 49 video modules, (C) 139 ungraded problem modules, and (D) 5 open assessment modules from the lowest level of course structure that were used by learners. Vertical dashed lines indicate the last module of a given course section. Circle area size is linearly scaled to show the percentage of a group that interacted with a module.
Fig 6
Fig 6
Correlation of different variables with final grade (left) and number of events (right). The plots compare: (A) grade and events, (B) grade and unique modules accessed in the course, (C) grade and open assessment events, (D) events to cumulative user sessions, (E) events and problem attempts, (F) events and open assessment events.
See this image and copyright information in PMC

References

    1. Richey MC, Hamilton M, Madni AM. Boeing’s Investment and Returns in Learning, Knowledge Creation and Profitability Conference on Systems Engineering Research (CSER); Redondo Beach, CA: International Council on Systems Engineering; 2014.
    1. Shah D. By The Numbers: MOOCS in 2017—Class Central: Class Central, Inc; 2018. [Available from: https://www.class-central.com/report/mooc-stats-2017/.
    1. Engineering technology education in the United States. Washington, DC: National Academies Press; 2016 2016.
    1. Bogdan R, Holotescu C, Andone D, Grosseck G. How MOOCS are Being Used for Corporate Training? eLearning & Software for Education. 2017;2.
    1. Hogle P. Hot or Not: How Do MOOCs Fit into Corporate eLearning? Santa Rosa, CA: eLearning Guild; 2017. [Available from: https://www.learningsolutionsmag.com/articles/2228/hot-or-not-how-do-moo....

Publication types