Cyber Teaming and Role Specialization in a Cyber Security Defense Competition

Abstract

A critical requirement for developing a cyber capable workforce is to understand how to challenge, assess, and rapidly develop human cyber skill-sets in realistic cyber operational environments. Fortunately, cyber team competitions make use of simulated operational environments with scoring criteria of task performance that objectively define overall team effectiveness, thus providing the means and context for observation and analysis of cyber teaming. Such competitions allow researchers to address the key determinants that make a cyber defense team more or less effective in responding to and mitigating cyber attacks. For this purpose, we analyzed data collected at the 12th annual Mid-Atlantic Collegiate Cyber Defense Competition (MACCDC, http://www.maccdc.org), where eight teams were evaluated along four independent scoring dimensions: maintaining services, incident response, scenario injects, and thwarting adversarial activities. Data collected from the 13-point OAT (Observational Assessment of Teamwork) instrument by embedded observers and a cyber teamwork survey completed by all participants were used to assess teamwork and leadership behaviors and team composition and work processes, respectively. The scores from the competition were used as an outcome measure in our analysis to extract key features of team process, structure, leadership, and skill-sets in relation to effective cyber defense. We used Bayesian regression to relate scored performance during the competition to team skill composition, team experience level, and an observational construct of team collaboration. Our results indicate that effective collaboration, experience, and functional role-specialization within the teams are important factors that determine the success of these teams in the competition and are important observational predictors of the timely detection and effective mitigation of ongoing cyber attacks. These results support theories of team maturation and the development of functional team cognition applied to mastering cybersecurity.

Keywords: collaboration; computer personnel selection; cyber defense; cybersecurity; expertise; skill composition; team development; teamwork.

Figure 1

The 10-region National Collegiate Cyber Defense Competition with Mid-Atlantic Region circled.

Figure 2

“Man vs. Machine” Internet-of-Things scenario description given to participants.

Figure 3

The network diagram of the We-B-Smart network.

Figure 4

The composite overall team score in the MACCDC 2017 competition was composed of five performance metrics. These included: (1) Maintaining Services, (2) CEO Reporting, (3) Scenario Injects, (4) Incident Response, and (5) Red Team Activity. Performance scores are normalized z-scores and the teams arrayed with increasing composite performance by team number.

Figure 5

(Top) Heatmap representation depicting the skill composition of the various cybersecurity teams with darker colors indicating the proportion of team members endorsing a particular skill or role. (Bottom) Histogram of average years experience by each team.

Figure 6

Heatmap representation depicting the skill composition of the various cybersecurity tasks.

Figure 7

(A) Posterior distributions of regression parameters for Model 1 predicting Maintaining Services score with β1 (Communication & Collaboration Factor), β2 (Years Experience), and β3 (Number of Roles) as predictors of team performance. Strong predictors (Number of Roles) indicated by red type. (B) Posterior distributions of difference among parameters indicate unique predictive quality of β1, β2 and β3 to Maintaining Services scored performance.

Figure 8

(A) Posterior distributions of regression parameters for the simultaneous model predicting Incident Response score with β1 (Communication and Collaboration Factor), β2 (Leadership Factor), and β3 (Number of Skill Roles) as predictors of team performance. Strong predictors indicated by red type. (B) Posterior distributions of difference among parameters indicates unique predictive quality of β1 and β3 to Incident Response scores.

Figure 9

Comparison of Bayesian Multiple Linear Regression Models predictors across the MACCDC 2016 (Top) and MACCDC 2017 (Bottom) events for the scored performance dimensions of (Left) Maintaining Services, (Middle) Scenario Injects, (Right) Incident Response.