A hull reconstruction-reprojection method for pose estimation of free-flying fruit flies

Abstract

Understanding the mechanisms of insect flight requires high-quality data of free-flight kinematics, e.g. for comparative studies or genetic screens. Although recent improvements in high-speed videography allow us to acquire large amounts of free-flight data, a significant bottleneck is automatically extracting accurate body and wing kinematics. Here, we present an experimental system and a hull reconstruction-reprojection algorithm for measuring the flight kinematics of fruit flies. The experimental system can automatically record hundreds of flight events per day. Our algorithm resolves a significant portion of the occlusions in this system by a reconstruction-reprojection scheme that integrates information from all cameras. Wing and body kinematics, including wing deformation, are then extracted from the hulls of the wing boundaries and body. This model-free method is fully automatic, accurate and open source, and can be readily adjusted for different camera configurations or insect species.

Keywords: Drosophila; Biolocomotion; Insect flight; Kinematics; Motion capture; Tracking.

Fig. 1.

Experimental system and 2D segmentation. (A) Experimental setup. (B) Quantifying wing visibility across an ensemble of fly-model poses, used for comparisons between four camera configurations. Three-camera visibility threshold: the minimum percentage of the wing boundary that must be seen in at least three cameras to qualify the identification as successful. (C) Definition of the fly's body axes and degrees of freedom. (D) Definition of the wing Euler angles and stroke plane. (E) Motion-based segmentation of the fly's body. (Ei) When summing the binary images during one wingbeat (73 frames), the most intense pixels correspond to the body. White crosses show the body center of mass (CM) in each summed frame. (Eii) After compensating for body motion, body segmentation and CM estimation improve. Scale on the right of the images in Ei and Eii indicates number of frames.

Fig. 2.

Pose estimation by hull reconstruction and reprojection. (A) Combining 2D image to obtain body hull and wing–body expanded hulls. Body hull is obtained by reconstructing the body images in all four views. Expanded wing hulls are obtained by reconstructing a wing-only image in one view (here, j=1) and the entire fly image in the remaining three views. Expanded body hulls, are obtained similarly. (B) The body hull (dark green) with head and tail blobs (light green). Wing hulls are divided into top and bottom halves (blue/cyan, red/magenta), with the stripes to first approximate wing center of mass (CM; black/gray). (C) Reprojection of onto the image plains allows us to identify otherwise-occluded pixels. Here, the raw image is superposed with the span vector (black), the identified reprojected wing pixels and their boundaries, divided into leading edge (LE) and trailing edge (TE), as well as the body pixels reprojected from . (D) Body hull with the wing-boundary hulls divided into LE (blue/red) and TE (cyan/magenta) voxels. Also shown are the body axes x_b, and the stroke plane (pink) with its normal (magenta). (E) Quantifying wing deformation by calculating five local chord vectors along the wing span, based on the wing boundary voxels (red/blue).

Fig. 3.

Pose estimation results. (A) Sample wing kinematics data for five wingbeats: stroke angle φ, elevation angle θ and the pitch angle ψ, for the left (blue) and right (red) wings. Body angles for the entire flight event are shown in green. (B) Local wing pitch angle in four cross sections ψ₁…ψ₄ for the leading (Bi) and trailing (Bii) edges, showing wing deformation. (C) 2D histogram of wingbeat-averaged r versus p. The negative correlation (fitted Gaussian contours) indicates that the fly's turns are typically coordinated, along the ‘yoll’ axis shown in D. (E) Histograms of the difference between the front stroke angles of both wings (φ_f,L–φ_f,R) per wingbeat. Histograms are split by the value of . For >3×10⁵ deg s^–2 (orange), the left wing reaches more forward than the right wing and vice versa, indicating that moderate roll acceleration is obtained by front φ asymmetry. (F) Histograms of the mean front stoke angle per wingbeat, split . For >3×10⁵ deg s^–2 (orange), the front φ distribution is skewed to the left, indicating that pitch acceleration is achieved by symmetric control of the front stroke angle.