Experimental setup

The experimental setup consisted of a camera (IL5 high-speed camera, Fastec), a lens (LM25HC, Kowa), an arena, a light plate, and a thermal insulation layer (Fig. 1A). The camera above the arena captured images at 1100 × 1100 pixels at 24 frames/second. The light plate was set under the arena (Fig. 1B). A backlighting design was applied to separate objects from the background smoothly. Since a long recording duration would cause a temperature increase in the environment, a thermal insulation layer was used to isolate the heat from the light plate.

Fig. 1

Equipment setup, arena design, and analysis procedure. A Experimental setup. The CCD captures frames above the arena. A glass plate was set between the arena and the lighting plate as a thermal insulation layer. B In the arena design, the fly can move in a cylindrical space of 11 mm in diameter and 3 mm in height. All four tests can be performed simultaneously. C An overview of the flow chart of the software

Software overview

A series of steps were proposed in this study to characterize the flies (Fig. 1C). It is important to note that when the details of the experimental parameter settings for this study are described in the Supplementary Information (Additional file 1), users have the flexibility to adjust these values in the program settings based on their specific circumstances.

(1)

Frame reading: The frame was loaded into the system first.

(2)

Region of interest (ROI) detection: There were 4 regions in one frame. The Hough transform was applied to detect these four circles in the frame.

(3)

Background subtraction: This step yielded the silhouettes of flies in each arena.

(4)

Identification: While two flies overlapped, which can be detected by the number of silhouettes, spectral clustering was applied to separate the overlapping regions. The separated regions can then be assigned to the corresponding identities by rectilinear motion prediction.

(5)

Wing/torso extraction: Next, the wing and torso were extracted from each silhouette and characterized.

(6)

Wing/torso characterization: The position of the fly, the heading direction, and the wing extension angles can be determined in this step.

(7)

Identity checkup: During overlapping events, the identities were assigned by rectilinear motion prediction in the (4) identification step. After the two flies were separated, torso shape matching was executed to verify the identity of the flies. The torso was segmented into head, thorax, and abdomen parts and can be used to assign the identities of flies directly.

(8)

Behavior recognition: The final step was to apply defined parameters to identify behavioral elements.

Background subtraction

The background subtraction method was applied to obtain the initial silhouette of the flies. Figure 2A shows the raw data of a frame. A clear background was generated by a spatial maximum filter [4] (Fig. 2B). As the subtraction process was applied (Fig. 2C), a clear silhouette could be collected by thresholding (Fig. 2D).

Fig. 2

The demonstration of each step in wing/torso detection. A Raw image. B Background generated by a spatial maximum filter. C Background subtraction results from (B) and (C). D Binary image of (C). E Foreground image of (D). F Wing extraction result. G Torso extraction result

Wing/torso extraction

The foreground image (Fig. 2E) was obtained by multiplying the silhouette (Fig. 2D) and the raw image (Fig. 2A). Wings can be extracted by applying the following equation with morphological processing (Fig. 2F):

$$Region\,of\,wings=round\left[\frac\times \frac\times \frac\right]>0$$

R, G, and B in the equation denote the pixel intensities in the red, green, and blue color spaces, respectively. The torso is then obtained by subtracting the region of the wings from the silhouette via morphological processing (Fig. 2G).

Wing/torso characterization - Fly’s position and heading (torso processing)

After extraction of the torso and wings, the position and the heading of a fly were defined as shown in Fig. 3A–D. Figure 3A shows a fly model. The centroid of the torso was defined as the position of the fly (the blue cross in Fig. 3B). The major axis can be calculated by performing ellipse fitting to the region of the torso. The two ends of the major axis were defined as candidates for the head and tail (the green points in Fig. 3B). The red cross in Fig. 3C indicates the centroid position of the fly’s body. A candidate point farther from this red cross was defined as the head point (illustrated by the blue dot in Fig. 3D), while the other candidate point became the tail (depicted by the red dot in Fig. 3D). The heading of the fly can be obtained from the centroid of the torso to the head point (the blue vector in Fig. 3D).

Fig. 3

Wing/body characterization and spectral clustering. A–D Fly position and heading definition. A A schematic of a fly. B Torso ellipse fitting; the centroid of the torso (blue cross) denotes the position of the fly. The two ends of the major axis (green points) were assigned as the candidates for the head and tail. C The centroid of the whole body (red cross). D A candidate further to the centroid of the whole body is designated the head (blue point), and the other candidate is assigned to the tail (red point). The heading is the vector pointed from the centroid of the torso to the head point. E–G Wing extension angle calculation. E Edge of the wing calculation (red edge). F The wing is separated into right and left sides by a cross-product. The wing tips are defined as the farthest edge points from the centroid of the torso (the green and yellow points). G The wing extension angles (θ and θ’) between the wing tips and the major axis of the torso can thus be calculated. H The process of spectral clustering

Wing/torso characterization - the wing extension angles (wing processing)

The wing extension angles were defined as shown in Fig. 3E–G. Edge detection was used to collect the pixels on the edge of the wing (red boundary in Fig. 3E). The wing vector was defined as pointing from the centroid of the torso to each edge pixel (red vector in Fig. 3F). The edge pixels can be divided into left and right sides by the cross product of the heading and wing vectors. The wing vector of the left edge pixels led to positive cross-product results with the heading vector (green boundary in Fig. 3F). Conversely, negative results are denoted on the right side (yellow boundary in Fig. 3F). After separating the edge points into two sides, the point farthest from the centroid of the torso on each side was defined as the wingtip (green point and yellow point in Fig. 3F). The wing extension angles (θ and θ’) between the wing tips and the major axis of the torso can thus be calculated (Fig. 3G).

Identification and identity checkup

Three different identification modules were proposed to maintain high tracking accuracy in this system.

Rectilinear motion prediction

This study predicted the position of the fly by the rectilinear motion model according to the position of the fly in the previous two frames [4]. Tracking identity involved calculating the movement speed at known positions in two consecutive frames and using this speed to estimate where the fly might be in the third frame. Since the time between these three consecutive frames was very short (1/24 s × 2 = 1/12 s), the fly’s position closest to the predicted position was considered its true location. While both male and female flies had predicted positions, the closest fly to the predicted position was labeled the same identity.

Spectral clustering

When flies touched each other, the spectral clustering method was applied to differentiate the overlapping region according to the corresponding identity [27] (Fig. 3H).

Torso shape matching (head, thorax, and abdomen detection)

To achieve greater accuracy when flies touch each other, the torso shape matching method was developed in this study (Fig. 4). The torso was segmented into head, thorax, and abdomen parts by watershed transform, and the identities were subsequently matched by the higher Dice similarity coefficient. This process (six steps) was executed every time the flies were separated after overlapping. Headed and headless tests are shown in Fig. 4A, with two frames in each test. The ‘headless test’ was used to monitor the courtship behavior of male flies toward females with severed heads. In comparison to normal females, females with severed heads exhibited no resistance to courtship by male flies, helping to minimize variables related to female rejection of male mating attempts. Figure 4B shows the matching results according to the Dice similarity coefficient.

Step 1: Crop the binary image of the torso. Then rotate and align it horizontally.

Step 2: Inverse the binary information.

Step 3: Calculate the euclidean distance between each non-zero pixel and its nearest non-zero pixel.

Step 4: Make an additive inverse of the result in Step 3. Thus far, the result was similar to an elevation map. Darker pixels indicated lower altitudes, such as valley regions. Relatively greater amounts of land can be considered to constitute watersheds that can separate nearby basins.

Step 5: The head, thorax, and abdomen can be segmented based on the results of the watershed transformation.

Step 6: The Dice similarity coefficient was computed between the corresponding part of the torso. Three similarity coefficients indicated the similarity of pairs on the head, thorax, and abdomen. A higher value denoted a more similar shape. The sum of the maximum and minimum of these three values must be greater than 1.6 to be considered the same identity.

Fig. 4

The steps of torso shape matching for head, thorax, and abdomen detection. A The process for a headed test and a headless test with two frames for each test is demonstrated. The different colors in step 5 denote the head, thorax, and abdomen. The female fly had a severed head in the headless test, while the female fly retained its head in the headed test. B The Dice similarity coefficient result of the comparison

Together, shape matching ensures the correct identity of flies when they overlap, and this approach can even be applied to headless flies without additional manual settings. It is important to emphasize that the torso matching method has certain limitations when dealing with abnormal fly postures during identity tracking. In instances where the fly was not in a normal standing position, such as when climbing a wall, torso matching may be unable to compute similarity indices. Although identity errors in such situations may not be immediately detected or corrected, flies do not continuously maintain abnormal postures, such as climbing on walls. Consequently, these errors in identity are identified and rectified during subsequent torso matching steps. Despite these limitations, the precision of identity tracking can be ensured through iterative identity verification processes. It is worth noting that the method proposed in this study can automatically detect the threshold utilized for watershed segmentation, alleviating the need for users to set additional parameters for torso segmentation.

Behavior recognition

The image processing methods mentioned above can generate a correct position and enough feature descriptions of each fly. These parameters were used to identify each behavioral element, including singing, orientation, tapping, attempted copulation, and copulation during the courtship ritual.

Singing

Singing refers to the behavior in which the wing is spread open accompanied by vibrations, and the resulting wing sound resembles the behavior of singing for courtship. Due to the difficulty of audio recording, most studies monitored wing vibration to reflect singing. In accordance with previous reports [16, 28], a male with a wing extension angle greater than 30° was considered to be singing in this study.

Orientation

The vector length from the centroid of the torso to the head was extended 2.5 times. Two expanding lines were determined by swinging the extending line ± 10°. A sector was defined as the field of view between these two expanding lines (red boundary in Fig. 5A). When the female was in this sector, the male was recognized as exhibiting orientation behavior.

Fig. 5

The solution to identify orientation and tapping behavior. A The definition of the orientation. B A flow chart of tapping detection. Six conditional steps are used to judge tapping behavior. C–P Demonstration of processing in each step. C Binarization of the body. D Binarization of the torso. E Leg-only skeletonization; the minimum branch length L of the skeletonization is set to 5 (L = 5). F Body with leg skeleton, the combination of (E) and the torso. G The erosion result of (F). H Completed skeleton, the skeletonization of the body (L = infinity). I The subtraction of the complete skeleton from leg-only skeletonization. J Torso and wings area, morphology processing result of (I). K Leg branch, the subtraction of the torso and wings area from the body with leg-skeleton. L Branched torso, the combination of the leg branch and torso. M Denoising result for a branched torso. N The multiplication of the binarization of the body and the raw image. O Double thresholding result of (N). P The binarizing and denoising results for the cyan area in (O)

Tapping

Tapping behavior was defined as the time when a male touched a female’s abdomen with his leg. As shown in Fig. 5B, we gave a solution to determine the leg connection between two flies when tapping occurred. Figures 5C–P present the process at each step. Three tests are shown in Fig. 6 to display how different connection situations were identified via our method. Six conditional steps were applied to determine the leg connection (Fig. 5B).

Conditional step 1: The body touching event must occur when tapping behavior occurs. The number of body regions was counted to capture touching events (Fig. 5C).

Conditional step 2: The torso touching event does not occur when tapping occurs. The number of torso regions was counted to detect touching events (Fig. 5D).

Conditional step 3: The tapping distance between the male’s head and the female’s tail (red and yellow points in Fig. 5D) was defined as less than 1 mm.

Fig. 6

Demonstration of three tests with different connection situations. A Situation 1: Male touched female’s abdomen with his leg. Tapping behavior is recognized. B Situation 2: The male touched the female’s abdomen with his leg, but the event was under the wing of the female. Tapping behavior is recognized. C Situation 3: Male-touched female’s wing. Behavior is not recognized as tapping

When the above criteria were met, there were only 3 connection situations. The tapping events were recognized in connection situations 1 and 2 but not in connection situation 3:

Connection situation 1- Only the leg touched the female’s torso (Fig. 6A).

Connection situation 2- The leg touched the female’s torso but under the wings (Fig. 6B).

Connection situation 3- The leg touches the female’s wing but not the torso (Fig. 6C).

Conditional step 4: Skeletonization was performed to reduce the object to branch. The minimum branch length L of the skeletonization was set to 5 to obtain the branch of the leg and maintain the shape of the wings and torso (Fig. 5E). Figure 5F shows the union of Fig. 5E and the torso (Fig. 5D). Subsequently, the skeletonized leg can be eliminated by using the erosion algorithm (Fig. 5G). While the connection disappeared after erosion, causing the region to separate, the connection part was considered the leg. The two possible connection situations involved the leg touching the torso or the leg touching the wing (Fig. 6A or C). On the other hand, while the region was still connected after erosion, two possible outcomes were wings covering the leg connections (Fig. 6B) or no touching (tapping) at all.

Conditional step 5: This step was applied to determine if the leg touched the torso or wings (Fig. 6A or C). The minimum branch length L of the skeletonization was set to infinity to obtain the complete skeleton of the body (Fig. 5H). The entire skeletonization process simplified the fly’s body, primarily emphasizing its skeletal structure, akin to the process of focusing solely on the legs (Fig. 5E), while still retaining full leg details. Given that both processes successfully retain leg information, subtracting the complete skeleton from the leg-only skeletonization yielded a generalized outline of fly sans legs (Fig. 5I). After morphology processing, a more complete view of the shape of the torso and wings was obtained (Fig. 5J). The branch information of the leg (Fig. 5K) can be obtained by subtracting the torso and wing areas from the body with the leg skeleton. A branched torso can be generated by coupling the leg branch to the torso (Fig. 5L). After removing the noise (Fig. 5M), if the two branched torsos could be connected, the process was considered tapping (Fig. 6A). Otherwise, it is considered as no tapping behavior (Fig. 6C).

Conditional step 6: This step was used to judge whether the wings were covering the leg (Fig. 6C) or if tapping was not occurring. Figure 5N is made by multiplying the binary image of the body by the original image. Otsu’s method was subsequently used to segment Fig. 5N with double thresholds to detect the connection of the legs (the cyan area in Fig. 5O). After binarizing and denoising the cyan area in Fig. 5O (Fig. 5P), if the two binarized regions are connected, tapping is considered to occur (Fig. 6B). If they were not connected, it was considered no tapping behavior. This method demonstrated robustness in obtaining leg information, with errors typically occurring only in scenarios where the original frame lacked complete leg information, such as when the fly’s legs were obstructed by the body or wings.

Attempted copulation

In this study, ellipse fitting was performed on the torso of the male, and eccentricity was used to determine whether a male tried to mate with a female. Since males bend their abdomen and try to climb the female’s body during attempted copulation, the body of males changes from a long ellipse shape to a nearly round shape. Its torso eccentricity is thus much lower than the original shape. In each experiment, the user first selected a male without bending its body as a standard (Fig. 7A) and multiplied the eccentricity of the torso by 0.9 to obtain a standard value. In the following analysis, whether the torsos were connected or not, as long as the eccentricity was smaller than the standard value and the distance between the male’s head and the female’s tail was less than 1 mm, was regarded as attempted copulation (Fig. 7B and C). A number greater than the standard value was considered to indicate no attempted copulation (Fig. 7D).

Fig. 7

Attempted copulation was determined by the eccentricity of the male torso. A A frame in which the male does not bend his body is assigned as a standard figure. The standard value is defined as 0.9 times the eccentricity of a male’s torso. B, C The eccentricity of the torso is less than the standard value when a male bends his abdomen, whether the torsos are connected (B) or not connected (C). D If the male does not try to copulate with the female, the eccentricity of his torso will be greater than the standard value

Copulation

When the duration of attempted copulation was greater than half a minute, copulation behavior was determined. Since we did not study behavior after copulation in this report, the behavior of males in the rest of the test was considered copulation.

Noise filter

The rule of denoising is shown in the following:

1.

All courtship behaviors were certified if the behavior was identified in more than 5 frames within 12 frames.

2.

No behavior was determined when there was no behavior recognized for 12 continuous frames. Like in other courtship behaviors, the designation of “no behavior” requires a certain duration of performance to be defined as such. If any moment without defined behavioral occurrences is labeled as an instance of “no behavior”, the precision of this label would be compromised.

3.

If one more behavior was recognized in the same frame, the priority of the recognition was copulation > attempted copulation > tapping > sing > orientation.

Fly stocks and environmental details for the courtship assay

D. simulans (D. sim.) and D. melanogaster (D. mel.) Canton-S (CS) were obtained from the fly core in Taiwan. All flies were kept at 25 °C and 60% RH with a 12 h light/12 h dark cycle and were raised in standard white food supplemented with yeast, corn powder, agar, antibiotics, and preservatives. The flies were transferred to new vials with fresh food every 2–3 days until the behavioral assays were performed.

Fly collection for courtship assay

Virgin CS females serving as courtship targets were collected within 8 h after emergence under CO2 anesthesia. Thirty female flies were housed in one vial for 7 days before the behavioral assays. For the dead females, the flies were decapitated immediately before the courtship test. Unless otherwise noted, all male flies were collected within 8 h after emergence under CO2 anesthesia. Thirty male flies were housed in one vial for 7 days before the courtship test. For flies of different ages, the flies were transferred to new vials every 2–3 days for 2, 8, 14, 21, 29, 35, 42, or 49 days before the courtship assay.

Courtship assay

The diameter of the courtship arena was 11 mm. The wall of the arena and cover glass were coated with water repellent to prevent fly climbing. For courtship recording, 1 male and 1 female fly were placed into the arena under CO2 anesthesia. After a 5-min adaptation, the interaction between the males and females was recorded for 20 min. The recorded video was further analyzed to identify different male courtship behavioral elements.

Total courtship time The sum of all the behavioral elements, including orientation, tapping, singing, and attempted copulation. Data for males that had mated within 20 min were excluded.

The proportion of behavioral elements The time of each behavioral element divided by the total observation time. For analyzing males of different ages, the observation time was considered the time from the beginning to copulation for mated males or 20 min for unmated males. To compare mated and unmated males, considering that behaviors may change as a function of time, we analyzed mated and unmated males within a similar duration by analyzing mated males from the beginning of copulation and analyzing unmated males for 484.75 s, which is the mean duration for the copulation of mated males.

The transition of behavioral elements The number of changes from behavioral element A to B divided by the total number of changes from A to others. Again, since the transition probabilities may change as a function of time, we analyzed mated males from the beginning of the experiment to the time of copulation and analyzed unmated males for 484.75 s, the mean duration for the copulation of mated males.

Statistics for courtship assay

All the statistical analyses were completed with Prism and MATLAB software. An unpaired Student’s t test was used to compare data between male courtship to live and dead females, between D. mel and D. sim, and between mated and unmated males. For the effect of age on courtship, regression analysis and quadratic fitting were performed via QR decomposition. For nonnormally distributed data, the Kruskal‒Wallis test followed by Dunn’s test was used to compare total courtship time, tapping, and the attempted copulation ratios. For normally distributed data, one-way ANOVA followed by Tukey’s test was used to compare the singing and orientation ratios.