Since Allen et al. proved through experiments in 1992 that Laguerre Gaussian (LG) beams carry orbital angular momentum (OAM) [1], vortex light with spiral phase wavefront has attracted the attentions of many researchers. Vortex light with spiral phase term can be expressed by exp(ilφ), where l represents the OAM mode, φ represents the azimuth angle, and each photon carries the OAM of lℏ, ℏ is the reduced Planck constant. Theoretically, the OAM mode can take any values, and different OAM modes are orthogonal to each other. Therefore, vortex beams carrying OAM modes have been widely used in optical information manipulation [2], electromagnetism [3], [4], [5] and acoustics [6], [7].

With the further development of OAM beam in the field of free space optical (FSO) communication [8], [9], [10], [11], [12], [13], the efficient demodulation of OAM beam has always been the research focus. At present, common OAM mode demodulation schemes mainly have spiral phase plate [14], computational hologram [15], [16], interferometer method [17], [18], diffraction method [19], [20], [21] and optical geometric transformation [22], [23]. However, with the application of OAM-FSO communications, its transmission in free space is obviously affected by the actual environments, and atmospheric turbulence (AT) is one of the main effects. During propagations, the AT will cause serious distortion on the OAM beam spiral phase, leading to crosstalk between different OAM modes, which will affect the demodulation accuracy of the OAM mode at the demodulation part. At present, researchers often use adaptive optics [24], [25], [26], [27] and digital signal processing [28], [29], [30] to alleviate AT effects. However, Since the intensity of OAM mode is “doughnut” distribution, the intensity distribution among different OAM modes only has the characteristics of different beam radii, which make the detecting system complex for longer propagation distances. And meanwhile the conjugate modes have the same light intensity distribution, which make the recognition of the conjugate modes difficult. With increasing AT intensity and long transmission distance, the influence on the vortex beam will also increase, and the intensity field of the “doughnut” distribution of the vortex beam will be destroyed totally. The demodulation accuracy of traditional demodulation schemes, which based on different radii of vortex beams to identify different OAM modes, have obvious limitations. Therefore, it is imperative to reduce AT influence on transmitting OAM mode.

Due to powerful information processing capabilities, machine learning has been widely used in many fields such as image recognition, computer vision and machine data mining. In recent years, machine learning has been used as OAM mode demodulation technology, which can greatly improve the demodulation accuracy of OAM mode [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43]. As a model in machine learning, CNN (Convolutional Neural Network) has the characteristics of weight sharing and local connection, which has attracted the attention in the application of OAM mode demodulation scheme. Li et al. proposed a beam adaptive demodulator based on CNN [34], which can be used to distinguish the OAM modes of LG beams propagating under different AT intensities, after which the team further proposed a joint AT detection and adaptive demodulation technology based on CNN for the OAM-FSO communication [41]. In Ref. [43], an improved residual network of the CNN model has been proposed to identify the OAM mode, which greatly improved the demodulation accuracy of the OAM mode. The above studies based on CNN have improved the demodulation performance of OAM mode, but because the light intensity map of OAM mode is a “doughnut” distribution, it does not have obvious characteristic distribution. So, in the case of long-distance with strong AT, the demodulation accuracy of OAM mode still has a certain improving space.

In order to increase the characteristics of the OAM beam in light intensity distribution, researchers often use interferometry to interfere with the OAM beam [44], which can generally improve the demodulation accuracy of the OAM mode, but require additional optical devices. In this paper, we propose a phase-compression (PC) method for compressing the phase distribution of the OAM beam at the transmitting part to obtain a hybrid OAM beam with outstanding light intensity distribution characteristics, and then the OAM mode can be recognized by an improved CNN model. In addition, our system is different from the previous CNN-based OAM demodulation system, which can distinguish the positive and negative OAM modes from the rotation directions of the mixed vortex intensity, greatly increasing the number of available OAM modes. We have explored the dependence of the OAM modes’ demodulation accuracy on different transmission distances, AT intensities and PC ratios respectively. Then, we have also discussed the recognition accuracy of the multiplexed hybrid vortex beam, which show that the recognition accuracy can reach 99.75% in the case of longer distances and strong ATs. We also discuss the recognition of the same OAM mode but different PC ratios, and the simulation results show that the multiplexed hybrid vortex beams with the same OAM mode but different PC ratios can still be recognized with a very high accuracy at the receiver. In addition, in order to prove that our model has good generalization ability under unknown turbulence conditions, we use the trained model to test multiple sets of OAM modes under unknown ATs, which demonstrate a good generalization ability.