Vortex beams (VBs) with orbital angular momentum (OAM) modes hold great potential in optical communications for offering additional multiplexing dimensions [[1], [2], [3], [4], [5], [6], [7], [8], [9]]. Although extensive research on OAM mode (de)multiplexing, signal demodulation and noise monitoring of these multiplexed channels remain challenging due to the absence of effective methods for analyzing the OAM spectrum, which characterizes the mode components and intensity weights. Spatial separation schemes, including Mach-Zender interferometry [[10], [11], [12], [13]], Damman gratings [[14], [15], [16]], and optical geometric transformations [[17], [18], [19], [20]], enable the analysis of OAM mode components by sorting multiple VBs into distinct spatial positions. However, they are ineffective for directly measuring intensity weights of the modes. While interferometry has been proposed as a solution, it often involves multiple measurements of interference fields, making the process tedious and inefficient [[21], [22], [23]]. Recently, deep neural networks have emerged as efficient tools for extracting OAM mode features from the intensity distribution of VBs, thanks to their powerful information processing capabilities [[24], [25], [26], [27], [28], [29]]. However, these methods have limited ability to perform intricate OAM spectrum analyses, and it can be challenging to extract signal and noise power distributions for optical signal-to-noise ratio monitoring solely through qualitative mode component analysis. Moreover, previous studies lack development for multidimensional multiplexing communication. Consequently, there is an unexplored avenue for a quantitative method to analyze OAM modes, specifically for signal demodulation and noise monitoring purposes.

Herein, we present a novel OAM spectrum analysis approach that combines a residual neural network with interference preprocessing. The target vortex beam is interfered with a Gaussian spherical wave, resulting in the mapping of spectral information onto the stripe features observed in the interferogram. The number and rotation direction of the stripes, as well as the presence of bifurcations at the central and periphery of the stripes, are indicative of the mode components and intensity weights. This interference preprocessing technique not only highlights implicit OAM spectral features for easy extraction, but also effectively expands the range of available modes by enabling chiral analysis of OAM. To accurately analyze the OAM mode spectrum details from interference fringes, we design a residual neural network with deep structure and shortcut connections. It can effectively construct a nonlinear mapping relationship between the input interference pattern and the output mode spectrum through a gradient descent optimisation algorithm [30]. By introducing this residual mechanism, the spectral feature information can be rapidly propagated from the shallow layer to the deep layer through interlayer shortcut connections, which endows the neural network with excellent feature processing capability. Therefore, this neural network approach embedded with interference techniques is capable of analysing intricate OAM modes and their spectra.

To demonstrate the feasibility of this approach, we performed OAM mode spectrum analysis for signal demodulation and noise monitoring of multiplexed channels. Leveraging the advantages of deep structures and shortcut connections, the residual neural network can accurately extract the mode and weight features directly from a single interferogram, eliminating the need for additional complex measurement steps. Consequently, the network can effectively compose OAM mode spectra, contributing to simplified and efficient analysis of OAM modes in optical communication systems. We conducted simulations to analyze the spectral features of 15 superimposed modes, achieving an impressive mean-square-error (MSE) of less than 2.3 × 10−4. Additionally, experimental results from the spectral analysis of 5 superimposed modes indicate an MSE of only 1.23 × 10−4. Building upon these findings, we developed an 80-channel (five OAM modes, two polarizations, and eight wavelengths) three-dimensional multiplexing communication link with a high data rate of 4 Tbit/s. Through the assistance of a residual neural network, we achieved the effective demodulation of quadrature-phase shift-keying (QPSK) signals. The monitored optical signal-to-noise ratio (OSNR) reached 11.5 dB. The QPSK signals exhibited a bit-error-rate (BER) of 3.34 × 10−6 and an error-vector-magnitude (EVM) of 14.19% at a received power of −19 dBm. These results underscore the effectiveness of the proposed approach in extracting the spectrum features of OAM modes, which holds significant potential for applications in signal demodulation and noise monitoring applications.