With booming business and applications such as cloud platforms, the Internet of Things, online high-definition video playback, and artificial intelligence, the demand for global data traffic is growing exponentially, this has led to the development of fiber optic communication systems and networks as the backbone of information transmission towards high-speed and large-capacity [1]. Different innovative technologies, including high-order modulation format, wavelength division multiplexing (WDM), orthogonal frequency division multiplexing (OFDM), and space division multiplexing, have been widely used or investigated in order to fulfill the rising capacity need. The implementation of new technology not only improves the transmission capacity and spectral efficiency of the system, but also significantly increases the optical launch power, which leads to more and more serious nonlinear distortion caused by the nonlinear effect of optical fiber, which in turn becomes the main factor limiting the improvement of system performance [2,3]. The nonlinear frequency division multiplexing (NFDM) technology proposed in recent years takes the nonlinear effect as an inherent property of optical fiber communication systems and theoretically avoids the performance degradation caused by fiber nonlinear effects by designing a new communication system architecture [[4], [5], [6]]. This technology is to achieve linearization of nonlinear optical fiber channels through the mathematical tool of nonlinear Fourier transform (NFT), which utilizes the important feature of nonlinear spectrum (NS) obtained by NFT evolving linearly in nonlinear optical fiber channels, then information can be modulated onto the NS and transmitted in optical fiber. Depending on which part of NS is exploited [7], NFDM systems can be divided into discrete spectrum modulation, continuous spectrum modulation, and full spectrum modulation [[8], [9], [10]]. On the basis, they can be further subdivided into single-polarization transmission and dual-polarization transmission according to the number of polarizations utilized [7,10,11].

Although the NFDM system theoretically resolves the nonlinear effects-related capacity limitation issue in high-speed long-distance fiber optic communication, its channel model must satisfy the requirement of lossless and noise-free conditions in order for the signal's NS to satisfy the linear evolution rule and the NFT algorithm at the receiver (Rx) to effectively recover the signal from the transmitter (Tx) [11]. However, in actual fiber optic links, loss and noise are unavoidable. The loss problem is generally solved by the loss-less path average (LPA) model [12], and the shorter the chosen fiber span distance, the more accurate the model gets. The noise in the NFDM system cannot be eliminated using a straightforward and unified approach due to its inherent randomness. The inverse nonlinear Fourier transform (INFT)/NFT algorithm is the primary contributor of processing noise. Due to its inherent nonlinearity, this algorithm has high computational complexity and implementation difficulty. Moreover, there are significant computational errors when processing high-power signals, which is not conducive to the modulation of multiple eigenvalues in the discrete spectrum system and more subcarriers in the continuous spectrum system, thereby limiting the transmission performance and spectral efficiency of NFDM systems. And physical noise, mainly the amplifier spontaneous emission (ASE) noise introduced by erbium-doped fiber amplifier (EDFA), breaks the integrability condition of the nonlinear Schrödinger equation, causing the NS to no longer be orthogonal and has crosstalk. In addition, ASE is no longer a Gaussian white noise model in the nonlinear spectral domain (NSD), but has strong correlation with signal [13,14], further limiting the transmission performance and spectral efficiency of NFDM systems.

To address processing noise, Sander Wahls proposed using the Wiener Hopf method to achieve CS INFT calculation, avoiding the error propagation problem of the frequency-domain Ablowitz Ladik classical algorithm [15]. Sergey Medvedev et al. proposed the “multi index” algorithm, which achieved fourth order accuracy for NFT algorithm [16,17]. However, the above methods still have high computational complexity and large errors in the INFT/NFT algorithm when processing high-power signals. Wenqi Zhang directly used convolutional neural networks (CNN) to demodulate information from NFDM time-domain waveforms, replacing the original NFT demodulation [18] and reducing numerical calculation errors. However, the CNN complexity in this scheme is relatively high. Currently, there is no solution that can simultaneously reduce complexity and improve accuracy in dealing with processing noise. For ASE noise, Oleksandr Kotlyar used a feed forward artificial neural network (FFNN) to equalize the noise in the NSD. Through numerical simulation, an order of magnitude improvement for bit error rate (BER) was demonstrated [19]. Subsequently, the team proposed a bidirectional long short-term memory (BLSTM) gated recurrent neural network equalization scheme, which significantly outperformed the previously proposed FFNN equalization scheme in terms of performance improvement. However, the complexity of this scheme significantly increases [20] and it is only applicable to single-polarization continuous spectrum nonlinear frequency division multiplexing (SP-CS NFDM) systems. Xinyu Chen proposed a two-stage artificial neural network (ANN) that can equalize the interference between NFDM pulses in the time-domain and the crosstalk between subcarriers in the NSD [21]. Although this scheme achieves joint equalization of noise on multiple dimensions, its complexity increases significantly. Jiacheng Wei proposed a noise equalization joint scheme, which using probabilistic shaping (PS) at the Tx and utilizing a real-valued neural network (r-ANN) at the Rx [22]. However, the scheme only studies noise equalization in SP-CS NFDM systems and the highest order modulation format is 64QAM.

In order to further improve the transmission capacity and performance of NFDM systems, we will propose a dual-polarization transmission noise equalization scheme based on PS and complex-valued neural network (c-ANN) for high-power signals. This scheme utilizes PS technology to reduce the distribution probability of high-power constellation points in high-order modulation format signals, decrease the peak-to-average power ratio (PAPR) of continuous spectrum subcarriers and their average power, and achieve suppression and equalization of ASE noise and processing noise. In addition, the scheme uses the c-ANN to effectively decrease the correlation between subcarriers caused by noise, improve system performance, and significantly reduce the implementation complexity of noise equalization.

The rest of the paper is organized as follows: In Section 2, the transmission model of DP-CS NFDM system and the principle of dual-polarization transmission noise equalization based on PS and c-ANN are discussed. Section 3 describes the parameter settings and optimization of the DP-CS NFDM system. Section 4 verifies the proposed noise equalization scheme and discusses the results in depth. Section 5 is complexity analysis. Section 6 is the conclusion.