Magnetic Resonance Imaging (MRI) has been widely used in clinical diagnosis of various diseases due to its superior contrast on soft tissues and high resolution of the acquired MR images compared with other medical imaging modalities such as Computerized Tomography (CT) and ultrasound. In addition, MRI is a non-invasive imaging method, which can avoid the potential damage to health from radiation. Nevertheless, the imaging process is time-consuming due to the sequential collection of a complete data matrix in k-space. As a result, the possibility of containing motion artifacts in acquired images will increase a lot. Moreover, the painfully slow imaging process will cause uncomfortable healthcare experiences and limit the total throughput in hospitals. To this end, developing algorithms to reduce the required duration of MRI holds significant practical value.

The excessively long scan time of MRI is strongly correlated to the large number of frequency and phase encoding steps when collecting data in k-space. Intuitively, an effective acceleration method is to reconstruct MR images from partially sampled k-space data. However, this simple method will generate images with unacceptable artifacts that significantly affect accurate diagnosis because of the disobeying of the Nyquist Sampling Theorem. Previous researchers have proposed effective methods such as Parallel Imaging (PI) [[1], [2], [3]] and Compressed Sensing (CS) [[4], [5], [6]] to accelerate MRI. Parallel imaging methods collect data with multiple receive coils simultaneously and achieve acceleration by exploiting the redundant information contained in different receive coils. However, the obtained MR images suffer from low Signal-to-Noise Ratios (SNRs) and incompletely eliminated artifacts, especially when utilizing relatively high acceleration factors. Compressed Sensing (CS) methods treat the reconstruction task as an inverse optimization problem and solve it iteratively. By enforcing sparsity in some transformation domains, this category of methods can generate outstanding results. Similarly, it also suffers from several limitations such as the time-intensive iterative process, unsatisfying reconstruction results at high acceleration rates, and excess sensitivity of noise and motion. Therefore, it is still pressing to develop more efficient and robust acceleration methods in this field.

Recently, deep learning has shown promising ability in extracting patterns and learning complicated mappings from large-scale datasets. Tremendous outstanding works have been proposed to solve challenging tasks in the Computer Vision (CV) field such as image classification, object detection, and image segmentation. In terms of fast MRI reconstruction, a pioneering work done by Wang et al. [7] where a Convolutional Neural Network is developed to approximate the mapping function between undersampled and fully-sampled MR images. Afterward, more deep learning-based methods [[8], [9], [10], [11], [12], [13], [14]], have been proposed for fast MRI reconstruction. It is worth mentioning that several strategies and mechanisms are successfully applied to help the reconstruction network extract more discriminative and comprehensive features. Specifically, some works [15,16] focused on exploiting the complex nature of the sampled MR data by equally treating the real and the imaginary parts of the complex-valued MR image. Apart from that, dual-domain learning has been extensively investigated and proven effective in boosting the quality of reconstructed MR images by extracting local and global features in the spatial and frequency domain simultaneously to reduce artifacts and recover detailed structures. Compared to the aforementioned strategies, deep cascaded networks [[17], [18], [19]], which refine the intermediate reconstructions progressively using a group of sequentially arranged subnetworks, can probably bring the most prominent performance improvements. Among them, Schlemper et al [20] iteratively deployed CNNs and Data Consistency (DC) layers to reconstruct aggressively Cartesian undersampled MR images and obtained outstanding results. More recently, some works [21,22] incorporated the recurrent mechanism into the design of networks, which essentially equals increasing the cascading number of subnetworks.

However, it should be noticed and emphasized that most existing deep cascaded networks employ relatively large cascading numbers to boost the reconstruction performance by increasing refinement steps, which may cause unbearable computation and memory burdens. Moreover, as shown in Fig. 1, the improvement brought by the newly added subnetwork can shrink rapidly as the growth of the cascading number. More undesirably, due to the lack of proper guidance and clear optimization direction, excessively enlarging the cascading number may lead to optimization hurdles and induce performance saturation or even degradation in some cases, which runs counter to the initial purpose of cascading. This undesirable issue leads to a question: Is it possible to guide each subnetwork in pointing out the appropriate refinement direction, thereby assisting the overall network to obtain superior results with fewer cascades?

Motivated by this idea, in this paper, we introduce a novel Reconstruction Residual-Based Feature Modulation Mechanism (RRFMM) to guide each cascaded subnetwork at multiple feature levels. The idea is inspired by the underlying meaning of reconstruction residual that can be computed by subtracting the input image from the corresponding reconstructed image of a subnetwork as shown in Fig. 2. It can be seen that the reconstruction residual map of a subnetwork represents the differences between the input image and the output image and also indicates the locations of those areas that have been refined by the current subnetwork. Additionally, in a progressive reconstruction process, those aliasing areas can not be fixed all at once and still need some polishment in the next subnetwork. Based on this analysis, we believe that the reconstruction residual of the former stage can guide the following stage by highlighting those potential regions that need to be polished. Specifically, we arrange an additional subnetwork parallelly for each reconstruction subnetwork to extract residual features which are then used to generate attention maps to modulate the corresponding image features. In this way, each reconstruction subnetwork will emphasize regions that still require improvement rather than aimlessly updating its parameters. To further boost the overall performance, we propose two kinds of connections to fully leverage the knowledge of previous subnetworks and facilitate network training. Specifically, we introduce the Reconstruction Dense Connection (RDC) that concatenates all the intermediate reconstructions of previous subnetworks as the input for the current subnetwork. Additionally, we fuse the features of the previous subnetwork's decoder with those of the current subnetwork's encoder by the Cross-Stage Feature Reuse Connection (CSFRC). The obtained reconstruction network DIRECTION achieves consistently superior results on the commonly used fastMRI [23] dataset.

To summarize, the innovations and contributions of this work are as follows:

• We propose a novel Reconstruction Residual-Based Feature Modulation Mechanism (RRFMM) that can provide proper guidance to each subnetwork at multiple feature levels and enable each cascaded subnetwork to possess its own optimization direction and reconstruction emphasis. To the best of our knowledge, this is the first work that uses residual information from such a novel perspective to help rebuild the network and improve performance.

• To further facilitate the training of the deep cascaded network and boost the reconstruction performance, we introduce the Reconstruction Dense Connection (RDC) and the Cross-Stage Feature Reuse Connection (CSFRC) that can reduce information loss and enable sufficient feature utilization.

• We conduct comprehensive experiments on the large-scale fastMRI dataset and the experimental results demonstrate that our proposed DIRECTION can consistently outperform other state-of-the-art methods at multiple setups.