Imaging plays a crucial role in both daily life and various industries. The conventional imaging approaches, which capture the target light by a detector array and provide an image that faithfully resembles human vision, are typical “what you see is what you get” imaging methods. Throughout the evolution of imaging technology, there has been a persistent pursuit of surpassing human visual capabilities in terms of speed, field of view (FOV), and detectable spectrum, enabling selective information acquisition.

Despite these advancements, image post-processing1–31. J. Canny, IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 679 (1986). https://doi.org/10.1109/TPAMI.1986.47678512. R. M. Haralick and L. G. Shapiro, Comput. Vis. Graph. Image Process. 29, 100 (1985). https://doi.org/10.1016/S0734-189X(85)90153-73. S.-T. Lee, K.-S. Kim, T.-H. Yoon, J.-W. Lee, K.-D. Kim, and W.-S. Park, Trans. Korean Inst. Electr. Eng. 59, 1136 (2010). is still necessary to extract selective information from images captured by conventional imaging techniques. An alternative approach is to enhance selective information through labeling, such as ultraviolet fluorescence irradiation44. I. H. Chiu, S. Takeda, M. Kajino, A. Shinohara, M. Katsuragawa, S. Nagasawa, R. Tomaru, G. Yabu, T. Takahashi, S. Watanabe, S. Takeshita, Y. Miyake, and K. Ninomiya, Sci. Rep. 12, 5261 (2022). https://doi.org/10.1038/s41598-022-09137-5 or illumination schemes with infrared cutoff filters.55. M. J. Rust, M. Bates, and X. Zhuang, Nat. Methods 3, 793 (2006). https://doi.org/10.1038/nmeth929 No matter what process is employed to obtain selective visual information, current imaging technologies, as an extension of human vision, cannot capture the features of interest directly.Single-pixel imaging (SPI) is a computational imaging approach that reconstructs an image by processing correlations between illumination patterns and one-dimensional voltage signals from one non-spatial-resolution detector.6–86. M.-J. Sun, M. P. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, Nat. Commun. 7, 12010 (2016). https://doi.org/10.1038/ncomms120107. M. P. Edgar, G. M. Gibson, and M. J. Padgett, Nat. Photonics 13, 13 (2019). https://doi.org/10.1038/s41566-018-0300-78. G. M. Gibson, S. D. Johnson, and M. J. Padgett, Opt. Express 28, 28190 (2020). https://doi.org/10.1364/OE.403195 SPI offers advantages such as a wide imaging spectral bandwidth and beyond-visual-field imaging.9–129. Y.-H. He, A.-X. Zhang, M.-F. Li, Y.-Y. Huang, B.-G. Quan, D.-Z. Li, L.-A. Wu, and L.-M. Chen, APL Photonics 5, 056102 (2020). https://doi.org/10.1063/1.514032210. N. Radwell, K. J. Mitchell, G. M. Gibson, M. P. Edgar, R. Bowman, and M. J. Padgett, Optica 1, 285 (2014). https://doi.org/10.1364/OPTICA.1.00028511. G. M. Gibson, B. Sun, M. P. Edgar, D. B. Phillips, N. Hempler, G. T. Maker, G. P. A. Malcolm, and M. J. Padgett, Opt. Express 25, 2998 (2017). https://doi.org/10.1364/OE.25.00299812. X. Yang, Z. Tian, X. Chen, M. Hu, Z. Yi, C. Ouyang, J. Gu, J. Han, and W. Zhang, Appl. Phys. Lett. 116, 241106 (2020). https://doi.org/10.1063/5.0010531 Despite these benefits, current SPI still draws on traditional image post-processing methods for information acquisition, making it still a “what you see is what you get” type of imaging.1313. S. Li, Z. Zhang, X. Ma, and J. Zhong, Opt. Commun. 403, 257 (2017). https://doi.org/10.1016/j.optcom.2017.07.058 Efforts to improve SPI have been focused on different illumination pattern algorithms14,1514. Z. Gao, M. Li, P. Zheng, J. Xiong, Z. Tang, and H.-C. Liu, Opt. Express 30, 35923 (2022). https://doi.org/10.1364/OE.46462515. H. Deng, X. Gao, M. Ma, P. Yao, Q. Guan, X. Zhong, X. Zhong, and J. Zhang, Appl. Phys. Lett. 114, 221906 (2019). https://doi.org/10.1063/1.5097901 and operations with these patterns.16–1816. Y. Liu, P. Yu, X. Hu, Z. Wang, Y. Li, and L. Gong, Opt. Lett. 45, 4028 (2020). https://doi.org/10.1364/OL.39690317. T. Mao, Q. Chen, W. He, Y. Zou, H. Dai, and G. Gu, IEEE Photonics J. 8, 6900810 (2016). https://doi.org/10.1109/JPHOT.2016.257893418. H. Ren, S. Zhao, and J. Gruska, Opt. Express 26, 5501 (2018). https://doi.org/10.1364/OE.26.005501 Based on these attempts, feature extraction methods at a low sampling ratio with single-pixel imaging have been developed in the field of fast tracking19–2119. Q. Deng, Z. Zhang, and J. Zhong, Opt. Lett. 45(17), 4734–4737 (2020). https://doi.org/10.1364/OL.39920420. L. Zha, D. Shi, J. Huang, K. Yuan, W. Meng, W. Yang, R. Jiang, Y. Chen, and Y. Wang, Opt. Express 29(19), 30327–30336 (2021). https://doi.org/10.1364/OE.43634821. D. Shi, K. Yin, J. Huang, K. Yuan, W. Zhu, C. Xie, D. Liu, and Y. Wang, Opt. Commun. 440, 155–162 (2019). https://doi.org/10.1016/j.optcom.2019.02.006 and classification.22,2322. S. Jiao, J. Feng, Y. Gao, T. Lei, Z. Xie, and X. Yuan, Opt. Lett. 44(21), 5186 (2019). https://doi.org/10.1364/OL.44.00518623. P. Latorre-Carmona, V. J. Traver, J. S. Sánchez, and E. Tajahuerce, Image Vision Comput. 86, 28–37 (2019). https://doi.org/10.1016/j.imavis.2019.03.007 However, the one-dimensional signals from the detectors, which contain information about the modulation between illumination patterns and the image target, are still underutilized.In this Letter, we propose a framework for direct feature imaging without image post-processing, based on the processing of one-dimensional signals in SPI. Features refer to the information-containing parts of the image or scene. In this paper, features are sparsely distributed in the frequency domain despite being mixed together in the spatial domain. This means that image energy is concentrated in the large amplitude frequency domain coefficients, with each feature having a distinct distribution. Based on such principle, the framework is proposed and illustrated in Fig. 1.The framework starts by obtaining the spectrum of the scene through Fourier single-pixel imaging (FSPI), which measures the correlation between the scene and a series of Fourier patterns, resulting in a one-dimensional signal that can be translated to the Fourier frequency domain of the scene. A Fourier pattern P(x,y|fx,fy,ϕ) can be characterized by its spatial frequency pair (fx,fy) and initial phase ϕ, Px,y|fx,fy,ϕ=1/2+1/2cos[2π(fx+fy)+ϕ],(1)where (x, y) denotes the coordinate in spatial domain, fx is the spatial frequency along the x direction, fy is the spatial frequency along the y direction, and ϕ is the initial phase of the Fourier basis. By the four-step phase shifting method, FSPI can fully sample the Fourier spectrum of an M pixel × N pixel image from 2 × M × N measurements.2424. Z. Zhang, X. Ma, and J. Zhong, Nat. Commun. 6, 6225 (2015). https://doi.org/10.1038/ncomms7225 These four patterns share the same frequency pair (fx,fy) and only differ in their initial phases ϕ, which are 0,π2,π, and 3π/2, respectively.The modulated energy Eϕ(fx,fy) equals to the inner product of scene image I(x, y) and pattern Px,y|fx,fy,ϕ. The response of a single-pixel detector Dϕfx,fy, thus, consists of the linear response of detector to modulation intensity and noise in actual, Dϕ(fx,fy)=m∑x=1M∑y=1NI(x,y)Px,y|fx,fy,ϕ+Dn.(2)The latter term Dn denotes ambient noise, which is usually considered as a constant due to short duration of modulation for every pattern. The Fourier coefficient C(fx,fy) acquired by the four-step phase shifting method can, thus, be expressed as Cfx,fy=D0−Dπ+jDπ/2−D3π/2=∬Ix,ye−j2πfxx+fyydxdy.(3)The frequency domain acquired by FSPI is normalized and displayed in Fig. 1(b). Then, the frequency domain is separated into several frequency centers by an adaptive 2D variational modal decomposition (2D-VMD) method, which is a non-recursive approach designed for extracting required number of smooth sub-signals with limited bandwidths from two dimensional signal. The total constrained of 2D-VMD is expressed as follows: min∑k=1K∇uAS,kx→e−jωk←,x→22 s.t.∑k=1Kûk=Î.(4)Here, K is the number of modes to decompose, which should not be smaller than the number of features in the frequency domain, which is represented by red boxes in Fig. 1(b). UAS,k(x→) is the 2D analytic signal, and uk and ωk← are shorthand notations for the set of all modes and their center frequencies, respectively. The superscript ⋅̂ means the Fourier transform of variables and the ⟨·⟩ means inner product. Further details about the convergence condition and parameter setup of 2D-VMD are provided in the supplementary material. Note that the 2D-VMD is only conducted once for subsequent feature extraction, which is different from traditional post-image processing but somewhat similar to the pre-training process in machine learning. The comparison between the proposed method and traditional post-image processing in low sampling ratio condition is detailed in the supplementary material.With the aid of 2D-VMD, each separated mode gets its own frequency center in the frequency domain. The amplitude order map is sorted to determine the significance of each mode in Fig. 1(c). Illumination patterns are then generated based on the significance of each mode, and FSPI is conducted to acquire the most n% important Fourier coefficients, which ensures the subsequent feature extraction can be finished at a very low sampling ratio and redundancy. Eventually, the features are reconstructed by means of inverse Fourier transform (IFT).

The primary challenge in single-pixel imaging lies in balancing the trade-off between image resolution and reconstruction speed. The current implementation of FSPI, which employs a basis scanning strategy, results in a temporal resolution that is inversely proportional to the square of the spatial resolution. To address this challenge, the proposed framework utilizes a portion of the basis that contains the key features to reconstruct the image. Hence, the proposed feature imaging offers an alternative solution to the trade-off dilemma between resolution and reconstruction speed. To demonstrate the effectiveness of this approach, an experiment has been performed at static, dynamic, and real-world scenarios with ultra-low sampling ratios.

The single-pixel imaging experimental system, as shown in Fig. 2, includes an illumination source, a projector lens, a spatial light modulator (SLM), and a photoelectric detector. As for static situation, a 12-W flat square LED functions as the illumination source and is capable of producing both bright-field and dark-field illumination, depending on the requirements of the application. As for dynamic situation, a 13.3-in. screen displaying preloaded movie serves as the target. The projector lens collects the reflected or transmitted light from the target, which is then directed toward the surface of the SLM for binary amplitude modulation. In this system, a digital micromirror device (DMD) serves as the SLM, offering high-speed modulation through its programmable micro-mirrors. The modulated intensity is detected by a photomultiplier diode and converted into analog signals for processing.Figure 3 presents the experimental results obtained from the designed target when taking ambient noise into consideration. The results of the four modes, obtained by the 2D-VMD technique, are presented in Fig. 3(a), and the associated importance order maps have been omitted. The DMD approach utilizes masks that are preloaded with the most significant n% coefficients from the corresponding order map for each feature, where n% is selected as 0.01%, 0.025%, 0.5%, 1%, and 2.5%. The corresponding sampling ratios, obtained using the four-step phase shift method, are 0.02%, 0.05%, 1%, 2%, and 5%, which are well below the Nyquist sampling limit. It is noteworthy that the experiment was conducted under real-world conditions, where the environment was illuminated by two fluorescent lamps with a frequency of 100 Hz, rather than in a traditional darkroom setup for SPI, to assess the proposed approach in the presence of noise. The results show that the feature information of the four modes can be retrieved using an extremely low sampling ratio of 0.02% through the application of the coefficient selection strategy. In the case of the first mode, other modes begin to emerge as the order maps overlap in the zero-frequency area of the frequency domain, implying that the frequency centers are not entirely sparse in this domain. Conversely, the contours of the other three modes become clearer as the sampling ratios increase. This provides strong evidence that the proposed method can effectively reconstruct the features of a scene with a sparse frequency domain using ultra-low sampling ratios.The efficacy of the proposed feature imaging scheme in a static environment has been established through experimentation under stringent sampling ratios. However, in real-world scenarios, the location and orientation of features and backgrounds are constantly changing. To address this, a feature dynamic tracking experiment, as illustrated in Fig. 4, has been conducted. The results demonstrate the performance of the proposed method in situations where the target features are shifting in position or when additional, irrelevant features are introduced into the scene. The illumination patterns set from the former experiment are used to evaluate the transferability of the proposed feature imaging method. Here, transferability means the selected illumination patterns set can efficiently track the target feature even when it changes the position and background. The sampling ratio is set at 2%, which is low for FSPI.The first designed target, as depicted in Fig. 4(a), features a deviation in the location of all features from the ones depicted in Fig. 3(a). The reconstruction results, shown in Figs. 4(c)–4(f), demonstrate the ability of the illumination patterns obtained from Fig. 3 to extract features under translation conditions. The second designed target, shown in Fig. 4(g), retains the same positions and contours of all features as those in Fig. 3(a), but with altered frequencies for modes other than the second mode. The result presented in Fig. 4(j) indicates that the second mode can still be accessed despite the interference of the other three irrelevant features, as the illumination patterns used were obtained from a different image. Finally, the last designed target involves changes in both the locations and frequencies of the features. The feature reconstruction results, displayed in Fig. 4(p), demonstrate that the translation of the interested feature and the simultaneous presence of non-relative features do not hinder the extraction of the specific feature. The corresponding dynamic feature tracking results are provided in Movie S1. These results manage to demonstrate dynamic tracking of feature at a frame rate of 33.5 Hz and a spatial resolution of 128 × 128 pixels, which is a challenge for other existing single-pixel imaging methods.

Subsequently, a practical implementation of the proposed method in defects detection of integrated circuit (IC) wafers is demonstrated. Current method of defect detection in the IC wafer industry is based on image processing, resulting in a large amount of data for processing. Differently, we manage to directly imaging defects without the need for additional imaging processing.

The target for imaging in this study is a piece of normal IC wafer. Stripe features are commonly present on the surface of the IC wafers, and the defects in this study are open circuit or strained defects. Two modes are utilized for feature imaging, with mode I being utilized for defect imaging and mode II for stripe imaging.

The proposed feature imaging method is applied for the detection of defects in IC wafers as demonstrated in Fig. 5. For the defect-free area in Fig. 5(b), mode I captures the imaging with no discernible structure in Fig. 5(c), while mode II clearly captures the stripe structures in Fig. 5(d). For an area with open circuit defects, as shown in Fig. 5(e), the open circuit defect is captured by mode I in Fig. 5(f), and the stripe structures with slight discontinuity in the defect area are captured by mode II in Fig. 5(g). Similarly, for Fig. 5(h), mode I captures the imaging of the strained defect, while mode II captures the stripe imaging. Additionally, as the sampling ratio is only 2%, the data produced and processed by the proposed feature imaging method are significantly smaller than that produced by CCD, which may help to improve the efficiency of defect detection.The proposed scheme has been shown to be suitable for processing fringe pattern images, and to test its ability to handle naturally smooth objects, another simulation experiment was performed on chest x-ray images, details are given in Sec. S3 of the supplementary material. The results show that the proposed method has the potential to handle smooth images, but further research is needed.

In conclusion, we present a direct feature imaging approach that utilizes the one-dimensional signal to reconstruct the feature, enabling the rapid capture of features with ultra-small data. We make the assumption that the feature information is sparsely distributed in the Fourier frequency domain, which is inherent in the one-dimensional signal of Fourier single-pixel imaging. This assumption is reasonable for many industrial and medical scenes where the features of interest are primarily defects or foreign matter. Moreover, this study presents a framework to single-pixel imaging that diverges from conventional methods by directly and selectively capturing feature information. The framework is not restricted to the Fourier frequency domain, and further exploration is necessary to fully exploit its capabilities. This framework holds promise for the development of selective feature single-pixel imaging and has applications in detection and diagnosis due to its effectiveness in obtaining specific information from a scene.

See the supplementary material for the details of the adaptive 2D-VMD algorithm, a comparison between the proposed method and traditional post-image processing under low sampling ratio condition, the results of the proposed method dealing with a natural smooth picture, and results of the dynamic feature tracking experiment (Movie S1).

This study was supported by the National Natural Science Foundation of China (No. 12132016), the Fundamental Research Funds for the Central Universities (Nos. WK2090000039 and WK2480000010), and the 100 Talents Program of Chinese Academy of Sciences (No. KY2090000077).

Conflict of Interest

The authors have no conflicts to disclose.

Author Contributions

Mingyang Ni: Conceptualization (lead); Data curation (lead); Formal analysis (lead); Investigation (equal); Methodology (lead); Software (lead); Validation (lead); Visualization (lead); Writing – original draft (lead). Huaxia Deng: Conceptualization (equal); Funding acquisition (equal); Writing – review & editing (equal). Xingzhi Xiao: Methodology (equal). Yu Cai: Formal analysis (equal). Xinglong Gong: Funding acquisition (equal); Project administration (equal); Resources (equal); Supervision (equal).

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

REFERENCES

1. J. Canny, IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8, 679 (1986). https://doi.org/10.1109/TPAMI.1986.4767851, Google ScholarCrossref, ISI2. R. M. Haralick and L. G. Shapiro, Comput. Vis. Graph. Image Process. 29, 100 (1985). https://doi.org/10.1016/S0734-189X(85)90153-7, Google ScholarCrossref3. S.-T. Lee, K.-S. Kim, T.-H. Yoon, J.-W. Lee, K.-D. Kim, and W.-S. Park, Trans. Korean Inst. Electr. Eng. 59, 1136 (2010). Google Scholar4. I. H. Chiu, S. Takeda, M. Kajino, A. Shinohara, M. Katsuragawa, S. Nagasawa, R. Tomaru, G. Yabu, T. Takahashi, S. Watanabe, S. Takeshita, Y. Miyake, and K. Ninomiya, Sci. Rep. 12, 5261 (2022). https://doi.org/10.1038/s41598-022-09137-5, Google ScholarCrossref5. M. J. Rust, M. Bates, and X. Zhuang, Nat. Methods 3, 793 (2006). https://doi.org/10.1038/nmeth929, Google ScholarCrossref, ISI6. M.-J. Sun, M. P. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, Nat. Commun. 7, 12010 (2016). https://doi.org/10.1038/ncomms12010, Google ScholarCrossref7. M. P. Edgar, G. M. Gibson, and M. J. Padgett, Nat. Photonics 13, 13 (2019). https://doi.org/10.1038/s41566-018-0300-7, Google ScholarCrossref8. G. M. Gibson, S. D. Johnson, and M. J. Padgett, Opt. Express 28, 28190 (2020). https://doi.org/10.1364/OE.403195, Google ScholarCrossref9. Y.-H. He, A.-X. Zhang, M.-F. Li, Y.-Y. Huang, B.-G. Quan, D.-Z. Li, L.-A. Wu, and L.-M. Chen, APL Photonics 5, 056102 (2020). https://doi.org/10.1063/1.5140322, Google ScholarScitation, ISI10. N. Radwell, K. J. Mitchell, G. M. Gibson, M. P. Edgar, R. Bowman, and M. J. Padgett, Optica 1, 285 (2014). https://doi.org/10.1364/OPTICA.1.000285, Google ScholarCrossref11. G. M. Gibson, B. Sun, M. P. Edgar, D. B. Phillips, N. Hempler, G. T. Maker, G. P. A. Malcolm, and M. J. Padgett, Opt. Express 25, 2998 (2017). https://doi.org/10.1364/OE.25.002998, Google ScholarCrossref12. X. Yang, Z. Tian, X. Chen, M. Hu, Z. Yi, C. Ouyang, J. Gu, J. Han, and W. Zhang, Appl. Phys. Lett. 116, 241106 (2020). https://doi.org/10.1063/5.0010531, Google ScholarScitation, ISI13. S. Li, Z. Zhang, X. Ma, and J. Zhong, Opt. Commun. 403, 257 (2017). https://doi.org/10.1016/j.optcom.2017.07.058, Google ScholarCrossref14. Z. Gao, M. Li, P. Zheng, J. Xiong, Z. Tang, and H.-C. Liu, Opt. Express 30, 35923 (2022). https://doi.org/10.1364/OE.464625, Google ScholarCrossref15. H. Deng, X. Gao, M. Ma, P. Yao, Q. Guan, X. Zhong, X. Zhong, and J. Zhang, Appl. Phys. Lett. 114, 221906 (2019). https://doi.org/10.1063/1.5097901, Google ScholarScitation, ISI16. Y. Liu, P. Yu, X. Hu, Z. Wang, Y. Li, and L. Gong, Opt. Lett. 45, 4028 (2020). https://doi.org/10.1364/OL.396903, Google ScholarCrossref17. T. Mao, Q. Chen, W. He, Y. Zou, H. Dai, and G. Gu, IEEE Photonics J. 8, 6900810 (2016). https://doi.org/10.1109/JPHOT.2016.2578934, Google ScholarCrossref18. H. Ren, S. Zhao, and J. Gruska, Opt. Express 26, 5501 (2018). https://doi.org/10.1364/OE.26.005501, Google ScholarCrossref19. Q. Deng, Z. Zhang, and J. Zhong, Opt. Lett. 45(17), 4734–4737 (2020). https://doi.org/10.1364/OL.399204, Google ScholarCrossref20. L. Zha, D. Shi, J. Huang, K. Yuan, W. Meng, W. Yang, R. Jiang, Y. Chen, and Y. Wang, Opt. Express 29(19), 30327–30336 (2021). https://doi.org/10.1364/OE.436348, Google ScholarCrossref21. D. Shi, K. Yin, J. Huang, K. Yuan, W. Zhu, C. Xie, D. Liu, and Y. Wang, Opt. Commun. 440, 155–162 (2019). https://doi.org/10.1016/j.optcom.2019.02.006, Google ScholarCrossref22. S. Jiao, J. Feng, Y. Gao, T. Lei, Z. Xie, and X. Yuan, Opt. Lett. 44(21), 5186 (2019). https://doi.org/10.1364/OL.44.005186, Google ScholarCrossref23. P. Latorre-Carmona, V. J. Traver, J. S. Sánchez, and E. Tajahuerce, Image Vision Comput. 86, 28–37 (2019). https://doi.org/10.1016/j.imavis.2019.03.007, Google ScholarCrossref24. Z. Zhang, X. Ma, and J. Zhong, Nat. Commun. 6, 6225 (2015). https://doi.org/10.1038/ncomms7225, Google ScholarCrossref© 2023 Author(s). Published under an exclusive license by AIP Publishing.