The concept describes a system that combines high-speed microscopy, image analysis, and high throughput cell isolation in one device. To address the demand of existing and future biomedical applications, the technology needs to reach both high throughput and high precision for cell analysis and isolation. Experiments have shown that high-speed microscopy can scan a complete microtiter plate in less than a minute with a resolution of 1.69 μm/Pixel. A fast and highly efficient process can be achieved in combination with LIFT, a pure optical and contact-free transfer technology, with process times in the microsecond regime for one single cell transfer.

Hardware integration

The integration of analysis and isolation of cells into one process is achieved by physically combining the necessary hardware in one microscopic setup. Therefore, a laser system (Nanolevante, APE Berlin, Germany) for cell isolation is added as an external unit to the microscope. The beam is led through the optical path of the inverse microscope (microscope: Eclipse Ti 2, Nikon, Japan; objective lens: CFI Plan Fluor DL 4XF, Nikon, Japan; magnification: 4, NA: 0.13) using its rear ports [14]. The concept is proven in a demonstrator (Fig. 1). By using the filter cube turret the system can switch between camera observation (camera: pco Edge 4.2, pco, Germany) and laser application. When laser mode is selected, the beam passes through the microscope’s optical path into the objective (selfbuilt objective, NA 0,2) focusing on the exact position of the cell to be transferred [15]. The moveable x,y-microscope stage (ScanPlus IM 130 × 85, Märzhäuser Wetzlar, Germany) can be used for high-speed image acquisition and cell positioning for the LIFT. The microscope can take phase-contrast images besides regular bright-field images [16]. The focal plane is adjusted by a piezo z-stage (SPS-D90500-001, nanoFaktur, Germany). All moveable components (x,y-microscope stage, piezo z-stage, filter cube turret, objective revolver) are controlled by a motion controller (TANGO 4 Dektop, Märzhäuser Wetzlar, Germany). The whole process is managed by one central computer next to the demonstrator.

Fig. 1

A Photo of the demonstrator. The demonstrator uses a Nikon Eclipse Ti 2 microscope body and a Märzhäuser Wetzlar Scan IM stage. It is controlled by the accompanying computer on the right in the picture. The laser currently used for the LIFT is a prototype system. In the final LIFTOSCOPE it can be replaced by a commercial laser source. B Schematics of a laser beam integrated into the light path of the inverse microscope. LIFT Laser and optical path for high-speed microscope are combined using the filter cube turret. Additionally, the standard eyepiece can be used

High-speed microscopy

A continuous scan process is used as proposed by Schenk [8]. Cell images are acquired during a continuous moving process. The camera allows up to 100 frames per second with a resolution of 2048 × 2048 pixels. The individual images are stitched to one image for each well, which can then be automatically analyzed.

To obtain sharp images throughout despite the microscope’s low depth of field and the uneven MTP bottom, an autofocus system is used. Contrary to the process by Schenk et al. [8], a software autofocus is implemented by acquiring a z-stack of images at multiple positions equally spread over each well [17]. At each position, the motor moves vertically to a predefined position that corresponds approximately to the focal plane. Then images are continuously taken while the piezo travels its full range. The sharpest of those images corresponds to the focal plane. Using multiple images, the surface of the well bottom can be interpolated and extrapolated to generate a focal map. During the main image acquisition process, the piezo positioning system adjusts the sample height in real time based on the previously created focal map. To further speed up the focusing process, a future implementation of the setup can include the hardware autofocus system presented by Schenk [8].

Time-effective LIFT process

The key feature of the LIFTOSCOPE concept is the fast transfer by utilizing the continuous moving process of high-speed microscopy. The transfer process works equivalent to the image acquisition process, but instead of taking images, a laser is triggered to perform a LIFT. Therefore, the microscope stage moves along predefined paths. Synchronous to the movement of the x,y-stage, the z-stage moves according to the focal map acquired at the beginning of the microscopic scan process (see section III.B). The position of cells to be transferred is known to the algorithm as a result of the AI-based image analysis. When a target cell is positioned at the microscope’s focal point along the stage movement, a path synchronous TTL trigger signal (transistor-transistor logic) is initiated by the motion controller to start the laser for the LIFT process. The exact microscope stage position is determined by its encoder and processed by the motion controller, that also sends commands to the z-stage using an analog output signal.

Since the LIFT itself only takes around 200 μs, multiple transfers can be conducted within a second. A target of 100 cell transfers per second is set for efficient overall performance yielding a time of 100 s for an assumed 10,000 transferred cells, which is in the order of magnitude of image acquisition.

Instead of the continuous moving process described above, a ‘stop-and-go’ algorithm could more easily be implemented. Thereby, the microscope stage would move to each cell position individually and stop for every cell transfer. However, the multitude of accelerations results in the sloshing of the cell culture medium in the wells. In order to circumvent that, stop times in the order of 1 s are needed for each transfer, which drastically increases the total time.

Consequently, the continuous moving method is used. To make this reliable, it has to be taken into consideration that LIFT affects an extended area around the laser focus from which cells will be transferred [18]. Assuming a radius for guaranteed cell transfer of 25 μm, this can be used to optimize the path such that not every position will precisely be reached, but only within this tolerance. Subsequently, the microscope stage will move in straight lines 50 μm apart and trigger the laser beam in an area of up to 25 μm away from the target positions. If one of the lines does not come into the tolerance area of a transferable cell, it will be skipped. This process is shown schematically in Fig. 2.

Fig. 2

Schematic scanning path for single cell LIFT over a six-well MTP (Distance Δy not to scale). Black circles represent target cells. The laser moves along the parallel lines and is triggered if a target lies within the LIFT tolerance (blue circles around red crosses). If no target lies within the tolerance of a line, the line is skipped completely (dashed lines)

The total time for this procedure using an MTP with a usable area of 120 mm ∙ 80 mm is calculated as follows. To reduce acceleration time and achieve a higher average velocity, the number of lines must be minimized. Thus, the stage will move continuously along the long axis of the rectangular MTP. Since the area of interest is Ly = 80 mm wide and the spacing between two lines is ΔLy = 50 μm, there are N = 80 mm/50 μm = 1600 lines in total. The movement time for each line can be calculated assuming constant acceleration using Newton’s equations of motion [19], which yields

$$_=2 \frac}+\frac_}}\\ for\ _x\ge 2 _,\end}$$

(1)

where vmax represents the maximum velocity and a the acceleration of the microscopic stage, respectively. Lx, accel stands for the distance needed to accelerate to the maximum velocity and decelerate from the maximum velocity to a standstill. It is calculated using the following formula:

$$_=\fraca_^2=\fraca}\right)}^2=\frac\frac^2}.$$

(2)

The LIFTOSCOPE demonstrator reaches vmax = 120 mm/s and a = 500mm/s2. To reduce sloshing, a maximum acceleration of a = 100mm/s2 is used, which thereby reduces the maximum reached velocity to vmax = 109.5 mm/s. These values are used for calculations, resulting in tline = 2.26 s, plus the time to switch lines (lswitch = 0.04 s) and a pause to calm down the liquid. Conclusively, the total time is

$$_=N_+\left(N-1\right)_\\ \approx N\left(_+_\right).\end}$$

(3)

The resulting time is now compared between this algorithm and a regular stop-and-go for benchmarking. To generate enough data without screening too many real cell cultures, a Python script has been implemented to randomly distribute cells within a six-well-plate and store their positions as XML file. This file is read into two further Python scripts, calculating the transfer time with the stop-and-go and the continuous method, respectively. Studies are carried out with cell numbers of 100; 1000; 10,000; and 100,000; where five random distributions have been created each. The stop-and-go algorithm is implemented such that it works through one well after another, starting at the cell with the lowest x-value each and then moving to the coordinate with the next higher x-value. Results of this simulation are shown in section IV.

MTP adapter

For LIFT, in addition to the MTPs containing cells to be examined, receivers are needed to which the selected cells are transferred. From there, the cells can be cultivated or examined with different methods [20, 21].

As receivers, glass plates are chosen, positioned 2 mm above the bottom of each well. Successful high-precision transfers over this distance and greater have previously been shown [11]. Since there is no hardware to position the receiver plates for six-well MTPs, a new holder has been designed for LIFTOSCOPE experiments. It is depicted in Fig. 3 and consists of a plastic housing that fits tightly on top of a six-well MTP. Each well contains four sheet metal springs with hooks at the lower end, spaced equally around the circumference. This assembly allows the round glass plate to be mounted at a fixed distance to the well bottoms so that cells can reproducibly be transferred.

Fig. 3

Rendering of the MTP adapter with section view. Glass plates are inserted into each well

Bottom-up LIFT with mid-infrared radiation

A pulsed laser source with pulse durations of 9 ns, pulse energies ranging from 5 μJ to 50 μJ, and a wavelength of 2940 nm is used for the LIFTOSCOPE prototype. The advantage of this wavelength is twofold. On the one hand, most polymers used for standard MTP plates show low absorption in this regime, allowing the laser to penetrate the MTP material and reach the probe. On the other hand, this wavelength matches the water absorption peak for electromagnetic radiation, hence the laser radiation is absorbed strongly in the cell culture medium surrounding the cell. This avoids the need for an additional metal or polymer absorption layer. Thus standard MTP plates can be used for the process with little preparation. The processing time of a single transfer event for an individual cell or a cell cluster in cell culture medium ranges from 10 μs to 200 μs for the transfer itself and up to 1 ms to return to the initial state [22]. This processing time increases with larger transferred volumes and distances and decreases with higher laser pulse energies [19,20,21]. The LIFT process is well investigated for the transfer of living cells with a cell survival rate close to 100% [23].

Cell culture and live/dead assay

Henrietta Lacks (HeLa) epithelial cells were used to test influence of the LIFT on adherently growing cells. The cells were cultivated with Dulbecco’s modified eagle medium (DMEM) supplemented with 10% fetal bovine serum (FBS) and 100 U/mL penicillin-streptomycinat at 100% humidity, 37 °C, and 5% CO2.

The cells were stained directly after the cell removal using a solution of 8 μL fluorescein diacetate (FDA; 5 mg/mL in acetone) and 50 μL propidium iodide (PI; 2 mg/mL in Dulbecco’s phosphate-buffered saline (DPBS)) in 5 mL DPBS. After printing, the cells were rinsed with DPBS and incubated for 5 min at room temperature in the staining solution. Finally the cells were rinsed with DPBS again.

Cell removal experiment using LIFT

In a first cell study, the influenced area for LIFT was tested in a cell removal experiment using adherent HeLa cells without a separation layer in six-well MTPs. By omitting the separation layer, no hydrogel is needed, while it is expected that the direct irradiation of the cells causes high mortality in the transferred cells.

After cultivation in a six-well MTP, the amount of cell culture medium was reduced to a thin layer of roughly 150 μm thickness to form a transfer layer. The lid of the MTP was replaced with the MTP-Adapter and cell clusters were removed using individual laser pulses. A live/dead staining of the cell culture in the MTP was conducted using fluorescein diacetate (FDA) and propidium iodide (PI). Results of the experiments are presented in section IV.

AI-based cell analysis

Since the LIFTOSCOPE aims to analyze and separate a large variety of cells, the specific cell recognition and classification algorithms depend on the application. For this purpose, training data must be obtained and classified to train a neural network. Potentially, hyperparameters should be tuned to optimize the training results [9, 24, 25].

The analysis algorithm must be given images acquired by the high-speed microscope and return an array of two-dimensional target parameters for the LIFT. Thus, the algorithm must be able to select specific cells and calculate a central point, ideally the center of gravity. The neural network can either directly output a list of cells to be transferred or an XML-based property list, which is then used by a rule-based algorithm to select appropriate targets. Figure 4 shows an example of using a pre-trained algorithm to identify induced pluripotent stem cells (iPSC) [9]. It utilizes the PHANTAST algorithm to generate a foreground segmented image, which is in turn transferred into a U-Net architecture [10]. It then provides an image where all iPSCs are marked and an xml file containing data for each colony. For a proof of concept, the first task will be to identify all iPSC colonies within the region of interest and transfer them to the attached receiver plate [6].

Fig. 4

Steps of image processing for transfer preparation: A iPSC culture in one well of a six-well-MTP in phase-contrast microscopy with 4x objective and a resolution of 1.3 μm/px. B Section of the phase contrast area with multi-class segmentation via deep learning. Grey: iPSC colonies, blue: single iPSCs, black: background, purple: detached cells, red: differentiated cells, pink: dead cells. C Zoomed view of the colonies, border of the colonies highlighted in green, center as a red cross. The unique colony number and the roundness are written into the image. D XML representation using colony 221 as an example. centerX: center point X-coordinate in px, centerY: center point Y-coordinate in px, size: area of the colony in px2, length: circumference in px, roundness: 4π ∙ size/length, childrenSize: area of inclusions of a different class within the colony, colonyPercent: Percentage of the colony compared to the total area including inclusions

To reduce calculation and training time of the algorithm, the image of the whole well is cropped into multiple smaller sections. The algorithm outputs local positions of cell colonies within those image sections in pixels. The algorithm for the cell transfer, however, needs absolute positions within the MTP in milimeters. Therefore, the absolute position in milimeters is calculated from the position within the image section in pixels, the section’s offset to the boundary, and the camera chip pixel size and magnification. Due to the stitching algorithm, the single frames within an image might slightly be shifted, but only in the range of less than 10 μm. The LIFT process, however, allows tolerances of up to 50 μm.

Results and further challenges

The initial prototype setup consisting of the laser (including its periphery), the microscopic stage, and an optical path made of multiple mirrors has been assembled. It has been used to prove the general suitability of its components for a full demonstrator. Most importantly removal of cells from the MTP could be shown as described below. Therefore, this assembly is suitable for incorporating into the final LIFTOSCOPE device.

It was shown that cells can be removed reliably. Previous experiments using LIFT have shown that cells can be transferred reliably even if the laser beam is focused with a lateral deviation of 25 μm relative to the cell position [26]. This can be taken advantage of when calculating a time-optimized path for the continuously moving MTP, as presented above. However, the precision of LIFT during continuous movement must be evaluated in future tests. Moreover, studying a new algorithm which is able to transfer cells with smaller tollerances and short process times is one of the most important remaining challenges of LIFTOSCOPE.

The efficient transfer method introduced in section III.C has been compared with a stop-and-go process. The results are shown in Table 1. It can be deducted that the continuous process is more time efficient for a larger number of cells, whereas it underperformes for cell numbers up to a few thousand. This is due to the fact that the chance of multiple cells lying in a line in the direction of motion increases with the cell count. Therefore, the time to transfer 100,000 cells is only slightly higher than to transfer 10,000 cells, while this relationship is almost linear for the stop-and-go process. Moreover, a continuous process is advantages if a long pause (in the order of 1 s) is required to reduce sloshing of the liquids. For the targeted 10,000 cells and assumed 1 s pause, the continuous process is around 3.5 times faster than stop-and-go. This ratio is increased if a microscope stage with higher maximum velocity and lower acceleration is used.

Table 1 Comparison between stop-and-go method and continuous lift method using simulated cell positions. The average time in seconds to transfer cells of five different distributions for each number of points is shown. The influence of the pause time after each stop is examined by using two different pause times. The ratio of the resulting times (stop-and-go time divided by continuous time) is also given. The maximum velocity is assumed to be 120 mm/s, a linear acceleration of 100 mm/s2 is used

Another point that has to be investigated is the transfer of adherently growing cells. To evaluate the influenced area for LIFT adherently growing HeLa cell clusters were removed from the bottom of a MTP using individual laser pulses. On the MTP, each laser pulse created an empty area of between 80 μm and 100 μm in diameter around its focus (depicted in Fig. 5). Cells outside this area were unharmed by the process, while some cells died at the edge of the area. The cells targeted for removal died during transfer, as seen in Fig. 5, first column, middle row, and third column top and bottom row. For the future, transfer of living adherent growing cells a thin hydrogel layer will be added at the bottom of the MTP plate, to avoid direct irradiation of the cells. Therefore, hydrogels and other separating layers are under investigation that are both inert and structurally stable to support the cells during cultivation and, at the same time, allow for cell adhesion. Possible candidates include, but are not limited to, Matrigel [27, 28], crosslinked gelatin [29], and chitosan [30, 31]. This separation layer will be coated on the bottom of the MTPs before the cells are cultivated and guarantees that no irradiation or thermal damage is introduced to the cells during the LIFT process. This will reduce cell damage and decrease the area of influence in the cell layer [26, 32].

Fig. 5

Cell culture of adherently growing HeLa cells in a six-well MTP with a live/dead staining using FDA and PI. Empty areas were induced using laser pulses with 46 μJ pulse energy and 25 μm focus diameter

The influenced area also scales with the used laser pulse energy. A larger area allows for a faster process since it gives the path-finding algorithm more freedom if a high cell transfer accuracy is not required.

Furthermore, the influence of sample movement on the LIFT process has not yet been tested. Assuming a stable velocity of 120 mm/s and a standard LIFT time of 200 μs/jet, the sample would move 24 μm during the LIFT process. A jet diameter of 20 μm to 50 μm will most likely result in an oblique jet. Trials to quantify this effect on the LIFT quality are currently conducted.

The AI-based cell analysis software has proven to detect iPSCs reliably and will therefore be used for the LIFTOSCOPE prototype as is [9].