This system can still accurately locate liver tumor, even if breathing has caused the liver’s position to be different from when the CT was taken. These three key technologies mentioned above can be used to achieve this goal.

Identifying and puncturing liver tumors directly in ultrasound images is the most direct and effective surgical option. However, liver tumors may not be clear on ultrasound images and have various shapes, and it is difficult to accurately identify all kinds of liver tumors by one model or algorithm. Blood vessels inside the liver are abundant and have clear outlines in ultrasound image, making them easy to accurately identify. Breathing affects the position of the liver in the abdominal cavity and can cause deformation of both blood vessels and tumors. Tumors and blood vessels are fixed inside the liver, and their relative positional relationship is unchanged. Therefore, easily identifiable vascular structures under ultrasound can be used to accurately locate tumors.

In ultrasound images, blood vessels will appear in a hypoechoic state, with obvious grayscale differences between the blood vessel wall and surrounding tissue. Blood vessels in the liver can be identified by a variety of methods [10,11,12]. To reduce misidentification, we use improved U-Net model and add preprocessing during input, data filtering, and judgment during output. Only classes with a confidence level above 0.5 are considered correct outputs, and the others are considered background. In addition, continuous regions with a number of pixels less than 40 or greater than 50,000 are also considered misidentifications. Using this method, we obtained two AI models for phantom and pig liver blood vessel recognition using the training set and validation set as shown in Table 2. The test results of the two models on the test set show that the average recognition Intersection over Union (IOU) of pig liver vessels and phantom vessels were 0.73 and 0.82. In Fig. 2, the green contour lines (width about 1 pixel) are annotation results of blood vessels and considered ground true. The red masks are the recognition results of blood vessels. The contour lines and masks of other colors represent the recognition results of other tissues (the yellow area represents the gallbladder, and the purple area represents the tumor).

Liver blood vessel identification results

The identification of liver blood vessels is the basis for further reconstruction of blood vessels point cloud. In this robotic system, the ultrasound probe is rigidly attached to the end of the robotic arm, and robot arm has the function to get the positional information of ultrasound probe. Therefore, the spatial position information of all ultrasound images acquired by this system can be obtained. Briefly, the point cloud reconstruction steps: identify the blood vessels in the ultrasound image, obtain the real-time positional information of ultrasound images, and then construct point cloud of liver blood vessels according to the reconstruction algorithm.

The liver is rich in blood vessel, each part of the blood vessel has obvious characteristics and is easy to identify. There are different methods to obtain blood vessel point clouds from CT data [13,14,15]. We developed preoperative planning software based on the Medical Imaging Interaction Toolkit (MITK) for CT data segmentation and needle insertion trajectory planning.

A feature point-based registration method based on improved ICP algorithm [16] is used in this system. The first step is to calculate feature points from CT point cloud and ultrasound point cloud. The root-mean-square average distance (RMS-A) of paired feature points is used for registration evaluation. This method not only fully considers the characteristics of liver blood vessels, making tumor prediction more accurate, but also reduces the number of calculation points, speeds up the calculation, and improves registration efficiency. The efficiency of this method is more than twice that of traditional solutions, and the average distance of the feature point groups after registration is about 0.1 mm. Figure 3b shows the extraction of registration feature points and the registration process of point clouds. After finishing the registration, the system will automatically calculate the CT slices that match the current ultrasound image based on the registration matrix. Then, the CT slices are scaled according to the proportion of the ultrasound image and fused with the current ultrasound image.

Schematic of point cloud registration based on feature points

After the above registration and fusion process are completed, robot arm carries puncture guide mechanism to the planned puncture site. The accuracy of the puncture guide mechanism will determine the success of puncture surgery. To ensure the rigidity, accuracy, and stability of it, we abandoned the traditional parallel multi-link structure and adopted a customized design based on high-strength arc guide rails, high-strength linear guide rails, and high-precision servo motors. The theoretical control accuracy of the needle tip position can reach 0.1 mm by using this design. Since the dimensions of each component of the puncture guide mechanism are known and the installation position of the ultrasound probe is fixed, the pre-pass trajectory of the puncture needle can be predicted based on the puncture angle and depth. The pre-pass trajectory the needle is marked in green dotted line. Figure 4 shows the accuracy test of puncture guide mechanism, the results indicate that the average puncture accuracy of it in water (the deviation between the needle tip and the end of the guide wire) is within 0.4 mm, and the average puncture accuracy in silicone tissue (used to simulate human body) is within 0.6 mm.

Navigation accuracy of puncture guidance mechanism