In modern surgical practice, computer-generated modelling has emerged as a potentially transformative tool, revolutionising the planning and execution of complex surgical procedures. This chapter details 3D reconstruction methods and their multifaceted applications in surgery, underscoring a significant advancement in medical technology.

3D reconstruction, a cornerstone of computer-generated modelling in surgery, entails creating a three-dimensional model of anatomical structures from two-dimensional image data. This process typically begins with acquiring medical imaging data, such as computed tomography (CT) scans, magnetic resonance imaging (MRI), or ultrasound images. A comprehensive 3D representation of the patient's internal anatomy can be extracted from imaging data using a variety of computer-assisted and advanced algorithmic solutions. These methods have evolved from basic renderings to highly sophisticated models with exceptional detail and accuracy.

This chapter will explore various techniques for 3D reconstruction, including volume rendering, surface rendering, and voxel-based methods. Each technique has its unique advantages and limitations, which will be discussed regarding suitability for different surgical planning and intervention types.

The applications of computer-generated modelling in surgery are vast and varied. Surgeons utilise these models for preoperative planning, allowing for precise visualisation of the operative field before making any incisions. These models may improve surgical accuracy and efficiency, reduce intraoperative risks, and ultimately improve patient outcomes. In complex procedures, such as craniofacial surgery or organ transplantation, 3D models provide unprecedented detail, aiding surgeons in navigating critical structures.

Furthermore, this chapter will address the role of computer-generated models in surgical simulation and training. These models offer a safe and effective environment for surgeons to hone their skills and practice intricate procedures without risk to patients.

Segmenting Digital Imaging and Communications in Medicine (DICOM) is the standard format used to visualise medical data. This involves categorising image data into distinct structures and assigning labels to each voxel based on its membership to a particular anatomical structure.

Different methods and tools exist for segmenting images, ranging from entirely manual to partially or fully automated approaches. The degree of automation employed typically hinges on the quality and nature of the image being analysed.

Manual segmentation can be helpful even in difficult imaging situations due to its simple implementation. With this technique, a trained technician, surgeon, or radiologist labels structures on each slice of the imaging scan. However, it is time-consuming and needs more reproducibility since the segmentation accuracy relies heavily on the skill and consistency of the individual performing the task. Its quality is also highly dependent on the ability of the operator to identify structures correctly and comprehensively. Given these limitations, an automated solution to segmentation is desirable. Algorithmic approaches aim to yield fast, accurate and reproducible results. Some well-established methods include:

Thresholding: This method involves setting a global or interval lower and upper threshold to create a binary image, classifying voxels as either part of the target structure or background. This is commonly used in CT data segmentation for bone. However, thresholding may result in incomplete segmentation due to the partial volume effect1.

Region-Based Segmentation: This technique is predicated on homogeneity, assuming target structures have similar, clustered pixels. It is similar to threshold-based segmentation but focuses on one connected component. The process starts from user-selected points (seeds) and aggregates neighbouring voxels recursively until specific criteria are met. This method is often used for segmenting contrast-enhanced vascular structures but may yield incomplete results due to the partial volume effect2.

Edge-Based Segmentation: This approach relies on image data discontinuities, typically indicated by rapid changes in pixel signal intensity between two different structures (1).

Atlas-Based Segmentation: Here, the geometry and features of organs, blood vessels, and soft tissues are compiled into an extensive database to form a statistical atlas. This atlas, representing anatomical variations in patient populations, is used to iteratively deform Statistical Shape Models (SSMs) to fit new target structures based on shapes derived from the atlas' training set of labelled data1,3.

However, these techniques are incomplete owing to the inherent challenges in imaging modalities, such as signal noise, partial volume effect (signal loss in small structures due to the imaging system's limited resolution), and other imaging artefacts (e.g., motion, ring, intensity inhomogeneity). Machine learning techniques have emerged as an exciting technique that may facilitate automatic image segmentation of complex medical imaging.

Convolutional neural networks (CNNs) are specialised neural networks that have become a dominant technique for medical image segmentation.

They are empowered by their aptitude for automatically learning spatial hierarchies of features from pixel data. Their architecture uniquely suits grid-like inputs, as found in medical imaging data (see Figure 2, Figure 3), with layers organised spatially and connected sparsely4.

CNNs contain three primary types of layers: convolutional, pooling, and activation layers (figure 4). Convolutional layers apply a set of filters to the input, enabling feature extraction. Pooling layers downsample the feature maps output by convolutional layers, reducing computational requirements and expanding the receptive field (figure 5). Activation layers introduce non-linearities through functions like ReLU (figure 6), enabling the network to learn complex data patterns5.

The essence of CNNs lies in convolution, which involves applying small kernels or filters (e.g., 3 × 3 or 5 × 5 matrices) to the image via dot product operations. This transforms local pixels into feature map values, with the kernel sliding across the image by a specified stride parameter (figure 6)6. The resulting feature map indicates the presence and intensity of detected features at different spatial locations. Each kernel detects particular features, with deeper convolutional layers learning more complex feature hierarchies through this process7.

During training, backpropagation optimises the kernel weights so the network can identify task-relevant features effectively8. Convolution is thus a dynamic feature extraction procedure, converting visual inputs into analytical data critical for tasks like classification and segmentation9.

For segmentation, CNN architectures like U-Net shine. U-Net's unique symmetrical "U" shape integrates a contracting path to capture context and an expansive path, enabling precise localisation10. Skip connections retain spatial information, while max pooling extracts feature hierarchies optimally (figure 7)11.

CNNs have been applied successfully in diverse medical segmentation challenges. For lung tumour segmentation in CT scans, a modified 3D U-Net approach achieved a Dice score of 0.83 (a Dice score, also known as F1 score or Sørensen–Dice coefficient, is a standard evaluation metric used to measure the pixel-wise overlap accuracy of a segmentation model)12. A multilevel convolutional network called V-Net produced state-of-the-art performance for prostate MRI segmentation13. Deep CNNs have also excelled in neuron reconstruction from electron microscopy imagery14.

Careful architecture tuning balances overfitting and underfitting risks (figure 8). Augmentation, regularisation, early stopping, and appropriate model complexity help reduce overfitting. More data, longer training, and enhanced features counter underfitting. The ultimate aim is optimising generalisation - where the model excels equally on training and unseen data5.

Deep learning has enabled remarkable advances in medical image analysis, but successfully applying these powerful techniques requires thoughtful data pre-processing. Appropriate conditioning of the input data is critical to enabling the neural networks to achieve optimal performance and generalisation15.

Common pre-processing steps help normalise variations, enhance features of interest, reduce noise, standardise geometry, and expand limited datasets. For example, pixel value normalisation through rescaling scans to a standard range helps account for differences in imaging equipment and protocols16. Contrast enhancement selectively amplifies pertinent structures like lesions while suppressing confounding noise17. Spatial registration corrects for inconsistencies in patient positioning by mapping all scans to a standard anatomical coordinate space18.

Data augmentation artificially expands datasets by applying realistic transformations like rotations, flips, zooms, and elastic warping19. This guards against overfitting and improves the network's generalisation ability20. Carefully designed pre-processing pipelines tailored to the imaging modality, network architecture, and diagnostic goals are indispensable for training robust models capable of expert-level feature extraction and analysis5. By explicitly conditioning the data to showcase features of interest best, pre-processing enables deep neural networks to maximise their recognition potential21. When thoughtfully implemented, even relatively simple pre-processing techniques, such as appropriate image windowing can unlock substantial performance gains22. The future possibilities for healthcare are exciting as ever-more powerful deep learning and pre-processing techniques reinforce each other synergistically23.

Volumetric or direct volume rendering is one of the two primary techniques for visualising medical imaging data in its raw, unprocessed form24. This approach bypasses any intermediate representation, projecting volumetric data directly onto the viewing plane via sophisticated optical models that simulate light interaction with materials25.

There are two primary optical models. The absorption-only model assumes light within the volume can only be absorbed without scattering or emission26. This simplification enables faster rendering but trades off realism. In contrast, the more realistic emission-absorption model accounts for the light's absorption, scattering, and emission as it traverses the volume, producing more lifelike images at the cost of greater computation27.

The foundation of direct volume rendering is the volume rendering integral (VRI) - an integral equation describing light accumulation along a ray through the volume (25). It incorporates the light's intensity, the material's absorption/emission properties, and the ray's path length within the material (25).

Ray casting is the predominant VRI solution method28. Rays originate from the viewer's perspective, penetrating each pixel on the image plane and traversing the volume's depth. As rays propagate, data samples are gathered at regular intervals24. These samples determine colour and opacity relative to the optical model, compositing an image along the ray29.

Advanced optical enhancements like phase functions for scattering (27), shadow algorithms30, and ambient occlusion boost realism. Other acceleration strategies include early ray termination, space leaping, and empty space skipping to curtail sampling31. The culmination is high-fidelity volumetric images valuable for medical analysis and diagnosis32.

While effective, direct volume rendering lacks extensive lighting, shading, and textural detail. This is addressed by cinematic rendering, which employs advanced computer graphics principles to create photorealistic visualisations33. Directional path tracing simulates light interactions. Materials definition assigns physically-based properties, enabling nuanced tissue differentiation34. Depth of field, camera lens effects, and cinematic lighting heighten realism further35.

Together, these elements enhance understanding of anatomical contexts, leveraging realistic illumination, shading, focus, and motion effects familiar to the real world or cinema36. Cinematic rendering builds upon direct volume rendering's unprocessed data projection to impart more perceptually meaningful cues through physics-based photorealism.

In contrast to direct rendering methods, surface rendering relies on creating a polygon mesh intermediary form of the dataset, which is subsequently rendered. This approach is labelled 'indirect' because it relies on a representation separate from the original dataset. Unlike volume rendering, which directly converts volume data into images, surface rendering uses an indirect approach by creating a polygon mesh representation. This mesh can be derived by isolating an isosurface from the original volume data or modifying segmentation results. A polygon mesh comprises an interconnected network of vertices, edges, and faces, shaping the structure of a polyhedral object. Vertices represent a position in three-dimensional Euclidian space. When a straight line links two vertices, they form an edge, the basic mesh geometry unit. The faces of the mesh, which are the flat surfaces, are constructed by connecting these edges. Typically, a face is either a triangle, formed by three vertices and their respective edges, or a quadrangle (often referred to as 'quads'), which involves four vertices and four edges37.

The initial appeal of surface rendering in medical imaging stemmed from two key factors. First, it was substantially quicker than volume rendering because a mesh requires less memory than a full-volume dataset. Second, it produces images with depth cues and clear illumination, making them easier to interpret. However, the rise of GPU rendering, which supports all phases of direct volume rendering and can handle medical datasets in real time, has lessened these advantages. Despite this, surface rendering remains widely used in radiologic and therapy planning systems due to its familiarity among users38.

Surface rendering has gained renewed importance in recent years due to several developments. First, the growing application of biophysical simulations has created a demand for surface meshes to form the basis of volume grids used in these simulations. Second, the rise of interactive 3D visualisations in web browsers and mobile devices has favoured surface renderings for their lower memory requirements despite the feasibility of direct volume rendering in these settings. Lastly, the construction of surface meshes is required for 3D printing, where specific anatomical parts are modelled and printed for patient-specific planning and simulation purposes.

The marching cubes algorithm ingeniously reconstructs surfaces from 3D volumetric datasets. Developed in 1987 by William Lorensen and Harvey Cline, this computer graphics technique efficiently extracts an isosurface as a triangular mesh model39.

At its core, marching cubes divides the volume into cubic cells containing eight voxel corners. It processes these cells one by one to determine how the isosurface passes through each cell based on the voxel values at the corners. The algorithm reconstructs a triangular facet of the isosurface within each cell by analysing voxel intersections along the cube edges40.

Specifically, each voxel corner is classified as above or below the desired isovalue threshold. This classification pattern indexes into a precomputed lookup table specifying the edge intersections41. The exact coordinates of these intersections are calculated via linear interpolation of the edge voxel values42. The points are connected into triangles using another precomputed triangulation table. As the triangular patches from each cell merge, they knit together into a cohesive mesh43.

A key innovation is using lookup tables to encode possible triangle topologies, avoiding expensive geometric calculations efficiently. Normal vectors generated per vertex enable smooth shading44.

While the original algorithm had ambiguity at cell boundaries, later improvements addressed this, using strategies like trilinear interpolation. The algorithm's simplicity and scalability make it well-suited for large volumetric datasets across medical imaging and other fields.

The marching cubes algorithm also adeptly handles previously segmented data. As described above, each voxel contains a discrete label rather than continuous density. To reconstruct a surface, voxel labels are checked against the label of interest rather than an isovalue threshold38.

The core algorithm is unchanged. Cubes are processed cell-by-cell, analysing labelling patterns at the corners. Instead of density gradients, intersections now lie at label boundaries. These transitions are triangulated into surface facets within each cell38.

Pre-segmentation simplifies surface selection and reduces noise. It also isolates individual structures without visualising all tissue boundaries. But the critical process remains identical - marching through the volume in cubes, tabulating edge intersections via lookup tables, and connecting triangular patches.

This versatility to handle raw and segmented data makes marching cubes vital for medical imaging. As segmentation algorithms improve, it provides an efficient path to visualise labelled volumes in 3D.

Converting volumetric scan data into 3D surface models involves key post-processing steps to optimise visual appearance, functionality, and performance.

Two essential techniques are decimation and smoothing. However, where clinical applications demand high anatomical accuracy, post-processing should be limited owing to the potential distortion of the original geometry.

The application of colour is essential for distinguishing anatomical structures and aiding in user comprehension. Colours should be selected based on colour theory principles, including:

Hue - Unique hues like red, blue, and yellow are more easily discriminated than shades.

Value - Lighter and brighter colours stand out against darker backgrounds.

Complementary colours - Pairing opposite hues like blue/orange maximises contrast.

Context - Consideration of the semantic meaning colour can hold.

Adhering to established medical illustration colour conventions also capitalises on existing user familiarity. For example, arteries are red, veins are blue, nerves are yellow, etc. Selective colourisation focusing on structures of interest reduces visual clutter. The judicious use of colour is critical for effectively communicating complex 3D relationships45.

Decimation reduces the polygon count of a surface mesh to improve rendering efficiency. The marching cubes algorithm can produce meshes with hundreds of thousands of polygons from high-resolution scans39. This becomes problematic for real-time manipulation in interactive 3D applications.

Decimation simplifies the geometry while preserving the overall shape (figure 9)46. Taking a target polygon count as input, algorithms iteratively assess vertex or triangle importance based on criteria like curvature and proximity47. Less essential elements are collapsed or removed until the target simplification level is achieved.

Greedy algorithms offer faster performance by collapsing edges locally but can distort geometry48. Optimization-based approaches like quadric error metrics yield higher quality by considering global error but have higher computational costs47. Hybrid methods balance these trade-offs.

Decimation is crucial for web-based and mobile medical visualisation to prevent latency. Targeting 60 FPS rendering requires simplifying models to maintain responsive, fluid interactions49. However, aggressive simplification can negatively impact intricate structures like vasculature. Domain-specific strategies like vascular skeletons can help preserve critical details50.

Smoothing eliminates jagged surfaces and staircase artefacts to improve visual quality. The marching cubes produce noticeable sharp discontinuities between adjacent cubes. Smoothing blends these seams for more natural surfaces51. Most smoothing approaches do not alter the number of vertices but instead change their position and are therefore deemed topology-preserving38.

Common approaches iteratively adjust vertex positions based on surrounding geometry. For example, Laplacian smoothing moves vertices to the average location of their neighbours51. Low-pass filters can also be applied in the normal or domains52. These methods improve appearance but can shrink and distort geometry (figure 10).

Alternative edge-preserving filters like bilateral smoothing better maintain fidelity53. GUIDOS also shows promise for selective smoothing of medical models54. Smoothing must balance visual improvement against anatomical accuracy requirements in medical use cases.

Decimation and smoothing are indispensable post-processing steps for optimising extracted medical surfaces. However, implementation requires careful consideration of clinical visualisation goals versus real-time interface constraints. It is essential to maintain an awareness of the distortion processing methods confer on the raw segmentation mesh, especially for applications requiring high levels of accuracy.

Advanced 3D modelling techniques from visual effects and gaming can produce highly detailed anatomical models for medical visualisation and education55. These pipelines leverage specialised software (e.g., ZBrush (Maxon, 2023), Maya (Autodesk, 2023), Blender (Blender Foundation, 2023)) with complementary toolsets.

Complex polygon modelling in specialised software provides powerful mesh manipulation with tools to separate, extrude, and animate components56. This enables dividing models into distinct regions and adding thickness to surface-only structures like vessels57. Animation tools are valuable for illustrating motion and function.

Digital sculpture facilitates detailed organic modelling ideal for anatomy58. Sculpting introduces fine surface details like muscle striations, challenging traditional techniques. Adaptive subdivision enables high-poly sculpting projected as normal maps onto low-poly models, combining detail and performance59. This workflow allows adding high-frequency details while retaining low polygon counts for efficiency.

Segmented medical models contain triangle meshes. Quad remeshing converts these into quad-based meshes better suited for optimisation, texturing, and animation60. Most software provides rapid retopology tools and quad-focused algorithms to rebuild the model with primarily quads instead of triangles (figure 11). This confers technical advantages for later stages60.

Manually created seams define how the 3D model is cut and laid out on the 2D UV grid. Seam placement is optimised for clean, minimal distortion layouts specific to each model region. UV maps should have uniform texel density (texel density refers to the level of detail present in a texture map, it is measured by the number of texels, or texture elements packed into a given area) and scale with efficient use of space. Grids verify correct unwrapping before texturing61 (figure 12).

Texturing is essential for achieving photorealism in anatomical models. Physically based rendering (PBR) aims to simulate realistic material interactions with light62. It uses advanced shader programming and texture maps to define material properties.

PBR maps are defined in table 1.

PBR uses two common workflows - metal/roughness and specular/glossiness. The metal/roughness approach provides a more realistic material definition.

Base colour and metallic/roughness maps define fundamental material qualities. Normal maps add high-frequency detail without topological changes63. Bump maps emulate fine wrinkles and height variations64. Displacement provides actual surface height changes.

Advanced effects like subsurface scattering are vital for anatomical realism, simulating light penetration and diffusion in translucent materials like skin65.

Combined skilfully, PBR empowers unmatched realism and educational value for anatomical visualisations (figure 13).

Various platforms and technologies exist for visualising computer-generated 3D models. Each solution offers different capabilities and trade-offs between ease of use, interactive features, and development requirements.

Increasingly sophisticated 3D model viewing can be enabled through standard web JavaScript libraries like Three.js66. This allows models to be embedded in web pages and manipulated through the browser, including control over colourisation and element transparency (figure 14)(66).

While useful for education and basic visualisation, such platforms need more advanced clinical functionalities. Game engines, including Unity (Unity Technologies, 202367) and Unreal Engine (Epic Games, 202368), offer far greater flexibility and interactivity through customised development, physics simulations, and support for virtual reality, albeit with greater workflow complexity.

Specialised medical software like 3D Slicer69, Mimics (Materialise, 202370), Innersight labs71 and Visible Patient72 allows direct use of DICOM and other medical imaging formats, providing an ecosystem focused on clinical utility rather than visual polish. However, advanced rendering capabilities still need to be improved compared to entertainment-focused game engines.

The choice of visualisation platform is guided by the specific clinical context and use cases. While visual realism can be essential and facilitate user comprehension, this should not be at the expense or neglect of required clinical functionalities.

The crucial consideration should be enabling accurate anatomy and functional simulations to inform the surgical approach, resection planning, implant design, and other clinically driven tasks.

Virtual reality (VR) for 3D model interaction has gathered recent attention, given its immersive and intuitive spatial visualisation capacity. Workflow integration of VR/AR remains limited, presently. The practical advantage over traditional 3D planning user desktop interfaces remains debatable, and wearing an occlusive headset in the operating room poses logistical challenges73.

In summary, various tools and platforms exist for creating interactive 3D visualisations from medical imaging, each with different strengths and weaknesses. But the most crucial factor is that the technology and visualisation style enable the delivery of patient-specific clinical value rather than just visually impressive models.

Further technological advances will continue expanding the possibilities of 3D visualisation, but these should be assessed against tangible improvements in surgical care and outcomes.

3D models have been utilised in most surgical specialties for both planning and navigation applications. Planning activities can broadly be characterised as the following:

Anatomical Understanding: 3D models, both virtual and printed, provide a deeper perspective into patient anatomy compared to traditional 2D imaging. They are particularly beneficial in understanding complex pathologies and anatomical relationships crucial for surgical decision-making74, 75, 76, 77. Studies like those by Awan et al. have shown improvement in orthopaedic trainees' abilities with 3D models, although the correlation between anatomical understanding and surgical performance remains unclear74

Patient-Specific Simulation: Patient-specific simulation can be subdivided into process and outcome simulation. In process simulation, the model is used to recreate aspects of the surgery as a form of rehearsal78. Outcome simulation attempts to predict operative results and can include aesthetic outcomes post-reconstruction, blood flow or organ function79.The ability to realistically rehearse procedures on a patient-specific basis is a significant advantage of both 3D virtual and printed models for preoperative planning. Physical models especially allow tactile simulation of technical steps like osteotomies, resections, and implant contouring80. Surgeons can refine surgical techniques and identify potential pitfalls to optimise intraoperative approaches.

Resection Planning: Achieving negative (R0) tumour margins is critical for oncologic surgery, and 3D modelling has proven valuable for preoperative planning in speciality contexts like hepatic and renal surgery81. Accurately assessing tumour location and relationships to vessels and other structures is essential for balancing resection margins versus morbidity and function. Studies suggest that 3D visualisation provides a superior appreciation of patient-specific anatomy compared to standard imaging82. The ability to perform virtual resections provides quantitative data like estimated resection volumes and residual parenchymal volume for hepatic resections – critical metrics for assessing feasibility and surgical strategies83.

Reconstruction Planning: 3D modelling is pivotal in reconstructive surgeries, such as facial reconstructions, where precision and aesthetics are vital. These models facilitate the conceptualization of structural corrections and simulation of outcomes, thereby improving accuracy and minimizing operating time84, 85, 86

Intraoperative Navigation and Augmented Reality: 3D models have shown promise in enhancing surgical intraoperative navigation. From displaying models on screens to advanced augmented reality (AR) techniques, these models aid in intraoperative decision-making87, 88, 89. AR navigation overlay digital elements on real world scenes. They can be divided into video-based and optical see-through systems, offers real-time “X-ray vision” to identify key structures89. However, there remain significant technical and conceptual challenges if AR navigation is in tracking soft tissue deformation and the need for updated intraoperative imaging for registration accuracy are ongoing issues90,91.