Publications by authors named "Hongen Liao"

121 Publications

TypeSeg: A type-aware encoder-decoder network for multi-type ultrasound images co-segmentation.

Comput Methods Programs Biomed 2021 Dec 17;214:106580. Epub 2021 Dec 17.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, China.

Purpose: As a portable and radiation-free imaging modality, ultrasound can be easily used to image various types of tissue structures. It is important to develop a method which supports the multi-type ultrasound images co-segmentation. However, state-of-the-art ultrasound segmentation methods commonly only focus on the single type images or ignore the type-aware information.

Methods: To solve the above problem, this work proposes a novel type-aware encoder-decoder network (TypeSeg) for the multi-type ultrasound images co-segmentation. First, we develop a type-aware metric learning module to find an optimum latent feature space where the ultrasound images of the same types are close and that of the different types are separated by a certain margin. Second, depending on the extracted features, a decision module decides whether the input ultrasound images have the common tissue type or not, and the encoder-decoder network produces a segmentation mask accordingly.

Results: We evaluate the performance of the proposed TypeSeg model on the ultrasound dataset that contains four types of tissues. The proposed TypeSeg model achieves the overall best results with the mean IOU score of 87.51% ± 3.93% for the multi-type ultrasound images.

Conclusion: The experimental results indicate that the proposed method outperforms all the compared state-of-the-art algorithms for the multi-type ultrasound images co-segmentation task.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.cmpb.2021.106580DOI Listing
December 2021

Pneumatic System Capable of Supplying Programmable Pressure States for Soft Robots.

Soft Robot 2021 Dec 16. Epub 2021 Dec 16.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China.

Pneumatic soft robots are of great interest in varieties of potential applications due to their unique capabilities compared with rigid structures. As a part of the soft robotic system, the pneumatic system plays a very important role as all motion performance is ultimately related to the pressure control in air chambers. With the increasing flexibility and complexity of robotic tasks, diverse pneumatic robots driven by positive, negative, or even hybrid pressure are developed, and this comes with higher requirements of pneumatic system and air pressure control precision. In this study, we aim to propose a simplified pneumatic design capable of generating programmable pressure states ranging from negative to positive pressure in each air branch. Based on the design concept and system configuration, special inflation and deflation strategies and closed-loop feedback control strategy are proposed to achieve precise pressure control. Then, a prototype of the pneumatic system with six independent air supply branches is designed and fabricated. Experimental results show that the pneumatic system can achieve a wide range of pressure from -59 to 112 kPa. The speed of inflation and deflation is controllable. Finally, we demonstrate three robotic applications and design the related algorithms to verify the feasibility and practicability of the pneumatic system. Our proposed pneumatic design can satisfy the pressure control requirements of a variety of soft robots driven by both positive and negative pressure. It can be used as a universal pneumatic platform, which is inspiring for actuation and control in the soft robotic field.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1089/soro.2021.0016DOI Listing
December 2021

A Smoke Removal Method Based on Combined Data and Modified U-Net for Endoscopic Images.

Annu Int Conf IEEE Eng Med Biol Soc 2021 11;2021:3783-3786

In minimally invasive surgery, the ablation of human tissue will produce a lot of smoke, which will interfere with the surgeon's operation. We propose a smoke removal method based on combined data and modified U-net for endoscopic images. The real dataset and the synthetic dataset are built using a small amount of images with smoke. The real dataset is combined with the synthetic dataset successively. Qualitative evaluation shows that the quality of the output smoke-free image is the best when training using the combined data, compared to using only either the real dataset or the synthetic dataset above. Quantitative evaluation shows that the effect of smoke removal is still the best when training using the combined data in our method.Clinical Relevance-A real-time smoke removal method suitable for endoscopic surgery is proposed to help surgeons get clear images in real time and make the operation go smoothly.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/EMBC46164.2021.9630222DOI Listing
November 2021

Image-Based 3D Ultrasound Reconstruction with Optical Flow via Pyramid Warping Network.

Annu Int Conf IEEE Eng Med Biol Soc 2021 11;2021:3539-3542

3D Ultrasound (US) contains rich spatial information which is helpful for medical diagnosis. However, current reconstruction methods with tracking devices are not suitable for clinical application. The sensorless freehand methods reconstruct based on US images which is less accuracy. In this paper, we proposed a network which reconstructs the US volume based on US images features and optical flow features. We proposed the pyramid warping layer which merges the image features and optical flow features with warping operation. To fuse the warped features of different scales in different pyramid levels, we adopted the fusion module using the attention mechanism. Meanwhile, we adopted the channel attention and spatial attention to our network. Our method was evaluated in 100 freehand US sweeps of human forearms which exhibits the efficient performance on volume reconstruction compared with other methods.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/EMBC46164.2021.9630853DOI Listing
November 2021

An Unsupervised Convolution Neural Network for Deformable Registration of Mono/Multi-Modality Medical Images.

Annu Int Conf IEEE Eng Med Biol Soc 2021 11;2021:3455-3458

Image registration is a fundamental and crucial step in medical image analysis. However, due to the differences between mono-mode and multi-mode registration tasks and the complexity of the corresponding relationship between multimode image intensity, the existing unsupervised methods based on deep learning can hardly achieve the two registration tasks simultaneously. In this paper, we proposed a novel approach to register both mono- and multi-mode images $\color{blue}{\text{in a differentiable }}$. By approximately calculating the mutual information in a $\color{blue}{\text{differentiable}}$ form and combining it with CNN, the deformation field can be predicted quickly and accurately without any prior information about the image intensity relationship. The registration process is implemented in an unsupervised manner, avoiding the need for the ground truth of the deformation field. We utilize two public datasets to evaluate the performance of the algorithm for mono-mode and multi-mode image registration, which confirms the effectiveness and feasibility of our method. In addition, the experiments on patient data also demonstrate the practicability and robustness of the proposed method.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/EMBC46164.2021.9630731DOI Listing
November 2021

Correction to: Automatic virtual reconstruction of maxillofacial bone defects assisted by ICP (iterative closest point) algorithm and normal people database.

Clin Oral Investig 2021 Nov 27. Epub 2021 Nov 27.

Department of Oral and Maxillofacial Surgery, Peking University School and Hospital of Stomatology, 22 Zhongguancun South Road, Beijing, 100081, People's Republic of China.

View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1007/s00784-021-04246-3DOI Listing
November 2021

Motion parallax and lossless resolution autostereoscopic 3D display based on a binocular viewpoint tracking liquid crystal dynamic grating adaptive screen.

Opt Express 2021 Oct;29(22):35456-35473

The autostereoscopic 3D display has two important indicators, both the number of viewpoints and display resolution. However, it's a challenge to improve both the viewpoint and the resolution. Here, we develop a fixed-position multiview and lossless resolution autostereoscopic 3D display system that includes the dynamic liquid crystal (LC) grating screen. This display system consists of an LC display panel and an LC grating screen. The synchronization of the frame switching of the LC display panel and the LC grating screen shutter enables the preserved resolution. The "eye space" design makes the viewpoint dense enough and determines the LC grating screen's parameters. We use binocular viewpoint tracking technology to realize the LC grating screen's adaptive control based on the above work. Different binocular views are rendered in real-time according to the different positions of a single pair of stereoscopic viewpoints in the eye space, making the motion parallax possible. We present the working principle and mathematical analysis. We implement a prototype for verifying the principle. According to the experiment results analysis, this prototype can achieve viewpoint tracking and motion parallax based on resolution lossless and viewpoint dense enough.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1364/OE.439111DOI Listing
October 2021

Collagen crosslinking: effect on structure, mechanics and fibrosis progression.

Biomed Mater 2021 10 19;16(6). Epub 2021 Oct 19.

Department of Biomedical Engineering, School of Medicine, Tsinghua-Peking Center for Life Sciences, Tsinghua University, Beijing 100084, People's Republic of China.

Biophysical properties of extracellular matrix (ECM), such as matrix stiffness, viscoelasticity and matrix fibrous structure, are emerging as important factors that regulate progression of fibrosis and other chronic diseases. The biophysical properties of the ECM can be rapidly and profoundly regulated by crosslinking reactions in enzymatic or non-enzymatic manners, which further alter the cellular responses and drive disease progression. In-depth understandings of crosslinking reactions will be helpful to reveal the underlying mechanisms of fibrosis progression and put forward new therapeutic targets, whereas related reviews are still devoid. Here, we focus on the main crosslinking mechanisms that commonly exist in a plethora of chronic diseases (e.g. fibrosis, cancer, osteoarthritis) and summarize current understandings including the biochemical reaction, the effect on ECM properties, the influence on cellular behaviors, and related studies in disease model establishment. Potential pharmaceutical interventions targeting the crosslinking process and relevant clinical studies are also introduced. Limitations of pharmaceutical development may be due to the lack of systemic investigations related to the influence on crosslinking mechanism from micro to macro level, which are discussed in the last section. We also propose the unclarified questions regarding crosslinking mechanisms and potential challenges in crosslinking-targeted therapeutics development.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1088/1748-605X/ac2b79DOI Listing
October 2021

Automatic virtual reconstruction of maxillofacial bone defects assisted by ICP (iterative closest point) algorithm and normal people database.

Clin Oral Investig 2021 Sep 25. Epub 2021 Sep 25.

Department of Oral and Maxillofacial Surgery, Peking University School and Hospital of Stomatology, 22 Zhongguancun South Road, Beijing, 100081, China.

Objectives: The aim of this study was to propose and validate an automatic approach based on iterative closest point algorithm for virtual complement and reconstruction for maxillofacial bone defects.

Materials And Methods: A 3D craniomaxillofacial database of normal Chinese people including 500 skull models was established. Modified iterative closest point (ICP) algorithm was developed to complete bone defects automatically. The performances were evaluated by two approaches: (1) model experiment, virtual bony defects were created on 30 intact normal skull models not included in the database. For each defect model, the algorithm was applied to select the reference skull model from the database. 3-Dimensional and 2-dimensional comparison were conducted to evaluate the error between reference skull model with original intact model. Root mean square error (RMSE) and processing time were calculated. (2) Clinical application, the algorithm was utilized to assist reconstruction of 5 patients with maxillofacial bone defects. The symmetry of post-operative skull model was evaluated by comparing with its mirrored model.

Results: The algorithm was tested on an CPU with 1.80 GHz and average processing time was 493.5 s. (1) Model experiment, the average root-mean-square deviation of defect area was less than 2 mm. (2) Clinical application, the RMSE of post-operative skull and its mirrored model was 1.72 mm.

Conclusion: It is feasible using iterative closest point algorithm based on normal people database to automatically predict the reference data of missing maxillofacial bone.

Clinical Relevance: An automated approach based on ICP algorithm and normal people database for maxillofacial bone defect reconstruction has been proposed and validated.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1007/s00784-021-04181-3DOI Listing
September 2021

Acceleration of reconstruction for compressed sensing based synthetic transmit aperture imaging by using in-phase/quadrature data.

Ultrasonics 2022 Jan 6;118:106576. Epub 2021 Sep 6.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China. Electronic address:

Compressed sensing-based synthetic transmit aperture (CS-STA) was previously proposed to recover the full radio-frequency (RF) channel dataset of synthetic transmit aperture (STA) from that of a smaller number of randomly apodized plane wave (PW) transmissions. In this way, the imaging frame rate (FR) and contrast are improved with maintained spatial resolution, compared with those of STA. Because CS-STA reconstruction is repeated for all receive elements and RF samples (with a high sampling frequency), the recovery of STA dataset in RF domain is time-consuming. In the meantime, a large amount of RF data needs to be transferred and stored, resulting in an increase of system complexity and required memory space. In this study, CS-STA is extended to in-phase/quadrature (IQ) domain (with lower sampling frequency) for the recovery of baseband STA IQ dataset to accelerate the CS-STA reconstruction by reducing the amount of data to be processed. More importantly, CS-STA reconstruction using IQ data is of practical importance, as clinical ultrasound systems typically record baseband IQ signal instead of RF signal. Simulations, phantom and in vivo experiments verify the feasibility of CS-STA in IQ domain for the recovery of STA dataset. More specifically, CS-STA using IQ data achieves similar image quality and appreciably improves reconstruction speed (by ∼3 times) compared with that using RF data. These findings demonstrate that IQ-domain CS-STA is capable of relieving the computational and storage burdens, which may facilitate the implementation of CS-STA in practical ultrasound systems.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.ultras.2021.106576DOI Listing
January 2022

Quantitative Analysis of Pleural Line and B-Lines in Lung Ultrasound Images for Severity Assessment of COVID-19 Pneumonia.

IEEE Trans Ultrason Ferroelectr Freq Control 2022 01 31;69(1):73-83. Epub 2021 Dec 31.

Specific patterns of lung ultrasound (LUS) images are used to assess the severity of coronavirus disease 2019 (COVID-19) pneumonia, while such assessment is mainly based on clinicians' qualitative and subjective observations. In this study, we quantitatively analyze the LUS images to assess the severity of COVID-19 pneumonia by characterizing the patterns related to the pleural line (PL) and B-lines (BLs). Twenty-seven patients with COVID-19 pneumonia, including 13 moderate cases, seven severe cases, and seven critical cases, are enrolled. Features related to the PL, including the thickness (TPL) and roughness of the PL (RPL), and the mean (MPLI) and standard deviation (SDPLI) of the PL intensities are extracted from the LUS images. Features related to the BLs, including the number (NBL), accumulated width (AWBL), attenuation coefficient (ACBL), and accumulated intensity (AIBL) of BLs, are also extracted. The correlations of these features with the disease severity are evaluated. The performances of the binary severe/non-severe classification are assessed for each feature and support vector machine (SVM) classifiers with various combinations of features as input. Several features, including the RPL, NBL, AWBL, and AIBL, show significant correlations with disease severity (all ). The classification performance is optimal using the SVM classifier using all the features as input (area under the receiver operating characteristic (ROC) curve = 0.96, sensitivity = 0.93, and specificity = 1). These findings demonstrate that the proposed method may be a promising tool for automatic grading diagnosis and follow-up of patients with COVID-19 pneumonia.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/TUFFC.2021.3107598DOI Listing
January 2022

Force-guided autonomous robotic ultrasound scanning control method for soft uncertain environment.

Int J Comput Assist Radiol Surg 2021 Dec 9;16(12):2189-2199. Epub 2021 Aug 9.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China.

Purpose: Autonomous ultrasound imaging by robotic ultrasound scanning systems in complex soft uncertain clinical environments is important and challenging to assist in therapy. To cope with the complex environment faced by the ultrasound probe during the scanning process, we propose an autonomous robotic ultrasound (US) control method based on reinforcement learning (RL) model to build the relationship between the environment and the system. The proposed method requires only contact force as input information to achieve robot control of the posture and contact force of the probe without any a priori information about the target and the environment.

Methods: First, an RL agent is proposed and trained by a policy gradient theorem-based RL model with the 6-degree-of-freedom (DOF) contact force of the US probe to learn the relationship between contact force and output force directly. Then, a force control strategy based on the admittance controller is proposed for synchronous force, orientation and position control by defining the desired contact force as the action space.

Results: The proposed method was evaluated via collected US images, contact force and scan trajectories by scanning an unknown soft phantom. The experimental results indicated that the proposed method differs from the free-hand scanned approach in the US images within 3 ± 0.4%. The analysis results of contact forces and trajectories indicated that our method could make stable scanning processes on a soft uncertain skin surface and obtained US images.

Conclusion: We propose a concise and efficient force-guided US robot scanning control method for soft uncertain environment based on reinforcement learning. Experimental results validated our method's feasibility and validity for complex skin surface scanning, and the volunteer experiments indicated the potential application value in the complex clinical environment of robotic US imaging system especially with limited visual information.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1007/s11548-021-02462-6DOI Listing
December 2021

Intelligent optical diagnosis and treatment system for automated image-guided laser ablation of tumors.

Int J Comput Assist Radiol Surg 2021 Dec 7;16(12):2147-2157. Epub 2021 Aug 7.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China.

Purpose: For tumor resections near critical structures, accurate identification of tumor boundaries and maximum removal are the keys to improve surgical outcome and patient survival rate, especially in neurosurgery. In this paper, we propose an intelligent optical diagnosis and treatment system for tumor removal, with automated lesion localization and laser ablation.

Methods: The proposed system contains a laser ablation module, an optical coherence tomography (OCT) unit, and a robotic arm along with a stereo camera. The robotic arm can move the OCT sample arm and the laser ablation front-end to the suspected lesion area. The corresponding diagnosis and treatment procedures include computer-aided lesion segmentation using OCT, automated ablation planning, and laser control. The ablation process is controlled by a deflectable mirror, and a non-common-path ablation planning algorithm based on the transformation from lesion positions to mirror deflection angles is presented.

Results: Phantom and animal experiments are carried out for system verification. The robot could reach the planned position with high precision, which is approximately 1.16 mm. Tissue classification with OCT images achieves 91.7% accuracy. The error of OCT-guided automated laser ablation is approximately 0.74 mm. Experiments on mouse brain tumors show that the proposed system is capable of clearing lesions efficiently and precisely. We also conducted an ex vivo porcine brain experiment to verify the whole process of the system.

Conclusion: An intelligent optical diagnosis and treatment system is proposed for tumor removal. Experimental results show that the proposed system and method are promising for precise and intelligent theranostics. Compared to conventional cancer diagnosis and treatment, the proposed system allows for automated operations monitored in real-time, with higher precision and efficiency.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1007/s11548-021-02457-3DOI Listing
December 2021

Augmented reality-based autostereoscopic surgical visualization system for telesurgery.

Int J Comput Assist Radiol Surg 2021 Nov 7;16(11):1985-1997. Epub 2021 Aug 7.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China.

Purpose: The visualization of remote surgical scenes is the key to realizing the remote operation of surgical robots. However, current non-endoscopic surgical robot systems lack an effective visualization tool to offer sufficient surgical scene information and depth perception.

Methods: We propose a novel autostereoscopic surgical visualization system integrating 3D intraoperative scene reconstruction, autostereoscopic 3D display, and augmented reality-based image fusion. The preoperative organ structure and the intraoperative surface point cloud are obtained from medical imaging and the RGB-D camera, respectively, and aligned by an automatic marker-free intraoperative registration algorithm. After registration, preoperative meshes with precalculated illumination and intraoperative textured point cloud are blended in real time. Finally, the fused image is shown on a 3D autostereoscopic display device to achieve depth perception.

Results: A prototype of the autostereoscopic surgical visualization system was built. The system had a horizontal image resolution of 1.31 mm, a vertical image resolution of 0.82 mm, an average rendering rate of 33.1 FPS, an average registration rate of 20.5 FPS, and average registration errors of approximately 3 mm. A telesurgical robot prototype based on 3D autostereoscopic display was built. The quantitative evaluation experiments showed that our system achieved similar operational accuracy (1.79 ± 0.87 mm) as the conventional system (1.95 ± 0.71 mm), while having advantages in terms of completion time (with 34.11% reduction) and path length (with 35.87% reduction). Post-experimental questionnaires indicated that the system was user-friendly for novices and experts.

Conclusion: We propose a 3D surgical visualization system with augmented instruction and depth perception for telesurgery. The qualitative and quantitative evaluation results illustrate the accuracy and efficiency of the proposed system. Therefore, it shows great prospects in robotic surgery and telesurgery.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1007/s11548-021-02463-5DOI Listing
November 2021

Augmented reality navigation for minimally invasive knee surgery using enhanced arthroscopy.

Comput Methods Programs Biomed 2021 Apr 24;201:105952. Epub 2021 Jan 24.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China. Electronic address:

Purpose: During the minimally invasive knee surgery, surgeons insert surgical instruments and arthroscopy through small incisions, and implement treatment assisted by 2D arthroscopic images. However, this 2D arthroscopic navigation faces several problems. Firstly, the guidance information is displayed on a screen away from the surgical area, which makes hand/eye coordination difficult. Secondly, the small incision limits the surgeons to view the internal knee structures only from an arthroscopic camera. In addition, arthroscopic images commonly appear obscure visions.

Methods: To solve these problems, we proposed a novel in-situ augmented reality navigation system with the enhanced arthroscopic information. Firstly, intraoperative anatomical locations were obtained by using arthroscopic images and arthroscopy calibration. Secondly, tissue properties-based model deformation method was proposed to update the 3D preoperative knee model with anatomical location information. Then, the updated model was further rendered with glasses-free real 3D display for achieving the global in-situ augmented reality view. In addition, virtual arthroscopic images were generated from the updated preoperative model to provide the anatomical information of the operation area.

Results: Experimental results demonstrated that virtual arthroscopic images could reflect the correct structure information with a mean error of 0.32 mm. Compared with 2D arthroscopic navigation, the proposed augmented reality navigation reduced the targeting errors by 2.10 mm and 2.70 mm for the experiments of knee phantom and in-vitro swine knee, respectively.

Conclusion: Our navigation method is helpful for minimally invasive knee surgery since it can provide the global in-situ information and detail anatomical information.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.cmpb.2021.105952DOI Listing
April 2021

A comparative analysis framework of 3T and 7T TOF-MRA based on automated cerebrovascular segmentation.

Comput Med Imaging Graph 2021 04 28;89:101830. Epub 2021 Jan 28.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China. Electronic address:

Purpose: High field strength 3T and 7T Time-Of-Flight Magnetic Resonance Angiography (TOF- MRA) achieves better visualization of intracranial vessels, so it attracts much attention. However, quantitative comparison between 3T and 7T MRA is lacking in the aspects of image quality and the practical application of cerebrovascular diseases.

Methods: In this paper, a quantitative framework of 3T and 7T TOF-MRA comparison is proposed, which contains two steps including the automated cerebrovascular segmentation and statistical analysis. Firstly, the whole vascular structures on both 3T and 7T TOF-MRA images are segmented automatically, especially those small blood vessels in 7T MRA. The skeleton extraction-based automatic seed point detection is implemented to ensure the segmented vascular structure complete and precise. Secondly, the statistical analysis of the differences between 3T and 7T MRA is carried out in the aspects of image quality and the characteristics of some important vessels. The objects of statistical analysis are achieved and analyzed automatically without needing the time- consuming human beings' participation, therefore, it is efficient and objective.

Results: The comparison experiments on seven pairs of 3T and 7T TOF MRA images validated that about image quality, the contrast-to-noise ratio of 7T MRA was about 4.53 ± 0.95 times as much as that of 3T MRA. About the cerebrovascular information, small vessels were more abundant in 7T MRA compared with 3T MRA (branches number: 462.0 ± 58.5 vs 393.1 ± 63.3).

Conclusions: The proposed framework can segment the whole cerebrovascular structure automatically and compare TOF-MRA with different field strengths objectively and quantitatively. It is helpful for clinical cerebrovascular disease, especially cerebral small vessel diseases.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.compmedimag.2020.101830DOI Listing
April 2021

Autonomic Robotic Ultrasound Imaging System Based on Reinforcement Learning.

IEEE Trans Biomed Eng 2021 09 19;68(9):2787-2797. Epub 2021 Aug 19.

Objective: In this paper, we introduce an autonomous robotic ultrasound (US) imaging system based on reinforcement learning (RL). The proposed system and framework are committed to controlling the US probe to perform fully autonomous imaging of a soft, moving and marker-less target based only on single RGB images of the scene.

Methods: We propose several different approaches and methods to achieve the following objectives: real-time US probe controlling, soft surface constant force tracking and automatic imaging. First, to express the state of the robotic US imaging task, we proposed a state representation model to reduce the dimensionality of the imaging state and encode the force and US information into the scene image space. Then, an RL agent is trained by a policy gradient theorem based RL model with the single RGB image as the only observation. To achieve adaptable constant force tracking between the US probe and the soft moving target, we propose a force-to-displacement control method based on an admittance controller.

Results: In the simulation experiment, we verified the feasibility of the integrated method. Furthermore, we evaluated the proposed force-to-displacement method to demonstrate the safety and effectiveness of adaptable constant force tracking. Finally, we conducted phantom and volunteer experiments to verify the feasibility of the method on a real system.

Conclusion: The experiments indicated that our approaches were stable and feasible in the autonomic and accurate control of the US probe.

Significance: The proposed system has potential application value in the image-guided surgery and robotic surgery.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/TBME.2021.3054413DOI Listing
September 2021

Learning Two-View Correspondences and Geometry Using Order-Aware Network.

IEEE Trans Pattern Anal Mach Intell 2020 Dec 29;PP. Epub 2020 Dec 29.

Establishing correct correspondences between two images should consider both local and global spatial context. Given putative correspondences of feature points in two views, in this paper, we propose Order-Aware Network, which infers the probabilities of correspondences being inliers and regresses the relative pose encoded by the essential or fundamental matrix. Specifically, this proposed network is built hierarchically and comprises three operations. First, to capture the local context of sparse correspondences, the network clusters unordered input correspondences by learning a soft assignment matrix. These clusters are in canonical order and invariant to input permutations. Next, the clusters are spatially correlated to encode the global context of correspondences. After that, the context-encoded clusters are interpolated back to the original size and position to build a hierarchical architecture. We intensively experiment on both outdoor and indoor datasets. The accuracy of the two-view geometry and correspondences are significantly improved over the state-of-the-arts. Besides, based on the proposed method and advanced local feature, we won the first place in CVPR 2019 image matching workshop challenge and also achieve state-of-the-art results in the Visual Localization benchmark.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/TPAMI.2020.3048013DOI Listing
December 2020

Roadmap on 3D integral imaging: sensing, processing, and display.

Opt Express 2020 Oct;28(22):32266-32293

This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1364/OE.402193DOI Listing
October 2020

DBAN: Adversarial Network With Multi-Scale Features for Cardiac MRI Segmentation.

IEEE J Biomed Health Inform 2021 06 3;25(6):2018-2028. Epub 2021 Jun 3.

With the development of medical artificial intelligence, automatic magnetic resonance image (MRI) segmentation method is quite desirable. Inspired by the power of deep neural networks, a novel deep adversarial network, dilated block adversarial network (DBAN), is proposed to perform left ventricle, right ventricle, and myocardium segmentation in short-axis cardiac MRI. DBAN contains a segmentor along with a discriminator. In the segmentor, the dilated block (DB) is proposed to capture, and aggregate multi-scale features. The segmentor can produce segmentation probability maps while the discriminator can differentiate the segmentation probability map, and the ground truth at the pixel level. In addition, confidence probability maps generated by the discriminator can guide the segmentor to modify segmentation probability maps. Extensive experiments demonstrate that DBAN has achieved the state-of-the-art performance on the ACDC dataset. Quantitative analyses indicate that cardiac function indices from DBAN are similar to those from clinical experts. Therefore, DBAN can be a potential candidate for short-axis cardiac MRI segmentation in clinical applications.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/JBHI.2020.3028463DOI Listing
June 2021

Improved 3D Catheter Shape Estimation Using Ultrasound Imaging for Endovascular Navigation: A Further Study.

IEEE J Biomed Health Inform 2020 12 4;24(12):3616-3629. Epub 2020 Dec 4.

Objective: Two-dimensional fluoroscopy is the standard guidance imaging method for closed endovascular intervention. However, two-dimensional fluoroscopy lacks depth perception for the intervention catheter and causes radiation exposure for both surgeons and patients. In this paper, we extend our previous study and develop the improved three-dimensional (3D) catheter shape estimation using ultrasound imaging. In addition, we perform further quantitative evaluations of endovascular navigation.

Method: First, the catheter tracking accuracy in ultrasound images is improved by adjusting the state vector and adding direction information. Then, the 3D catheter points from the catheter tracking are further optimized based on the 3D catheter shape optimization with a high-quality sample set. Finally, the estimated 3D catheter shapes from ultrasound images are overlaid with preoperative 3D tissue structures for the intuitive endovascular navigation.

Results: the tracking accuracy of the catheter increased by 24.39%, and the accuracy of the catheter shape optimization step also increased by approximately 17.34% compared with our previous study. Furthermore, the overall error of catheter shape estimation was further validated in the catheter intervention experiment of in vitro cardiovascular tissue and in a vivo swine, and the errors were 2.13 mm and 3.37 mm, respectively.

Conclusion: Experimental results demonstrate that the improved catheter shape estimation using ultrasound imaging is accurate and appropriate for endovascular navigation.

Significance: Improved navigation reduces the radiation risk because it decreases use of X-ray imaging. In addition, this navigation method can also provide accurate 3D catheter shape information for endovascular surgery.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/JBHI.2020.3026105DOI Listing
December 2020

Gait acquisition and analysis system for osteoarthritis based on hybrid prediction model.

Comput Med Imaging Graph 2020 10 27;85:101782. Epub 2020 Aug 27.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, China. Electronic address:

Osteoarthritis (OA) is the most common type of joint-related diseases, which affects millions of people worldwide. Expensive and time-consuming medical imaging can provide precise structural description of knee joints, but lacks the functional descriptions. Gait analysis can provide functional descriptions of knee joints. However, orthopedic surgeons always observe the patient's gait qualitatively and perform subjective assessments through rating scales at present due to the lack of a quantitative gait analysis system. To solve these problems, a gait acquisition and analysis system is developed to provide a cheap, easy-to-use solution for quantitative recording and functional description of OA patients. Firstly, an automatic gait acquisition platform is designed for the clinical setting based on the RGB-D camera and the developed software of gait data recording. In addition, the effective working space of gait acquisition platform is evaluated for clinical applications by comparing with the ground-truth from infrared optical trackers. Secondly, the acquired gait data is analyzed with a novel hybrid prediction model to assess the gait anomalies quantitatively and objectively. In the hybrid model, the extracted features of gait data contain the manually-extracted features and the automatically-extracted features from Long Short-Term Memory network. Experimental results on real patients demonstrate that the proposed gait analysis system can quantitatively predict gait anomalies with a high accuracy of 98.77 %. Therefore, this gait acquisition and analysis system achieves quantitative recording and objective assessment of gait anomalies for clinical OA treatments.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.compmedimag.2020.101782DOI Listing
October 2020

Single Volume Image Generator and Deep Learning-Based ASD Classification.

IEEE J Biomed Health Inform 2020 11 4;24(11):3044-3054. Epub 2020 Nov 4.

Autism spectrum disorder (ASD) is an intricate neuropsychiatric brain disorder characterized by social deficits and repetitive behaviors. Deep learning approaches have been applied in clinical or behavioral identification of ASD; most erstwhile models are inadequate in their capacity to exploit the data richness. On the other hand, classification techniques often solely rely on region-based summary and/or functional connectivity analysis of functional magnetic resonance imaging (fMRI). Besides, biomedical data modeling to analyze big data related to ASD is still perplexing due to its complexity and heterogeneity. Single volume image consideration has not been previously investigated in classification purposes. By deeming these challenges, in this work, firstly, we design an image generator to generate single volume brain images from the whole-brain image by considering the voxel time point of each subject separately. Then, to classify ASD and typical control participants, we evaluate four deep learning approaches with their corresponding ensemble classifiers comprising one amended Convolutional Neural Network (CNN). Finally, to check out the data variability, we apply the proposed CNN classifier with leave-one-site-out 5-fold cross-validation across the sites and validate our findings by comparing with literature reports. We showcase our approach on large-scale multi-site brain imaging dataset (ABIDE) by considering four preprocessing pipelines, which outperforms the state-of-the-art methods. Hence, it is robust and consistent.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/JBHI.2020.2998603DOI Listing
November 2020

An accurate and universal approach for short-exposure-time microscopy image enhancement.

Comput Med Imaging Graph 2020 07 5;83:101743. Epub 2020 Jun 5.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, China. Electronic address:

Fluorescence microscopy imaging has become an essential technique in the biology and biomedical science which can provide comprehensive visualization of many biological processes, and the exposure time is one of the most critical parameters for fluorescence microscopy imaging. Short-exposure-time (SET) imaging overcomes the limitations of photo-bleaching and photo-toxicity, allowing comprehensive visualization of the biological processes. Unfortunately, SET images deteriorate the signal to noise ratio and the image quality when compared with the long-exposure-time (LET) images. Therefore, we introduce a data-driven microscopy image enhancement network (MIEN) to improve the quality of SET microscopy images without requiring any manual intervention, facilitating the production of high-resolution and high contrast images. The universal property and accuracy of the proposed network are validated on 38,500 real fluorescence microscopy images, which contain different object contents and are collected from various exposure time ratios and fluorescence microscopes platforms. Experimental results demonstrate that the proposed MIEN model is effective to enhance the quality of SET fluorescence microscopy images, and can be used to observe delicate changes in cells, tissues and organs with low photo-bleaching and photo-toxicity.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.compmedimag.2020.101743DOI Listing
July 2020

The Role of Imaging in the Detection and Management of COVID-19: A Review.

IEEE Rev Biomed Eng 2021 22;14:16-29. Epub 2021 Jan 22.

Coronavirus disease 2019 (COVID-19) caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is spreading rapidly around the world, resulting in a massive death toll. Lung infection or pneumonia is the common complication of COVID-19, and imaging techniques, especially computed tomography (CT), have played an important role in diagnosis and treatment assessment of the disease. Herein, we review the imaging characteristics and computing models that have been applied for the management of COVID-19. CT, positron emission tomography - CT (PET/CT), lung ultrasound, and magnetic resonance imaging (MRI) have been used for detection, treatment, and follow-up. The quantitative analysis of imaging data using artificial intelligence (AI) is also explored. Our findings indicate that typical imaging characteristics and their changes can play crucial roles in the detection and management of COVID-19. In addition, AI or other quantitative image analysis methods are urgently needed to maximize the value of imaging in the management of COVID-19.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/RBME.2020.2990959DOI Listing
February 2021

Real-time and multimodality image-guided intelligent HIFU therapy for uterine fibroid.

Theranostics 2020 26;10(10):4676-4693. Epub 2020 Mar 26.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, China.

: High-intensity focused ultrasound (HIFU) therapy represents a noninvasive surgical approach to treat uterine fibroids. The operation of HIFU therapy relies on the information provided by medical images. In current HIFU therapy, all operations such as positioning of the lesion in magnetic resonance (MR) and ultrasound (US) images are manually performed by specifically trained doctors. Manual processing is an important limitation of the efficiency of HIFU therapy. In this paper, we aim to provide an automatic and accurate image guidance system, intelligent diagnosis, and treatment strategy for HIFU therapy by combining multimodality information. : In intelligent HIFU therapy, medical information and treatment strategy are automatically processed and generated by a real-time image guidance system. The system comprises a novel multistage deep convolutional neural network for preoperative diagnosis and a nonrigid US lesion tracking procedure for HIFU intraoperative image-assisted treatment. In the process of intelligent therapy, the treatment area is determined from the autogenerated lesion area. Based on the autodetected treatment area, the HIFU foci are distributed automatically according to the treatment strategy. Moreover, an image-based unexpected movement warning and other physiological monitoring are used during the intelligent treatment procedure for safety assurance. : In the experiment, we integrated the intelligent treatment system on a commercial HIFU treatment device, and eight clinical experiments were performed. In the clinical validation, eight randomly selected clinical cases were used to verify the feasibility of the system. The results of the quantitative experiment indicated that our intelligent system met the HIFU clinical tracking accuracy and speed requirements. Moreover, the results of simulated repeated experiments confirmed that the autodistributed HIFU focus reached the level of intermediate clinical doctors. Operations performed by junior- or middle-level operators with the assistance of the proposed system can reach the level of operation performed by senior doctors. Various experiments prove that our proposed intelligent HIFU therapy process is feasible for treating common uterine fibroid cases. : We propose an intelligent HIFU therapy for uterine fibroid which integrates multiple medical information processing procedures. The experiment results demonstrated that the proposed procedures and methods can achieve monitored and automatic HIFU diagnosis and treatment. This research provides a possibility for intelligent and automatic noninvasive therapy for uterine fibroid.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.7150/thno.42830DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7150484PMC
February 2021

Immunohistochemical index prediction of breast tumor based on multi-dimension features in contrast-enhanced ultrasound.

Med Biol Eng Comput 2020 Jun 30;58(6):1285-1295. Epub 2020 Mar 30.

Ultrasonography Department, Nanjing Drum Tower Hospital, Nanjing, 210000, China.

Breast cancer is the leading killer of Chinese women. Immunohistochemistry index has great significance in the treatment strategy selection and prognosis analysis for breast cancer patients. Currently, histopathological examination of the tumor tissue through surgical biopsy is the gold standard to determine immunohistochemistry index. However, this examination is invasive and commonly causes discomfort in patients. There has been a lack of noninvasive method capable of predicting immunohistochemistry index for breast cancer patients. This paper proposes a machine learning method to predict the immunohistochemical index of breast cancer patients by using noninvasive contrast-enhanced ultrasound. A total of 119 breast cancer patients were included in this retrospective study. Each patient implemented the pathological examination of immunohistochemical expression and underwent contrast-enhanced ultrasound imaging of breast tumor. The multi-dimension features including 266 three-dimension features and 837 two-dimension dynamic features were extracted from the contrast-enhanced ultrasound sequences. Using the machine learning prediction method, 21 selected multi-dimension features were integrated to generate a model for predicting the immunohistochemistry index noninvasively. The immunohistochemical index of human epidermal growth factor receptor-2 (HER2) was predicted based on multi-dimension features in contrast-enhanced ultrasound sequence with the sensitivity of 71%, and the specificity of 79% in the testing cohort. Therefore, the noninvasive contrast-enhanced ultrasound can be used to predict the immunohistochemical index. To our best knowledge, no studies have been reported about predicting immunohistochemical index by using contrast-enhanced ultrasound sequences for breast cancer patients. Our proposed method is noninvasive and can predict immunohistochemical index by using contrast-enhanced ultrasound in several minutes, instead of relying totally on the invasive and biopsy-based histopathological examination. Graphical abstract Immunohistochemical index prediction of breast tumor based on multi-dimension features in contrast-enhanced ultrasound.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1007/s11517-020-02164-2DOI Listing
June 2020

Automatic estimation of morphological characteristics of proximal tibia for precise plate treatment using model matching.

Comput Med Imaging Graph 2020 04 19;81:101714. Epub 2020 Mar 19.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China.

Plate treatment is currently the standard treatment of proximal tibia fracture. Morphological characteristics can help orthopedic surgeons understand anatomic information of tibia and choose well-matched plate for reducing difficulties in plate treatment. However, manual measurement of morphological characteristics of patient's tibia is time-consuming and labor-intensive. Therefore, this study proposes an automatic method to accurately estimate the morphological characteristics of patient's tibia for assisting plate treatment. In the off-line stage, an average shape with typical characteristics was computed from 422 tibia models, and the morphological characteristics of the average shape were measured by the orthopedic surgeon. In the on-line stage, the point's correspondence between the average shape and every tibia model was created by the proposed morphable model matching method firstly. Then, the morphological characteristics of tibia for every patient were estimated automatically based on the point's correspondence and characteristics of average shape. The effectiveness of the method was validated by comparing the manual measured and automatic-estimated characteristics. In addition, the basic experiments of virtual and real plate implantation preliminarily confirmed that the automatic-estimated morphological characteristics were helpful for plate treatment. In all, we propose an automatic and accurate estimation method of morphological characteristics for a large-scale library of Chinese tibia models, which provides orthopedic surgeons with scientific and quantitative description of tibia.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1016/j.compmedimag.2020.101714DOI Listing
April 2020

Tissue Structure Updating for In Situ Augmented Reality Navigation Using Calibrated Ultrasound and Two-Level Surface Warping.

IEEE Trans Biomed Eng 2020 11 16;67(11):3211-3222. Epub 2020 Mar 16.

Objective: In minimally invasive surgery (MIS), in situ augmented reality (AR) navigation systems are usually implemented using a glasses-free 3D display to represent the preoperative tissue structure, and can provide intuitive see-through guidance information. However, due to changes in intraoperative tissue, the preoperative tissue structure is not able to exactly correspond to reality, which influences the precision of in situ AR navigation. To solve this problem, we propose a method to update the tissue structure for in situ AR navigation in such way to reflect changes in intraoperative tissue.

Methods: The proposed method to update the tissue structure is based on the calibrated ultrasound and two-level surface warping technologies. Firstly, the particle filter-based calibration is implemented to perform ultrasound calibration and obtain intraoperative position of anatomical points. Secondly, intraoperative positions of anatomical points are inputted in the two-level surface warping method to update the preoperative tissue structure. Finally, the glasses-free real 3-D display of the updated tissue structure is finished, and is superimposed onto a patient by a translucent mirror for in situ AR navigation.

Results: we validated the proposed method by simulating liver tissue intervention, and achieved the tissue updating accuracy of 92.86%. Furthermore, the targeting error of AR navigation based on the proposed method was also evaluated through minimally invasive liver surgery, and the acquired mean targeting error was 1.92 mm.

Conclusion: The results demonstrate that the proposed AR navigation method is effective.

Significance: The proposed method can facilitate MIS, as it provides accurate 3D navigation.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1109/TBME.2020.2979535DOI Listing
November 2020

Probability analysis of axillary lymph node metastasis in breast cancer patients using particle space-time distribution model.

Healthc Technol Lett 2019 Dec 26;6(6):266-270. Epub 2019 Nov 26.

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 10084, People's Republic of China.

The possibility of axillary lymph node metastasis differs in different breast cancer patients and is the strongest prognostic indicator in breast cancer. The existing studies mainly explored the relationship of axillary ultrasound imaging and axillary lymph node metastasis, without exploring whether ultrasound imaging of breast tumour can affect and perform axillary lymph node prediction. Therefore, this Letter proposes a novel particle space-time distribution model to find the correlation between contrast-enhanced ultrasonography of breast tumour and axillary lymphatic metastasis. Starting from the imaging principle of dynamic contrast-enhanced ultrasonography, the particle space-time distribution model not only comprises space-time features of contrast-enhanced ultrasonography with an encoder-decoder network, but also the flow field information of microbubble particles is integrated into the space-time features that better serves the metastasis prediction by enhancing the particle distribution information. Extensive experiments on real patients have demonstrated that dynamic contrast-enhanced ultrasonography of breast tumour can be used to predict the probability of lymphatic metastasis. This conclusion can be interpretable from the clinical and pathological perspectives.
View Article and Find Full Text PDF

Download full-text PDF

Source
http://dx.doi.org/10.1049/htl.2019.0072DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC6952258PMC
December 2019
-->