Intelligent medical assistant diagnosis method based on data fusion

ZHANG Tao-hong; FAN Su-li; GUO Xu-xu; LI Qian-qian

doi:10.13374/j.issn2095-9389.2021.01.12.003

Volume 43 Issue 9

Sep. 2021

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Engineering > 2021 > 43(9): 1197-1205

ZHANG Tao-hong, FAN Su-li, GUO Xu-xu, LI Qian-qian. Intelligent medical assistant diagnosis method based on data fusion[J]. Chinese Journal of Engineering, 2021, 43(9): 1197-1205. doi: 10.13374/j.issn2095-9389.2021.01.12.003

Citation:

ZHANG Tao-hong, FAN Su-li, GUO Xu-xu, LI Qian-qian. Intelligent medical assistant diagnosis method based on data fusion[J]. Chinese Journal of Engineering, 2021, 43(9): 1197-1205. doi: 10.13374/j.issn2095-9389.2021.01.12.003

Citation:

PDF( 1031 KB)

Intelligent medical assistant diagnosis method based on data fusion

doi: 10.13374/j.issn2095-9389.2021.01.12.003

ZHANG Tao-hong^{1, 2
,
,},
FAN Su-li^{1, 2},
GUO Xu-xu^{1, 2},
LI Qian-qian^{1, 2}

1.
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China
2.
Beijing Key Laboratory of Knowledge Engineering for Materials Science., Beijing 100083, China

More Information

Corresponding author: E-mail: zth_ustb@163.com
Received Date: 2021-01-12
Available Online: 2021-03-01
Publish Date: 2021-09-18

Abstract

Abstract

In the field of medicine, in order to diagnose a patient’s condition more efficiently and conveniently, image classification has been widely leveraged. It is well established that when doctors diagnose a patient’s condition, they not only observe the patient’s image information (such as CT image) but also make final decisions incorporating the patient’s clinical diagnostic information. However, current medical image classification only puts the image into a convolution neural network to obtain the diagnostic result and does not use the clinical diagnosis information. In intelligent auxiliary diagnosis, it is necessary to combine clinical symptoms with other imaging data for comprehensive diagnosis. This paper presented a new method of assistant diagnosis for the medical field. This method combined information from patients’ imaging with numerical data (such as clinical diagnosis information) and used the combined information to automatically predict the patient’s condition. Based on this method, a medical assistant diagnosis model based on deep learning was proposed. The model takes images and numerical data as input and outputs the patient’s condition. Thus, this method is comprehensive and helps improve the accuracy of automatic diagnosis and reduce diagnostic error. Moreover, the proposed model can simultaneously process multiple types of data, thus saving diagnosis time. The effectiveness of the proposed method was verified in two groups of experiments designed in this paper. The first group of experiments shows that if the unrelated data are fused for classification, the proposed method cannot enhance the classification ability of the model, although it is able to predict multiple diseases at one time. The second group of experiments show that the proposed method could significantly improve classification results if the interrelated data are fused.
- image classification,
- convolution neural network,
- feature fusion,
- medical diagnosis,
- deep learning

FullText(HTML)

References(31)

References

[1]	Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proc IEEE, 1998, 86(11): 2278 doi: 10.1109/5.726791
[2]	Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84 doi: 10.1145/3065386
[3]	Deng J, Dong W, Socher R, et al. ImageNet: A large-scale hierarchical image database // 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, 2009: 248
[4]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [J/OL]. ArXiv Preprint (2014-09-04) [2021-01-12]. https://arxiv.org/abs/1409.1556
[5]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770
[6]	Szegedy C, Liu W, Jia Y Q, et al. Going deeper with convolutions // 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, 2015: 1
[7]	Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift [J/OL]. ArXiv Preprint (2015-02-11) [2021-01-12]. https://arxiv.org/abs/1502.03167
[8]	Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 2818
[9]	Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, inception-resnet and the impact of residual connections on learning [J/OL]. ArXiv Preprint (2016-02-24) [2021-01-12]. https://arxiv.org/abs/1602.07261
[10]	Iandola F N, Han S, Moskewicz M W, et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size [J/OL]. ArXiv Preprint (2016-02-24) [2021-01-12]. https://arxiv.org/abs/1602.07360
[11]	Howard A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications [J/OL]. ArXiv Preprint (2016-02-24) [2021-01-12]. https://arxiv.org/abs/1704.04861v1
[12]	Zhang X Y, Zhou X Y, Lin M X, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 6848
[13]	Sandler M, Howard A, Zhu M L, et al. MobileNetV2: inverted residuals and linear bottlenecks// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 4510
[14]	Ma N N, Zhang X Y, Zheng H T, et al. ShuffleNet V2: practical guidelines for efficient CNN architecture design // 2018 European Conference on Computer Vision (ECCV). Munich, 2018: 122
[15]	Howard A, Sandler M, Chen B, et al. Searching for MobileNetV3//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, 2019: 1314
[16]	Li L, Xu M, Liu H R, et al. A large-scale database and a CNN model for attention-based glaucoma detection. IEEE Trans Med Imaging, 2020, 39(2): 413 doi: 10.1109/TMI.2019.2927226
[17]	Yang H, Kim J Y, Kim H, et al. Guided soft attention network for classification of breast cancer histopathology images. IEEE Trans Med Imaging, 2020, 39(5): 1306 doi: 10.1109/TMI.2019.2948026
[18]	Xu X Y, Wang C D, Guo J X, et al. MSCS-DeepLN: Evaluating lung nodule malignancy using multi-scale cost-sensitive neural networks. Med Image Anal, 2020, 65: 101772 doi: 10.1016/j.media.2020.101772
[19]	Mobiny A, Lu H Y, Nguyen H V, et al. Automated classification of apoptosis in phase contrast microscopy using capsule network. IEEE Trans Med Imaging, 2020, 39(1): 1 doi: 10.1109/TMI.2019.2918181
[20]	Zhou Y, Li G Q, Li H Q. Automatic cataract classification using deep neural network with discrete state transition. IEEE Trans Med Imaging, 2020, 39(2): 436 doi: 10.1109/TMI.2019.2928229
[21]	Wang Y, Wang N, Xu M, et al. Deeply-supervised networks with threshold loss for cancer detection in automated breast ultrasound. IEEE Trans Med Imaging, 2020, 39(4): 866 doi: 10.1109/TMI.2019.2936500
[22]	Liu T J, Guo Q Q, Lian C F, et al. Automated detection and classification of thyroid nodules in ultrasound images using clinical-knowledge-guided convolutional neural networks. Med Image Anal, 2019, 58: 101555 doi: 10.1016/j.media.2019.101555
[23]	姚超, 趙基淮, 馬博淵, 等. 基于深度學習的宮頸癌異常細胞快速檢測方法. 工程科學學報, https://doi.org/10.13374/j.issn2095-9389.2021.01.12.001 Yao C, Zhao J Z, Ma B Y, et al. Fast detection method for cervical cancer abnormal cells based on deep learning. Chin J Eng, https://doi.org/10.13374/j.issn2095-9389.2021.01.12.001
[24]	Zeng Y W, Liu X K, Xiao N, et al. Automatic diagnosis based on spatial information fusion feature for intracranial aneurysm. IEEE Trans Med Imaging, 2020, 39(5): 1448 doi: 10.1109/TMI.2019.2951439
[25]	Wang L T, Zhang L, Zhu M J, et al. Automatic diagnosis for thyroid nodules in ultrasound images by deep neural networks. Med Image Anal, 2020, 61: 101665 doi: 10.1016/j.media.2020.101665
[26]	Kumar A, Fulham M, Feng D G, et al. Co-learning feature fusion maps from PET?CT images of lung cancer. IEEE Trans Med Imaging, 2020, 39(1): 204 doi: 10.1109/TMI.2019.2923601
[27]	Joyseeree R, Otálora S, Müller H, et al. Fusing learned representations from Riesz filters and deep CNN for lung tissue classification. Med Image Anal, 2019, 56: 172 doi: 10.1016/j.media.2019.06.006
[28]	Kingma D, Ba J. Adam: A method for stochastic optimization[J/OL]. ArXiv Preprint (2014-12-22) [2021-01-12]. https://arxiv.org/abs/1412.6980
[29]	Bio. Heart Disease UCI [J/OL ]. Kaggle (2018-06-25) [2021-01-12]. https://www.kaggle.com/ronitf/heart-disease-uci
[30]	Società Italiana di Radiologia Medica e Interventistica. Covid−19 Database[J/OL]. Database Online (2020-03-18) [2021-01-12]. https://www.sirm.org/category/senza-categoria/covid-19
[31]	Zhao J, Zhang Y, He X, et al. COVID−CT−Dataset: A CT scan dataset about COVID−19[J/OL]. ArXiv Preprint (2020-03-30) [2021-01-12]. https://github.com/UCSD-AI4H/COVID-CT