Defocus spread effect elimination method in multiple multi-focus image fusion for microscopic images

YIN Xiang; MA Bo-yuan; BAN Xiao-juan; HUANG Hai-you; WANG Yu; LI Song-yan

doi:10.13374/j.issn2095-9389.2021.01.12.002

Volume 43 Issue 9

Sep. 2021

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Engineering > 2021 > 43(9): 1174-1181

YIN Xiang, MA Bo-yuan, BAN Xiao-juan, HUANG Hai-you, WANG Yu, LI Song-yan. Defocus spread effect elimination method in multiple multi-focus image fusion for microscopic images[J]. Chinese Journal of Engineering, 2021, 43(9): 1174-1181. doi: 10.13374/j.issn2095-9389.2021.01.12.002

Citation:

YIN Xiang, MA Bo-yuan, BAN Xiao-juan, HUANG Hai-you, WANG Yu, LI Song-yan. Defocus spread effect elimination method in multiple multi-focus image fusion for microscopic images[J]. Chinese Journal of Engineering, 2021, 43(9): 1174-1181. doi: 10.13374/j.issn2095-9389.2021.01.12.002

Citation:

YIN Xiang, MA Bo-yuan, BAN Xiao-juan, HUANG Hai-you, WANG Yu, LI Song-yan. Defocus spread effect elimination method in multiple multi-focus image fusion for microscopic images[J]. Chinese Journal of Engineering, 2021, 43(9): 1174-1181. doi: 10.13374/j.issn2095-9389.2021.01.12.002

PDF( 1019 KB)

Defocus spread effect elimination method in multiple multi-focus image fusion for microscopic images

doi: 10.13374/j.issn2095-9389.2021.01.12.002

YIN Xiang^{1, 2, 3},
MA Bo-yuan^{1, 2, 3, 4},
BAN Xiao-juan^{1, 2, 3, 4},
HUANG Hai-you^{1, 4, 5},
WANG Yu^{6
,
,},
LI Song-yan⁷

1.
Beijing Advanced Innovation Center for Materials Genome Engineering, University of Science and Technology Beijing, Beijing 100083, China
2.
Beijing Key Laboratory of Knowledge Engineering for Materials Science, University of Science and Technology Beijing, Beijing 100083, China
3.
Institute of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China
4.
Shunde Graduate School, University of Science and Technology Beijing, Foshan 528300, China
5.
Institute for Advanced Materials and Technology, University of Science and Technology Beijing, Beijing 100083, China
6.
School of Cyberspace Science and Technology, Beijing Institute of Technology, Beijing 100081, China
7.
Clinical Medicine, Zhengzhou University, Zhengzhou 450052, China

More Information

Corresponding author: E-mail: hejohejo@126.com
Received Date: 2021-01-12
Available Online: 2021-03-01
Publish Date: 2021-09-18

Abstract

Abstract

For a microscopic imaging scene, an all-in-focus image of the observation object is needed. Because of the limitation of the depth of field of the camera and the typically uneven surface of the observation object, an all-in-focus image is obtained through one shot with relative difficulty. In this case, an alternative method for obtaining the all-in-focus image is usually used, which is to fuse several images focusing on different depths with the help of multi-focus image fusion technology. Multi-focus image fusion is an important branch in the field of computer vision. It aims to use image processing technology to fuse the clear regions of multiple images, focusing on different objects in the same scene, and finally to obtain an all-in-focus fusion result. With the breakthrough of machine learning theory represented by deep learning, the convolutional neural network is widely adopted in the field of multi-focus image fusion. However, most methods only focus on improving network structure and use the simple one-by-one serial fusion method, which reduces the efficiency of multiple image fusion. In addition, the defocus spread effect in the fusion process, which causes blurred artifacts in the areas near focus map boundaries, can severely affect the quality of fusion results. In the application of microscopic imaging analysis, we proposed a maximum spatial frequency in the feature map (MSFIFM) fusion strategy. By adding a post-processing module in the convolution neural network based on unsupervised learning, the redundant feature extraction process in the one-by-one serial fusion is avoided. Experiments demonstrate that this strategy can significantly improve the efficiency of multi-focus image fusion with multiple images. In addition, we presented a correction strategy that can effectively alleviate the effect of defocus spread on the fusion result under the condition of ensuring the efficiency of the algorithm fusion.
- multi-focus image fusion,
- defocusing diffusion,
- deep learning,
- convolutional neural network,
- multiple image fusion

FullText(HTML)

References(29)

References

[1]	Liu Y, Wang L, Cheng J, et al. Multi-focus image fusion: A Survey of the state of the art. Inf Fusion, 2020, 64: 71 doi: 10.1016/j.inffus.2020.06.013
[2]	Szeliski R. Computer vision: Algorithms and Applications. London: Springer, 2011
[3]	章毓晉. 圖像工程. 4版. 北京: 清華大學出版社, 2018 Zhang Y J. Image Engineering. 4th ed. Beijing: Tsinghua University Press, 2018
[4]	Burt P, Adelson E. The Laplacian pyramid as a compact image code. IEEE Trans Commun, 1983, 31(4): 532 doi: 10.1109/TCOM.1983.1095851
[5]	Toet A. Image fusion by a ratio of low-pass pyramid. Pattern Recognit Lett, 1989, 9(4): 245 doi: 10.1016/0167-8655(89)90003-2
[6]	Li H, Manjunath B S, Mitra S K. Multisensor image fusion using the wavelet transform. Graph Models Image Process, 1995, 57(3): 235 doi: 10.1006/gmip.1995.1022
[7]	Li S T, Kwok J T, Wang Y N. Combination of images with diverse focuses using the spatial frequency. Inf Fusion, 2001, 2(3): 169 doi: 10.1016/S1566-2535(01)00038-0
[8]	Li S T, Kang X D, Hu J W. Image fusion with guided filtering. IEEE Trans Image Process, 2013, 22(7): 2864 doi: 10.1109/TIP.2013.2244222
[9]	Zhou Z Q, Li S, Wang B. Multi-scale weighted gradient-based fusion for multi-focus images. Inf Fusion, 2014, 20: 60 doi: 10.1016/j.inffus.2013.11.005
[10]	Liu Y, Liu S P, Wang Z F. Multi-focus image fusion with dense SIFT. Inf Fusion, 2015, 23: 139 doi: 10.1016/j.inffus.2014.05.004
[11]	LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436 doi: 10.1038/nature14539
[12]	Liu Y, Chen X, Peng H, et al. Multi-focus image fusion with a deep convolutional neural network. Inf Fusion, 2017, 36: 191 doi: 10.1016/j.inffus.2016.12.001
[13]	Ma, B Y, Zhu Y, Yin X, et al. SESF?Fuse: An unsupervised deep model for multi-focus image fusion. Neural Comput Appl, 2021, 33: 5793 doi: 10.1007/s00521-020-05358-9
[14]	Xu H, Ma J Y, Jiang J J, et al. U2Fusion: A unified unsupervised image fusion network. IEEE Trans Pattern Anal Mach Intell, 10.1109/TPAMI.2020.3012548
[15]	Prabhakar K R, Srikar V S, Babu R V. DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs// IEEE International Conference on Computer Vision. Venice, 2017: 4724
[16]	Ma B Y, Yin X, Wu D, et al. Gradient Aware Cascade Network for Multi-Focus Image Fusion[J/OL]. ArXiv Preprint (2020-10-01) [2021-01-12]. https://arxiv.org/abs/2010.08751
[17]	Xu H, Fan F, Zhang H, et al. A deep model for multi-focus image fusion based on gradients and connected regions. IEEE Access, 2020, 8: 26316 doi: 10.1109/ACCESS.2020.2971137
[18]	Huang J, Le Z L, Ma Y, et al. A generative adversarial network with adaptive constraints for multi-focus image fusion. Neural Comput Appl, 2020, 32(18): 15119 doi: 10.1007/s00521-020-04863-1
[19]	王鏢堡. 基于深度學習的多聚焦圖像算法研究[學位論文]. 昆明: 云南大學, 2018 Wang B B. Research on Multi-Focus Image Fusion Algorithm Based on Deep Learning [Dissertation]. Kunming: Yunnan University, 2018
[20]	Ma H, Liao Q, Zhang J, et al. An α-Matte Boundary Defocus Model Based Cascaded Network for Multi-focus Image Fusion[J/OL]. ArXiv Preprint (2019-10-30) [2021-01-12]. https://arxiv.org/abs/1910.13136
[21]	何凱, 魏穎, 王陽, 等. 一種改進的非剛性圖像配準算法. 工程科學學報, 2019, 41(7):955 He K, Wei Y, Wang Y, et al. An improved non-rigid image registration approach. Chin J Eng, 2019, 41(7): 955
[22]	陳世偉, 張勝修, 楊小岡, 等. 基于橢圓對稱方向矩的可見光與紅外圖像配準算法. 工程科學學報, 2017, 39(7):1107 Chen S W, Zhang S X, Yang X G, et al. Registration of visual-infrared images based on ellipse symmetrical orientation moment. Chin J Eng, 2017, 39(7): 1107
[23]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 7132
[24]	Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common Objects in Context. Computer Vision – ECCV 2014. Cham: Springer International Publishing, 2014
[25]	馬博淵, 印象. SESF−Fuse的多聚焦圖像融合開源代碼[J/OL]. Github (2019-08-21) [2021-01-12]. https://github.com/Keep-Passion/SESF-Fuse Ma B Y, Yin X. The Code of SESF−Fuse for multi-focus image fusion [J/OL]. Github (2019-08-21) [2021-01-12]. https://github.com/Keep-Passion/SESF-Fuse
[26]	Kingma D, Ba J. Adam: A method for stochastic optimization[J/OL]. ArXiv Preprint (2017-01-30) [2021-01-12]. https://arxiv.org/abs/1412.6980
[27]	Paszke A, Gross S, Massa F, et al. Py Torch: An imperative style, high-performance deep learning library[J/OL]. ArXiv Preprint (2019-12-3) [2021-01-12]. https://arxiv.org/abs/1912.01703
[28]	毛星云. OpenCV3編程入門. 北京: 電子工業出版社, 2015 Mao X Y. Introduction to OpenCV3 Programming. Beijing: Electronics industry publishing house, 2015
[29]	Xu S, Ji L Z, Wang Z, et al. Towards reducing severe defocus spread effects for multi-focus image fusion via an optimization based strategy. IEEE Trans Comput Imaging, 2020, 6: 1561 doi: 10.1109/TCI.2020.3039564