Micro-expression recognition algorithm based on separate long-term recurrent convolutional network

LI Xue-han; HU Si-quan; SHI Zhi-guo; ZHANG Ming

doi:10.13374/j.issn2095-9389.2020.06.15.006

Volume 44 Issue 1

Jan. 2022

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Engineering > 2022 > 44(1): 104-113

LI Xue-han, HU Si-quan, SHI Zhi-guo, ZHANG Ming. Micro-expression recognition algorithm based on separate long-term recurrent convolutional network[J]. Chinese Journal of Engineering, 2022, 44(1): 104-113. doi: 10.13374/j.issn2095-9389.2020.06.15.006

Citation:

LI Xue-han, HU Si-quan, SHI Zhi-guo, ZHANG Ming. Micro-expression recognition algorithm based on separate long-term recurrent convolutional network[J]. Chinese Journal of Engineering, 2022, 44(1): 104-113. doi: 10.13374/j.issn2095-9389.2020.06.15.006

Citation:

PDF( 1362 KB)

Micro-expression recognition algorithm based on separate long-term recurrent convolutional network

doi: 10.13374/j.issn2095-9389.2020.06.15.006

LI Xue-han¹,
HU Si-quan^{1, 2
,
,},
SHI Zhi-guo^{1, 2, 3},
ZHANG Ming⁴

1.
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China
2.
Shunde Graduate School, University of Science and Technology Beijing, Foshan 528399, China
3.
Beijing Big Data Center, Beijing 100101, China
4.
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China

More Information

Corresponding author: E-mail: husiquan@ustb.edu.cn
Received Date: 2020-06-15
Available Online: 2020-07-23
Publish Date: 2022-01-01

Abstract

Abstract

With the rapid development of machine learning and deep neural network and the popularization of intelligent devices, face recognition technology has rapidly developed. At present, the accuracy of face recognition has exceeded that of the human eyes. Moreover, the software and hardware conditions of large-scale popularization are available, and the application fields are widely distributed. As an important part of face recognition technology, facial expression recognition has been a widely studied subject in the fields of artificial intelligence, security, automation, medical treatment, and driving in recent years. Expression recognition, an active research area in human–computer interaction, involves informatics and psychology and has good research prospect in teaching evaluation. Micro-expression, which has great research significance, is a kind of short-lived facial expression that humans unconsciously make when trying to hide some emotion. Different from the general static facial expression recognition, to realize micro-expression recognition, besides extracting the spatial feature information of facial expression deformation in the image, the temporal-motion information of the continuous image sequence also needs to be considered. In this study, given that static expression features lack temporal information, so that the subtle changes in expression cannot be fully reflected, facial dynamic expression sequences were used to fuse spatial features and temporal features, and neural networks were used to provide good features in the field of image classification. Expression sequences were processed, and a micro-expression recognition method based on separate long-term recurrent convolutional network (S-LRCN) was proposed. First, the micro-expression data set was selected to extract the facial image sequence, and the transfer learning method was introduced to extract the spatial features of the expression frame through the pre-trained convolution neural network model, to reduce the risk of overfitting in the network training, and the extracted features of the video sequence were inputted into long short-term memory (LSTM) to process the temporal-domain features. Finally, a small database of learners’ expression sequences was established, and the method was used to assist teaching evaluation.
- micro-expression recognition,
- spatial-temporal features,
- LRCN,
- LSTM,
- education evaluation

FullText(HTML)

References(35)

References

[1]	Mehrabian A. Nonverbal Communication. New York: Routledge, 2017
[2]	Ekman P. Facial expression and emotion. Am Psychol, 1993, 48(4): 384 doi: 10.1037/0003-066X.48.4.384
[3]	Ekman P, Friesen W V. Nonverbal leakage and clues to deception. Psychiatry, 1969, 32(1): 88 doi: 10.1080/00332747.1969.11023575
[4]	Yan W J, Wang S J, Liu Y J, et al. For micro-expression recognition: Database and suggestions. Neurocomputing, 2014, 136(136): 82
[5]	王思宇. 基于CNN-RNN的微表情識別[學位論文]. 哈爾濱: 哈爾濱工程大學, 2018 Wang S Y. CNN-RNN Based Micro-Expression Recognition [Dissertation]. Harbin: Harbin Engineering University, 2018
[6]	Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84 doi: 10.1145/3065386
[7]	Donahue J, Hendricks L A, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description // 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, 2015: 2625
[8]	Ekman P, Rosenberg E L. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). 2nd Ed. USA: Oxford University Press, 2005
[9]	Gunes H, Pantic M. Automatic, dimensional and continuous emotion recognition. Int J Synthetic Emotions, 2010, 1(1): 68 doi: 10.4018/jse.2010101605
[10]	Song Y L, Morency L P, Davis R. Learning a sparse codebook of facial and body microexpressions for emotion recognition // Proceedings of the 15th ACM International Conference on Multimodal Interaction. New York, 2013: 237
[11]	李丹, 解侖, 盧婷, 等. 基于光流方向信息熵統計的微表情捕捉. 工程科學學報, 2017(11):1727 Li D, Xie L, Lu T, et al. Capture of microexpressions based on the entropy of oriented optical flow. Chin J Eng, 2017(11): 1727
[12]	Polikovsky S, Kameda Y, Ohta Y. Facial micro-expression detection in Hi-speed video based on facial action coding system (FACS). IEICE Trans Inform Syst, 2013, E96-D(1): 81 doi: 10.1587/transinf.E96.D.81
[13]	Shreve M, Godavarthy S, Goldgof D, et al. Macro- and micro-expression spotting in long videos using spatio-temporal strain // 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG). Santa Barbara, 2011: 51
[14]	Pfister T, Li X B, Zhao G Y, et al. Recognising spontaneous facial micro-expressions // 2011 International Conference on Computer Vision. Barcelona, 2011: 1449
[15]	梁靜, 顏文靖, 吳奇, 等. 微表情研究的進展與展望. 中國科學基金, 2013(2):75 Liang J, Yan W J, Wu Q, et al. Recent advances and future trends in micro-expression research. Bull Natl Nat Sci Foundation China, 2013(2): 75
[16]	Wang Y D, See J, Phan R C W, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition // Asian Conference on Computer Vision ? ACCV2014. Switzerland, 2015: 525
[17]	Liong S T, See J, Phan R C W, et al. Spontaneous subtle expression detection and recognition based on facial strain. Signal Process Image Commun, 2016, 47: 170 doi: 10.1016/j.image.2016.06.004
[18]	Le Ngo A C, See J, Phan R C W. Sparsity in dynamics of spontaneous subtle emotion: analysis & application. IEEE Trans Affective Comput, 2017, 8(3): 396 doi: 10.1109/TAFFC.2016.2523996
[19]	Xu F, Zhang J P, Wang J Z. Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affective Comput, 2017, 8(2): 254 doi: 10.1109/TAFFC.2016.2518162
[20]	Patel D, Hong X P, Zhao G Y. Selective deep features for micro-expression recognition // 2016 23rd International Conference on Pattern Recognition (ICPR). Cancun, 2016: 2258
[21]	Kim D H, Baddar W J, Ro Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations // Proceedings of the 24th ACM international conference on Multimedia. Amsterdam, 2016: 382
[22]	Khor H Q, See J, Phan R C W, et al. Enriched long-term recurrent convolutional network for facial micro-expression recognition // 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi’an, 2018: 667
[23]	Verburg M, Menkovski V. Micro-expression detection in long videos using optical flow and recurrent neural networks // 2019 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019). Lille, 2019: 1
[24]	Itti L, Koch C. Computational modelling of visual attention. Nat Rev Neurosci, 2001, 2(3): 194 doi: 10.1038/35058500
[25]	Wang C Y, Peng M, Bi T, et al. Micro-attention for micro-expression recognition [J/OL]. arXiv Preprint (2019-08-27) [2020-04-21]. https://arxiv.org/abs/1811.02360.
[26]	Parkhi O M, Vedaldi A, Zisserman A. Deep face recognition // Proceedings of the British Machine Vision Conference (BMVC). Swansea, 2015: 45
[27]	Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011 doi: 10.1109/TPAMI.2019.2913372
[28]	Cao Q, Shen L, Xie W D, et al. VGGFace2: A dataset for recognising faces across pose and age // 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi’an, 2018: 67
[29]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770
[30]	Cootes T F, Edwards G J, Taylor C J. Active appearance models. IEEE Trans Pattern Anal Mach Intell, 2001, 23(6): 681 doi: 10.1109/34.927467
[31]	彭敏. 基于雙時間尺度卷積神經網絡的微表情識別[學位論文]. 重慶: 西南大學, 2017 Peng M. Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition [Dissertation]. Chongqing: Southwest University, 2017
[32]	Yan W J, Li X B, Wang S J, et al. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE, 2014, 9(1): e86041 doi: 10.1371/journal.pone.0086041
[33]	Zhou Z H, Zhao G Y, Pietikinen M. Towards a practical lipreading system // The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). Providence, RI, 2011: 137
[34]	Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell, 2007, 29(6): 915 doi: 10.1109/TPAMI.2007.1110
[35]	Huang X H, Zhao G Y, Hong X P, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing, 2016, 175: 564 doi: 10.1016/j.neucom.2015.10.096