Review on deep learning models for time series forecasting in industry

LI Xiao-rui; BAN Xiao-juan; YUAN Zhao-lin; QIAO Hao-ran

doi:10.13374/j.issn2095-9389.2021.12.02.004

Volume 44 Issue 4

Apr. 2022

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Engineering > 2022 > 44(4): 757-766

LI Xiao-rui, BAN Xiao-juan, YUAN Zhao-lin, QIAO Hao-ran. Review on deep learning models for time series forecasting in industry[J]. Chinese Journal of Engineering, 2022, 44(4): 757-766. doi: 10.13374/j.issn2095-9389.2021.12.02.004

Citation:

LI Xiao-rui, BAN Xiao-juan, YUAN Zhao-lin, QIAO Hao-ran. Review on deep learning models for time series forecasting in industry[J]. Chinese Journal of Engineering, 2022, 44(4): 757-766. doi: 10.13374/j.issn2095-9389.2021.12.02.004

Citation:

PDF( 850 KB)

Review on deep learning models for time series forecasting in industry

doi: 10.13374/j.issn2095-9389.2021.12.02.004

LI Xiao-rui^{1, 2, 3, 4},
BAN Xiao-juan^{1, 2, 3, 4
,
,},
YUAN Zhao-lin^{1, 2, 3, 4},
QIAO Hao-ran¹

1.
Beijing Advanced Innovation Center for Materials Genome Engineering, University of Science and Technology Beijing, Beijing 100083, China
2.
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China
3.
Beijing Key Laboratory of Knowledge Engineering for Materials Science, University of Science and Technology Beijing, Beijing 100083, China
4.
Institute of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China

More Information

Corresponding author: E-mail: banxj@ustb.edu.cn
Received Date: 2021-12-02
Available Online: 2022-03-14
Publish Date: 2022-04-02

Abstract

Abstract

This review outlined recent developments in deep learning time series forecasting technology for the needs of industrial applications. With the advances in industrial automation, the storage and analysis of massive production data have become possible. Traditional mechanism modeling methods based on statistics encounter difficulties in dealing with high-dimensional industrial problems. Thus, time series forecasting for complex processes and products has played an important role in industrial scenarios, such as device modeling, production forecasting, remaining life prediction, and precise control, thereby receiving considerable attention from both academia and industry. To methodically review the time series forecasting method and its industrial applications, this review first introduced the three types of time series forecasting, namely, statistical learning, integrated learning, and deep learning, and compared their ease of use, complexity, and applicability issues. Focusing on industrial data analysis and decision-making, this review analyzed the three types of deep learning models, namely, recurrent neural network, convolutional neural network, and encoder–decoder network. The advantages and disadvantages of these three types of models and the applicable industrial environment were given, and how they can be embedded in industrial application sites to reduce costs and improve production efficiency was described. Afterward, to evaluate the performance of different algorithms clearly and comprehensively, statistical metrics and loss functions for the point prediction sequence and shape (motif) prediction problems were presented. At the same time, this review compiled classic public datasets of the industry for researchers to quickly find authoritative assessment data for an industry sector or issue. Taking mining and metallurgy in the process industry as examples, this review presented some widespread problems in this field, such as nonlinearity, long time delays, and unobservability of variables. This review also showed how deep-learning-based time series forecasting techniques can solve the aforementioned problems, build soft sensors, create digital twins, and achieve the visualization of complex processes. This review revealed that the application of deep learning in the process industry requires highly robust and strongly interpretable or explainable algorithms. For the robustness problem, the use of the ordinary differential equation model and Kalman filter method to solve the modeling of irregularly sampled time series and the use of the deep learning method to detect online sensor abnormalities were proposed. For the interpretability problem, sample-based, structure-based interpretable, and external co-explanation methods were introduced. This review also analyzed how explainable techniques can be applied to industrial deep learning models. Finally, the future directions of time series research were discussed in terms of both deep learning methods and industrial applications.
- process industry,
- time series forecasting,
- deep learning,
- model interpretability,
- evaluation metric

FullText(HTML)

References(56)

References

[1]	袁小鋒, 王雅琳, 陽春華, 等. 深度學習在流程工業過程數據建模中的應用. 智能科學與技術學報, 2020, 2(2):107 Yuan X F, Wang Y L, Yang C H, et al. The application of deep learning in data-driven modeling of process industries. Chin J Intell Sci Technol, 2020, 2(2): 107
[2]	Lim B, Zohren S. Time-series forecasting with deep learning: A survey. Philos Trans Ser A Math Phys Eng Sci, 2021, 379(2194): 20200209
[3]	徐鋼, 黎敏, 徐金梧, 等. 基于函數型數字孿生模型的轉爐煉鋼終點碳控制技術. 工程科學學報, 2019, 41(4):521 Xu G, Li M, Xu J W, et al. Control technology of end-point carbon in converter steelmaking based on functional digital twin model. Chin J Eng, 2019, 41(4): 521
[4]	Chung J, Gulcehre C, Cho K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling [J/OL]. arXiv preprint (2014-12-11) [2021-12-02].https://arxiv.org/abs/ 1412.3555
[5]	Salinas D, Flunkert V, Gasthaus J, et al. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. Int J Forecast, 2020, 36(3): 1181 doi: 10.1016/j.ijforecast.2019.07.001
[6]	Wen R, Torkkola K, Narayanaswamy B, et al. A multi-horizon quantile recurrent forecaster [J/OL]. arXiv preprint (2018-06-28) [2021-12-02].https://arxiv.org/abs/1711.11053
[7]	Oord A, Dieleman S, Zen H, et al. Wavenet: A generative model for raw audio [J/OL]. arXiv preprint (2016-09-19) [2021-12-02].https://arxiv.org/abs/1609.03499
[8]	Bai S, Kolter J Z, Koltun V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling [J/OL]. arXiv preprint (2016-09-19) [2021-12-02].https://arxiv.org/abs/1803.01271
[9]	Zhang K P, Liu Z J, Zheng L. Short-term prediction of passenger demand in multi-zone level: Temporal convolutional neural network with multi-task learning. IEEE Trans Intell Transp Syst, 2020, 21(4): 1480 doi: 10.1109/TITS.2019.2909571
[10]	Zhao W T, Gao Y Y, Ji T X, et al. Deep temporal convolutional networks for short-term traffic flow forecasting. IEEE Access, 2019, 7: 114496 doi: 10.1109/ACCESS.2019.2935504
[11]	Zheng C P, Fan X L, Wang C, et al. Gman: A graph multi-attention network for traffic prediction // Proceedings of the AAAI Conference on Artificial Intelligence. New York, 2020, 34(1): 1234
[12]	Lea C, Flynn M D, Vidal R, et al. Temporal convolutional networks for action segmentation and detection // 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, 2017: 1003
[13]	Farha Y A, Gall J. Ms-tcn: Multi-stage temporal convolutional network for action segmentation // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, 2019: 3570
[14]	張永峰, 陸志強. 基于集成神經網絡的剩余壽命預測. 工程科學學報, 2020, 42(10):1372 Zhang Y F, Lu Z Q. Remaining useful life prediction based on an integrated neural network. Chin J Eng, 2020, 42(10): 1372
[15]	徐任超, 閻威武, 王國良, 等. 基于周期性建模的時間序列預測方法及電價預測研究. 自動化學報, 2020, 46(6):1136 Xu R C, Yan W W, Wang G L, et al. Time series forecasting based on seasonality modeling and its application to electricity price forecasting. Acta Autom Sin, 2020, 46(6): 1136
[16]	Yuan Z L, Hu J L, Wu D, et al. A dual-attention recurrent neural network method for deep cone thickener underflow concentration prediction. Sensors, 2020, 20(5): 1260 doi: 10.3390/s20051260
[17]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [J/OL]. arXiv preprint (2017-12-12) [2021-12-02].https://arxiv.org/abs/1706.03762
[18]	Li S Y, Jin X Y, Xuan Y, et al. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting//33rd Conference on Neural Information Processing Systems. Vancouver, 2019
[19]	Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale [J/OL]. arXiv preprint (2021-06-03) [2021-12-02].https://arxiv.org/abs/2010.11929
[20]	Child R, Gray S, Radford A, et al. Generating long sequences with sparse transformers [J/OL]. arXiv preprint (2019-04-23) [2021-12-02].https://arxiv.org/abs/1904.10509
[21]	Kitaev N, Kaiser Ł, Levskaya A. Reformer: the efficient transformer [J/OL]. arXiv preprint (2020-02-18) [2021-12-02].https://arxiv.org/abs/2001.04451
[22]	Beltagy I, Peters M E, Cohan A. Longformer: the long-document transformer [J/OL]. arXiv preprint (2020-12-02) [2021-12-02].https://arxiv.org/abs/2004.05150
[23]	Zhou H Y, Zhang S H, Peng J Q, et al. Informer: beyond efficient transformer for long sequence time-series forecasting [J/OL]. arXiv preprint (2020-03-28) [2021-12-02].https://arxiv.org/abs/2012.07436
[24]	Wu H X, Xu J H, Wang J M, et al. Autoformer: decomposition transformers with auto-correlation for long-term series forecasting [J/OL]. arXiv preprint (2021-06-24) [2021-12-02].https://arxiv.org/abs/2106.13008
[25]	Botchkarev A. Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology [J/OL]. arXiv preprint (2018-09-09) [2021-12-02].https://arxiv.org/abs/1809.03006
[26]	Cui Y, Xie J D, Zheng K. Historical inertia: A neglected but powerful baseline for long sequence time-series forecasting // Proceedings of the 30th ACM International Conference on Information & Knowledge Management. Gold Coast, 2021: 2965
[27]	Moor B D, Gersem P D, Schutter B D, et al. DAISY: A database for identification of systems. J A, 1997, 38(3): 4
[28]	Dua D, Graff C. UCI Machine Learning Repository [DB/OL]. University of California, School of Information and Computer Science (2021-06-05)[2021-12-02].https://archive.ics.uci.edu/ml/index.php
[29]	Godahewa R, Bergmeir C, Webb G I, et al. Monash time series forecasting archive [J/OL]. arXiv preprint (2021-05-14) [2021-12-02].https://arxiv.org/abs/2105.06643
[30]	王洪江, 王小林, 張璽, 等. 超細全尾砂深錐動態絮凝濃密試驗. 工程科學學報, 2022, 44(2):163 Wang H J, Wang X L, Zhang X, et al. Deep cone dynamic flocculation thickening of ultrafine full tailings. Chin J Eng, 2022, 44(2): 163
[31]	Sun Q Q, Ge Z Q. Probabilistic sequential network for deep learning of complex process data and soft sensor application. IEEE Trans Ind Inform, 2019, 15(5): 2700 doi: 10.1109/TII.2018.2869899
[32]	Yuan X F, Li L, Wang Y L. Nonlinear dynamic soft sensor modeling with supervised long short-term memory network. IEEE Trans Ind Inform, 2020, 16(5): 3168 doi: 10.1109/TII.2019.2902129
[33]	姚志賓, 牛文靜, 張宇, 等. 巖爆數據庫管理系統開發及應用. 工程科學學報,http://doi.org/10.13374/j.issn2095-9389.2021.08.12.002 Yao Z B, Niu W J, Zhang Y, et al. Development and application of a rockburst database management system. Chin J Eng, http://doi.org/10.13374/j.issn2095-9389.2021.08.12.002
[34]	Nú?ez F, Langarica S, Díaz P, et al. Neural network-based model predictive control of a paste thickener over an industrial Internet platform. IEEE Trans Ind Inform, 2020, 16(4): 2859 doi: 10.1109/TII.2019.2953275
[35]	Zhang H, Tang Z H, Xie Y F, et al. Feature reconstruction-regression network: A light-weight deep neural network for performance monitoring in the froth flotation. IEEE Trans Ind Inform, 2021, 17(12): 8406 doi: 10.1109/TII.2020.3046278
[36]	Yuan Z L, Li X R, Wu D, et al. Continuous-time prediction of industrial paste thickener system with differential ODE-net. IEEE/CAA J Autom Sin, 2022, 9(4): 686 doi: 10.1109/JAS.2022.105464
[37]	Fan D Y, Sun H, Yao J, et al. Well production forecasting based on ARIMA-LSTM model considering manual operations. Energy, 2021, 220: 119708 doi: 10.1016/j.energy.2020.119708
[38]	Zhang X G, Lei Y Y, Chen H, et al. Multivariate time-series modeling for forecasting sintering temperature in rotary kilns using DCGNet. IEEE Trans Ind Inform, 2021, 17(7): 4635 doi: 10.1109/TII.2020.3022019
[39]	Li J F, Yang C J, Li Y X, et al. A context-aware enhanced GRU network with feature-temporal attention for prediction of silicon content in hot metal. IEEE Trans Ind Inform,https://ieeexplore.ieee.org/document/9537301
[40]	Du S, Wu M, Chen L F, et al. Operating mode recognition of iron ore sintering process based on the clustering of time series data. Control Eng Pract, 2020, 96: 104297 doi: 10.1016/j.conengprac.2020.104297
[41]	袁兆麟, 何潤姿, 姚超, 等. 基于強化學習的濃密機底流濃度在線控制算法. 自動化學報, 2021, 47(7):1558 Yuan Z L, He R Z, Yao C, et al. Online reinforcement learning control algorithm for concentration of thickener underflow. Acta Autom Sin, 2021, 47(7): 1558
[42]	Long J Y, Sun Z Z, Pardalos P M, et al. A robust dynamic scheduling approach based on release time series forecasting for the steelmaking-continuous casting production. Appl Soft Comput, 2020, 92: 106271 doi: 10.1016/j.asoc.2020.106271
[43]	Han Y M, Zhou R D, Geng Z Q, et al. Production prediction modeling of industrial processes based on Bi-LSTM // 2019 34rd Youth Academic Annual Conference of Chinese Association of Automation (YAC). Jinzhou, 2019: 285
[44]	Wang K, Gopaluni R B, Chen J, et al. Deep learning of complex batch process data and its application on quality prediction. IEEE Trans Ind Inform, 2020, 16(12): 7233 doi: 10.1109/TII.2018.2880968
[45]	Essien A, Giannetti C. A deep learning model for smart manufacturing using convolutional LSTM neural network autoencoders. IEEE Trans Ind Inform, 2020, 16(9): 6069 doi: 10.1109/TII.2020.2967556
[46]	Antonopoulos I, Robu V, Couraud B, et al. Data-driven modelling of energy demand response behaviour based on a large-scale residential trial. Energy AI, 2021, 4: 100071 doi: 10.1016/j.egyai.2021.100071
[47]	Rojat T, Puget R, Filliat D, et al. Explainable artificial intelligence (XAI) on TimeSeries data: A survey [J/OL]. arXiv preprint (2021-04-02) [2021-12-02].https://arxiv.org/abs/2104.00950
[48]	Demeester T. System identification with time-aware neural sequence models [J/OL]. arXiv preprint (2019-11-21) [2021-12-02].https://arxiv.org/abs/1911.09431
[49]	Fatehi A, Huang B. Kalman filtering approach to multi-rate information fusion in the presence of irregular sampling rate and variable measurement delay. J Process Control, 2017, 53: 15 doi: 10.1016/j.jprocont.2017.02.010
[50]	Martí L, Sanchez-Pi N, Molina J M, et al. Anomaly detection based on sensor data in petroleum industry applications. Sens (Basel Switz), 2015, 15(2): 2774 doi: 10.3390/s150202774
[51]	Malhotra P, Ramakrishnan A, Anand G, et al. LSTM-based encoder-decoder for multi-sensor anomaly detection [J/OL]. arXiv preprint (2016-07-11) [2021-12-02].https://arxiv.org/abs/1607.00148
[52]	Siddiqui S A, Mercier D, Munir M, et al. Tsviz: demystification of deep learning models for time-series analysis. IEEE Access, 2019, 7: 67027 doi: 10.1109/ACCESS.2019.2912823
[53]	Schockaert C, Leperlier R, Moawad A. Attention mechanism for multivariate time series recurrent model interpretability applied to the ironmaking industry [J/OL]. arXiv preprint (2020-07-15) [2021-12-02].https://arxiv.org/abs/2007.12617
[54]	Ribeiro M T, Singh S, Guestrin C. “Why should I trust You?”: Explaining the predictions of any classifier // Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Francisco, 2016: 1135
[55]	Lundberg S M, Lee S I. A unified approach to interpreting model predictions [J/OL]. arXiv preprint (2017-11-25) [2021-12-02].https://arxiv.org/abs/1705.07874
[56]	Serradilla O, Zugasti E, Cernuda C, et al. Interpreting remaining useful life estimations combining explainable artificial intelligence and domain knowledge in industrial machinery // 2020 IEEE International Conference on Fuzzy Systems. Glasgow, 2020: 1