Deep Reinforcement Learning Based Transferable EMS for Hybrid Electric Trains

Majlesi Journal of Electrical Engineering

مقاله 18، دوره 17، شماره 3، آذر 2023، صفحه 165-179   

نوع مقاله: Reseach Article

شناسه دیجیتال (DOI): 10.30486/mjee.2023.1988714.1158

نویسندگان

Yogesh Wankhede^{* 1}؛ Sheetal Rana¹؛ Faruk Kazi²

¹Department of Electrical Engineering, Veermata Jijabai Technological Institute, Mumbai, India.

²Department of Electrical Engineering, Veermata Jijabai Technological Institute, Mumbai, India

چکیده

The hybrid electric train which operates without overhead wires or traditional power sources relies on hydrogen fuel cells and batteries for power. These fuel cell-based hybrid electric trains (FCHETs) are more efficient than those powered by diesel or electricity because they do not produce any tailpipe emissions making them an eco-friendly mode of transport. The target of this paper is to propose low-budget FCHETs that prioritize energy efficiency to reduce operating costs and minimize their impact on the environment. To this end, an energy management strategy [EMS] has been developed that optimizes the distribution of energy to reduce the amount of hydrogen required to power the train. The EMS achieves this by balancing battery charging and discharging. To enhance the performance of the EMS, proposes to use of a deep reinforcement learning (DRL) algorithm specifically the deep deterministic policy gradient (DDPG) combined with transfer learning (TL) which can improve the system's efficiency when driving cycles are changed. DRL-based strategies are commonly used in energy management and they suffer from unstable convergence, slow learning speed, and insufficient constraint capability. To address these limitations, an action masking technique to stop the DDPG-based approach from producing incorrect actions that go against the system's physical limits and prevent them from being generated is proposed.  The DDPG+TL agent consumes up to 3.9% less energy than conventional rule-based EMS while maintaining the battery's charge level within a predetermined range. The results show that DDPG+TL can sustain battery charge at minimal hydrogen consumption with minimal training time for the agent.

کلیدواژه‌ها

Fuel Cell؛ State of Charge؛ Energy Management Strategy؛ Deep Reinforcement Learning؛ Deep Deterministic Policy Gradient؛ Transfer Learning

مراجع

• Varan Navale , Timothy C. Havens, “Fuzzy logic controller for energy management of power split hybrid electrical vehicle transmission”, IEEE International Conference on Fuzzy Systems : 940-947, 2014.
• Ziaeinejad, Y. Sananse, A Mehrizi, “Fuel cell based auxiliary power unit: Ems. Sizing and current estimator based controller”, IEEE Trans. on Vehicular Tech 65: 4826-4835, 2016.
• Tashakori Abkenar, A. Nazari, S. D. G. Jayasinghe, A. Kapoor, and M. Negnevitsky, “Fuel cell power management using genetic expression programming in all-electric ships”, IEEE Transactions on Energy Conversion: 779-787, 2017.
• Rui Wang , Srdjan M. Lukic, “ Dynamic programming technique in hybrid electric vehicle optimization”, IEEE International Electric Vehicle Conference: 01-08, 2012.
• Ali Borhan , Ardalan Vahidi , Anthony M. Phillips, Ming L. Kuang , Ilya V. Kolmanovsky. “Predictive energy management of a power-split hybrid electric vehicle”, American Control Conference: 3970-3976, 2009.
• Hassan Khalil, “Nonlinear System analysis”, Pearson Publication, 2002
• Ettihir, L. Boulon, and K. Agbossou, “Energy management strategy for a fuel cell hybrid vehicle based on maximum efficiency and maximum power identification”, IET Electrical Systems in Transportation 6(4): 261-268, 2016.
• Mane, F. Kazi, and N. M. Singh, “Fuel cell and ultra-capacitor based hybrid energy control using ida-pbc methodology”, In International Conference on Industrial Instrumentation and Control (ICIC): 879-884, 2015.
• Henni-A. Abo M. Wack M.Ayad, M. Becherif , “Energy management of a fuel cell and supercapacitors by passivity-based control and sliding mode control”, Power journal 2(4): 1-7, 2011.
• Thounthong, S. Pierfederici, J. P. Martin, M. Hinaje, and B. Davat, “Modeling and control of fuel cell/supercapacitor hybrid source based on differential flatness control”, IEEE Transactions on Vehicular Technology 59(6): 2700-2710, 2010.
• J Snoussi, S. Ben Elghali, R. Outbib, M.F. Mimouni , “Sliding mode control for frequency-based energy management strategy of hybrid Storage System in vehicular application”, International Symposium on Power Electronics, Electrical Drives, Automation and Motion (SPEEDAM): 1109-1114, 2016.
• Chao-Ming Lee, Shin-Han Han, Chen-Hong Zheng, and We-Song Lin, “ Power split of fuel cell/ultracapacitor hybrid power system by backstepping sliding mode control”, In IPEC 2012 Conference on Power Energy: 538-543, 2012.
• Daming, Z., AI-Durra, A., Fei, G., Simoes, M.G, “Online energy management strategy of fuel cell hybrid electric vehicles based on  data fusion approach”, Journal of Power Sources 366 :278–29, 2017.
• Zheng, C., Xu, G., Xu, K., Pan, Z., Liang, Q , “An energy management approach of hybrid vehicles using traffic preview information for energy saving”, Energy Convers. Manag., (105) :  462–470, 2015.
• Liu, T., Zou Y., Liu D., Sun F, “Reinforcement learning-based energy management strategy for a hybrid electric tracked vehicle”, Journal of Energies (8): 7243–7260, 2015.
• Heeyun Lee, Changbeom Kang, Yeong Park, Namwook K, “Online data-driven energy management of a hybrid electric vehicle using model-based Q-Learning”, IEEE Access (8) : 84444-84454, 2020.
• Hu, Y., Li, W., Xu, H., Xu, G, “An online learning control strategy for hybrid electric vehicle based on fuzzy Q-learning”, Energies (8): 11167–1118, 2015.
• Zhongping Yang, Feiqin Zhu, Fei Lin, “ Deep-Reinforcement-Learning-Based Energy Management Strategy for Supercapacitor Energy Storage Systems in Urban Rail Transit”, 22,(2): 1150-1160 , 2021.
• Sehang  , Yonghua Z  , Hamido Fujit , “ Deep reinforcement learning with reference system to handle constraints for energy-efficient train control”, Information Sciences Elsevier, 570:  708-721, 2021.
• Hyunsoo Lee, Seok-Youn Han, Keejun Park, Hoyoung Lee and Taesoo Kwon , “ Real-Time Hybrid Deep Learning-Based Train Running Safety Prediction Framework of Railway Vehicle” , Machines MDPI:. 1-18, 2021.
• Kai Deng ,  Yingxu Liu,  Di Hai ,  Hujun Peng, “ Deep reinforcement learning based energy management strategy of fuel cell hybrid railway vehicles considering fuel cell aging”, Energy conversion and management (251): 1-8, 2022.
• Saadi , M. Becherif , D. Hissel , H.S. Ramadan, “ Dynamic modeling and experimental analysis of PEMFCs: A comparative study”, International Journal of Hydrogen Energy,  V 42(2): 1544-1557, 2017.
• Zekeriya Ender Eger, “Reinforcement learning based energy management strategy for fuel cell hybrid vehicles”, Sabanci University: 1- 56, 2022.
• Xiaowei Guo, Teng Liu, Bangbei Tang, Xiaolin Tang, Jinwei Zhang, Wenhao Tan, Shufeng Jin , “Transfer Deep Reinforcement Learning-enabled Energy Management Strategy for Hybrid Tracked Vehicle”, arXiv:2007.08690: 1-11, 2020.
• Yogesh E. Wankhede, Sheetal Rana, Faruk Kazi , “SoC Estimation of Battery in FCHEVs Using Reformulated Constrained Unscented Kalman Filter”, 1st International Conference on Sustainable Technology for Power and Energy Systems (STPES): 1-6, 2022.
• Yuecheng Li,  Hongwen He, “Deep Reinforcement Learning-based Energy Management for Hybrid Electric Vehicles”, Springer Cham : 1-123, 2022.