刘俊峰;陈剑龙;王晓生;曾君;黄倩颖;

• 1:华南理工大学自动化科学与工程学院
• 2:华南理工大学电力学院

摘要(Abstract)：

面向多种可再生能源接入的微能源网,提出一种基于深度强化学习(deep reinforcement learning,DRL)的微能源网能量管理与优化方法。该方法使用深度Q网络(deepQ network,DQN)对预测负荷、风/光等可再生能源功率输出和分时电价等环境信息进行学习,通过习得的策略集对微能源网进行能量管理,是一种模型无关基于价值的智能算法。首先,基于能量总线模型,建立了微能源网研究框架及设备模型。在深入阐述强化学习的框架、Q学习算法和DQN算法的基础理论的基础上,分析了提升DQN性能的经验回放机制与冻结参数机制,并以经济性为目标完成了微能源网能量管理与优化。通过对比不同参数的DQN算法及Q学习算法在微能源网能量管理中的表现,仿真结果展示了继承策略集后算法性能的提升,验证了深度强化学习相比启发式算法在微能源网能量管理应用的可行性和优越性。

关键词(KeyWords)： 微能源网;能量管理;深度强化学习;Q学习;深度Q网络

Abstract:

Keywords:

基金项目(Foundation): 国家自然科学基金(61573155,51877085)~~

作者(Author): 刘俊峰;陈剑龙;王晓生;曾君;黄倩颖;

Email:

DOI: 10.13335/j.1000-3673.pst.2020.0144

参考文献(References)：

• [1]马腾飞，吴俊勇，郝亮亮．含冷热电三联供的微能源网能量流计算及综合仿真[J]．电力系统自动化，2016,40(23):22-27.Ma Tengfei,Wu Junyong,Hao Liangliang,et al.Energy flow calculation and integrated simulation of micro-energy grid with combined cooling,heating and power[J].Automation of Electric Power Systems,2016,40(23):22-27(in Chinese).
• [2]张涛，张福兴，张彦．面向能源互联网的能量管理系统研究[J]．电网技术，2016,40(1):146-155.Zhang Tao,Zhang Fuxing,Zhang Yan.Study on energy management system of energy internet[J].Power System Technology,2016,40(1):146-155(in Chinese).
• [3] Olivares D E,Canizares C A,Kazerani M.A centralized energy management system for isolated microgrids[J].IEEE Transactions on Smart Grid,2017,5(4):1864-1875.
• [4] Li Y,Yang Z,Li G,et al.Optimal scheduling of an isolated microgrid with battery storage considering load and renewable generation uncertainties[J].IEEE Transactions on Industrial Electronics,2019,66(2):1565-1575.
• [5] Li Y,Wang C,Li G,et al.Improving operational flexibility of integrated energy system with uncertain renewable generations considering thermal inertia of buildings[J].Energy Conversion and Management,2020(207):112526.
• [6] Askarzadeh A.A memory-based genetic algorithm for optimization of power generation in a microgrid[J].IEEE Transactions on Sustainable Energy,2017,PP(99):1-1.
• [7]吴定会，高聪，纪志成．混合粒子群算法在微电网经济优化运行的应用[J]．控制理论与应用，2018,35(4):457-467.Wu Dinghui, Gao Cong, Ji Zhicheng. Economic optimization operation of the microgrid using the hybrid particle swarm optimization algorithm[J].Control Theory&Applications,2018,35(4):457-467(in Chinese).
• [8]曾君，徐冬冬，刘俊峰，等．考虑负荷满意度的微电网运行多目标优化方法研究[J]．中国电机工程学报，2016,36(12):3325-3333.Zeng Jun,Xu Dongdong,Liu Junfeng,et al.Multi-objective optimal operation of microgrid considering dynamic loads[J].Proceedings of the CSEE,2016,36(12):3325-3333(in Chinese).
• [9]王鲁浩，李歧强，王桂荣，等．多可再生能源冷热电联供微网系统环境经济优化调度[J]．控制与决策，2016,31(5):913-918.Wang Luhao, Li Qiqiang, Wang Guirong, et al. Optimal environmental economic scheduling of combined cooling heating and power microgrid system with multi-renewable energy[J].Control and Decision,2016,31(5):913-918(in Chinese).
• [10]刘国静，韩学山，王尚，等．基于强化学习方法的风储合作决策[J]．电网技术，2016,40(9):2729-2736.Liu Guojing,Han Xueshan,Wang Shang,et al.Optimal decisionmaking in the cooperation of wind power and energy storage based on reinforcement learning algorithm[J].Power System Technology,2016,40(9):2729-2736(in Chinese).
• [11] Tang Y,He H,Wen J,et al.Power system stability control for a wind farm based on adaptive dynamic programming[J].IEEE Transactions on Smart Grid,2015,6(1):166-177.
• [12] Ruelens F,Claessens B,Vandael S,et al.Residential demand response of thermostatically controlled loads using batch Reinforcement Learning[J]. IEEE Transactions on Smart Grid, 2017, 8(5):2149-2159.
• [13] Foruzan E,Soh L K,Asgarpoor S.Reinforcement learning approach for optimal distributed energy management in a microgrid[J].IEEE Transactions on Power Systems,2018,33(5):5749-5758.
• [14] Mnih V,Kavukcuoglu K,Silver D,et al.Human-level control through deep reinforcement learning[J].Nature,2015,518(7540):529.
• [15] Silver D,Huang A,Maddison C J,et al.Mastering the game of Go with deep neural networks and tree search[J]. Nature, 2016,529(7587):484-489.
• [16]唐振韬，邵坤，赵冬斌，等．深度强化学习进展：从AlphaGo到AlphaGo Zero[J]．控制理论与应用，2017,34(12):1529-1546.Tang Zhentao,Shao Kun,Zhao Dongbin,et al.Recent progress of deep reinforcement learning:from Alphago to AlphaGo Zero[J].Control Theory&Applications,2017,34(12):1529-1546(in Chinese).
• [17]赵冬斌，邵坤，朱圆恒，等．深度强化学习综述：兼论计算机围棋的发展[J]．控制理论与应用，2016,33(6):701-717.Zhao Dongbin,Shao Kun,Zhu Yuanheng,et al.Review of deep reinforcement learning and discussions on the development of computer Go[J].Control Theory&Applications,2016,33(6):701-717(in Chinese).
• [18]张自东，邱才明，张东霞，等．基于深度强化学习的微电网复合储能协调控制方法[J]．电网技术，2019,43(6):1914-1921.Zhang Zidong,Qiu Caiming,Zhang Dongxia,et al.A coordinated control method for hybrid energy storage system in microgrid based on deep reinforcement learning[J].Power System Technology,2019,43(6):1914-1921(in Chinese).
• [19] Fran?oislavet V,Taralla D,Ernst D,et al.Deep reinforcement learning solutions for energy microgrids management[C]//European Workshop on Reinforcement Learning(EWRL 2016).Barcelona,Spain:Pompeu Fabra University,2016:1-7.
• [20] Haochen H,Yuchao Q,et al.Optimal energy management strategies for energy internet via deep reinforcement learning approach[J].Applied Energy,2019(239):598-609.
• [21] Watkins C J C H.Learning from delayed rewards[J].Robotics&Autonomous Systems,1989,15(4):233-235.
• [22] Jaakkola T,Jordan M I,Singh S P.On the convergence of stochastic iterative dynamic programming algorithms[J].Neural Computation,1993,6(6):1185-1201.
• [23] Pan S J,Yang Q.A survey on transfer learning[J].IEEE Transactions on Knowledge&Data Engineering,2010,22(10):1345-1359.