基于时空依赖关系多智能体强化学习的多路口交通信号协同控制方法

doi:10.7523/j.ucas.2023.076

摘要/Abstract

摘要： 面对日益严重的交通拥堵现象，智能交通信号控制已成为提升城市道路网络性能必不可少的手段。提出一种基于时空依赖关系多智能体强化学习算法的多路口交通信号控制方法STLight(spatiotemporal traffic light control)。通过基于注意力机制的时空依赖模块STDM(spatiotemporal dependent module)，STLight可将初始交通观测数据提取为时空特征，以有效捕获各交叉路口间的时空依赖关系。此外，基于所提取的时空特征，STLight在基于集中训练分散执行框架的多智能体强化学习算法基础之上进一步为各个智能体引入全局时空信息，从而进一步提升多智能体之间的协作能力。实验结果表明，STLight在提升城市道路网络的性能方面具有显著的优势，有助于缓解当前大规模城市道路网络的交通拥堵问题。

关键词: 多智能体强化学习, 多路口交通信号控制, 注意力机制, 马尔可夫博弈, 时空依赖

Abstract: In the face of increasingly serious traffic congestion, intelligent traffic signal control has become an indispensable means to improve the performance of urban road network. In this paper, a spatiotemporal traffic light control (STLight) based on multi-agent reinforcement learning algorithm is proposed. Through the spatiotemporal dependent module (STDM) based on the attention mechanism, STLight can extract the initial traffic observation data as spatiotemporal features, so as to effectively capture the spatiotemporal dependence relationship between intersections. In addition, based on the extracted spatiotemporal characteristics, STLight further introduces global spatiotemporal information to each agent on the basis of the multi-agent reinforcement learning algorithm based on the centralized training decentralized execution framework, so as to further improve the cooperation ability among multi-agents. The experimental results show that STLight has significant advantages in improving the performance of urban road networks, and helps to alleviate the traffic congestion problem of current large-scale urban road networks.

Key words: multi-agent reinforcement learning, multi-intersection traffic signal control, attention mechanism, Markov game, spatiotemporal dependent

中图分类号:

TP39

王兆瑞, 岩延, 张宝贤. 基于时空依赖关系多智能体强化学习的多路口交通信号协同控制方法[J]. 中国科学院大学学报, 2024, 41(3): 398-410.

WANG Zhaorui, YAN Yan, ZHANG Baoxian. Cooperative traffic signal control method for multi-intersection: an approach based on spatiotemporal dependence multi-agent reinforcement learning[J]. Journal of University of Chinese Academy of Sciences, 2024, 41(3): 398-410.

参考文献

[1] Lowrie P. SCATS: sydney co-ordinated adaptive traffic system: a traffic responsive method of controlling urban traffic[R]. Transportation Research Board, 1990.
[2] Hunt P, Robertson D, Bretherton R, et al. SCOOT-a traffic responsive method of coordinating signals[R]. Transportation Research Board, 1981.
[3] Varaiya P. Max pressure control of a network of signalized intersections[J]. Transportation Research Part C: Emerging Technologies, 2013, 36: 177-195. DOI: 10.1016/j.trc.2013.08.014.
[4] Cools S B, Gershenson C, D’Hooghe B. Self-organizing traffic lights: a realistic simulation[M]//Advances in Applied Self-Organizing Systems. London: Springer, 2013: 45-55. DOI: 10.1007/978-1-4471-5113-5_3.
[5] Koonce P, Rodegerdts L. Traffic signal timing manual[R/OL]: United States. Federal Highway Administration, 2008. (2008-06-01) [2023-09-05]. https://rosap.ntl.bts.gov/view/dot/800.
[6] Robertson D. TRANSYT: a traffic network study tool[R]. Transportation Research Board, 1969.
[7] Sutton R S, Barto A G. Reinforcement learning: an introduction[M]. 2nd ed. Cambridge, Massachusetts: MIT press, 2018.
[8] Wei H A, Zheng G J, Yao H X, et al. IntelliLight: a reinforcement learning approach for intelligent traffic light control[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. London United Kingdom. New York, NY, USA: ACM, 2018: 2496-2505. DOI: 10.1145/3219819. 3220 096.
[9] Wei H A, Xu N, Zhang H C, et al. CoLight: learning network-level cooperation for traffic signal control[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management. Beijing China. New York, NY, USA: ACM, 2019: 1913-1922. DOI: 10.1145/3357384.3357902.
[10] Wei H A, Chen C C, Zheng G J, et al. PressLight: learning max pressure control to coordinate traffic signals in arterial network[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Anchorage AK USA. New York, NY, USA: ACM, 2019: 1290-1298. DOI: 10.1145/3292500.3330949.
[11] Zang X S, Yao H X, Zheng G J, et al. MetaLight: Value-based meta-reinforcement learning for traffic signal control[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(1): 1153-1160. DOI: 10.1609/aaai.v34i01.5467.
[12] Joo H, Ahmed S H, Lim Y. Traffic signal control for smart cities using reinforcement learning[J]. Computer Communications, 2020, 154: 324-330. DOI: 10.1016/j.comcom.2020.03.005.
[13] Ge H W, Gao D W, Sun L, et al. Multi-agent transfer reinforcement learning with multi-view encoder for adaptive traffic signal control[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8): 12572-12587. DOI: 10.1109/TITS.2021.3115240.
[14] Timur M I A, Dharmawan A, Istiyanto J E, et al. A3C and A2C performance comparison in intelligent traffic signal controller using Indonesian traffic rules simulation[C]//AIP Conference Proceedings. AIP Publishing LLC, 2023: 020014. DOI: 10.1063/5.0119098.
[15] Yang H, Zhao H, Wang Y, et al. Deep reinforcement learning based strategy for optimizing phase splits in traffic signal control[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 2329-2334. DOI: 10.1109/ITSC55140.2022.9922531.
[16] Yang J C, Zhang J P, Wang H H. Urban traffic control in software defined internet of things via a multi-agent deep reinforcement learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(6): 3742-3754. DOI: 10.1109/TITS.2020.3023788.
[17] Huang L B, Qu X H. Improving traffic signal control operations using proximal policy optimization[J]. IET Intelligent Transport Systems, 2023, 17(3): 592-605. DOI: 10.1049/itr2.12286.
[18] Chin Y K, Kow W Y, Khong W L, et al. Q-learning traffic signal optimization within multiple intersections traffic network[C]//2012 Sixth UKSim/AMSS European Symposium on Computer Modeling and Simulation. IEEE, 2012: 343-348. DOI: 10.1109/EMS.2012.75.
[19] Li S E. Deep reinforcement learning[M]//Reinforcement Learning for Sequential Decision and Optimal Control. Singapore: Springer, 2023: 365-402. DOI: 10.1007/978-981-19-7784-8_10.
[20] Prabuchandran K J, Hemanth Kumar A N, Bhatnagar S. Multi-agent reinforcement learning for traffic signal control[C]//17th International IEEE Conference on Intelligent Transportation Systems (ITSC). October 8-11, 2014, Qingdao, China. IEEE, 2014: 2529-2534. DOI: 10.1109/ITSC.2014.6958095.
[21] Lyu X G, Xiao Y C, Daley B, et al. Contrasting centralized and decentralized critics in multi-agent reinforcement learning[EB/OL]. 2021: arXiv: 2102.04402. (2021-02-08) [2023-09-06]. https://arxiv.org/abs/2102.04402.
[22] Li X S, Li J C, Shi H B. A multi-agent reinforcement learning method with curriculum transfer for large-scale dynamic traffic signal control[J]. Applied Intelligence, 2023: 1-15. DOI: 10.1007/s10489-023-04652-y.
[23] Bokade R, Jin X N, Amato C. Multi-agent reinforcement learning based on representational communication for large-scale traffic signal control[J]. IEEE Access, 2023, 11: 47646-47658. DOI: 10.1109/ACCESS.2023.3275883.
[24] Yan L P, Zhu L L, Song K, et al. Graph cooperation deep reinforcement learning for ecological urban traffic signal control[J]. Applied Intelligence, 2023, 53(6): 6248-6265. DOI: 10.1007/s10489-022-03208-w.
[25] Nishi T, Otaki K, Hayakawa K, et al. Traffic signal control based on reinforcement learning with graph convolutional neural nets[C]//2018 21st International Conference on Intelligent Transportation Systems (ITSC). November 4-7, 2018, Maui, HI, USA. IEEE, 2018: 877-883. DOI: 10.1109/ITSC.2018.8569301.
[26] Yau K L A, Qadir J, Khoo H L, et al. A survey on reinforcement learning models and algorithms for traffic signal control[J]. ACM Computing Surveys， 2018, 50(3): 1-38. DOI: 10.1145/3068287.
[27] Zhang Z, Yang J C, Zha H. Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization[C]//Proceedings of the 19th Inter-national Conference on Autonomous Agents and MultiAgent Systems. 2020: 2083-2085. DOI: 10.5555/3398761.339 9082.
[28] Li L, Lv Y S, Wang F Y. Traffic signal timing via deep reinforcement learning[J]. IEEE/CAA Journal of Automatica Sinica, 2016, 3(3): 247-254. DOI: 10.1109/JAS.2016.7508798.
[29] Genders W, Razavi S. Using a deep reinforcement learning agent for traffic signal control[EB/OL]. 2016: arXiv: 1611.01142. (2016-11-03) [2023-09-06]. https://arxiv.org/abs/1611.01142.
[30] Aslani M, Mesgari M S, Wiering M. Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events[J]. Transportation Research Part C: Emerging Technologies, 2017, 85: 732-752. DOI: 10.1016/j.trc.2017.09.020.
[31] Chen X Y, Xiong G, Lv Y S, et al. A collaborative communication-qmix approach for large-scale networked traffic signal control[C]//2021 IEEE International Intelligent Transportation Systems Conference (ITSC). September 19-22, 2021, Indianapolis, IN, USA. IEEE, 2021: 3450-3455. DOI: 10.1109/ITSC48978.2021.9564683.
[32] Rashid T, Samvelyan M, de Witt C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[EB/OL].2020, arXiv: 2003.08839. (2020-03-19) [2023-09-06]. https://doi.org/10.48550/arXiv.2003.08839.
[33] Wang Y N, Xu T, Niu X, et al. STMARL: a spatio-temporal multi-agent reinforcement learning approach for cooperative traffic light control[J]. IEEE Transactions on Mobile Computing, 2022, 21(6): 2228-2242. DOI: 10.1109/TMC.2020.3033782.
[34] Wu L B, Wang M, Wu D, et al. DynSTGAT: dynamic spatial-temporal graph attention network for traffic signal control[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management. Virtual Event Queensland Australia. New York, NY, USA: ACM, 2021: 2150-2159. DOI: 10.1145/3459637.3482254.
[35] Zhao C, Dai X Y, Wang X, et al. Learning transformer-based cooperation for networked traffic signal control[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 3133-3138. DOI: 10.1109/ITSC55140.2022.9921995.
[36] Kong A Y, Lu B X, Yang C Z, et al. A deep reinforcement learning framework with memory network to coordinate traffic signal control[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 3825-3830. DOI: 10.1109/ITSC55140.2022.9921752.
[37] Littman M L. Markov games as a framework for multi-agent reinforcement learning[M]//Machine learning Proceedings 1994. Amsterdam: Elsevier, 1994: 157-163. DOI: 10.1016/b978-1-55860-335-6.50027-1.
[38] Wang X Q, Ke L J, Qiao Z M, et al. Large-scale traffic signal control using a novel multiagent reinforcement learning[J]. IEEE Transactions on Cybernetics, 2021, 51(1): 174-187. DOI: 10.1109/TCYB.2020.3015811.
[39] Mao F, Li Z H, Li L. A comparison of deep reinforcement learning models for isolated traffic signal control[J]. IEEE Intelligent Transportation Systems Magazine, 2023, 15(1): 160-180. DOI: 10.1109/MITS.2022.3144797.
[40] Lowe R, Wu Y, Tamar A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[J/OL]. Advances in Neural Information Processing Systems, 2017: 6379-6390. (2017-12-04) [2023-09-05]. https://proceedings.neurips.cc/paper/2017/hash/68a9750337a418a86fe06c1991a1d64c-Abstract.html.
[41] Behrisch M, Bieker L, Erdmann J, et al. SUMO-simulation of urban mobility: an overview[C/OL]//Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation. ThinkMind, 2011. (2011-10-23) [2023-09-05]. https://elib.dlr.de/71460.
[42] Yi C L, Wu J, Ren Y Y, et al. A spatial-temporal deep reinforcement learning model for large-scale centralized traffic signal control[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 275-280. DOI: 10.1109/ITSC55140.2022.9922459.
[43] Zeng J, Xin J, Cong Y, et al. HALight: hierarchical deep reinforcement learning for cooperative arterial traffic signal control with cycle strategy[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 479-485. DOI: 10.1109/ITSC55140.2022. 9921819.