Cooperative traffic signal control method for multi-intersection: an approach based on spatiotemporal dependence multi-agent reinforcement learning

doi:10.7523/j.ucas.2023.076

Abstract

Abstract: In the face of increasingly serious traffic congestion, intelligent traffic signal control has become an indispensable means to improve the performance of urban road network. In this paper, a spatiotemporal traffic light control (STLight) based on multi-agent reinforcement learning algorithm is proposed. Through the spatiotemporal dependent module (STDM) based on the attention mechanism, STLight can extract the initial traffic observation data as spatiotemporal features, so as to effectively capture the spatiotemporal dependence relationship between intersections. In addition, based on the extracted spatiotemporal characteristics, STLight further introduces global spatiotemporal information to each agent on the basis of the multi-agent reinforcement learning algorithm based on the centralized training decentralized execution framework, so as to further improve the cooperation ability among multi-agents. The experimental results show that STLight has significant advantages in improving the performance of urban road networks, and helps to alleviate the traffic congestion problem of current large-scale urban road networks.

Key words: multi-agent reinforcement learning, multi-intersection traffic signal control, attention mechanism, Markov game, spatiotemporal dependent

CLC Number:

TP39

WANG Zhaorui, YAN Yan, ZHANG Baoxian. Cooperative traffic signal control method for multi-intersection: an approach based on spatiotemporal dependence multi-agent reinforcement learning[J]. Journal of University of Chinese Academy of Sciences, 2024, 41(3): 398-410.

References

[1] Lowrie P. SCATS: sydney co-ordinated adaptive traffic system: a traffic responsive method of controlling urban traffic[R]. Transportation Research Board, 1990.
[2] Hunt P, Robertson D, Bretherton R, et al. SCOOT-a traffic responsive method of coordinating signals[R]. Transportation Research Board, 1981.
[3] Varaiya P. Max pressure control of a network of signalized intersections[J]. Transportation Research Part C: Emerging Technologies, 2013, 36: 177-195. DOI: 10.1016/j.trc.2013.08.014.
[4] Cools S B, Gershenson C, D’Hooghe B. Self-organizing traffic lights: a realistic simulation[M]//Advances in Applied Self-Organizing Systems. London: Springer, 2013: 45-55. DOI: 10.1007/978-1-4471-5113-5_3.
[5] Koonce P, Rodegerdts L. Traffic signal timing manual[R/OL]: United States. Federal Highway Administration, 2008. (2008-06-01) [2023-09-05]. https://rosap.ntl.bts.gov/view/dot/800.
[6] Robertson D. TRANSYT: a traffic network study tool[R]. Transportation Research Board, 1969.
[7] Sutton R S, Barto A G. Reinforcement learning: an introduction[M]. 2nd ed. Cambridge, Massachusetts: MIT press, 2018.
[8] Wei H A, Zheng G J, Yao H X, et al. IntelliLight: a reinforcement learning approach for intelligent traffic light control[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. London United Kingdom. New York, NY, USA: ACM, 2018: 2496-2505. DOI: 10.1145/3219819. 3220 096.
[9] Wei H A, Xu N, Zhang H C, et al. CoLight: learning network-level cooperation for traffic signal control[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management. Beijing China. New York, NY, USA: ACM, 2019: 1913-1922. DOI: 10.1145/3357384.3357902.
[10] Wei H A, Chen C C, Zheng G J, et al. PressLight: learning max pressure control to coordinate traffic signals in arterial network[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Anchorage AK USA. New York, NY, USA: ACM, 2019: 1290-1298. DOI: 10.1145/3292500.3330949.
[11] Zang X S, Yao H X, Zheng G J, et al. MetaLight: Value-based meta-reinforcement learning for traffic signal control[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(1): 1153-1160. DOI: 10.1609/aaai.v34i01.5467.
[12] Joo H, Ahmed S H, Lim Y. Traffic signal control for smart cities using reinforcement learning[J]. Computer Communications, 2020, 154: 324-330. DOI: 10.1016/j.comcom.2020.03.005.
[13] Ge H W, Gao D W, Sun L, et al. Multi-agent transfer reinforcement learning with multi-view encoder for adaptive traffic signal control[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8): 12572-12587. DOI: 10.1109/TITS.2021.3115240.
[14] Timur M I A, Dharmawan A, Istiyanto J E, et al. A3C and A2C performance comparison in intelligent traffic signal controller using Indonesian traffic rules simulation[C]//AIP Conference Proceedings. AIP Publishing LLC, 2023: 020014. DOI: 10.1063/5.0119098.
[15] Yang H, Zhao H, Wang Y, et al. Deep reinforcement learning based strategy for optimizing phase splits in traffic signal control[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 2329-2334. DOI: 10.1109/ITSC55140.2022.9922531.
[16] Yang J C, Zhang J P, Wang H H. Urban traffic control in software defined internet of things via a multi-agent deep reinforcement learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(6): 3742-3754. DOI: 10.1109/TITS.2020.3023788.
[17] Huang L B, Qu X H. Improving traffic signal control operations using proximal policy optimization[J]. IET Intelligent Transport Systems, 2023, 17(3): 592-605. DOI: 10.1049/itr2.12286.
[18] Chin Y K, Kow W Y, Khong W L, et al. Q-learning traffic signal optimization within multiple intersections traffic network[C]//2012 Sixth UKSim/AMSS European Symposium on Computer Modeling and Simulation. IEEE, 2012: 343-348. DOI: 10.1109/EMS.2012.75.
[19] Li S E. Deep reinforcement learning[M]//Reinforcement Learning for Sequential Decision and Optimal Control. Singapore: Springer, 2023: 365-402. DOI: 10.1007/978-981-19-7784-8_10.
[20] Prabuchandran K J, Hemanth Kumar A N, Bhatnagar S. Multi-agent reinforcement learning for traffic signal control[C]//17th International IEEE Conference on Intelligent Transportation Systems (ITSC). October 8-11, 2014, Qingdao, China. IEEE, 2014: 2529-2534. DOI: 10.1109/ITSC.2014.6958095.
[21] Lyu X G, Xiao Y C, Daley B, et al. Contrasting centralized and decentralized critics in multi-agent reinforcement learning[EB/OL]. 2021: arXiv: 2102.04402. (2021-02-08) [2023-09-06]. https://arxiv.org/abs/2102.04402.
[22] Li X S, Li J C, Shi H B. A multi-agent reinforcement learning method with curriculum transfer for large-scale dynamic traffic signal control[J]. Applied Intelligence, 2023: 1-15. DOI: 10.1007/s10489-023-04652-y.
[23] Bokade R, Jin X N, Amato C. Multi-agent reinforcement learning based on representational communication for large-scale traffic signal control[J]. IEEE Access, 2023, 11: 47646-47658. DOI: 10.1109/ACCESS.2023.3275883.
[24] Yan L P, Zhu L L, Song K, et al. Graph cooperation deep reinforcement learning for ecological urban traffic signal control[J]. Applied Intelligence, 2023, 53(6): 6248-6265. DOI: 10.1007/s10489-022-03208-w.
[25] Nishi T, Otaki K, Hayakawa K, et al. Traffic signal control based on reinforcement learning with graph convolutional neural nets[C]//2018 21st International Conference on Intelligent Transportation Systems (ITSC). November 4-7, 2018, Maui, HI, USA. IEEE, 2018: 877-883. DOI: 10.1109/ITSC.2018.8569301.
[26] Yau K L A, Qadir J, Khoo H L, et al. A survey on reinforcement learning models and algorithms for traffic signal control[J]. ACM Computing Surveys， 2018, 50(3): 1-38. DOI: 10.1145/3068287.
[27] Zhang Z, Yang J C, Zha H. Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization[C]//Proceedings of the 19th Inter-national Conference on Autonomous Agents and MultiAgent Systems. 2020: 2083-2085. DOI: 10.5555/3398761.339 9082.
[28] Li L, Lv Y S, Wang F Y. Traffic signal timing via deep reinforcement learning[J]. IEEE/CAA Journal of Automatica Sinica, 2016, 3(3): 247-254. DOI: 10.1109/JAS.2016.7508798.
[29] Genders W, Razavi S. Using a deep reinforcement learning agent for traffic signal control[EB/OL]. 2016: arXiv: 1611.01142. (2016-11-03) [2023-09-06]. https://arxiv.org/abs/1611.01142.
[30] Aslani M, Mesgari M S, Wiering M. Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events[J]. Transportation Research Part C: Emerging Technologies, 2017, 85: 732-752. DOI: 10.1016/j.trc.2017.09.020.
[31] Chen X Y, Xiong G, Lv Y S, et al. A collaborative communication-qmix approach for large-scale networked traffic signal control[C]//2021 IEEE International Intelligent Transportation Systems Conference (ITSC). September 19-22, 2021, Indianapolis, IN, USA. IEEE, 2021: 3450-3455. DOI: 10.1109/ITSC48978.2021.9564683.
[32] Rashid T, Samvelyan M, de Witt C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[EB/OL].2020, arXiv: 2003.08839. (2020-03-19) [2023-09-06]. https://doi.org/10.48550/arXiv.2003.08839.
[33] Wang Y N, Xu T, Niu X, et al. STMARL: a spatio-temporal multi-agent reinforcement learning approach for cooperative traffic light control[J]. IEEE Transactions on Mobile Computing, 2022, 21(6): 2228-2242. DOI: 10.1109/TMC.2020.3033782.
[34] Wu L B, Wang M, Wu D, et al. DynSTGAT: dynamic spatial-temporal graph attention network for traffic signal control[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management. Virtual Event Queensland Australia. New York, NY, USA: ACM, 2021: 2150-2159. DOI: 10.1145/3459637.3482254.
[35] Zhao C, Dai X Y, Wang X, et al. Learning transformer-based cooperation for networked traffic signal control[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 3133-3138. DOI: 10.1109/ITSC55140.2022.9921995.
[36] Kong A Y, Lu B X, Yang C Z, et al. A deep reinforcement learning framework with memory network to coordinate traffic signal control[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 3825-3830. DOI: 10.1109/ITSC55140.2022.9921752.
[37] Littman M L. Markov games as a framework for multi-agent reinforcement learning[M]//Machine learning Proceedings 1994. Amsterdam: Elsevier, 1994: 157-163. DOI: 10.1016/b978-1-55860-335-6.50027-1.
[38] Wang X Q, Ke L J, Qiao Z M, et al. Large-scale traffic signal control using a novel multiagent reinforcement learning[J]. IEEE Transactions on Cybernetics, 2021, 51(1): 174-187. DOI: 10.1109/TCYB.2020.3015811.
[39] Mao F, Li Z H, Li L. A comparison of deep reinforcement learning models for isolated traffic signal control[J]. IEEE Intelligent Transportation Systems Magazine, 2023, 15(1): 160-180. DOI: 10.1109/MITS.2022.3144797.
[40] Lowe R, Wu Y, Tamar A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[J/OL]. Advances in Neural Information Processing Systems, 2017: 6379-6390. (2017-12-04) [2023-09-05]. https://proceedings.neurips.cc/paper/2017/hash/68a9750337a418a86fe06c1991a1d64c-Abstract.html.
[41] Behrisch M, Bieker L, Erdmann J, et al. SUMO-simulation of urban mobility: an overview[C/OL]//Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation. ThinkMind, 2011. (2011-10-23) [2023-09-05]. https://elib.dlr.de/71460.
[42] Yi C L, Wu J, Ren Y Y, et al. A spatial-temporal deep reinforcement learning model for large-scale centralized traffic signal control[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 275-280. DOI: 10.1109/ITSC55140.2022.9922459.
[43] Zeng J, Xin J, Cong Y, et al. HALight: hierarchical deep reinforcement learning for cooperative arterial traffic signal control with cycle strategy[C]//2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022: 479-485. DOI: 10.1109/ITSC55140.2022. 9921819.