基于强化学习的车载边缘计算服务更新与资源分配联合优化算法*

doi:10.7523/j.ucas.2025.038

摘要/Abstract

摘要： 车载边缘计算是一种新兴的计算范式,它将移动边缘计算的计算能力与车辆网络的通信能力相结合,可以有效增强智能交通系统的用户体验。本文构建了一个面向城市公交系统的车载边缘计算系统架构,将服务资源和任务时延约束下的任务卸载率最大化问题建模为一个随机混合整数非线性规划问题。通过将强化学习与公交车辆的车载算力及其无线通信能力相结合,提出了一种基于在线最大后验策略优化的区段性服务更新与资源分配联合优化算法。大量仿真结果表明,与多种基线算法相比较,所提算法在提升任务卸载率方面具有显著优势。

关键词: 车载边缘计算, 服务部署, 资源分配, 强化学习

Abstract: Vehicle edge computing is a new computing paradigm, which combines the ability of mobile edge computing with vehicle network, and can effectively enhance the quality of user experience in intelligent transportation systems. In this paper, a vehicle edge computing system architecture based on urban public transport system is constructed, and the problem of maximizing the task offloading rate under service resource and task delay constraints is modeled as a stochastic mixed integer nonlinear programming problem. By combining reinforcement learning with buses' carried computing resources and also wireless communication capabilities, a joint segmental service update and resource allocation joint algorithm based on online maximum a posteriori strategy optimization is proposed. Extensive simulation results show that our proposed algorithm has significant advantages in improving the task offloading rate as compared with baseline algorithms.

Key words: vehicle edge computing, service deployment, resource allocation, reinforcement learning

中图分类号:

TP39

赵润, 姚郑, 张宝贤. 基于强化学习的车载边缘计算服务更新与资源分配联合优化算法^*[J]. 中国科学院大学学报, DOI: 10.7523/j.ucas.2025.038.

ZHAO Run, YAO Zheng, ZHANG Baoxian. Reinforcement learning based joint service update and resource allocation optimization algorithm for vehicular edge computing[J]. Journal of University of Chinese Academy of Sciences, DOI: 10.7523/j.ucas.2025.038.

参考文献

[1] Guan Y Y, Song Q Y, Qi W J, et al.Multidimensional resource fragmentation-aware virtual network embedding for IoT applications in MEC networks[J]. IEEE Internet of Things Journal, 2023, 10(24): 22223-22232. DOI: 10.1109/JIOT.2023.3304976.
[2] Wu M R, Song Q Y, Guo L, et al.Energy-efficient secure computation offloading in wireless powered mobile edge computing systems[J]. IEEE Transactions on Vehicular Technology, 2023, 72(5): 6907-6912. DOI: 10.1109/TVT.2023.3236327.
[3] Bitam S, Mellouk A, Zeadally S.Bio-inspired routing algorithms survey for vehicular ad hoc networks[J]. IEEE Communications Surveys & Tutorials, 2015, 17(2): 843-867. DOI: 10.1109/COMST.2014.2371828.
[4] Tabatabaee Malazi H, Chaudhry S R, Kazmi A, et al.Dynamic service placement in multi-access edge computing: A systematic literature review[J]. IEEE Access, 2022, 10: 32639-32688. DOI: 10.1109/ACCESS.2022.3160738.
[5] Sarkar I, Adhikari M, Kumar N, et al.Dynamic task placement for deadline-aware IoT applications in federated fog networks[J]. IEEE Internet of Things Journal, 2022, 9(2): 1469-1478. DOI: 10.1109/JIOT.2021.3088227.
[6] Sugawara S.Implementing a dynamic-static hybrid fog computing system for content sharing on local networks[C]// 2024 IEEE International Conference on Consumer Electronics (ICCE). January 6-8, 2024, Las Vegas, NV, USA. IEEE, 2024: 1-4. DOI: 10.1109/ICCE59016.2024.10444138.
[7] Azizi S, Farzin P, Shojafar M, et al.A scalable and flexible platform for service placement in multi-fog and multi-cloud environments[J]. The Journal of Supercomputing, 2024, 80(1): 1109-1136. DOI: 10.1007/s11227-023-05520-9.
[8] Lee G, Saad W, Bennis M.An online secretary framework for fog network formation with minimal latency[C]// 2017 IEEE International Conference on Communications (ICC) . May 21-25, 2017, Paris, France. IEEE, 2017: 1-6. DOI: 10.1109/ICC.2017.7996574.
[9] Yousefpour A, Patil A, Ishigaki G, et al. Qos-aware dynamic fog service provisioning [EB/OL].2018:1802.00800. https://arxiv.org/abs/1802.00800v2.
[10] Mahmud R, Ramamohanarao K, Buyya R.Latency-aware application module management for fog computing environments[J]. ACM Transactions on Internet Technology (TOIT), 2019, 19(1): 1-21. DOI: 10.1145/3186592.
[11] Hou Y J, Zhang K S, Chen Z B, et al.Joint server activation and network slice deployment in mobile edge computing networks[C]// 2024 IEEE 35th International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC). September 2-5, 2024, Valencia, Spain. IEEE, 2024: 1-7. DOI: 10.1109/PIMRC59610.2024.10817430.
[12] Masoumi M, de Miguel I, Brasca F G, et al. Leveraging load balance metrics to unravel the impact of multi-access edge computing locations on online dynamic network performance[J]. IEEE Open Journal of the Communications Society, 2024, 5: 5635-5651. DOI:10.1109/OJCOMS.2024.3453972.
[13] Hu M L, Wang H, Xu X H, et al.Joint optimization of microservice deployment and routing in edge via multi-objective deep reinforcement learning[J]. IEEE Transactions on Network and Service Management, 2024, 21(6): 6364-6381. DOI:10.1109/TNSM.2024.3443872.
[14] Liang J Y, Feng Z H, Gao H, et al.Deep reinforcement learning based reliability-aware resource placement and task offloading in edge computing[C]// 2024 IEEE International Conference on Web Services (ICWS). July 7-13, 2024, Shenzhen, China. IEEE, 2024: 686-695. DOI:10.1109/ICWS62655.2024.00088.
[15] Xue H, Xia Y.Profit-aware edge server placement based on all-pay auction for edge offloading[C]// 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS). June 19-21, 2024, Guangzhou, China. IEEE, 2024: 1-2. DOI:10.1109/IWQoS61813.2024.10682876.
[16] Chai H Y, Wang H D, Li T, et al.Generative AI-driven digital twin for mobile networks[J]. IEEE Network, 2024, 38(5): 84-92. DOI:10.1109/MNET.2024.3420702.
[17] Velasquez K, Abreu D P, Curado M, et al.Service placement for latency reduction in the Internet of Things[J]. Annals of Telecommunications, 2017, 72(1): 105-115. DOI:10.1007/s12243-016-0524-9.
[18] Gong Y D.Optimal edge server and service placement in mobile edge computing[C]// 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC). December 11-13, 2020, Chongqing, China. IEEE, 2020: 688-691. DOI: 10.1109/ITAIC49862.2020.9339180.
[19] Kim W S, Chung S H.User-participatory fog computing architecture and its management schemes for improving feasibility[J]. IEEE Access, 2018, 6: 20262-20278. DOI: 10.1109/ACCESS.2018.2815629
[20] Yu R Z, Xue G L, Zhang X.Application provisioning in FOG computing-enabled internet-of-things: A network perspective[C]// IEEE INFOCOM 2018-IEEE Conference on Computer Communications. April 16-19, 2018, Honolulu, HI, USA. IEEE, 2018: 783-791. DOI: 10.1109/INFOCOM.2018.8486269.
[21] Ouyang T, Zhi Z, Xu C.Follow me at the edge: Mobility-aware dynamic service placement for mobile edge computing[J]. IEEE Journal on Selected Areas in Communications, 2018, 36(10): 2333-2345. DOI: 10.1109/JSAC.2018.2869954.
[22] Yakubu A B, Abd El-Malek A H, Abo-Zahhad M, et al. Task Offloading and Resource Allocation in an RIS-assisted NOMA-based Vehicular Edge Computing[J]. IEEE Access, 2024, 12: 124330-124348. DOI: 10.1109/ACCESS.2024.3454810
[23] Ning Z L, Dong P R, Wang X J, et al.Distributed and dynamic service placement in pervasive edge computing networks[J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 32(6): 1277-1292. DOI: 10.1109/TPDS.2020.3046000.
[24] Farhadi V, Mehmeti F, He T, et al.Service placement and request scheduling for data-intensive applications in edge clouds[J]. IEEE/ACM Transactions on Networking, 2021, 29(2): 779-792. DOI:10.1109/TNET.2020.3048613.
[25] Cao T, Wang Q H, Zhang Y H, et al.Walking on two legs: Joint service placement and computation configuration for provisioning containerized services at edges[J]. Computer Networks, 2024, 239: 110144. DOI:10.1016/j.comnet.2023.110144.
[26] Tang Z H, Huang A W, Wang Y H, et al.Edge servers on wheels: deployment and route planning of mobile servers for Internet of vehicles[C]// 2023 19th International Conference on Mobility, Sensing and Networking (MSN). December 14-16, 2023, Nanjing, China. IEEE, 2023: 707-713. DOI: 10.1109/MSN60784.2023.00103.
[27] Natesha B V, Guddeti R M R. Adopting elitism-based Genetic Algorithm for minimizing multi-objective problems of IoT service placement in fog computing environment[J]. Journal of Network and Computer Applications, 2021, 178: 102972. DOI:10.1016/j.jnca.2020.102972.
[28] Sarrafzade N, Entezari-Maleki R, Sousa L.A genetic-based approach for service placement in fog computing[J]. The Journal of Supercomputing, 2022, 78(8): 10854-10875. DOI:10.1007/s11227-021-04254-w.
[29] Maia A M, Ghamri-Doudane Y, Vieira D, et al.Dynamic service placement and load distribution in edge computing[C]//2020 16th International Conference on Network and Service Management (CNSM). November 2-6, 2020, Izmir, Turkey. IEEE, 2020: 1-9. DOI: 10.23919/CNSM50824.2020.9269059.
[30] Ayoubi M, Ramezanpour M, Khorsand R.An autonomous IoT service placement methodology in fog computing[J]. Software: Practice and Experience, 2021, 51(5): 1097-1120. DOI:10.1002/spe.2939.
[31] Eyckerman R, Mercelis S, Marquez-Barja J, et al.Requirements for distributed task placement in the fog[J]. Internet of Things, 2020, 12: 100237. DOI: 10.1016/j.iot.2020.100237.
[32] Azizi S, Shojafar M, Farzin P, et al.DCSP: A delay and cost-aware service placement and load distribution algorithm for IoT-based fog networks[J]. Computer Communications, 2024, 215: 9-20. DOI:10.1016/j.comcom.2023.12.016.
[33] Kayal P, Liebeherr J.Distributed service placement in fog computing: An iterative combinatorial auction approach[C]// 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). July 7-10, 2019, Dallas, TX, USA. IEEE, 2019: 2145-2156. DOI: 10.1109/ICDCS.2019.00211.
[34] Aloqaily M, Kantarci B, Mouftah H T.Fairness-aware game theoretic approach for service management in vehicular clouds[C]// 2017 IEEE 86th Vehicular Technology Conference (VTC-Fall). September 24-27, 2017, Toronto, ON, Canada. IEEE, 2017: 1-5. DOI: 10.1109/VTCFall.2017.8288282.
[35] Lera I, Guerrero C, Juiz C.Availability-aware service placement policy in fog computing based on graph partitions[J]. IEEE Internet of Things Journal, 2019, 6(2): 3641-3651. DOI: 10.1109/JIOT.2018.2889511.
[36] Talpur A, Gurusamy M.DRLD-SP: A deep-reinforcement-learning-based dynamic service placement in edge-enabled Internet of vehicles[J]. IEEE Internet of Things Journal, 2022, 9(8): 6239-6251. DOI: 10.1109/JIOT.2021.3110913.
[37] Sharma A, Thangaraj V.Intelligent service placement algorithm based on DDQN and prioritized experience replay in IoT-Fog computing environment[J]. Internet of Things, 2024, 25: 101112. DOI:10.1016/j.iot.2024.101112.
[38] Ibn-Khedher H, Laroui M, Moungla H, et al.Next-generation edge computing assisted autonomous driving based artificial intelligence algorithms[J]. IEEE Access, 2022, 10: 53987-54001. DOI: 10.1109/ACCESS.2022.3174548.
[39] Liu T, Ni S G, Li X Q, et al.Deep reinforcement learning based approach for online service placement and computation resource allocation in edge computing[J]. IEEE Transactions on Mobile Computing, 2023, 22(7): 3870-3881. DOI: 10.1109/TMC.2022.3148254.
[40] Zare M, Elmi Sola Y, Hasanpour H.Towards distributed and autonomous IoT service placement in fog computing using asynchronous advantage actor-critic algorithm[J]. Journal of King Saud University-Computer and Information Sciences, 2023, 35(1): 368-381. DOI:10.1016/j.jksuci.2022.12.006.
[41] Song H F, Abdolmaleki A, Springenberg J T, et al. V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control[EB/OL].2019:1909.12238. https://arxiv.org/abs/1909.12238v1.
[42] Christodoulou P. Soft actor-critic for discrete action settings[EB/OL].2019:1910.07207. https://arxiv.org/abs/1910.07207v2.
[43] Mnih V, Kavukcuoglu K, Silver D, et al.Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533. DOI: 10.1038/nature14236.

基于强化学习的车载边缘计算服务更新与资源分配联合优化算法^*

Reinforcement learning based joint service update and resource allocation optimization algorithm for vehicular edge computing

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

访问统计

联系我们

[1]	盛家华, 洪佩琳, 王航. 在网计算中资源受限的流汇聚算法[J]. 中国科学院大学学报, 2025, 42(2): 248-259.
[2]	王雪帆, 李宗旺, 梁旭文. 基于DQN的VDES异构星座兼容策略[J]. 中国科学院大学学报, 2024, 41(4): 550-557.
[3]	王兆瑞, 岩延, 张宝贤. 基于时空依赖关系多智能体强化学习的多路口交通信号协同控制方法[J]. 中国科学院大学学报, 2024, 41(3): 398-410.
[4]	李雪源, 韩丛英. Actor-critic框架下的二次指派问题求解方法[J]. 中国科学院大学学报, 2024, 41(2): 275-284.
[5]	杨光开, 陈皓, 张茗奕, 尹奇跃, 黄凯奇. 基于不确定度的多智能体信用分配方法[J]. 中国科学院大学学报, 2024, 41(2): 231-240.
[6]	陈皓, 杨立昆, 尹奇跃, 黄凯奇. 面向Ad-Hoc协作的局部观测重建方法[J]. 中国科学院大学学报, 2024, 41(1): 117-126.
[7]	杨特, 洪佩琳, 李润洲. 蜂窝网络下的SWIPT-D2D通信资源分配[J]. 中国科学院大学学报, 2022, 39(6): 845-852.
[8]	张华明, 李强. 基于深度强化学习的低轨卫星下行功率分配方案[J]. 中国科学院大学学报, 2022, 39(4): 543-550.
[9]	靳晓晗, 岩延, 张宝贤. 基于深度强化学习方法的无线多跳网络能量高效机会路由[J]. 中国科学院大学学报, 2022, 39(3): 410-420.
[10]	王静, 陈岚, 张贺, 王海永. 基于EDA仿真软件的多资源调度算法[J]. 中国科学院大学学报, 2021, 38(5): 696-701.
[11]	李维谦, 邱玲. 支持D2D多播的蜂窝网络分簇策略与资源分配[J]. 中国科学院大学学报, 2019, 36(1): 137-143.
[12]	王鑫, 邱玲. H2H与M2M共存场景的准入控制及资源分配[J]. 中国科学院大学学报, 2016, 33(3): 427-432.
[13]	李艾芸, 邱玲. 异构网络中基于比例速率限制的网络选择和资源分配算法[J]. 中国科学院大学学报, 2014, 31(6): 821-826.
[14]	曹家燕, 邱玲. 一种基于有效容量的混合业务资源分配算法[J]. 中国科学院大学学报, 2014, 31(5): 685-690.
[15]	舒锦, 卫国. 多用户MIMO-OFDM系统中基于迫零波束成型的自适应资源分配算法[J]. 中国科学院大学学报, 2009, 26(3): 419-425.