Reinforcement learning based joint service update and resource allocation optimization algorithm for vehicular edge computing

doi:10.7523/j.ucas.2025.038

Abstract

Abstract: Vehicle edge computing is a new computing paradigm, which combines the ability of mobile edge computing with vehicle network, and can effectively enhance the quality of user experience in intelligent transportation systems. In this paper, a vehicle edge computing system architecture based on urban public transport system is constructed, and the problem of maximizing the task offloading rate under service resource and task delay constraints is modeled as a stochastic mixed integer nonlinear programming problem. By combining reinforcement learning with buses' carried computing resources and also wireless communication capabilities, a joint segmental service update and resource allocation joint algorithm based on online maximum a posteriori strategy optimization is proposed. Extensive simulation results show that our proposed algorithm has significant advantages in improving the task offloading rate as compared with baseline algorithms.

Key words: vehicle edge computing, service deployment, resource allocation, reinforcement learning

CLC Number:

TP39

ZHAO Run, YAO Zheng, ZHANG Baoxian. Reinforcement learning based joint service update and resource allocation optimization algorithm for vehicular edge computing[J]. Journal of University of Chinese Academy of Sciences, DOI: 10.7523/j.ucas.2025.038.

References

[1] Guan Y Y, Song Q Y, Qi W J, et al.Multidimensional resource fragmentation-aware virtual network embedding for IoT applications in MEC networks[J]. IEEE Internet of Things Journal, 2023, 10(24): 22223-22232. DOI: 10.1109/JIOT.2023.3304976.
[2] Wu M R, Song Q Y, Guo L, et al.Energy-efficient secure computation offloading in wireless powered mobile edge computing systems[J]. IEEE Transactions on Vehicular Technology, 2023, 72(5): 6907-6912. DOI: 10.1109/TVT.2023.3236327.
[3] Bitam S, Mellouk A, Zeadally S.Bio-inspired routing algorithms survey for vehicular ad hoc networks[J]. IEEE Communications Surveys & Tutorials, 2015, 17(2): 843-867. DOI: 10.1109/COMST.2014.2371828.
[4] Tabatabaee Malazi H, Chaudhry S R, Kazmi A, et al.Dynamic service placement in multi-access edge computing: A systematic literature review[J]. IEEE Access, 2022, 10: 32639-32688. DOI: 10.1109/ACCESS.2022.3160738.
[5] Sarkar I, Adhikari M, Kumar N, et al.Dynamic task placement for deadline-aware IoT applications in federated fog networks[J]. IEEE Internet of Things Journal, 2022, 9(2): 1469-1478. DOI: 10.1109/JIOT.2021.3088227.
[6] Sugawara S.Implementing a dynamic-static hybrid fog computing system for content sharing on local networks[C]// 2024 IEEE International Conference on Consumer Electronics (ICCE). January 6-8, 2024, Las Vegas, NV, USA. IEEE, 2024: 1-4. DOI: 10.1109/ICCE59016.2024.10444138.
[7] Azizi S, Farzin P, Shojafar M, et al.A scalable and flexible platform for service placement in multi-fog and multi-cloud environments[J]. The Journal of Supercomputing, 2024, 80(1): 1109-1136. DOI: 10.1007/s11227-023-05520-9.
[8] Lee G, Saad W, Bennis M.An online secretary framework for fog network formation with minimal latency[C]// 2017 IEEE International Conference on Communications (ICC) . May 21-25, 2017, Paris, France. IEEE, 2017: 1-6. DOI: 10.1109/ICC.2017.7996574.
[9] Yousefpour A, Patil A, Ishigaki G, et al. Qos-aware dynamic fog service provisioning [EB/OL].2018:1802.00800. https://arxiv.org/abs/1802.00800v2.
[10] Mahmud R, Ramamohanarao K, Buyya R.Latency-aware application module management for fog computing environments[J]. ACM Transactions on Internet Technology (TOIT), 2019, 19(1): 1-21. DOI: 10.1145/3186592.
[11] Hou Y J, Zhang K S, Chen Z B, et al.Joint server activation and network slice deployment in mobile edge computing networks[C]// 2024 IEEE 35th International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC). September 2-5, 2024, Valencia, Spain. IEEE, 2024: 1-7. DOI: 10.1109/PIMRC59610.2024.10817430.
[12] Masoumi M, de Miguel I, Brasca F G, et al. Leveraging load balance metrics to unravel the impact of multi-access edge computing locations on online dynamic network performance[J]. IEEE Open Journal of the Communications Society, 2024, 5: 5635-5651. DOI:10.1109/OJCOMS.2024.3453972.
[13] Hu M L, Wang H, Xu X H, et al.Joint optimization of microservice deployment and routing in edge via multi-objective deep reinforcement learning[J]. IEEE Transactions on Network and Service Management, 2024, 21(6): 6364-6381. DOI:10.1109/TNSM.2024.3443872.
[14] Liang J Y, Feng Z H, Gao H, et al.Deep reinforcement learning based reliability-aware resource placement and task offloading in edge computing[C]// 2024 IEEE International Conference on Web Services (ICWS). July 7-13, 2024, Shenzhen, China. IEEE, 2024: 686-695. DOI:10.1109/ICWS62655.2024.00088.
[15] Xue H, Xia Y.Profit-aware edge server placement based on all-pay auction for edge offloading[C]// 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS). June 19-21, 2024, Guangzhou, China. IEEE, 2024: 1-2. DOI:10.1109/IWQoS61813.2024.10682876.
[16] Chai H Y, Wang H D, Li T, et al.Generative AI-driven digital twin for mobile networks[J]. IEEE Network, 2024, 38(5): 84-92. DOI:10.1109/MNET.2024.3420702.
[17] Velasquez K, Abreu D P, Curado M, et al.Service placement for latency reduction in the Internet of Things[J]. Annals of Telecommunications, 2017, 72(1): 105-115. DOI:10.1007/s12243-016-0524-9.
[18] Gong Y D.Optimal edge server and service placement in mobile edge computing[C]// 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC). December 11-13, 2020, Chongqing, China. IEEE, 2020: 688-691. DOI: 10.1109/ITAIC49862.2020.9339180.
[19] Kim W S, Chung S H.User-participatory fog computing architecture and its management schemes for improving feasibility[J]. IEEE Access, 2018, 6: 20262-20278. DOI: 10.1109/ACCESS.2018.2815629
[20] Yu R Z, Xue G L, Zhang X.Application provisioning in FOG computing-enabled internet-of-things: A network perspective[C]// IEEE INFOCOM 2018-IEEE Conference on Computer Communications. April 16-19, 2018, Honolulu, HI, USA. IEEE, 2018: 783-791. DOI: 10.1109/INFOCOM.2018.8486269.
[21] Ouyang T, Zhi Z, Xu C.Follow me at the edge: Mobility-aware dynamic service placement for mobile edge computing[J]. IEEE Journal on Selected Areas in Communications, 2018, 36(10): 2333-2345. DOI: 10.1109/JSAC.2018.2869954.
[22] Yakubu A B, Abd El-Malek A H, Abo-Zahhad M, et al. Task Offloading and Resource Allocation in an RIS-assisted NOMA-based Vehicular Edge Computing[J]. IEEE Access, 2024, 12: 124330-124348. DOI: 10.1109/ACCESS.2024.3454810
[23] Ning Z L, Dong P R, Wang X J, et al.Distributed and dynamic service placement in pervasive edge computing networks[J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 32(6): 1277-1292. DOI: 10.1109/TPDS.2020.3046000.
[24] Farhadi V, Mehmeti F, He T, et al.Service placement and request scheduling for data-intensive applications in edge clouds[J]. IEEE/ACM Transactions on Networking, 2021, 29(2): 779-792. DOI:10.1109/TNET.2020.3048613.
[25] Cao T, Wang Q H, Zhang Y H, et al.Walking on two legs: Joint service placement and computation configuration for provisioning containerized services at edges[J]. Computer Networks, 2024, 239: 110144. DOI:10.1016/j.comnet.2023.110144.
[26] Tang Z H, Huang A W, Wang Y H, et al.Edge servers on wheels: deployment and route planning of mobile servers for Internet of vehicles[C]// 2023 19th International Conference on Mobility, Sensing and Networking (MSN). December 14-16, 2023, Nanjing, China. IEEE, 2023: 707-713. DOI: 10.1109/MSN60784.2023.00103.
[27] Natesha B V, Guddeti R M R. Adopting elitism-based Genetic Algorithm for minimizing multi-objective problems of IoT service placement in fog computing environment[J]. Journal of Network and Computer Applications, 2021, 178: 102972. DOI:10.1016/j.jnca.2020.102972.
[28] Sarrafzade N, Entezari-Maleki R, Sousa L.A genetic-based approach for service placement in fog computing[J]. The Journal of Supercomputing, 2022, 78(8): 10854-10875. DOI:10.1007/s11227-021-04254-w.
[29] Maia A M, Ghamri-Doudane Y, Vieira D, et al.Dynamic service placement and load distribution in edge computing[C]//2020 16th International Conference on Network and Service Management (CNSM). November 2-6, 2020, Izmir, Turkey. IEEE, 2020: 1-9. DOI: 10.23919/CNSM50824.2020.9269059.
[30] Ayoubi M, Ramezanpour M, Khorsand R.An autonomous IoT service placement methodology in fog computing[J]. Software: Practice and Experience, 2021, 51(5): 1097-1120. DOI:10.1002/spe.2939.
[31] Eyckerman R, Mercelis S, Marquez-Barja J, et al.Requirements for distributed task placement in the fog[J]. Internet of Things, 2020, 12: 100237. DOI: 10.1016/j.iot.2020.100237.
[32] Azizi S, Shojafar M, Farzin P, et al.DCSP: A delay and cost-aware service placement and load distribution algorithm for IoT-based fog networks[J]. Computer Communications, 2024, 215: 9-20. DOI:10.1016/j.comcom.2023.12.016.
[33] Kayal P, Liebeherr J.Distributed service placement in fog computing: An iterative combinatorial auction approach[C]// 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). July 7-10, 2019, Dallas, TX, USA. IEEE, 2019: 2145-2156. DOI: 10.1109/ICDCS.2019.00211.
[34] Aloqaily M, Kantarci B, Mouftah H T.Fairness-aware game theoretic approach for service management in vehicular clouds[C]// 2017 IEEE 86th Vehicular Technology Conference (VTC-Fall). September 24-27, 2017, Toronto, ON, Canada. IEEE, 2017: 1-5. DOI: 10.1109/VTCFall.2017.8288282.
[35] Lera I, Guerrero C, Juiz C.Availability-aware service placement policy in fog computing based on graph partitions[J]. IEEE Internet of Things Journal, 2019, 6(2): 3641-3651. DOI: 10.1109/JIOT.2018.2889511.
[36] Talpur A, Gurusamy M.DRLD-SP: A deep-reinforcement-learning-based dynamic service placement in edge-enabled Internet of vehicles[J]. IEEE Internet of Things Journal, 2022, 9(8): 6239-6251. DOI: 10.1109/JIOT.2021.3110913.
[37] Sharma A, Thangaraj V.Intelligent service placement algorithm based on DDQN and prioritized experience replay in IoT-Fog computing environment[J]. Internet of Things, 2024, 25: 101112. DOI:10.1016/j.iot.2024.101112.
[38] Ibn-Khedher H, Laroui M, Moungla H, et al.Next-generation edge computing assisted autonomous driving based artificial intelligence algorithms[J]. IEEE Access, 2022, 10: 53987-54001. DOI: 10.1109/ACCESS.2022.3174548.
[39] Liu T, Ni S G, Li X Q, et al.Deep reinforcement learning based approach for online service placement and computation resource allocation in edge computing[J]. IEEE Transactions on Mobile Computing, 2023, 22(7): 3870-3881. DOI: 10.1109/TMC.2022.3148254.
[40] Zare M, Elmi Sola Y, Hasanpour H.Towards distributed and autonomous IoT service placement in fog computing using asynchronous advantage actor-critic algorithm[J]. Journal of King Saud University-Computer and Information Sciences, 2023, 35(1): 368-381. DOI:10.1016/j.jksuci.2022.12.006.
[41] Song H F, Abdolmaleki A, Springenberg J T, et al. V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control[EB/OL].2019:1909.12238. https://arxiv.org/abs/1909.12238v1.
[42] Christodoulou P. Soft actor-critic for discrete action settings[EB/OL].2019:1910.07207. https://arxiv.org/abs/1910.07207v2.
[43] Mnih V, Kavukcuoglu K, Silver D, et al.Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533. DOI: 10.1038/nature14236.