为减少移动边缘计算（mobile edge computing， MEC）网络中移动用户的长期任务开销，利用强化学习的马尔科夫决策过程，将用户的移动性与系统的动态信息建模为随机优化问题。依据系统信息的状态，将问题分为系统信息已知、系统信息未知2种情况。在系统信息已知时，提供了问题的最优解；系统信息未知时，基于在线学习提出2个任务卸载策略。一个策略能够收敛到系统最优解，但收敛速度较慢；另一个策略能以更快的收敛速度，达到接近最优解的表现，可用于更复杂的系统。最后在仿真中展示算法的有效性。
To reduce the costs of task processing, task offloading is put forward as a promising technology in mobile edge computing. In this paper, in order to lessen the burden of long-term tasks of the moving user, we utilize Markov decision process to formulate the task offloading problem as a stochastic programming problem, when taking the user mobility and system dynamics into account. According to the system information state, the problem can be categorized as the one with fully known system information and the other one with limited system information. We provide the optimal and learning-based task offloading algorithms under these two kinds of systems respectively. Furthermore, two learning-based algorithms, one with optimality and another with faster convergence rate, are proposed. The performance is verified in simulations.
[1] Li Y T, Cheng Q F, Liu X M, et al. A secure anonymous identity-based scheme in new authentication architecture for mobile edge computing[J]. IEEE Systems Journal, 2021, DOI: 10.1109/JSYST.2020.2979006.
[2] Dinh T Q, Tang J H, La Q D, et al. Offloading in mobile edge computing: task allocation and computational frequency scaling[J]. IEEE Transactions on Communications, 2017, 65(8): 3571-3584.DOI:10.1109/TCOMM.2017.2699660.
[3] Mao Y Y, You C S, Zhang J, et al. A survey on mobile edge computing: the communication perspective[J]. IEEE Communications Surveys & Tutorials, 2017, 19(4): 2322-2358.DOI:10.1109/COMST.2017.2745201.
[4] Mach P, Becvar Z. Mobile edge computing: a survey on architecture and computation offloading[J]. IEEE Communications Surveys & Tutorials, 2017, 19(3): 1628-1656.DOI:10.1109/COMST.2017.26823.
[5] You C S, Huang K B, Chae H, et al. Energy-efficient resource allocation for mobile-edge computation offloading[J]. IEEE Transactions on Wireless Communications, 2017, 16(3): 1397-1411.DOI:10.1109/TWC.2016.2633522.
[6] Mao Y Y, Zhang J, Song S H, et al. Stochastic joint radio and computational resource management for multi-user mobile-edge computing systems[J]. IEEE Transactions on Wireless Communications, 2017, 16(9): 5994-6009.DOI:10.1109/TWC.2017.2717986.
[7] Chen X F, Zhang H G, Wu C, et al. Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning[J]. IEEE Internet of Things Journal, 2019, 6(3): 4005-4018.DOI:10.1109/JIOT.2018.2876279.
[8] Min M H, Xiao L, Chen Y, et al. Learning-based computation offloading for IoT devices with energy harvesting[J]. IEEE Transactions on Vehicular Technology, 2019, 68(2): 1930-1941.DOI:10.1109/TVT.2018.2890685.
[9] Huang L, Bi S Z, Zhang Y J A. Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks[J]. IEEE Transactions on Mobile Computing, 2020,19(11):2581-2593. DOI: 10.1109/TMC.2019.2928811.
[10] 余翀,邱其文. 基于栅格地图的分层式机器人路径规划算法[J]. 中国科学院大学学报, 2013, 30(4): 528-538, 546.DOI:10.7523/j.issn.2095-6134.2013.04.015.
[11] Cheung M H, Huang J W. DAWN: delay-aware Wi-Fi offloading and network selection[J]. IEEE Journal on Selected Areas in Communications, 2015, 33(6): 1214-1223.DOI:10.1109/JSAC.2015.2416989.
[12] Nicholson A J, Noble B D. BreadCrumbs: forecasting mobile connectivity[C]//MobiCom ′08:Proceedings of the 14th ACM International Conference on Mobile Computing and Networking. 2008: 46-57.DOI:10.1145/1409944.1409952.
[13] Gambs S, Killijian M O, del Prado Cortez M N. Next place prediction using mobility Markov chains[C]//MPM ′12:Proceedings of the First Workshop on Measurement, Privacy, and Mobility. 2012: 1-6.DOI:10.1145/2181196.2181199.
[14] Sutton R S, Barto A G. Reinforcement learning: an introduction[M]. 2nd ed.Cambridge,MA:The MIT Press,2018.
[15] Watkins C J C H, Dayan P. Q-learning[J]. Machine Learning, 1992, 8(3/4): 279-292.DOI:10.1007/BF00992698.
[16] Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533.DOI:10.1038/nature14236.
[17] Zhu Z W, Jin S D, Yang Y, et al. Time reusing in D2D-enabled cooperative networks[J]. IEEE Transactions on Wireless Communications, 2018, 17(5): 3185-3200.DOI:10.1109/twc.2018.2808259.
[18] Abadi M, Barham P, Chen J M, et al. Tensorflow: a system for large-scale machine learning[J]. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2016,abs/1605.08695.
[19] Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks[C]//Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. 2011: 315-323.
