针对无人机自组网等高动态飞行自组织网络中，网络拓扑的快速变化导致通信链路断裂和路由重建频繁的问题，研究一种基于Q-learning的QoS （quality of service）路由方法。该方法以Q-learning强化学习框架为基础，将邻居节点数量、链路持续时间和链路可用带宽作为路由度量信息，设计一种提供QoS保证的Q-learning奖励函数。网络节点通过广播Hello消息交互各自的本地路由度量信息，邻居节点接收到Hello分组或者数据分组，根据奖励函数计算并更新Q值，待转发数据分组的节点根据其维护的Q值表智能选择下一跳转发节点。EXata无线网络仿真环境中的仿真结果表明，该方法能为高动态飞行自组织网络中的数据传输提供稳定性好、服务质量高的通信链路。
黄鑫陈
,
陈光祖
,
郑敏
,
谭冲
,
刘洪
. 基于Q-learning的飞行自组织网络QoS路由方法[J]. 中国科学院大学学报, 2022
, 39(1)
: 134
-143
.
DOI: 10.7523/j.ucas.2020.0001
In high dynamic flying ad hoc networks (FANETs), such as UAV (unmanned aerial vehicle) ad hoc networks, the rapid change of network topology leads to the breakage of communication links and the frequent reconstruction of routes. To solve this problem, a QoS (quality of service) routing method based on Q-learning is studied. Based on the basic Q-learning framework, this method takes the number of neighbor nodes, link duration and link available bandwidth as routing metrics, and designs a Q-learning reward function to provide QoS guarantee. All nodes exchange local routing metrics information with neighbor nodes by broadcasting Hello messages and forwarding data packets. After receiving Hello packets or data packets, neighbor nodes calculate and update the Q value according to the reward function. Then one of neighbor nodes selects a next hop node to forward data packets intelligently according to the Q value table that it maintains. The simulation results in EXata simulator show that this method can provide stable and high QoS communication links for high dynamic flying ad hoc networks.
[1] Bekmezci I[DD(-*2]·, Sahingoz O K, Temel Ş. Flying ad-hoc networks (FANETs):a survey[J]. Ad Hoc Networks, 2013, 11(3):1254-1270.
[2] Moussaoui A, Boukeream A. A survey of routing protocols based on link-stability in mobile ad hoc networks[J]. Journal of Network and Computer Applications, 2015, 47:1-10.
[3] Fan X R, Cai W L, Lin J Y. A survey of routing protocols for highly dynamic mobile ad hoc networks[C]//2017 IEEE 17th International Conference on Communication Technology (ICCT). October 27-30, 2017, Chengdu, China. IEEE, 2017:1412-1417.
[4] 陈峥, 单剑锋, 俞能海. 移动Ad hoc网络的分簇后择路由协议[J]. 中国科学技术大学学报, 2008, 38(12):1372-1375.
[5] 张福全, 吴寅, 杨绪兵. 移动性度量的移动时延容忍网络路由策略[J]. 中国科学技术大学学报, 2019, 49(2):132-137.
[6] Chaudhari S S, Biradar R C. Survey of bandwidth estimation techniques in communication networks[J]. Wireless Personal Communications, 2015, 83(2):1425-1476.
[7] Chettibi S, Chikhi S. A survey of reinforcement learning based routing protocols for mobile ad-hoc networks[C]//Recent Trends in Wireless and Mobile Networks. CoNeCo 2011, WiMo 2011. Springer, Berlin, Heidelberg, 2011, 162:1-13.
[8] Alsheikh M A, Lin S W, Niyato D, et al. Machine learning in wireless sensor networks:algorithms, strategies, and applications[J]. IEEE Communications Surveys & Tutorials, 2014, 16(4):1996-2018.
[9] 张德干, 葛辉, 刘晓欢, 等. 一种基于Q-Learning策略的自适应移动物联网路由新算法[J]. 电子学报, 2018, 46(10):2325-2332.
[10] 李荥, 王芳, 景栋盛, 等. 一种基于Q学习的无线传感网络路由方法[J]. 计算技术与自动化, 2017, 36(2):155-160.
[11] 王月娟, 张苏宁, 吴水明, 等. 基于秩的Q-路由选择算法[J]. 计算机与现代化, 2018(10):1-5.
[12] Liang X D, Balasingham I, Byun S S. A reinforcement learning based routing protocol with QoS support for biomedical sensor networks[C]//2008 First International Symposium on Applied Sciences on Biomedical and Communication Technologies. October 25-28, 2008, Aalborg, Denmark. IEEE, 2008:1-5.
[13] Hu T S, Fei Y S. QELAR:A machine-learning-based adaptive routing protocol for energy-efficient and lifetime-extended underwater sensor networks[J]. IEEE Transactions on Mobile Computing, 2010, 9(6):796-809.
[14] 朱明, 雷磊, 朱钢, 等. 多跳无线DCF自组网MAC层可用带宽预测[J]. 小型微型计算机系统, 2016, 37(10):2217-2221.
[15] Chen Y J, Wang G J, Peng S C. Link lifetime-based segment-by-segment routing protocol in MANETs[C]//2008 IEEE International Symposium on Parallel and Distributed Processing with Applications. December 10-12, 2008, Sydney, NSW, Australia. IEEE, 2008:387-392.
[16] Waheed A, Wahid A, Shah M A. LAOD:link aware on demand routing in flying Ad-Hoc networks[C]//2019 IEEE International Conference on Communications Workshops (ICC Workshops). May 20-24, 2019, Shanghai, China. IEEE, 2019:1-5.
[17] Wang H P, Cui L. An enhanced AODV for mobile ad hoc network[C]//2008 International Conference on Machine Learning and Cybernetics. July 12-15, 2008, Kunming, China. IEEE, 2008, 2:1135-1140.
