Q-learning based QoS routing for high dynamic flying Ad Hoc networks

doi:10.7523/j.ucas.2020.0001

Abstract

Abstract: In high dynamic flying ad hoc networks (FANETs), such as UAV (unmanned aerial vehicle) ad hoc networks, the rapid change of network topology leads to the breakage of communication links and the frequent reconstruction of routes. To solve this problem, a QoS (quality of service) routing method based on Q-learning is studied. Based on the basic Q-learning framework, this method takes the number of neighbor nodes, link duration and link available bandwidth as routing metrics, and designs a Q-learning reward function to provide QoS guarantee. All nodes exchange local routing metrics information with neighbor nodes by broadcasting Hello messages and forwarding data packets. After receiving Hello packets or data packets, neighbor nodes calculate and update the Q value according to the reward function. Then one of neighbor nodes selects a next hop node to forward data packets intelligently according to the Q value table that it maintains. The simulation results in EXata simulator show that this method can provide stable and high QoS communication links for high dynamic flying ad hoc networks.

Key words: FANETs, QoS routing, Q-learning, link available bandwidth, link expiration time

CLC Number:

TN929.5

HUANG Xinchen, CHEN Guangzu, ZHENG Min, TAN Chong, LIU Hong. Q-learning based QoS routing for high dynamic flying Ad Hoc networks[J]. Journal of University of Chinese Academy of Sciences, 2022, 39(1): 134-143.

References

[1] Bekmezci I[DD(-*2]·, Sahingoz O K, Temel Ş. Flying ad-hoc networks (FANETs):a survey[J]. Ad Hoc Networks, 2013, 11(3):1254-1270.
[2] Moussaoui A, Boukeream A. A survey of routing protocols based on link-stability in mobile ad hoc networks[J]. Journal of Network and Computer Applications, 2015, 47:1-10.
[3] Fan X R, Cai W L, Lin J Y. A survey of routing protocols for highly dynamic mobile ad hoc networks[C]//2017 IEEE 17th International Conference on Communication Technology (ICCT). October 27-30, 2017, Chengdu, China. IEEE, 2017:1412-1417.
[4] 陈峥, 单剑锋, 俞能海. 移动Ad hoc网络的分簇后择路由协议[J]. 中国科学技术大学学报, 2008, 38(12):1372-1375.
[5] 张福全, 吴寅, 杨绪兵. 移动性度量的移动时延容忍网络路由策略[J]. 中国科学技术大学学报, 2019, 49(2):132-137.
[6] Chaudhari S S, Biradar R C. Survey of bandwidth estimation techniques in communication networks[J]. Wireless Personal Communications, 2015, 83(2):1425-1476.
[7] Chettibi S, Chikhi S. A survey of reinforcement learning based routing protocols for mobile ad-hoc networks[C]//Recent Trends in Wireless and Mobile Networks. CoNeCo 2011, WiMo 2011. Springer, Berlin, Heidelberg, 2011, 162:1-13.
[8] Alsheikh M A, Lin S W, Niyato D, et al. Machine learning in wireless sensor networks:algorithms, strategies, and applications[J]. IEEE Communications Surveys & Tutorials, 2014, 16(4):1996-2018.
[9] 张德干, 葛辉, 刘晓欢, 等. 一种基于Q-Learning策略的自适应移动物联网路由新算法[J]. 电子学报, 2018, 46(10):2325-2332.
[10] 李荥, 王芳, 景栋盛, 等. 一种基于Q学习的无线传感网络路由方法[J]. 计算技术与自动化, 2017, 36(2):155-160.
[11] 王月娟, 张苏宁, 吴水明, 等. 基于秩的Q-路由选择算法[J]. 计算机与现代化, 2018(10):1-5.
[12] Liang X D, Balasingham I, Byun S S. A reinforcement learning based routing protocol with QoS support for biomedical sensor networks[C]//2008 First International Symposium on Applied Sciences on Biomedical and Communication Technologies. October 25-28, 2008, Aalborg, Denmark. IEEE, 2008:1-5.
[13] Hu T S, Fei Y S. QELAR:A machine-learning-based adaptive routing protocol for energy-efficient and lifetime-extended underwater sensor networks[J]. IEEE Transactions on Mobile Computing, 2010, 9(6):796-809.
[14] 朱明, 雷磊, 朱钢, 等. 多跳无线DCF自组网MAC层可用带宽预测[J]. 小型微型计算机系统, 2016, 37(10):2217-2221.
[15] Chen Y J, Wang G J, Peng S C. Link lifetime-based segment-by-segment routing protocol in MANETs[C]//2008 IEEE International Symposium on Parallel and Distributed Processing with Applications. December 10-12, 2008, Sydney, NSW, Australia. IEEE, 2008:387-392.
[16] Waheed A, Wahid A, Shah M A. LAOD:link aware on demand routing in flying Ad-Hoc networks[C]//2019 IEEE International Conference on Communications Workshops (ICC Workshops). May 20-24, 2019, Shanghai, China. IEEE, 2019:1-5.
[17] Wang H P, Cui L. An enhanced AODV for mobile ad hoc network[C]//2008 International Conference on Machine Learning and Cybernetics. July 12-15, 2008, Kunming, China. IEEE, 2008, 2:1135-1140.