International Journal of Computer Networks and Applications

Open Access Open Access  Restricted Access Subscription Access

QMRNB: Design of an Efficient Q-Learning Model to Improve Routing Efficiency of UAV Networks via Bioinspired Optimizations


Affiliations
1 School of Computer Science and Engineering, Lovely Professional University, Punjab, India
2 Menzies Institute of Technology, Melbourne, Australia
 

The design of efficient routing strategies for Unmanned Aerial Vehicle (UAV) Networks is a multidomain task that involves analysis of node-level & network-level parameters, and mapping them with communication & contextual conditions. Existing path planning optimization models either showcase higher complexity or cannot be scaled for larger network scenarios. Moreover, the efficiency of these models also reduces w.r.t. the number of communication requests, which limits their scalability levels. To get a better result over these challenges, this article provides an idea to design an efficient Q-Learning model to improve the routing efficiency of UAV networks via bioinspired optimizations. The model initially collects temporal routing performance data samples for individual nodes and uses them to form coarse routes via Q-Learning optimizations. These routes are further processed via a Mayfly Optimization (MO) Model, which assists in the selection of optimal routing paths for high Quality of Service (QoS) even under large-scale routing requests. The MO Model can identify alternate paths via the evaluation of a high-density routing fitness function that assists the router in case the selected paths are occupied during current routing requests. This assists in improving temporal routing performance even under dense network conditions. Due to these optimizations, the model is capable of reducing the routing delay by 8.5%, improving energy efficiency by 4.9%, and reducing the routing jitter by 3.5% when compared with existing routing techniques by taking similar routing conditions.

Keywords

UAV, Routing, Delay, Energy, Mayfly, Optimization, Jitter, efficiency, Complexity.
User
Notifications
Font Size

  • N. Toorchi, F. Hu, E. S. Bentley, and S. Kumar, "Skeleton-Based Swarm Routing (SSR): Intelligent Smooth Routing for Dynamic UAV Networks," in IEEE Access, vol. 9, pp. 1286-1303, 2021, doi: 10.1109/ACCESS.2020.3043672.

  • L. Hong, H. Guo, J. Liu, and Y. Zhang, "Toward Swarm Coordination: Topology-Aware Inter-UAV Routing Optimization," in IEEE Transactions on Vehicular Technology, vol. 69, no. 9, pp. 10177-10187, Sept. 2020, doi: 10.1109/TVT.2020.3003356.

  • A. Mukherjee, S. Misra, V. S. P. Chandra, and N. S. Raghuwanshi, "ECoR: Energy-Aware Collaborative Routing for Task Offload in Sustainable UAV Swarms," in IEEE Transactions on Sustainable Computing, vol. 5, no. 4, pp. 514-525, 1 Oct.-Dec. 2020, doi: 10.1109/TSUSC.2020.2976453.

  • L. Zhang, F. Hu, Z. Chu, E. Bentley, and S. Kumar, "3D Transformative Routing for UAV Swarming Networks: A Skeleton-Guided, GPS-Free Approach," in IEEE Transactions on Vehicular Technology, vol. 70, no. 4, pp. 3685-3701, April 2021, doi: 10.1109/TVT.2021.3061911.

  • Z. Yang, H. Liu, Y. Chen, X. Zhu, Y. Ning, and W. Zhu, "UEE-RPL: A UAV-Based Energy Efficient Routing for Internet of Things," in IEEE Transactions on Green Communications and Networking, vol. 5, no. 3, pp. 1333-1344, Sept. 2021, doi: 10.1109/TGCN.2021.3085897.

  • S. Jiang, Z. Huang, and Y. Ji, "Adaptive UAV-Assisted Geographic Routing with Q-Learning in VANET," in IEEE Communications Letters, vol. 25, no. 4, pp. 1358-1362, April 2021, doi: 10.1109/LCOMM.2020.3048250.

  • T. Li et al., "A mean field game-theoretic cross-layer optimization for multi-hop swarm UAV communications," in Journal of Communications and Networks, vol. 24, no. 1, pp. 68-82, Feb. 2022, doi: 10.23919/JCN.2021.000035.

  • X. Tan, Z. Zuo, S. Su, X. Guo, X. Sun and D. Jiang, "Performance Analysis of Routing Protocols for UAV Communication Networks," in IEEE Access, vol. 8, pp. 92212-92224, 2020, doi: 10.1109/ACCESS.2020.2995040.

  • H. Fatemidokht, M. K. Rafsanjani, B. B. Gupta, and C. -H. Hsu, "Efficient and Secure Routing Protocol Based on Artificial Intelligence Algorithms with UAV-Assisted for Vehicular Ad Hoc Networks in Intelligent Transportation Systems," in IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 7, pp. 4757-4769, July 2021, doi: 10.1109/TITS.2020.3041746.

  • O. Thakoor, J. Garg and R. Nagi, "Multiagent UAV Routing: A Game Theory Analysis with Tight Price of Anarchy Bounds," in IEEE Transactions on Automation Science and Engineering, vol. 17, no. 1, pp. 100-116, Jan. 2020, doi: 10.1109/TASE.2019.2902360.

  • H. Song, L. Liu, S. M. Pudlewski and E. S. Bentley, "Random Network Coding Enabled Routing Protocol in Unmanned Aerial Vehicle Networks," in IEEE Transactions on Wireless Communications, vol. 19, no. 12, pp. 8382-8395, Dec. 2020, doi: 10.1109/TWC.2020.3022399.

  • M. Y. Arafat and S. Moh, "A Q-Learning-Based Topology-Aware Routing Protocol for Flying Ad Hoc Networks," in IEEE Internet of Things Journal, vol. 9, no. 3, pp. 1985-2000, 1 Feb.1, 2022, doi: 10.1109/JIOT.2021.3089759.

  • F. Xiong et al., "Energy-Saving Data Aggregation for Multi-UAV System," in IEEE Transactions on Vehicular Technology, vol. 69, no. 8, pp. 9002-9016, Aug. 2020, doi: 10.1109/TVT.2020.2999374.

  • X. Qiu, L. Xu, P. Wang, Y. Yang, and Z. Liao, "A Data-Driven Packet Routing Algorithm for an Unmanned Aerial Vehicle Swarm: A MultiAgent Reinforcement Learning Approach," in IEEE Wireless Communications Letters, vol. 11, no. 10, pp. 2160-2164, Oct. 2022, doi: 10.1109/LWC.2022.3195963.

  • M. Zhang, C. Dong, P. Yang, T. Tao, Q. Wu, and T. Q. S. Quek, "Adaptive Routing Design for Flying Ad Hoc Networks," in IEEE Communications Letters, vol. 26, no. 6, pp. 1438-1442, June 2022, doi: 10.1109/LCOMM.2022.3152832.

  • P. K. Deb, A. Mukherjee, and S. Misra, "XiA: Send-It-Anyway QRouting for 6G-Enabled UAV-LEO Communications," in IEEE Transactions on Network Science and Engineering, vol. 8, no. 4, pp. 2722-2731, 1 Oct.-Dec. 2021, doi: 10.1109/TNSE.2021.3086484.

  • P. K. Deb, A. Mukherjee, and S. Misra, "XiA: Send-It-Anyway QRouting for 6G-Enabled UAV-LEO Communications," in IEEE Transactions on Network Science and Engineering, vol. 8, no. 4, pp. 2722-2731, 1 Oct.-Dec. 2021, doi: 10.1109/TNSE.2021.3086484.

  • J. Yoon, S. Doh, O. Gnawali, and H. Lee, "Time-Dependent Ad-Hoc Routing Structure for Delivering Delay-Sensitive Data Using UAVs," in IEEE Access, vol. 8, pp. 36322-36336, 2020, doi: 10.1109/ACCESS.2020.2974553.

  • J. Fu, A. Núñez and B. De Schutter, "Real-Time UAV Routing Strategy for Monitoring and Inspection for Postdisaster Restoration of Distribution Networks," in IEEE Transactions on Industrial Informatics, vol. 18, no. 4, pp. 2582-2592, April 2022, doi: 10.1109/TII.2021.3098506.

  • S. G. Manyam, K. Sundar and D. W. Casbeer, "Cooperative Routing for an Air–Ground Vehicle Team—Exact Algorithm, Transformation Method, and Heuristics," in IEEE Transactions on Automation Science and Engineering, vol. 17, no. 1, pp. 537-547, Jan. 2020, doi: 10.1109/TASE.2019.2931894.

  • S. G. Manyam, K. Sundar and D. W. Casbeer, "Cooperative Routing for an Air–Ground Vehicle Team—Exact Algorithm, Transformation Method, and Heuristics," in IEEE Transactions on Automation Science and Engineering, vol. 17, no. 1, pp. 537-547, Jan. 2020, doi: 10.1109/TASE.2019.2931894.

  • D. Shumeye Lakew, U. Sa’ad, N. -N. Dao, W. Na, and S. Cho, "Routing in Flying Ad Hoc Networks: A Comprehensive Survey," in IEEE Communications Surveys & Tutorials, vol. 22, no. 2, pp. 1071-1120, Second quarter 2020, doi: 10.1109/COMST.2020.2982452.

  • A. Rovira-Sugranes, F. Afghah, J. Qu and A. Razi, "Fully-Echoed QRouting with Simulated Annealing Inference for Flying Adhoc Networks," in IEEE Transactions on Network Science and Engineering, vol. 8, no. 3, pp. 2223-2234, 1 July-Sept. 2021, doi: 10.1109/TNSE.2021.3085514.

  • G. Sun, D. Qin, T. Lan, and L. Ma, "Research on Clustering Routing Protocol Based on Improved PSO in FANET," in IEEE Sensors Journal, vol. 21, no. 23, pp. 27168-27185, 1 Dec.1, 2021, doi: 10.1109/JSEN.2021.3117496.

  • Y. Cui, Q. Zhang, Z. Feng, Z. Wei, C. Shi, and H. Yang, "TopologyAware Resilient Routing Protocol for FANETs: An Adaptive QLearning Approach," in IEEE Internet of Things Journal, vol. 9, no. 19, pp. 18632-18649, 1 Oct.1, 2022, doi: 10.1109/JIOT.2022.3162849.

  • C. Wang et al., "Elastic Routing Mechanism for Flying Ad Hoc Network," in IEEE Access, vol. 10, pp. 98712-98723, 2022, doi: 10.1109/ACCESS.2022.3206767.

  • O. T. Abdulhae, J. S. Mandeep and M. Islam, "Cluster-Based Routing Protocols for Flying Ad Hoc Networks (FANETs)," in IEEE Access, vol. 10, pp. 32981-33004, 2022, doi: 10.1109/ACCESS.2022.3161446.

  • O. T. Abdulhae, J. S. Mandeep and M. Islam, "Cluster-Based Routing Protocols for Flying Ad Hoc Networks (FANETs)," in IEEE Access, vol. 10, pp. 32981-33004, 2022, doi: 10.1109/ACCESS.2022.3161446.

  • Z. Du et al., "A Routing Protocol for UAV-Assisted Vehicular Delay Tolerant Networks," in IEEE Open Journal of the Computer Society, vol. 2, pp. 85-98, 2021, doi: 10.1109/OJCS.2021.3054759.

  • B. Liu, W. Zhang, W. Chen, H. Huang and S. Guo, "Online Computation Offloading and Traffic Routing for UAV Swarms in EdgeCloud Computing," in IEEE Transactions on Vehicular Technology, vol. 69, no. 8, pp. 8777-8791, Aug. 2020, doi: 10.1109/TVT.2020.2994541.

  • J. Baek, S. I. Han, and Y. Han, "Energy-Efficient UAV Routing for Wireless Sensor Networks," in IEEE Transactions on Vehicular Technology, vol. 69, no. 2, pp. 1741-1750, Feb. 2020, doi: 10.1109/TVT.2019.2959808.

  • Souto, Anderson, Rodrigo Alfaia, Evelin Cardoso, Jasmine Araújo, and Carlos Francês. 2023. "UAV Path Planning Optimization Strategy: Considerations of Urban Morphology, Microclimate, and Energy Efficiency Using Q-Learning Algorithm" Drones 7, no. 2: 123. https://doi.org/10.3390/drones702012

  • Alam, Muhammad Morshed, and Sangman Moh. 2022. "Survey on QLearning-Based Position-Aware Routing Protocols in Flying Ad Hoc Networks" Electronics 11, no. 7: 1099. https://doi.org/10.3390/electronics11071099

  • Balasubramanian, E., Elangovan, E., Tamilarasan, P. et al. Optimal energy efficient path planning of UAV using hybrid MACO-MEA* algorithm: theoretical and experimental approach. J Ambient Intell Human Comput (2022). https://doi.org/10.1007/s12652-022-04098-z

  • Fotouhi A, Ding M, Hassan M. Deep Q-Learning for Two-Hop Communications of Drone Base Stations. Sensors (Basel). 2021 Mar 11;21(6):1960. doi: 10.3390/s21061960. PMID: 33799546; PMCID: PMC7999891.

  • Mannan, A, Obaidat, MS, Mahmood, K, Ahmad, A, Ahmad, R. Classical versus reinforcement learning algorithms for unmanned aerial vehicle network communication and coverage path planning: A systematic literature review. Int J Commun Syst. 2023; 36(5):e5423. doi:10.1002/dac.5423 How to cite this article:

  • A. Vashisth and R. S. Batth, "An Overview, Survey, and Challenges in UAVs Communication Network," 2020 International Conference on Intelligent Engineering and Management (ICIEM), London, UK, 2020, pp. 342-347, doi: 10.1109/ICIEM48762.2020.9160197.

  • A. Vashisth, R. Singh Batth, and R. Ward, "Existing Path Planning Techniques in Unmanned Aerial Vehicles (UAVs): A Systematic Review," 2021 International Conference on Computational Intelligence and Knowledge Economy (ICCIKE), Dubai, United Arab Emirates, 2021, pp. 366-372, doi: 10.1109/ICCIKE51210.2021.9410787.


Abstract Views: 227

PDF Views: 1




  • QMRNB: Design of an Efficient Q-Learning Model to Improve Routing Efficiency of UAV Networks via Bioinspired Optimizations

Abstract Views: 227  |  PDF Views: 1

Authors

Anshu Vashisth
School of Computer Science and Engineering, Lovely Professional University, Punjab, India
Balraj Singh
School of Computer Science and Engineering, Lovely Professional University, Punjab, India
Ranbir Singh Batth
Menzies Institute of Technology, Melbourne, Australia

Abstract


The design of efficient routing strategies for Unmanned Aerial Vehicle (UAV) Networks is a multidomain task that involves analysis of node-level & network-level parameters, and mapping them with communication & contextual conditions. Existing path planning optimization models either showcase higher complexity or cannot be scaled for larger network scenarios. Moreover, the efficiency of these models also reduces w.r.t. the number of communication requests, which limits their scalability levels. To get a better result over these challenges, this article provides an idea to design an efficient Q-Learning model to improve the routing efficiency of UAV networks via bioinspired optimizations. The model initially collects temporal routing performance data samples for individual nodes and uses them to form coarse routes via Q-Learning optimizations. These routes are further processed via a Mayfly Optimization (MO) Model, which assists in the selection of optimal routing paths for high Quality of Service (QoS) even under large-scale routing requests. The MO Model can identify alternate paths via the evaluation of a high-density routing fitness function that assists the router in case the selected paths are occupied during current routing requests. This assists in improving temporal routing performance even under dense network conditions. Due to these optimizations, the model is capable of reducing the routing delay by 8.5%, improving energy efficiency by 4.9%, and reducing the routing jitter by 3.5% when compared with existing routing techniques by taking similar routing conditions.

Keywords


UAV, Routing, Delay, Energy, Mayfly, Optimization, Jitter, efficiency, Complexity.

References





DOI: https://doi.org/10.22247/ijcna%2F2023%2F220740