Subarea Collaborative Deep Q-Network With Online Learning: Advanced Multi-Agent Routing Strategies for Dynamic Maritime Networks

As the shipping industry evolves, communication-limited non-terrestrial networks (NTN) like maritime communication networks face increasing challenges, particularly due to frequent and large-scale changes in network topology. To address these challenges effectively, we have developed a subarea-based collaborative deep q-network (Subarea_CDQN) enhanced with online learning. Our approach employs a multi-agent system to detect the positions of neighboring nodes and to manage the status of their send and receive queues, ensuring safe operational distances at sea. It utilizes integrated sensing and communication (ISAC) technology to mitigate communication constraints within the network. Additionally, considering the extensive coverage requirements of maritime networks, we model the network topology as a graph and partition it into multiple subareas. Within each subarea, multi-point collaboration employs CDQN to intelligently select the next-hop transmission paths, thus minimizing global computational loads and improving the efficiency of routing decisions. Furthermore, to cope with the dynamic topology changes caused by node mobility within maritime networks, we have devised global and subarea online learning strategies. These strategies dynamically adjust routing policies and optimize data forwarding paths in response to network topology changes, substantially reducing processing scales and computational costs. Simulation results show that our proposed Subarea_CDQN with online learning outperforms both the traditional shortest path and global CDQN algorithm in terms of packet arrival rate, retransmission rate, and packet loss rate, while maintaining a relatively optimal arrival time, thereby achieving a more balanced overall performance. This performance demonstrates their efficacy in communication-limited non-terrestrial networks, particularly in managing the complexities of dynamic maritime communication networks.

Recommended citation: X. Sun, T. Yang, C. Han, G. C. Lee, and S. Sun, “Subarea collaborative deep Q-network with online learning: Advanced multi-agent routing strategies for dynamic maritime networks,” IEEE Open Journal of the Communications Society, 2025.