Deep Reinforcemnet Learning for Robust Beamforming in Integrated Sensing, Communication and Power Transmission Systems

Abstract

A communication network integrating multiple modes can effectively support the sustainable development of next-generation wireless communications. Integrated sensing, communication, and power transfer (ISCPT) represents an emerging technological paradigm that not only facilitates information transmission but also enables environmental sensing and wireless power transfer. To achieve optimal beamforming in transmission, it is crucial to satisfy multiple constraints, including quality of service (QoS), radar sensing accuracy, and power transfer efficiency, while ensuring fundamental system performance. The presence of multiple parametric constraints makes the problem a non-convex optimization challenge, underscoring the need for a solution that balances low computational complexity with high precision. Additionally, the accuracy of channel state information (CSI) is pivotal in determining the achievable rate, as imperfect or incomplete CSI can significantly degrade system performance and beamforming efficiency. Deep reinforcement learning (DRL), a machine learning technique where an agent learns by interacting with its environment, offers a promising approach that can dynamically optimize system performance through adaptive decision-making strategies. In this paper, we propose a DRL-based ISCPT framework, which effectively manages complex environmental states and continuously adjusts variables related to sensing, communication, and energy harvesting to enhance overall system efficiency and reliability. The achievable rate upper bound can be inferred through robust, learnable beamforming in the ISCPT system. Our results demonstrate that DRL-based algorithms significantly improve resource allocation, power management, and information transmission, particularly in dynamic and uncertain environments with imperfect CSI.

Keywords: communication; deep reinforcement learning; imperfect channel state information; integrating sensing; multi-user; power transfer; robust beamforming.

Figure 1

Illustration of the integrated IoT devices and sensing target communication model.

Figure 2

The framework of DRL for the ISCPT system.

Figure 3

Training performance with different learning rate.

Figure 4

Training performance with different CSI.

Figure 5

Average rate vs. transmit power under K = 4, $E_k^{\min }$ = 0 dBm, $ϵ$ = 0.1.

Figure 6

Average rate vs. CSI uncertainty under K = 4, $P_{max}$ = 10 dBm, $E_k^{\min }$ = 0 dBm, $ϵ$ = 0.1.

Figure 7

Average rate vs. the number of IoT devices under $P_{max}$ = 10 dBm, $E_k^{\min }$ = 0 dBm, $ϵ$ = 0.1.

Figure 8

Impact of energy harvesting on transmission rate under $P_{\max }$ = 10 dBm, K = 4, $ϵ$ = 0.1, ${\sigma }^2$ = 0.

Figure 9

Impact of energy harvesting on transmission rate under $P_{\max }$ = 10 dBm, K = 4, $ϵ$ = 0.1, ${\sigma }^2$ = 0.1.

Figure 10

Impact of transmit power and energy on CRB.