Reinforcement Learning Approaches for Delay-Sensitive Network Scheduling

Abstract

Modern communication technologies have changed the way people communicate, such as real-time text messaging, audio and video calls, and video streaming services over the Internet. Therefore, for handheld communication terminals, low delay and low power consumption are basic requirements of users. For a typical communication system, the power cost is convex in transmission rate under fixed channel conditions. This means that as the transmission rate increases, the delay decreases at the cost of increased power consumption per bit. Therefore, there is a trade-off between delay and power. In this thesis, our goal is to develop lightweight reinforcement learning for resource-efficient wireless scheduling that is adaptive for dynamic changes and tackles challenges in reinforcement learning.

In Chapter 2, we adopt an RL-based approach to obtain the optimal trade-off between delay and power consumption for a given power constraint in a communication system whose conditions (e.g., channel conditions, traffic arrival rates) can change over time. To this end, we first formulate this problem as an infinite-horizon MDP, and then Q-learning is adopted to solve this problem. To handle the given power constraint, we apply the Lagrange multiplier method that transforms a constrained optimization problem into a non-constrained problem. Finally, via simulation, we show that Q-learning achieves the optimal policy.

In Chapter 3, we propose an RL algorithm to find an optimal scheduling policy to minimize the delay for a given energy constraint in a communication system where the environments, such as traffic arrival rates, are not known in advance and can change over time. For this purpose, this problem is formulated as an infinite-horizon CMDP. To handle the constrained optimization problem, we first adopt the Lagrangian relaxation technique to solve it. Then, we propose a variant of Q-learning, Q-greedyUCB, that combines ϵ-greedy and UCB algorithms to solve this constrained MDP problem. We mathematically prove that the Q-greedyUCB algorithm converges to an optimal solution. Simulation results show that Q-greedyUCB finds an optimal scheduling strategy and is more efficient than Q-learning with ϵ-greedy, Rlearning, and the Average payoff RL (ARL) algorithm in terms of the cumulative regret. We also show that our algorithm can learn and adapt to the changes in the environment so as to obtain an optimal scheduling strategy under a given power constraint for the new environment.

In Chapter 4, our goal is to minimize the average delay under the average energy consumption constraint in a single-queue and single-server wireless communication system with block fading channels. To this end, we formulate this problem as an infinite-horizon CMDP. In our CMDP, we jointly consider the queue length and channel condition as the state. We apply the Lagrange multiplier method to transform the constrained optimization problem into an unconstrained optimization problem. Then, we prove that an optimal scheduling strategy is non-decreasing with respect to queue length and channel state. To obtain an optimal scheduling policy, an efficient reinforcement learning algorithm, the Structural-Optimistic Q-learning algorithm (SOQ), is proposed, which exploits the non-decreasing property of optimal policies by using policy projection. Finally, we analyze how to control the average energy consumption to satisfy a given energy consumption constraint. The simulation results show that the performance of the SOQ surpasses that of the traditional Q-learning algorithm in terms of the average cost during the learning phase.