Successive Over-Relaxation ${Q}$ -Learning

Kamanchi, Chandramouli ; Diddigi, Raghuram Bharadwaj ; Bhatnagar, Shalabh (2020) Successive Over-Relaxation ${Q}$ -Learning IEEE Control Systems Letters, 4 (1). pp. 55-60. ISSN 2475-1456

Full text not available from this repository.

Official URL: http://doi.org/10.1109/LCSYS.2019.2921158

Related URL: http://dx.doi.org/10.1109/LCSYS.2019.2921158

Abstract

In a discounted reward Markov decision process (MDP), the objective is to find the optimal value function, i.e., the value function corresponding to an optimal policy. This problem reduces to solving a functional equation known as the Bellman equation and a fixed point iteration scheme known as the value iteration is utilized to obtain the solution. In literature, a successive over-relaxation (SOR)-based value iteration scheme is proposed to speed-up the computation of the optimal value function. The speed-up is achieved by constructing a modified Bellman equation that ensures faster convergence to the optimal value function. However, in many practical applications, the model information is not known and we resort to reinforcement learning (RL) algorithms to obtain optimal policy and value function. One such popular algorithm is Q -learning. In this letter, we propose SOR Q -learning. We first derive a modified fixed point iteration for SOR Q -values and utilize stochastic approximation to derive a learning algorithm to compute the optimal value function and an optimal policy. We then prove the almost sure convergence of the SOR Q -learning to SOR Q -values. Finally, through numerical experiments, we show that SOR Q -learning is faster compared to the standard Q -learning algorithm.

Item Type:Article
Source:Copyright of this article belongs to Institute of Electrical and Electronics Engineers.
Keywords:Machine Learning; Stochastic Optimal Control; Stochastic Systems.
ID Code:116439
Deposited On:12 Apr 2021 05:54
Last Modified:12 Apr 2021 05:54

Repository Staff Only: item control page