A sensitivity formula for risk-sensitive cost and the actor-critic algorithm

Abstract

We propose for risk sensitive control of finite Markov chains a counterpart of the popular 'actor-critic'algorithm for classical Markov decision processes. The algorithm is based on a 'sensitivity formula'for the risk sensitive cost; is shown to converge with probability one to the desired solution. The proof technique is an adaptation of the ordinary differential equations approach for the analysis of two time-scale stochastic approximation algorithms.