Identification and adaptive control of Markov chains

Abstract

Consider a countable state controlled Markov chain whose transition probability is specified up to an unknown parameter alpha taking values in a compact metric space A. To each alpha is associated a prespecified stationary control law zeta ( alpha ). The adaptive control law selects at each time t the control action zeta ( alpha sub(t), x sub(t)) where x sub(t) is the state and alpha sub(t) is the maximum likelihood estimate of alpha . The asymptotic behavior of this control scheme is investigated for the cases when the true parameter value alpha sub(0) does or does not belong to A, and for the case when zeta is chosen to minimize an average cost criterion. The analysis uses an appropriate extension of the notions of recurrence to nonstationary Markov chains.