Adaptive control of Markov chains. I. Finite parameter set

Abstract

Consider a controlled Markov chain whose transition probabilities depend upon an unknown parameter a taking values in finite setA. To each a is associated a prespecified stationary control law φ(α). The adaptive control law selects at each timetthe control action indicated byphi(alpha_{t})where atis the maximum likelihood estimate of α. It is shown that atconverges to a parameter α*such that the "closed-loop" transition probabilities corresponding to α*and φ(α^{ast}) are the same as those corresponding to α⁰ and φ(α)where α⁰ is the true parameter. The situation when a0does not belong to the model setAis briefly discussed.