Natural actor-critic algorithms

摘要: We present four new reinforcement learning algorithms based on actor-critic, natural-gradient and function-approximation ideas, we provide their convergence proofs. Actor-critic methods are online approximations to policy iteration in which the value-function parameters estimated using temporal difference updated by stochastic gradient descent. Methods gradients this way of special interest because compatibility with methods, needed handle large or infinite state spaces. The use is many applications it dramatically reduces variance estimates. natural can produce better conditioned parameterizations has been shown further reduce some cases. Our results extend prior two-timescale for actor-critic Konda Tsitsiklis actor incorporating gradients. empirical studies Peters, Vijayakumar Schaal providing first proofs fully incremental algorithms.