Polyathlon - Jose A. Martin H., Universidad Complutense de Madrid (slides)
Octopus - Martha White, University of Alberta (slides)
Acrobot - TBD

In this talk I give an overview of lessons learned through the participation of a graduate computer science class in this year's Reinforcement Learning Competition. The class, taught by Dr. Michael Littman at Rutgers University, split into groups, each entering the competition as a class project. Their experiences provide insight into how new users understand and interact with the competition software and changes that can made to the competition itself to encourage new participants. The class produced several successful algorithms, 3 of which topped the announced leaderboards in their respective domains. I will outline these algorithms as well as some of the less successful algorithms, many of which show promise and are being researched by the participants. The talk focuses on how the competition experience shapes the development of new ideas and the education of new RL researchers, and how the competition can adapt to further these goals.

In my talk, I will describe the Mario domain and the rationale behind the design of our agent for this domain. I will discuss a mechanism for reducing the size of the state space in this domain using ranking. Additionally, I will present empirical results from a model-free algorithm that used this reduced state in the competition. I will follow up my talk with a brief demo of our agent.

Helicopter hovering is an important challenge problem in the field of reinforcement learning. This presentation considers several neuroevolutionary approaches to discovering robust controllers for a generalized version of the problem used in the 2008 Reinforcement Learning Competition, in which wind in the helicopter's environment varies from run to run. A simple model-free strategy that won first place in the competition and also several more complex model-based approaches will be presented. The empirical results demonstrate that neuroevolution is effective at optimizing the weights of multi-layer perceptrons, that linear regression is faster and more effective than evolution for learning models, and that model-based approaches can outperform the simple model-free strategy, especially if prior knowledge is used to aid model learning. Finally, I will explain our strategy for the current 2009 Reinforcement Learning Competition and give an intuition about future work, wherein the goal is to efficiently learn an optimal controller in high risk domains.

One goal of the competition is to stimulate research in reinforcement learning that broadens the generality and applicability of our empirical results. Inspired by the Polyathlon domain, I will present some ideas for a benchmark problem for the reinforcement learning community that specifically measures the flexibility and generality of reinforcement learning agents on a suite of MDPs they have never seen before. The structure of the benchmark will emphasize research on agents that do not require MDP-specific tuning or require MDP-specific prior knowledge (eg. no a-priori models). In this session, I'll present some ideas, and solicit discussion from the community about how we might proceed together.

To wrap up the competition, we want to hear from you. This discussion is a chance for everyone to vent their frustrations, bestow their compliments, and contribute their ideas. How do you feel like this year's competition went? What can we change? What were the problems that you encountered (technical and other)? What do you think we could do to improve the competition in future years? Are there new domains you would suggest? Are there new formats for the competition that you think would be good (maybe a batch version?) How can we increase participation? Bring your thoughts!