Publications

Sort by: [year] [type] [author]

Using PCA to Efficiently Represent State Spaces

William Curran, Tim Brys, Matthew Taylor, and William D. Smart.
In "Proceedings of the 12th European Workshop on Reinforcement Learning (EWRL)", Lille, France, 2015.

Reinforcement learning algorithms need to deal with the exponential growth of states and actions when exploring optimal control in high-dimensional spaces. This is known as the curse of dimensionality. By projecting the agent’s state onto a low-dimensional manifold, we can represent the state space in a smaller and more efficient representation. By using this representation during learning, the agent can converge to a good policy much faster. We test this approach in the Mario Benchmarking Domain. When using dimensionality reduction in Mario, learning converges much faster to a good policy. But, there is a critical convergence-performance trade-off. By projecting onto a low-dimensional manifold, we are ignoring important data. In this paper, we explore this trade-off of convergence and performance. We find that learning in as few as 4 dimensions (instead of 9), we can improve performance past learning in the full dimensional space at a faster convergence rate.

Paper: [PDF]

@inproceedings{ewrl2015,
  author = {Curran, William and Brys, Tim and Taylor, Matthew E. and Smart, William D.},
  title = {Using {PCA} to Efficiently Represent State Spaces},
  booktitle = {Proceedings of the 12th European Workshop on Reinforcement Learning ({EWRL})},
  address = {Lille, France},
  year = {2015}
}