Abstraction and Hierarchy in Reinforcement Learning
The deadline for submission of technical contributions and
statements of interest is September 18, 1998. (Submissions after
September 18 may still be considered, but we cannot guarantee that the
names of the authors will appear in all of the printed program
material.)
The workshop will be held on December 5, 1998 in Breckenridge, CO.
Organizers
Doina Precup, University of Massachusetts, Amherst
Ron Parr, Stanford University
Leslie Kaelbling, Brown University
Tom Dietterich, Oregon State University
Motivation
When making everyday decisions, people are able to foresee the
consequences of their possible courses of action at multiple levels of
abstraction. Recent research in reinforcement learning (RL) has focused
on the way in which knowledge about abstract actions and abstract
representations can be incorporated into the framework of Markov
Decision Processes (MDPs). Several theoretical results and applications
suggest that these methods can improve significantly the scalability of
reinforcement learning systems by accelerating learning and by promoting
sharing and re-use of learned subtasks. However, most of the interesting
questions remain open:
Task formulation and automated task creation:
One of the most promising and appealing aspects of abstract actions
is that they may be learned as solutions to specific
subproblems. This framework facilitates knowledge transfer between
different problems. Although this is a very powerful idea, it is
not clear how it should be implemented. Several task formulations
have been introduced in the literature, such as additional rewards,
different dynamics and different terminal values for certain
states.
Degree and complexity of action models:
Some methods construct and apply models that describe the effects
of executing a temporally-extended action over an entire set of
possible starting states. Other methods do not learn explicit
models of actions at all. What are the advantages and disadvantages
of model-based and model-free methods when abstraction and
hierarchy are introduced?
Integrating abstraction methods:
Integrating state abstraction with temporal abstraction is a key
aspect of scaling up RL systems. However, state abstraction can
lead to many problems including loss of optimality and the need to
perform hierarchical credit assignment. What is the best way to
combine state abstraction with temporal abstraction? How can the
problems introduced by state abstraction be identified and resolved
automatically? Can they be avoided entirely by developing joint
methods that build state and temporal abstractions simultaneously?
Hidden state:
Abstract actions can, to some extent, guide agents through
sequences of intermediate states where sensory information is
ambiguous or missing. At a higher level of abstraction, the agent
may be able to treat the world as fully-observable even though at
lower levels it is only partially observable. How can an agent
acquire abstract behaviors and representations appropriate to
partially observable environment? Investigating the relationship
between temporal abstraction and existing POMDP methods could bring
useful insights for solving problems with hidden state.
Utility and computational efficiency considerations:
Introducing abstract actions in addition to primitive actions
increases the space of possible choices. Therefore, methods that
prune the space of possible actions or policies considered by the
agent become very important. A general theory of how this should be
done is not yet apparent. Should it be done according to some a
priori rules, a utility metric, or some on-line heuristic?
Incremental methods for learning in parallel about several possible
courses of action are also needed.
Scaling up to multi-layer abstractions:
Methods for abstraction and decomposition are often most easily
understood when a single abstract layer is placed on top of a
concrete layer. What issues arise when deeper hierarchies of
abstractions are constructed?
Temporally Extended Perception:
It is commonly recognized that action and perception are intimately
related. In real applications, "primitive" percepts are in fact
implemented by sequences of simpler percepts, actions, and
inferences. Conversely, abstract actions can be viewed as defining
new "abstract" percepts. For example, the set of states from which
a robot can successfully dock with a battery charger could be used
to define the perception of the charger. What is the interplay
between perception, temporal abstraction and state abstraction?
Format
The workshop will begin with a brief overview of the current state of
the field, and a couple of invited talks. The remaining time will be
organized as a series of group or panel discussions. For some topics,
discussion will be preceded by one or two brief presentations of new
research related to the discussion topic.
Participation
Those interested in leading discussion or participating in a panel,
should send a short statement of interest describing their experience in
the field and the areas of discussion to which they hope to
contribute. Links to relevant publications would also be
helpful. Authors who wish to present new technical results should submit
an extended abstract in postscript (5 page maximum), and be prepared to
submit a more detailed description of their work if requested.
Both technical contributions and statements of interest should be
submitted to parr@cs.stanford.edu and dprecup@cs.umass.edu by September
18, 1998. For registration fees and a full schedule of the conference
and workshops, please check the NIPS web page:
http://www.cs.cmu.edu/Groups/NIPS/