Kagan Tumer's Publications

Display Publications by [Year] [Type] [Topic]


Unifying Temporal and Structural Credit Assignment Problems. A. Agogino and K. Tumer. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, New York, NY, July 2004.

Abstract

Single-agent reinforcement learners in time-extended domains and multi-agent systems share a common difficulty known as the credit assignment problem. Multiagent systems have the structural credit assignment problemof determining the contributions of a particular agent to a common task. Instead, time-extended single-agent systems have the temporal credit assignment problem of determining the contribution of a particular action to the quality of the full sequence of actions. Traditionally these two problems are considered different and are handled in separate ways. In this article we showhowthese two forms of the credit assignment problem are equivalent. In this unified framework, a single-agent Markov decision process can be broken down into a single-time-step multiagent process. Furthermore we show that Monte Carlo estimation or Q-learning (depending on whether the values of resulting actions in the episode are known at the time of learning) are equivalent to different agent utility functions in a multi-agent system. This equivalence shows howan often neglected issue in multi-agent systems is equivalent to a well-known deficiency in multi-timestep learning and lays the basis for solving time-extended multi-agent problems, where both credit assignment problems are present.

Download

[PDF]239.9kB  

BibTeX Entry

@inproceedings{tumer-agogino_aamas04,
	author = {A. Agogino and K. Tumer},
	title = {Unifying Temporal and Structural Credit Assignment Problems},
	booktitle = {Proceedings of the Third International Joint Conference on
	        Autonomous Agents and Multiagent Systems},
	month = {July},
	address = {New York, NY},
	abstract = {Single-agent reinforcement learners in time-extended domains and multi-agent systems share a common difficulty known as the credit assignment problem. Multiagent systems have the structural credit assignment problemof determining the contributions of a particular agent to a common task. Instead, time-extended single-agent systems have the temporal credit assignment problem of determining the contribution of a particular action to the quality of the full sequence of actions. Traditionally these two problems are considered different and are handled in separate ways. In this article we showhowthese two forms of the credit assignment problem are equivalent. In this unified framework, a single-agent Markov decision process can be broken down into a single-time-step multiagent process. Furthermore we show that Monte Carlo estimation or Q-learning (depending on whether the values of resulting actions in the episode are known at the time of learning) are equivalent to different agent utility functions in a multi-agent system. This equivalence shows howan often neglected issue in multi-agent systems is equivalent to a well-known deficiency in multi-timestep learning and lays the basis for solving time-extended multi-agent problems, where both credit assignment problems are present.},
	bib2html_pubtype = {Refereed Conference Papers},
	bib2html_rescat = {Multiagent Systems, Reinforcement Learning},
	year = {2004}
}

Generated by bib2html.pl (written by Patrick Riley ) on Wed Apr 01, 2020 17:39:43