Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation

Sean Lamont; John Aslanides; Jan Leike; Marcus Hutter

doi:10.48550/arXiv.1703.01358

Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation

Sean Lamont, John Aslanides, Jan Leike, Marcus Hutter

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In recent years, work has been done to develop the theory of General Reinforcement Learning (GRL). However, there are no examples demonstrating the known results regarding generalised discounting. We have added to the GRL simulation platform (AIXIjs) the functionality to assign an agent arbitrary discount functions, and an environment which can be used to determine the effect of discounting on an agent's policy. Using this, we investigate how geometric, hyperbolic and power discounting affect an informed agent in a simple MDP. We experimentally reproduce a number of theoretical results, and discuss some related subtleties. It was found that the agent's behaviour followed what is expected theoretically, assuming appropriate parameters were chosen for the Monte-Carlo Tree Search (MCTS) planning algorithm.

Original language	English
Title of host publication	Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
Editors	E Durfee, M Winikoff, K Larson & S Das
Place of Publication	TBC
Publisher	IFAAMAS (International Foundation for Autonomous Agents and Multiagent Systems)
Pages	1589-1591
Edition	To be checked
ISBN (Print)	978-151085507-6
DOIs	https://doi.org/10.48550/arXiv.1703.01358
Publication status	Published - 2017
Event	16th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2017 - Sao Paulo, Brazil Duration: 1 Jan 2017 → …

Conference

Conference	16th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2017
Period	1/01/17 → …
Other	May 8-12 2017

Access to Document

10.48550/arXiv.1703.01358

Cite this

Lamont, S., Aslanides, J., Leike, J., & Hutter, M. (2017). Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation. In E. D. M. W. K. L. . S. D. (Ed.), Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS (To be checked ed., pp. 1589-1591). IFAAMAS (International Foundation for Autonomous Agents and Multiagent Systems). https://doi.org/10.48550/arXiv.1703.01358

Lamont, Sean ; Aslanides, John ; Leike, Jan et al. / Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS. editor / E Durfee, M Winikoff, K Larson & S Das. To be checked. ed. TBC : IFAAMAS (International Foundation for Autonomous Agents and Multiagent Systems), 2017. pp. 1589-1591

@inproceedings{0368f1d7db084d2ba16d5d8d7bfd08eb,

title = "Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation",

abstract = "In recent years, work has been done to develop the theory of General Reinforcement Learning (GRL). However, there are no examples demonstrating the known results regarding generalised discounting. We have added to the GRL simulation platform (AIXIjs) the functionality to assign an agent arbitrary discount functions, and an environment which can be used to determine the effect of discounting on an agent's policy. Using this, we investigate how geometric, hyperbolic and power discounting affect an informed agent in a simple MDP. We experimentally reproduce a number of theoretical results, and discuss some related subtleties. It was found that the agent's behaviour followed what is expected theoretically, assuming appropriate parameters were chosen for the Monte-Carlo Tree Search (MCTS) planning algorithm.",

author = "Sean Lamont and John Aslanides and Jan Leike and Marcus Hutter",

year = "2017",

doi = "10.48550/arXiv.1703.01358",

language = "English",

isbn = "978-151085507-6",

pages = "1589--1591",

editor = "{E Durfee, M Winikoff, K Larson & S Das}",

booktitle = "Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS",

publisher = "IFAAMAS (International Foundation for Autonomous Agents and Multiagent Systems)",

edition = "To be checked",

note = "16th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2017 ; Conference date: 01-01-2017",

}

Lamont, S, Aslanides, J, Leike, J & Hutter, M 2017, Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation. in EDMWKLSD (ed.), Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS. To be checked edn, IFAAMAS (International Foundation for Autonomous Agents and Multiagent Systems), TBC, pp. 1589-1591, 16th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2017, 1/01/17. https://doi.org/10.48550/arXiv.1703.01358

Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation. / Lamont, Sean; Aslanides, John; Leike, Jan et al.
Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS. ed. / E Durfee, M Winikoff, K Larson & S Das. To be checked. ed. TBC: IFAAMAS (International Foundation for Autonomous Agents and Multiagent Systems), 2017. p. 1589-1591.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation

AU - Lamont, Sean

AU - Aslanides, John

AU - Leike, Jan

AU - Hutter, Marcus

PY - 2017

Y1 - 2017

N2 - In recent years, work has been done to develop the theory of General Reinforcement Learning (GRL). However, there are no examples demonstrating the known results regarding generalised discounting. We have added to the GRL simulation platform (AIXIjs) the functionality to assign an agent arbitrary discount functions, and an environment which can be used to determine the effect of discounting on an agent's policy. Using this, we investigate how geometric, hyperbolic and power discounting affect an informed agent in a simple MDP. We experimentally reproduce a number of theoretical results, and discuss some related subtleties. It was found that the agent's behaviour followed what is expected theoretically, assuming appropriate parameters were chosen for the Monte-Carlo Tree Search (MCTS) planning algorithm.

AB - In recent years, work has been done to develop the theory of General Reinforcement Learning (GRL). However, there are no examples demonstrating the known results regarding generalised discounting. We have added to the GRL simulation platform (AIXIjs) the functionality to assign an agent arbitrary discount functions, and an environment which can be used to determine the effect of discounting on an agent's policy. Using this, we investigate how geometric, hyperbolic and power discounting affect an informed agent in a simple MDP. We experimentally reproduce a number of theoretical results, and discuss some related subtleties. It was found that the agent's behaviour followed what is expected theoretically, assuming appropriate parameters were chosen for the Monte-Carlo Tree Search (MCTS) planning algorithm.

U2 - 10.48550/arXiv.1703.01358

DO - 10.48550/arXiv.1703.01358

M3 - Conference contribution

SN - 978-151085507-6

SP - 1589

EP - 1591

BT - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS

A2 - null, E Durfee, M Winikoff, K Larson & S Das

PB - IFAAMAS (International Foundation for Autonomous Agents and Multiagent Systems)

CY - TBC

T2 - 16th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2017

Y2 - 1 January 2017

ER -

Lamont S, Aslanides J, Leike J, Hutter M. Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation. In EDMWKLSD, editor, Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS. To be checked ed. TBC: IFAAMAS (International Foundation for Autonomous Agents and Multiagent Systems). 2017. p. 1589-1591 doi: 10.48550/arXiv.1703.01358

Generalised Discount Functions applied to a Monte-Carlo AI(mu) Implementation

Abstract

Conference

Access to Document

Fingerprint

Cite this