Sparse kernel-SARSA(λ) with an eligibility trace

Matthew Robards; Peter Sunehag; Scott Sanner; Bhaskara Marthi

doi:10.1007/978-3-642-23808-6_1

Sparse kernel-SARSA(λ) with an eligibility trace

Matthew Robards^*, Peter Sunehag, Scott Sanner, Bhaskara Marthi

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Citations (Scopus)

Abstract

We introduce the first online kernelized version of SARSA(λ) to permit sparsification for arbitrary λ for 0 ≤ λ ≤ 1; this is possible via a novel kernelization of the eligibility trace that is maintained separately from the kernelized value function. This separation is crucial for preserving the functional structure of the eligibility trace when using sparse kernel projection techniques that are essential for memory efficiency and capacity control. The result is a simple and practical Kernel-SARSA(λ) algorithm for general 0 ≤ λ ≤ 1 that is memory-efficient in comparison to standard SARSA(λ) (using various basis functions) on a range of domains including a real robotics task running on a Willow Garage PR2 robot.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Proceedings
Pages	1-17
Number of pages	17
Edition	PART 3
DOIs	https://doi.org/10.1007/978-3-642-23808-6_1
Publication status	Published - 2011
Event	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2011 - Athens, Greece Duration: 5 Sept 2011 → 9 Sept 2011

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 3
Volume	6913 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2011
Country/Territory	Greece
City	Athens
Period	5/09/11 → 9/09/11

Access to Document

10.1007/978-3-642-23808-6_1

Cite this

Robards, M., Sunehag, P., Sanner, S., & Marthi, B. (2011). Sparse kernel-SARSA(λ) with an eligibility trace. In Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Proceedings (PART 3 ed., pp. 1-17). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6913 LNAI, No. PART 3). https://doi.org/10.1007/978-3-642-23808-6_1

Robards, Matthew ; Sunehag, Peter ; Sanner, Scott et al. / Sparse kernel-SARSA(λ) with an eligibility trace. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Proceedings. PART 3. ed. 2011. pp. 1-17 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 3).

@inproceedings{bb063ac4dae54688bb4f6f270acf9632,

title = "Sparse kernel-SARSA(λ) with an eligibility trace",

abstract = "We introduce the first online kernelized version of SARSA(λ) to permit sparsification for arbitrary λ for 0 ≤ λ ≤ 1; this is possible via a novel kernelization of the eligibility trace that is maintained separately from the kernelized value function. This separation is crucial for preserving the functional structure of the eligibility trace when using sparse kernel projection techniques that are essential for memory efficiency and capacity control. The result is a simple and practical Kernel-SARSA(λ) algorithm for general 0 ≤ λ ≤ 1 that is memory-efficient in comparison to standard SARSA(λ) (using various basis functions) on a range of domains including a real robotics task running on a Willow Garage PR2 robot.",

author = "Matthew Robards and Peter Sunehag and Scott Sanner and Bhaskara Marthi",

year = "2011",

doi = "10.1007/978-3-642-23808-6_1",

language = "English",

isbn = "9783642238079",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

number = "PART 3",

pages = "1--17",

booktitle = "Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Proceedings",

edition = "PART 3",

note = "European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2011 ; Conference date: 05-09-2011 Through 09-09-2011",

}

Robards, M, Sunehag, P, Sanner, S & Marthi, B 2011, Sparse kernel-SARSA(λ) with an eligibility trace. in Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Proceedings. PART 3 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 3, vol. 6913 LNAI, pp. 1-17, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2011, Athens, Greece, 5/09/11. https://doi.org/10.1007/978-3-642-23808-6_1

Sparse kernel-SARSA(λ) with an eligibility trace. / Robards, Matthew; Sunehag, Peter; Sanner, Scott et al.
Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Proceedings. PART 3. ed. 2011. p. 1-17 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6913 LNAI, No. PART 3).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Sparse kernel-SARSA(λ) with an eligibility trace

AU - Robards, Matthew

AU - Sunehag, Peter

AU - Sanner, Scott

AU - Marthi, Bhaskara

PY - 2011

Y1 - 2011

N2 - We introduce the first online kernelized version of SARSA(λ) to permit sparsification for arbitrary λ for 0 ≤ λ ≤ 1; this is possible via a novel kernelization of the eligibility trace that is maintained separately from the kernelized value function. This separation is crucial for preserving the functional structure of the eligibility trace when using sparse kernel projection techniques that are essential for memory efficiency and capacity control. The result is a simple and practical Kernel-SARSA(λ) algorithm for general 0 ≤ λ ≤ 1 that is memory-efficient in comparison to standard SARSA(λ) (using various basis functions) on a range of domains including a real robotics task running on a Willow Garage PR2 robot.

AB - We introduce the first online kernelized version of SARSA(λ) to permit sparsification for arbitrary λ for 0 ≤ λ ≤ 1; this is possible via a novel kernelization of the eligibility trace that is maintained separately from the kernelized value function. This separation is crucial for preserving the functional structure of the eligibility trace when using sparse kernel projection techniques that are essential for memory efficiency and capacity control. The result is a simple and practical Kernel-SARSA(λ) algorithm for general 0 ≤ λ ≤ 1 that is memory-efficient in comparison to standard SARSA(λ) (using various basis functions) on a range of domains including a real robotics task running on a Willow Garage PR2 robot.

UR - http://www.scopus.com/inward/record.url?scp=80052398950&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-23808-6_1

DO - 10.1007/978-3-642-23808-6_1

M3 - Conference contribution

SN - 9783642238079

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 1

EP - 17

BT - Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Proceedings

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2011

Y2 - 5 September 2011 through 9 September 2011

ER -

Robards M, Sunehag P, Sanner S, Marthi B. Sparse kernel-SARSA(λ) with an eligibility trace. In Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Proceedings. PART 3 ed. 2011. p. 1-17. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 3). doi: 10.1007/978-3-642-23808-6_1

Sparse kernel-SARSA(λ) with an eligibility trace

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this