Developing Data Processing Pipelines for Massive Sky Surveys - Lessons Learned from SkyMapper

Christian Wolf; Lance Luvaul; Christopher Onken; Jonathan Smillie; Marc White

Developing Data Processing Pipelines for Massive Sky Surveys - Lessons Learned from SkyMapper

Christian Wolf, Lance Luvaul, Christopher Onken, Jonathan Smillie, Marc White

Advanced Instrumentation and Technology Centre

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The SkyMapper Survey led by RSAA at ANU started in 2014 to map the Southern sky in six spectral passbands. It will identify over 100 million celestial objects and be the first deep and detailed digital resource covering the entire Southern sky. Here we report on lessons learned from the ongoing effort of developing its Science Data Pipeline. How reliably can the required functionality be anticipated? What issues are posed by dependencies? And, is a supercomputer the best platform?

Original language	English
Title of host publication	Astronomical Data Analysis Software and Systems XXV, ASP Conference Series, Vol. 512
Editors	N.P.F. Lorente, K. Shortridge, R. Wayth
Place of Publication	Michigan
Publisher	Astronomical Society of the Pacific
Pages	289-292
Edition	1
ISBN (Print)	978-1-58381-909-8
Publication status	Published - 2017
Event	25th Annual Conference on Astronomical Data Analysis Software and Systems (ADASS XXV) - Sydney, Australia, Australia Duration: 1 Jan 2015 → … https://adsabs.harvard.edu/cgi-bin/nph-toc_query?journal=ASPC.&volume=512&fulltoc=YES

Conference

Conference	25th Annual Conference on Astronomical Data Analysis Software and Systems (ADASS XXV)
Country/Territory	Australia
Period	1/01/15 → …
Other	25-29 October 2015
Internet address	https://adsabs.harvard.edu/cgi-bin/nph-toc_query?journal=ASPC.&volume=512&fulltoc=YES

Cite this

@inproceedings{879e08445f33409b8b226e598b03bc61,

title = "Developing Data Processing Pipelines for Massive Sky Surveys - Lessons Learned from SkyMapper",

abstract = "The SkyMapper Survey led by RSAA at ANU started in 2014 to map the Southern sky in six spectral passbands. It will identify over 100 million celestial objects and be the first deep and detailed digital resource covering the entire Southern sky. Here we report on lessons learned from the ongoing effort of developing its Science Data Pipeline. How reliably can the required functionality be anticipated? What issues are posed by dependencies? And, is a supercomputer the best platform?",

author = "Christian Wolf and Lance Luvaul and Christopher Onken and Jonathan Smillie and Marc White",

year = "2017",

language = "English",

isbn = "978-1-58381-909-8",

pages = "289--292",

editor = "{N.P.F. Lorente, K. Shortridge, R. Wayth}",

booktitle = "Astronomical Data Analysis Software and Systems XXV, ASP Conference Series, Vol. 512",

publisher = "Astronomical Society of the Pacific",

edition = "1",

note = "25th Annual Conference on Astronomical Data Analysis Software and Systems (ADASS XXV) ; Conference date: 01-01-2015",

url = "https://adsabs.harvard.edu/cgi-bin/nph-toc_query?journal=ASPC.&volume=512&fulltoc=YES",

}

Wolf, C, Luvaul, L, Onken, C, Smillie, J & White, M 2017, Developing Data Processing Pipelines for Massive Sky Surveys - Lessons Learned from SkyMapper. in NPFLKSRW (ed.), Astronomical Data Analysis Software and Systems XXV, ASP Conference Series, Vol. 512. 1 edn, Astronomical Society of the Pacific, Michigan, pp. 289-292, 25th Annual Conference on Astronomical Data Analysis Software and Systems (ADASS XXV), Australia, 1/01/15.

Developing Data Processing Pipelines for Massive Sky Surveys - Lessons Learned from SkyMapper. / Wolf, Christian; Luvaul, Lance; Onken, Christopher et al.
Astronomical Data Analysis Software and Systems XXV, ASP Conference Series, Vol. 512. ed. / N.P.F. Lorente, K. Shortridge, R. Wayth. 1. ed. Michigan: Astronomical Society of the Pacific, 2017. p. 289-292.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Developing Data Processing Pipelines for Massive Sky Surveys - Lessons Learned from SkyMapper

AU - Wolf, Christian

AU - Luvaul, Lance

AU - Onken, Christopher

AU - Smillie, Jonathan

AU - White, Marc

PY - 2017

Y1 - 2017

N2 - The SkyMapper Survey led by RSAA at ANU started in 2014 to map the Southern sky in six spectral passbands. It will identify over 100 million celestial objects and be the first deep and detailed digital resource covering the entire Southern sky. Here we report on lessons learned from the ongoing effort of developing its Science Data Pipeline. How reliably can the required functionality be anticipated? What issues are posed by dependencies? And, is a supercomputer the best platform?

AB - The SkyMapper Survey led by RSAA at ANU started in 2014 to map the Southern sky in six spectral passbands. It will identify over 100 million celestial objects and be the first deep and detailed digital resource covering the entire Southern sky. Here we report on lessons learned from the ongoing effort of developing its Science Data Pipeline. How reliably can the required functionality be anticipated? What issues are posed by dependencies? And, is a supercomputer the best platform?

M3 - Conference contribution

SN - 978-1-58381-909-8

SP - 289

EP - 292

BT - Astronomical Data Analysis Software and Systems XXV, ASP Conference Series, Vol. 512

A2 - null, N.P.F. Lorente, K. Shortridge, R. Wayth

PB - Astronomical Society of the Pacific

CY - Michigan

T2 - 25th Annual Conference on Astronomical Data Analysis Software and Systems (ADASS XXV)

Y2 - 1 January 2015

ER -

Developing Data Processing Pipelines for Massive Sky Surveys - Lessons Learned from SkyMapper

Abstract

Conference

Fingerprint

Cite this