Abstract
We review structure and frameworks behind the SkyMapper Science Data Pipeline (SDP), and consider the challenges of deploying on two disparate platforms: a publicly shared, massively parallel, queue-scheduled compute fabric, and a dedicated NUMA-based, multi-core, mini-supercomputer. Concepts reviewed include a) how to impose a layer of central operator control over hundreds of jobs of varying type and CPU/IO profile, all running concurrently and at different stages in their logic, b) how to maintain configuration control in an ever-changing algorithmic environment while not giving up ease of build and deployment, and c) how to configure a NUMA-architected machine for optimal cache buffer usage, process-to-memory locality, and user/system CPU cycle ratio.
| Original language | English |
|---|---|
| Pages (from-to) | 393-397 |
| Journal | Astronomical Data Analysis Software and Systems XXV, ASP Conference Series, Vol. 512 |
| Publication status | Published - 2017 |
| Event | 25th Annual Conference on Astronomical Data Analysis Software and Systems (ADASS XXV) - Sydney, Australia, Australia Duration: 1 Jan 2015 → … https://adsabs.harvard.edu/cgi-bin/nph-toc_query?journal=ASPC.&volume=512&fulltoc=YES |