Rapid CT reconstruction on GPU-enabled HPC clusters

Computed Tomography (CT) reconstruction is a computationally and data-intensive process applied across many fields of scientific endeavor, including medical and materials science, as a noninvasive imaging technique. A typical CT dataset obtained with a CCD-based X-ray detector, such as that at the Australian Synchrotron with 4K×4K pixels captured over multiple-view angles, is in the order of 128GB. The reconstructed output volume is in the order 256GB. CT data sizes increase at 1.5 times the number of pixels in the detector, while the data-processing load generally increases as the square of the number of pixels, hence data storage, management and throughput capabilities become paramount. From a computational perspective, CT reconstruction is particularly well suited to mass parallelisation whereby the problem can be decomposed into many smaller independent parts. We have achieved significant performance gains by adapting our XLI software algorithms to a two-level parallelisation scheme, utilising multiple CPU cores and multiple GPUs on a single machine. In turn, where data sizes become prohibitively large to be processed on a single machine, we have developed an integrated CT reconstruction software system that is able to scale up and be deployed onto large GPU-enabled HPC clusters. We present here the results of reconstructing large CT datasets using our XLI software on both the CSIRO GPU cluster and the new MASSIVE-1 cluster located at the Australian Synchrotron. Both of these clusters provide high-end compute nodes with multiple GPUs coupled by high-speed interconnect and IO capabilities which combine to allow rapid CT reconstruction. Provided in this paper are examples of the application of the developed tools to the reconstruction of large CT datasets collected both at synchrotrons and with laboratory-based CT scanners.