D1.2.3 Science Requirements Document (Mar '22)

S. Smartt

The LSST:UK Science Requirements Document, also known as the SRD, is maintained in a wiki page on the LSST:UK Science Centre’s Confluence wiki. Deliverable D1.2.3 comprises the SRD as of March 2022.

It lists the updates to the SRD from its deliverable D1.2.2 version. It also contains the PDF snapshot of the wiki that forms deliverable D1.2.3 following its review and subsequent update.

D2.1.5 Data Facilities Transition Plan

G. Beckett, P Clarke

This describes the work that needs to be done within LSST:UK, to ensure that the UK Consortium can deliver effectively on the in-kind commitment to complete 25% of the Rubin Observatory Data Release Processing (DRP) workload. It also documents the required roles and resource levels, during Rubin Operations, based on guidance provided by the Rubin Data Management team, US and French Data Facilities Coordinators, and contacts within the IRIS programme.

It begins by describing the context of this work and summarising initial activities that have been possible with in-kind effort from GridPP, working with peers in the USA and France, during the latter part of 2021.

It then proposes a high-level work plan for DRP commissioning contributions that are needed, from LSST:UK, in the remaining 12 months or so of Phase B. The work plan is aligned with a Rubin Observatory in-preparation document [RTN-021] and assumes telescope operations from April 2024.

Finally it describes the anticipated resource requirements (skills and effort levels) for during Rubin Operations (Phase C and D), plus the immediate resource requirements to complete the proposed Phase B work plan.

(Note: This deliverable is available only to LSST:UK project members at this time.)

D2.2.3 Ingestion software for ancillary surveys

Mike Read, George Beckett, Craig Manzi

The UK-DAC has been tasked with hosting and serving datasets from external surveys and those produced by LSST:UK work packages. This document describes the initial investigations into ingesting survey catalogues and details the workflows and software required. Two tests of preparing and ingesting data are discussed together with the priorities and next steps for this work.

Ancillary catalogues that need to be hosted alongside the LSST Data Releases will typically be hosted in Qserv, the bespoke, distributed, relational database being developed by the Rubin Observatory. While some archives provide a bulk-download facility, this is not universally the case. The DAC team is likely to need to collaborate with a small number of archive providers to obtain a copy of the ancillary catalogues they host.

There are likely to be differences between the form and format of ancillary catalogues from different archives (for example, related to treatment of null values) and an understanding of these differences is required to successfully ingest and publish the data.  

Catalogue ingestion is a resource-intensive and time-consuming task, thus it is worthwhile to invest effort to carefully prepare each catalogue to reduce the risk of problems being encountered late on in the process. The Qserv administrator needs to make decisions regarding the partitioning of data and indexing of the catalogue which will have long-term implications on the utility of the data. At the time of writing, a strategy for making such decisions is still being developed, and we expect this to be informed by the Rubin Data Management team as they scale up their use of Qserv to LSST-scale datasets.

At the time of writing, a catalogue needs to be available as (or converted to) a CSV/ TSV format, to be ingested, even though the CSV/TSV format is inefficient for transferring and handling substantial catalogues. This is related to a limitation in the underlying Qserv database architecture (MySQL).

The ingestion process has the potential to consume twice as much storage as the catalogue itself (during ingestion), though tactics exist for reducing the storage requirement.

The Rubin Data Management team is developing a suite of tools to help with ingestion of catalogues. These can simplify and automate the process. The UK DAC team has supplemented this suite with scripts to automate the preparation of ancillary datasets.

The UK DAC team plans to ingest and publish a number of survey catalogues, over the coming months, including ancillary surveys and previous, user-generated products from DEV activities. This work will provide the prerequisite experience for managing LSST-scale catalogues during Operations.