D1.2.3 Science Requirements Document (Mar '22)

S. Smartt

The LSST:UK Science Requirements Document, also known as the SRD, is maintained in a wiki page on the LSST:UK Science Centre’s Confluence wiki. Deliverable D1.2.3 comprises the SRD as of March 2022.

It lists the updates to the SRD from its deliverable D1.2.2 version. It also contains the PDF snapshot of the wiki that forms deliverable D1.2.3 following its review and subsequent update.

D2.1.5 Data Facilities Transition Plan

G. Beckett, P Clarke

This describes the work that needs to be done within LSST:UK, to ensure that the UK Consortium can deliver effectively on the in-kind commitment to complete 25% of the Rubin Observatory Data Release Processing (DRP) workload. It also documents the required roles and resource levels, during Rubin Operations, based on guidance provided by the Rubin Data Management team, US and French Data Facilities Coordinators, and contacts within the IRIS programme.

It begins by describing the context of this work and summarising initial activities that have been possible with in-kind effort from GridPP, working with peers in the USA and France, during the latter part of 2021.

It then proposes a high-level work plan for DRP commissioning contributions that are needed, from LSST:UK, in the remaining 12 months or so of Phase B. The work plan is aligned with a Rubin Observatory in-preparation document [RTN-021] and assumes telescope operations from April 2024.

Finally it describes the anticipated resource requirements (skills and effort levels) for during Rubin Operations (Phase C and D), plus the immediate resource requirements to complete the proposed Phase B work plan.

(Note: This deliverable is available only to LSST:UK project members at this time.)

D2.2.3 Ingestion software for ancillary surveys

Mike Read, George Beckett, Craig Manzi

The UK-DAC has been tasked with hosting and serving datasets from external surveys and those produced by LSST:UK work packages. This document describes the initial investigations into ingesting survey catalogues and details the workflows and software required. Two tests of preparing and ingesting data are discussed together with the priorities and next steps for this work.

Ancillary catalogues that need to be hosted alongside the LSST Data Releases will typically be hosted in Qserv, the bespoke, distributed, relational database being developed by the Rubin Observatory. While some archives provide a bulk-download facility, this is not universally the case. The DAC team is likely to need to collaborate with a small number of archive providers to obtain a copy of the ancillary catalogues they host.

There are likely to be differences between the form and format of ancillary catalogues from different archives (for example, related to treatment of null values) and an understanding of these differences is required to successfully ingest and publish the data.  

Catalogue ingestion is a resource-intensive and time-consuming task, thus it is worthwhile to invest effort to carefully prepare each catalogue to reduce the risk of problems being encountered late on in the process. The Qserv administrator needs to make decisions regarding the partitioning of data and indexing of the catalogue which will have long-term implications on the utility of the data. At the time of writing, a strategy for making such decisions is still being developed, and we expect this to be informed by the Rubin Data Management team as they scale up their use of Qserv to LSST-scale datasets.

At the time of writing, a catalogue needs to be available as (or converted to) a CSV/ TSV format, to be ingested, even though the CSV/TSV format is inefficient for transferring and handling substantial catalogues. This is related to a limitation in the underlying Qserv database architecture (MySQL).

The ingestion process has the potential to consume twice as much storage as the catalogue itself (during ingestion), though tactics exist for reducing the storage requirement.

The Rubin Data Management team is developing a suite of tools to help with ingestion of catalogues. These can simplify and automate the process. The UK DAC team has supplemented this suite with scripts to automate the preparation of ancillary datasets.

The UK DAC team plans to ingest and publish a number of survey catalogues, over the coming months, including ancillary surveys and previous, user-generated products from DEV activities. This work will provide the prerequisite experience for managing LSST-scale catalogues during Operations.

D2.5.5 Interface Requirements for DAC-DEV interactions

George Beckett, Mike Read

A unique element of the LSST:UK programme, and a particular advantage for UK astronomy, is a portfolio of research and development activities (commonly referred to as DEV activities), which are funded as part of the Phase B programme and are working to enhance and extend the astronomy potential of the baseline Rubin Observatory software, services, and data products, for priority astronomy topics within the UK.

Some of these DEV activities need to interface with the LSST:UK Data Access Centre (DAC), in order that their outputs can be made available to science users during operations. For example, a DEV activity may produce a piece of software that enables astronomers to analyse LSST data in a different way or may produce a User-generated Data Product (in Rubin Observatory nomenclature) to be co-hosted alongside official LSST Data Release products.

A goal of the WP2.5 team has been to engage early with the affected DEV teams to develop, test and document how the DEV products will interact with the DAC and be supported by the DAC team. An important element of this activity is the production of a set of DAC-DEV Interface Definitions, which will define the interfaces between the DAC and the DEV products in sufficient detail to ensure seamless operation, and to document where the responsibility for different aspects of the interface will lie (that is, with the DAC team or the DEV team).

The DAC team is working with these DEV activities to understand, agree and develop the required interfaces. The topical state of this work is captured in two documents:

• WP3.11 DAC-DEV Interface Requirements

• WP3.5 DAC-DEV Interface Requirements

and are summarised in this report.