Introduction

…Welcome to the first LSST:UK newsletter of 2021, which promises to be a crucial year for UK preparation for participation in the Rubin LSST - and, hopefully, an easier year for us all than was 2020.

In this newsletter we present an update on two important proposals: the UK’s proposed in-kind contributions to Rubin LSST operations, which will secure our data rights; and the community broker proposal that will provide the Lasair system with a copy of the full Rubin LSST alert stream. The outcomes from these two proposals will determine the UK role in the Rubin LSST and mark two important milestones due during 2021. Further items report on the latest challenge run by the Dark Energy Science Collaboration - testing methods for assigning galaxies to redshift bins to maximise the scientific return from future tomographic weak lensing analyses by DESC - and introduce the new logo of the Vera C. Rubin Observatory. We also solicit the community’s input to the planning of a virtual LSST:UK All-Hands Meeting to be held in April or May, and advertise the latest technical report arising for the STFC-funded LSST:UK Science Centre programme.

Those with ideas for future newsletter items should contact the LSST:UK Project Managers (George Beckett and Terry Sloan: lusc_pm@mlist.is.ed.ac.uk), while everyone is encouraged to subscribe to the Rubin Observatory Digest for more general news from the US observatory team. Given the Christmas and New Year break, the next newsletter will be a December/January issue, to be circulated towards the end of January.

Bob Mann

A new standard for photometry

Building the Vera C. Rubin Observatory is an immense engineering and scientific challenge and software engineering in particular is going to be a significant aspect of preparation and operations. Software development is of course well under way in both the core US teams and the UK. Due to the fact early software versions have already been used to process the HyperSuprimeCam imaging we have a large sky area of data on which to test performance and start to understand how to use the eventual LSST data products. For various reasons the LSST team decided to build a bespoke software stack built on C base functions and all tied together with Python. Starting from scratch in this way has the key benefit that the software can be relatively self contained and consistent. It does however remind me of an old XKCD comic that serves as a warning here:

Image Removed

There are however reasons to be optimistic that the LSST software stack will become widely adopted by the general astronomical community and hopefully 'the' standard for optical and near infrared photometry. LSST data will be used by the majority of working astronomers in the coming decade and the sheer scale of the project means many other telescopes may well choose to leverage the huge amount of work that has gone into the LSST photometry pipelines for their own data. For this reason myself, Manda Banerji, and a team based between Southampton and Cambridge, are working on using the stack to process near infrared imaging from the VISTA telescope using the LSST software stack. Given that other telescope teams such as the Dark Energy Survey have started to develop the LSST stack for their own data there is aso a great hope that by processing your imaging with the LSST code you not only open up fusion between your own data and LSST data but you also make other telescope data accesible in the same framework.

Image Removed

James Mullaney, at the University of Sheffield, has produced a general package called 'obs_necam' which helps teams such as ours start to to use the stack with 'Any Camera'. I am hopeful that the whole astronomy community can work towards using the LSST software as a kind of standard. This will mean that when LSST observations begin, utilising imaging from a large number of other telescopes will be possible and accessing VISTA imaging and photometry will be straightforward in the standard LSST formats people will be familiar with. This will enable the legacy of VISTA and other existing and indeed upcoming datasets to be optimally harnessed. Conducting forced photometry, multiband colour and shape measurements, in addition to standard cutout and imaging tasks might be simple tasks all within the same software framework. This will make multi-wavelength astronomy with Vera C. Rubin, VISTA, and other instruments easier and more widely used.

Raphael Shirley

Work Package highlights from STFC Project Assurance Report (PAR)

Work Package

Highlights

WP 1.4 Coordination of LSST:UK Contributions to Commissioning

Issued to the UK community an EoI Call for Rubin Observatory Commissioning. An accompanying briefing paper aims to assist colleagues in preparing for and contributing to commissioning. This is in the context of the current pause in Rubin construction, and the ongoing negotiations of the UK’s in-kind contributions to the Rubin Observatory and LSST Science Collaborations.  This call closed on October 30^th 2020.  

WP 2.1 DAC Management

Data Access Centre requirements have been integrated into the LSST:UK Science Requirements Document (Version 3.0, April 2020).

WP2.2 Data Ingestion and Publication

A set of data transfer and ingest experiments were undertaken, between IN2P3 (the Rubin data-processing facility in France) and the Royal Observatory Edinburgh. These are documented in D2.2.1. Following on from this, work is underway to ingest ancillary data for the next iteration of the DAC (UKDAC1), including ingestion of PanSTARSS DR2 and ZTF DR2.

WP2.4 Provision of the DAC Platform

The DAC team has begun collaborating with Rubin Observatory staff in the United States and France around the development of the Rubin Science Platform (RSP). WP2.4 staff are customising the baseline RSP for UK-specific ancillary datasets and IRIS IAM authentication, towards the roll-out of an updated RSP in the UK DAC in 2021Q1.

WP2.5 Science Support

Deliverable D2.5.1 “Training resources for LSST:UK DAC users” was completed. This describes an initial release of documentation for users of current and future services accessed via the UK’s LSST Data Access Centre (DAC). This documentation release is necessarily limited in scope given that the UK DAC is still being developed. It comprises existing documentation for the Lasair alert broker and a very preliminary set of documentation for the LSST Science Platform (LSP). The LSP is the set of data services to be provided by the Rubin Observatory to support analysis of LSST data products.

WP2.3/3.2 Lasair

The team has: completed and released Lasair v2.0; tested the Cassandra database architecture on IRIS and reviewed it as a future implementation for LSST scale alerts (Report in preparation at the time of writing); completed a science requirements and functionality review led by the LSST:UK PoCs; tested the RAPID light-curve classifier and published a technical summary paper.

WP3.5 LSST and near-infrared data fusion

The WP team has copied over onto the IRIS HPC infrastructure all publicly available VISTA imaging survey datasets and work is now ongoing to develop the pipeline to process the VISTA pixels through the Rubin stack. Current efforts in the WP are focused on understanding photometric calibration issues as well as better understanding the noise properties of the resulting images produced by the pipeline. The current plan is to have a test region in the SXDS field processed and available for scientific validation by December 2020.

WP3.7 Low-surface-brightness science using LSST

Team member Aaron Watkins has been appointed as Deputy Chair of the LSST Low Surface Brightness Working Group. He is working alongside Sarah Brough from the University of New South Wales Australia to help coordinate international community effort in infrastructure development related to intra cluster light and dwarf galaxy science. This includes, although not restricted to, activity related to international LoI proposals. Aaron also gave an update of his WP 3.7 work at the LSB Session of the Rubin Project and Community Workshop in August. He presented results quantifying the over subtraction in different versions of the current data pipeline to an audience of well over 100 participants. Aaron has also given virtual seminars at LJMU and Hertfordshire and to the LSST Galaxies forum and the LSB Challenge 1 meeting, of which he is co-chair.

WP3.9 LSST Point Spread Function, sensor characterisation and modelling

This WP team have established the following.

• Confirmation of optimal gate width settings in e2v CCDs – this WP were working on finding what setting of the CCD gate width parameter was optimal from the perspective of correlation and full well effects. The investigation has shown that one particular setting outperforms all others in most reasonable scenarios, and by coincidence this was the setting already used by LSST camera.

• Confirmation that silicon di-vacancies cause trap-related parallel CTI in e2v CCDs – investigations over the last year have produced strong evidence of the particular type of trap species that dominates in the operating conditions used in the LSST camera. From this, timing recommendations can be made to substantially reduce parallel CTI, an important performance degrading effect. Other CTI mechanisms than trapping exist in these devices and it is not yet fully clear what proportion of CTI is caused by trapping, but in a traditional CCD it is the dominant effect.

• Calibration Improvements in Oxford LSST testbench – due to renewed efforts to carefully track down systematics and calibration errors in our lab equipment and testbench, and thanks to the help of very strong undergraduate project students, this WP have been able to reliably reduce the illumination fluctuations in experiments performed on the OPMD lab testbench. This will aid in future WP efforts that require much more stable operation than was previously possible.

• Evidence that previous assumptions used in the theory of charge trapping in CCDs are incorrect – the WP have noted for quite a while that previous analyses of charge trapping in CCDs contained a somewhat unjustified assumption in the derivation of the equation governing trapping rate that essentially only applies to traps which happen to lie in high charge density areas of the device. Since traps are uniformly distributed in space this assumption does not hold in reality. The WP have made significant progress in developing an analysis method that relaxes this assumption. Results from the WP new analysis method are so far consistent with previous methods, yet have two advantages: a) they pave the way in future to use the same data to extract more detail about charge density than was previously possible, and b) they reduce the unexplained variance in individual trap energy levels that have been measured with the old methods

WP3.10 UK Contributions to DESC Operations

Deliverable D3.10.5 “Processing DC2 data using the LSST DM Stack on UK Facilities, was completed. This deliverable describes how to process the images generated by DESC's Data Challenge 2 using the LSST software pipeline. Instructions for installing the software, setting up a data repository and running each stage of the pipeline are provided. Possibilities for using container and workflow technologies to improve the process are also discussed.

WP3.11 Cross matching and astrometry at LSST depths

The WP team completed their first deliverable, D3.11.1, an investigation into the model for contamination of sources due to crowding at LSST depths. WP 3.11 are confident that they can model the astrometric perturbations and photometric contaminations of sources due to faint, blended objects within their PSFs and have algorithms in place that improve the accuracy of the simulated models across a wide range of signal-to-noise ratios, from bright, photon-dominated objects to (important for LSST) faint, sky background-dominated objects. In addition, this WP have started building the software framework for fully symmetric, many-to-many Bayesian cross-matches. This code will serve as the preliminary test bed for investigations in integrations with the DAC workflow, user interaction with the end products, profiling to LSST data sizes, and an analytic groundwork from which to probe reproducibility, recovery rates, etc.  The WP are also are working with the LSST:UK DAC team to establish a “data challenge” with the preliminary codebase. This involves a full end-to-end test — using Gaia and WISE, with WISE serving as a proxy for LSST in sources-per-PSF space — to verify and test a full-scale DAC-DEV product creation. Finally, this WP have also begun discussions with various US-based teams — both the DM team within LSSTC itself, and science collaborations — on coordinating efforts on understanding the LSST datasets. Links have also been made with the TVS SC, with potential collaboration efforts on the cross-matching of LSST alerts in real-time, the SMWLV SC, focussing on the importance of our cross-match algorithms in crowded Milky Way fields, and the Crowded Field Task Force in the LSSTC DM, again linking the WP 3.11 cross-match expertise but also working with them to ensure that the WP 3.11 implementation uses as robust and precise a description of the LSST data processing as possible.

If you are interested in more detail please contact Terry Sloan via lusc_pm@mlist.is.ed.ac.uk in the first instance.

Terry Sloan

LSST:UK leadership

Here’s a list of significant leadership positions held by members of the LSST:UK consortium in the project and international Science Collaborations. If you are aware of any corrections or additions please contact the LSST:UK Project Managers (George Beckett and Terry Sloan: lusc_pm@mlist.is.ed.ac.uk)

D. Alonso: co-convenor of the DESC Large Scale Structure WG and member of DESC Council;

D. Alonso: member of the LSST DESC Membership Committee

M. Banerji: co-chair of the LSST Galaxies Science Collaboration;

R. Bowler: co-chair of the SED fitting and Photometric Redshifts WG in the LSST Galaxies Science Collaboration;

B. Burningham: co-chair of solar neighbourhood WG in Stars, Milky Way and Local Volume Science Collaboration (from summer 2017);

T. Collett: co-convenor of the LSST DESC Strong Lensing Working Group;

P. Hatfield: co-chair of the Galaxy Environment WG in the LSST Galaxies Science Collaboration;

S. Kaviraj: co-chair of the LSST Galaxies Science Collaboration (from Summer 2018);

S. Kaviraj: member of the Rubin Observatory LSST Contribution Evaluation Committee representing the Galaxies Science Collaboration (from Spring 2020);

B Leistedt: member of the LSST DESC Equality, Diversity & Inclusion Committee;

D. Leonard: member of the LSST DESC Publication Board and Collaboration Council;

C. Lintott: leads the LSST EPO development of Zooniverse as a citizen science platform;

J. Mullaney: Co-Chair of the Active Galactic Nuclei WG in the LSST Galaxies Science Collaboration;

M. Schwamb: co-chair of Solar System Science Collaboration (re-appointed May 2020);

M. Schwamb: member of the Rubin Observatory LSST Contribution Evaluation Committee representing the Solar System Science Collaboration (from Spring 2020);

S. Smartt: member of the LSST Science Advisory Committee (from 2018);

G. Smith: co-chair of the LSST Strong Lensing Science Collaboration;

M. Sullivan: co-chair of DESC Follow-up Task Force;

M. Sullivan: co-lead of the DESC External Synergies Analysis Working Group;

A. Verma: chair of the Strong Lensing Working Group in the Galaxies Science Collaboration;

A. Verma: a member of the Rubin Observatory LSST Contribution Evaluation Committee representing the Strong Lensing Science Collaboration (from Spring 2020);

A. Watkins: co-lead of the LSST LSB challenge 1: "How do LSST algorithms do at detecting LSB sources?" (from March 2020);

A. Watkins: co-chair of the low-surface-brightness working group within the LSST Galaxies Science collaboration (from Autumn 2020);

J. Zuntz: Leader of Lensing/Largescale structure cross-correlation project.

The UK in-kind proposal

As mentioned in the September newsletter, the document outlining the UK’s proposed contribution to Rubin LSST operations was submitted on September 25th. The details of three of those contributions - to Commissioning and to the annual Data Release Processing, plus our operation of an Independent Data Access Centre - were not complete at that time, as they required further discussion with Rubin staff, so we took advantage of an extension mechanism provided by the Observatory and submitted our final proposal on November 6th. A copy of that proposal is available from the LSST:UK Science Working Group wiki page.

That proposal describes a broad range of significant contributions, reflecting the breadth of expertise within the UK community and the scale of our ambitions for UK participation in the Rubin LSST. They can be grouped under two headings:

• The DAC and DEV streams from the LSST:UK Science Centre programme, as originally outlined in our Phase A proposal to STFC in 2014: the DAC activity covers the development and operation of an LSST Data Access Centre in the UK, while the DEV activity comprises a set of software development work packages leading to the production of user-generated Products, to be published through the UK DAC. The in-kind proposal seeks credit for work done on this programme during Phases A and B, and includes their continuation through Phases C and D (i.e. survey operations).

• Several new activities that the Rubin leadership asked us to consider taking on and which, taken together, would help integrate us into the operations consortium, namely: (i) taking a 25% share of the annual Data Release Processing workload; (ii) supporting Education and Public Outreach software developed by the Zooniverse team; (iii) providing a half-time International Contributions Coordinator to the Rubin Director’s office; (iv) contributing to Rubin Commissioning; and (v) supporting a stream of Community Scientists secondments.

This proposal is now being assessed by the Contribution Evaluation Committee and by Observatory staff, which will lead to feedback in February and a recommendation to the US funding agencies of its value in terms of data rights for the UK community. That should, by April, yield an outline of our final agreement, which is then expected to be signed in mid-2021. So, in less than six months' time, we should have confirmation of the scale and scope of UK participation in the Rubin LSST.

Bob Mann

New logo for the Vera C. Rubin Observatory

In early December 2020, the Rubin Observatory unveiled its new logo.

Image Added

According to the press release, the new logo “is a visual representation of Rubin Observatory’s central purpose: to collect light from celestial objects and transform it into data for scientific discovery.” Rubin also released an infographic (see below) that explained the various design elements.

Image Added

Terry Sloan

The LSST:UK Community Alert Broker

R. D. Williams, K. W. Smith, G. Francis, A. Lawrence, S. Smartt, D. R. Young, M. Schwamb, C. Frohmaier, and T. Sloan

The LSST survey is focused on the dynamic sky: a source is not just brightness, but a history of brightness. One major product is the “alerts”, that report new sources and changes in brightness. Most observing nights are expected to produce millions of alerts, each perhaps 50 kB, so the data volume per night can be of order a terabyte. For a scientist to get what they want from this firehose, there will be “community brokers” that ingest the data and allow scientists to utilise it effectively. Only a limited number of brokers can be supported – because of the high data rates – so there is a competition. Thus in December 2020 the LSST:UK submitted a proposal to become a community broker.

Our broker is Lasair[1], the main partners being the University of Edinburgh and Queen’s University Belfast (Lasair means flame or flash in Scottish and Irish Gaelic). Although the LSST survey has not started, there is already a prototype transient stream – ZTF – on which we have built the first versions of Lasair. The architecture of the LSST stream will be similar to ZTF.

Lasair will provide a flexible and powerful platform that will enable worldwide users -- individual users, other projects, and citizen science -- to achieve their own science. Lasair will provide access to a rich variety of added value information and external data sources alongside the alert data, all of which can be interrogated, using queries, filters, watchlists, streaming queries and a programming interface. Lasair uses scalable technology and runs on the STFC-funded IRIS infrastructure. Although powerful, Lasair has an easy on-ramp for scientists from  web pages and simple SQL, then on to Jupyter notebooks on their own machines, and then to high-performance mining co-located with the data. Lasair offers direct access with a staged approach: scientists can start with a simple, immediate mechanism using familiar SQL-like languages. These SQL-like queries can be custom made or users can choose and modify one of our pre-built and tested queries. These queries return an initial selection of objects, based on our rich value-added data content, and users can then run their own local code on the results. Users can build up to running  their own code on both the stream and the database with high-throughput resources in the IRIS cloud. The SQL filters and code can be made public, shared with a group of colleagues, copied, modified, and excellent examples and their outputs are featured on the Lasair web page.

A broad overview of the Lasair design is shown in the figure below.

Image Added

Alerts arrive from the Rubin Observatory at left, and are cached and saved. Several “tagging” systems add value about sky context (Sherlock), external multi-messenger alerts, user-created watchlists of their own sources, classification engines, and featured of the light curves. User queries and filters are processed and results despatched, and the enriched alert stream kept in databases. The alerts can be utilised in several ways: by web, jupyter-style notebook, a programming interface (API), or received as real-time streams.

The Lasair team welcomes astronomers and technologists to have a try at the new (beta) version[1], and report comments and suggestions to lasair-help@lists.roe.ac.uk.

[1] https://lasair-iris.roe.ac.uk/

Roy Williams

DESC Tomographic Challenge

Last year the Dark Energy Science Collaboration (DESC) ran a semi-public analysis challenge, designed to find new methods for assigning galaxies to tomographic redshift bins as optimally as possible.

The main weak lensing methods split galaxies into separate radial bins before correlating those bins with themselves and each other. This is usually done by using a photometric redshift code to get a very approximate redshift and then comparing that to a set of pre-defined nominal bin edges. One challenge is of course to choose these edges in the first place as optimally as possible, to maximize the signal-to-noise of the resulting correlations, but a further complication arises when we use a galaxy shape measurement method called "metacalibration" to measure galaxy shapes in the first place.

This method allows us to find very accurate corrections for selection biases that arise whenever you split galaxies into groups (it turns out that almost any split, e.g. on magnitude, will correlate in some way with the shear of the galaxy, which is what we are trying to measure). But the cost is that we can only split galaxies using fluxes in bands with sufficiently precise PSF measurements, and the u, y, and possibly g bands will not be suitable. To get the advantages of this method we would have to split galaxies using only the riz and maybe g bands.

The tomography challenge asked entrants to attempt this, to maximise the constraining power of the correlations using only a restricted subset of bands and colours. More than twenty entrants submitted methods that first trained on a representative sample and then analysed a test sample to get a score. Methods ranged from machine learning tools with convolutional neural networks, self-organizing maps, or Gaussian processes, to more classic methods optimizing linear colour cut.

We have now analyzed the results, and found them encouraging - many methods managed impressively well-separated galaxy bins even using only the riz bands. This bodes very well for the use of metacalibration in early LSST data. We are now writing up the results in a paper, and evaluating the methods to see how the best ones succeeded.

Joe Zuntz

Preliminary plans for a virtual LSST:UK All-Hands Meeting in April/May 2021

The first LSST:UK All-Hands Meeting (AHM) took place in Cardiff in May 2019. It was a great success, and the consensus of opinion afterwards was that the level of LSST-related activity within the community motivated holding such a meeting annually. In the event, the LSST:UK Executive Group decided not to hold an AHM in 2020, preferring to invest in funding a strong UK presence at the LSST@Europe4 conference then planned to take place in Rome in June 2020. LSST@Europe4 may now take place later this year, but the Exec feel that 2021 is too important a year for UK involvement in the Rubin LSST to pass without a UK consortium meeting.

As noted above, we should know the outline of our in-kind package - and, hence, of our data rights position and our relationship to the operations consortium - by April, so the plan is to hold the LSST:UK AHM in late April or early May, once that information is known. The meeting will take place online, and will include plenary sessions, providing information to the community on our in-kind package and starting preparations for our Phase C funding proposal. It is also likely to include parallel science sessions, but its exact format is still TBD, and will be influenced by the outcome of bids to hold LSST-related sessions at NAM2021 and whether LSST@Europe4 will take place during 2021.

These plans will start to take more definite form in the next month or two, but, for the moment, the Exec are keen to get input from the community on what you would like to see at the 2021 AHM. To that end:

1. We have set up a Doodle poll - https://doodle.com/poll/2dtfzci3zqt4tipn - with dates in the second half of April and the first half of May, on which we invite you to record your availability - and, more importantly, periods of unavailability due to clashes with meetings likely to involve other members of the community. Please use the Comments box to annotate your entries, if appropriate.

2. We solicit suggestions for themes for parallel science sessions and for topics to be covered in the plenary sessions.

3. We seek LSST:UK consortium members willing to contribute some time to serving on the organising committee(s) for the meeting.

Please record any availability/unavailability information on the Doodle by February 19th and feel free to email me (rgm@roe.ac.uk) regarding items 2 and 3. Further information on the AHM plans will appear in later newsletters and/or messages to the lusc_announce mailing list.

Bob Mann

Recent LSST:UK outputs

LSST:UK has recently produced the following technical reports.

Terry Sloan