Introduction
The week of 22-26 July saw the first new style Rubin Community Workshop held at SLAC. The change in title, from the Project and Community Workshop of previous years, reflects a change in emphasis as the start of survey approaches: while these annual workshops will continue to include Rubin status update and provide an opportunity for interaction with Observatory staff, the workshops will now focus more on the data rights community and its science. Slides and recordings from sessions will be made available through the workshop website to those who registered to attend.
Closer to home, and a week earlier, NAM 2024 featured a pair of sessions on Preparing for UK involvement in early science with the Rubin LSST, which featured a range of introductory/background material aimed at the wider NAM audience, plus talks on several of the UK’s in-kind contributions and preparatory science work underway in the UK. Links to the presentations are presented below.
LSST Solar System Readiness Sprint – sign up now
The Solar System Science Collaboration (SSSC) is having our LSST Solar System Readiness Sprint at the University of Oxford this year on 25-27 September. Interested LSST:UK consortium members can sign up by 7 August.
The goal is to get the SSSC together to spend three dedicated days preparing for the start of Rubin Observatory science operations. After a series of introductory talks and unconference sessions, participants will break into working groups to work on mini-projects ranging from software development to proposal writing.
There is no registration fee. We do have a small amount of funding to help support travel and accommodation for collaboration members who need travel support for early-career researchers and researchers from historically under represented groups within astronomy and planetary science
Operations Rehearsal for Lasair: Lessons Learned
The LSST ran a rehearsal of the data pipeline, with simulated data sent from the summit to Santiago to California, with alerts distributed out to the community brokers in real time. Over three Chilean nights, 9.2 million alerts were sent, concerning 1.5 million unique objects. The difference is because most objects were ‘observed’ many times. There was 0.15 terabytes of cutout images sent. The Lasair broker received and processed all the data without failing. The trans-continental data flow and the replication subsystem (Mirrormaker) was able to keep up with a lag of at most a few minutes; however the Ingest subsystem – loading of data into the two Cassandra databases – was not as fast as the Lasair team would have liked. The Sherlock and Filter subsystems kept up well with the speed of the Ingest.
For six years, Lasair has been built and prototyped around the ZTF transient stream – from Palomar rather than Chile – but this Rehearsal was about ten times the rate of alerts, so a signal increase in bandwidth. The team deployed large Kafka and Cassandra clusters, and scaled up some of the instances from the minimal development deployment. We found that the cutout images will be a major storage cost – about 20 terabytes – even though only three months will be kept instead of ten years. However the LSST project will keep the full archive of cut outs, and we expect to be able to fetch from there in the rare cases where older cut outs are wanted.
We found that cut-out images will be a major storage cost, even though only three months of data will be kept
During the Chilean night, the trans-continental networks carrying the alerts will carry an average of 25 megabytes per second, with peak rates of hundreds of megabytes per second. However the internal network connecting the Lasair subsystems will be carrying an even higher load: receiving from Rubin, pushing to both cassandra databases, pushing back to Kafka, pushing to the relational database, pushing out Kafka streams resulting from user filters. In the figure below can be see the rates of the Ingest (green), Sherlock (yellow), and Filter (cyan) subsystems over the last day of the rehearsal. There is a clear change in behaviour; at first the Lasair processsing is happening in parallel with the trans-continental data flow, and after 11am the flow has stopped and it is just Lasair processing. There will be further investigations to understand this behaviour.
Lasair uses a NoSQL database called Cassandra. Even more so than relational databases, the key/index structure determines efficiency. For example, if we wish to delete cutout images that are more than three months old, then the timestamp of the cutout must be in the list of keys; but at the same time we want incoming cutouts to be spread between the multiple nodes of the Cassandra cluster, not all going to the same node. The first requirement involves the ‘clustering key’, and the second involves the ‘partition key’, requiring a deep understanding of the database internals.
The Lasair architecture can use multiple VMs to increase speed of any specific subsystem, each VM can have multiple CPUs, and the processing software is built with multiprocessing in mind. The objective is to have the ability to add cloud resources and obtain higher processing speed. Before this Rehearsal, the Sherlock VMs were redeployed with multiple CPU, resulting in higher performance. There is a lot of work ahead profiling and tuning the code and the cloud deployment to be able to reliably handle the enormous data deluge from Rubin.
Photometric galaxy clustering with HSC - update
The following is a brief summary and update on an in-kind contribution being carried out by myself (Thomas Cornish) and PI’d by David Alonso and Boris Leistedt . I also gave a talk on this topic last week at NAM 2024, the slides from which are linked below.
The focus of this contribution is to explore existing and novel techniques for mitigating systematics in galaxy clustering analyses, so that we are able to fully exploit the transformational sensitivity and survey area of LSST and achieve unbiased constraints on cosmology.
LSST:UK at NAM2024
We had a pair of two-hour sessions on the first day of NAM 2024. In addition to background information, primarily aimed at the wider NAM audience, talks covered several of the UK’s in-kind contributions to Rubin operations and a variety of preparatory science activities, with the day ending with a project update from Bob Blum, Rubin Director of Operations. Copies of the slides presented are listed below, while recordings of the sessions may be accessed by those who registered for NAM from this webpage.
Session 1:
Introduction and summary of UK involvement in the Rubin LSST (Bob Mann)
LSST survey strategy and early science plans (Stephen Smartt)
Preparations for Rubin Data Release productions (James Mullaney)
Photometric galaxy clustering with HSC DR3: an LSST precursor study (Tom Cornish)
Enabling early Rubin science with robust cross-matches in the crowded LSST sky (Tom Wilson)
Session 2:
Detecting the Galactic interstellar object population with LSST (Matthew Hopkins)
Eliciting astrophysical distributions from imperfect survey data (Dustin Mason)
Probing galaxy clusters with the XMM Cluster Survey and implications for LSST (Paul Giles)
Higher order statistics in Rubin data (Lea Harscouet)
Rubin Observatory update (Bob Blum)
OR4 reflections - James M to write
Elusive brown dwarfs are focus of latest Rubin science release
Capturing the faint light of distant brown dwarfs will help scientists better understand the formation of the Milky Way – that’s the message in the most recent Rubin press release.
LSST will detect a population of ancient brown dwarfs that is around 20 times bigger than previously seen.
Sometimes referred to as ‘failed stars’, brown dwarfs are too large to be considered planets but do not have sufficient mass to sustain nuclear fusion, which sets them apart from stars.
Despite their perceived shortcomings, brown dwarfs are excellent candidates for understanding the evolution of the Milky Way.
Read the full science release: NSF–DOE Rubin Observatory Will Detect Thousands of Elusive Brown Dwarfs, Unlocking Milky Way Mysteries
Recent LSST:UK Science Centre outputs
The LSST:UK Science Centre has recently produced the following technical reports.
ID | Title | Author(s) | Description |
|---|---|---|---|
D2.1.3: Operational plan for UK Data Facility | G.Beckett, R.Mann | This document contains a high-level, operational plan for the UK Data Facility, which encapsulates significant computational and staff contributions to the Rubin Observatory – focusing on the UK contribution to preparing science-ready data products from telescope observations and of serving those products to the astronomy community via a UK-based data facility. | |
D3.5.3 LSST+VISTA Forced Photometry Module | R.Shirley, M. Banerji | This summarises the work conducted of the LSST and Near IR Data Fusion workpackage in Phase B of the LSST UK project. | |
D2.6.6 Documentation for near-IR and crossmatch data products on UK RSP | G.Beckett, A.Ibsen, M. Read, S.Voutsinas | One of the advantages of LSST:UK operating an Independent Data Access Centre (IDAC) is the opportunity to host additional datasets and tools focused on UK science priorities. In addition to hosting these datasets, the UK IDAC team need to actively publicise and describe them, ensuring that astronomers can grasp the potential benefits of these datasets and confidently integrate them into their research. These datasets are expected to provide a valuable complement to Rubin data products published during the Early Science Program. The deliverable outlines efforts to document two specific UK-provided datasets. The first dataset involves a crossmatch catalogue for the Gaia EDR3 and CatWISE surveys (an early output of the Rubin in-kind contribution, UKD-S9). The second dataset comprises a near-infrared fused dataset that integrates HSC imaging with imaging from several overlapping VISTA surveys (VIKING, VIDEO and VHS). The described work not only enhances the accessibility and usability of the specific datasets but also establishes a methodology for documenting other ancillary datasets and supplementary services. | |
D2.5.3.1: Ingestions of ancillary surveys for crossmatch | M.Read, G.Beckett | This deliverable involves the preparation of input datasets, by the DAC team, to allow the UK In-kind contribution, UKD-S9, to progress experiments with the “Supercrossmatch” algorithm. | |
D1.9.1 LSST:UK Communications Plan | E.O’Kane | This document outlines how we will increase awareness of LSST:UK through external communication and defines our key audiences are. There is great potential for interest in the project beyond the astronomy community and several key audiences are outlined in this Communications Plan. Our plan also covers internal communication activities. A key part of the plan is to equip members of the research team, as ambassadors for LSST:UK, with the skills to talk about the global significance of LSST:UK. | |
D2.5.1.1: Provision of hosting platform for Lasair Version 6 | G.Blow, M.Holliman, G.Beckett | A key responsibility of the DAC team is to provision and support the hosting platform for Lasair. This is expected to be relatively stable, with a requirement for routine maintenance and day-to-day troubleshooting during Phase C, though two significant upgrades are likely to involve reconfiguration of the hosting platform. |
Secondary Mirror brought out of storage
In early July, the summit team brought Rubin’s Secondary Mirror (M2) out of storage to integrate it with the mirror cell. The full M2 assembly consists of the secondary mirror, mirror support system, mirror cell electronics and sensors, thermal control system, and the mirror control system. It is supported by 72 axial actuators and 6 tangential actuators. The M2 assembly will soon be installed on the telescope mount, after which the summit team will re-install the Rubin Commissioning Camera.
Eleanor O'Kane
Forthcoming meetings of interest
Dates, locations and links… The current list of forthcoming meetings is always available on the Relevant Meetings page. You may also wish to check information held on the LSST organisation website LSST-organised events and the LSST Corporation website.
Dates | Meeting Title / Event | Meeting Website/ Contact | Meeting location / venue |
|---|---|---|---|
16/Sep/24—20/Sep/24 | LSST@Europe6 | Spain | Island of La Palma | |
10/Nov/24—14/Nov/24 | Astronomical Data Analysis Software & Systems (ADASS | Malta | Virtual |
Members of the Consortium (not in receipt of travel funding through one of the Science Centre grants) may apply for travel support for meetings of this kind via the LSST:UK Pool Travel Fund. Details are available at Forthcoming LSST-related Meetings
If you have significant news or announcements that are directly relevant to LSST:UK and would like to share them in a future newsletter, contact Eleanor O'Kane (email eokane@roe.ac.uk)