• 1 Introduction
• 2 LSST Solar System Readiness Sprint – sign up now
• 3 Operations Rehearsal 4 update
• 4 Photometric galaxy clustering with HSC – update
• 5 Operations Rehearsal for Lasair: Lessons learned
• 6 LSST:UK at NAM 2024
• 7 Elusive brown dwarfs are focus of latest Rubin science release
• 8 Recent LSST:UK Science Centre outputs
• 9 Secondary Mirror brought out of storage
• 10 Forthcoming meetings of interest

Open

Introduction

The week of 22-26 July saw the first new style Rubin Community Workshop held at SLAC. The change in title, from the Project and Community Workshop of previous years, reflects a change in emphasis as the start of survey approaches: while these annual workshops will continue to include Rubin status update and provide an opportunity for interaction with Observatory staff, the workshops will now focus more on the data rights community and its science. Slides and recordings from sessions will be made available through the workshop website to those who registered to attend.

Closer to home, and a week earlier, NAM 2024 featured a pair of sessions on Preparing for UK involvement in early science with the Rubin LSST, which featured a range of introductory/background material aimed at the wider NAM audience, plus talks on several of the UK’s in-kind contributions and preparatory science work underway in the UK. Links to the presentations are presented below.

 

Open

Credit: Olivier Bonin/SLAC National Accelerator Laboratory

 

LSST Solar System Readiness Sprint – sign up now

The Solar System Science Collaboration (SSSC) is having our LSST Solar System Readiness Sprint at the University of Oxford this year on 25-27 September. Interested LSST:UK consortium members can sign up by 7 August.

The goal is to get the SSSC together to spend three dedicated days preparing for the start of Rubin Observatory science operations. After a series of introductory talks and unconference sessions, participants will break into working groups to work on mini-projects ranging from software development to proposal writing.

There is no registration fee. We do have a small amount of funding to help support travel and accommodation for collaboration members who need travel support for early-career researchers and researchers from historically under-represented groups within astronomy and planetary science.

@Meg Schwamb

Operations Rehearsal 4 update

With the Vera C. Rubin Observatory's Simonyi Survey Telescope nearing completion, and with the Commissioning Camera (ComCam) due to receive first light in around a month's time, the Data Management team are working hard to make sure that the data processing pipelines are ready to process data in real-time. And despite ComCam only containing 1/20th of the number of CCDs as the full camera, the sheer rate of image capture, combined with the need for real-time processing, makes such preparations a vital part of current operations.

One key way that the project is preparing for first light with ComCam is through a series of Operations Rehearsals. As the name suggests, during these rehearsals the observatory simulates night time operations – opening the dome, pointing the telescope, exposing the CCDs, processing the images etc. While no photons actually pass through the Simonyi telescope (after all, it's not yet fully completed!), telescope operators and observers are present at the observatory and perform a full night's worth of ‘observing’. The idea is that these rehearsals should be a close reproduction of real observation.

The latest Operations Rehearsal took place at the end of June, and was the fourth in a series of rehearsals that have taken place over the previous few years. Importantly, it was the final rehearsal prior to ComCam observations, making it the last opportunity to identify any operational problems or inefficiencies before real data is taken. Two different types of data were passed through the processing pipelines in real-time. The largest of these two datasets were simulated images, created in such a way to faithfully reproduce the depth, dimensions, instrument signatures etc. of real images taken with ComCam. But while the best efforts were made to simulate ComCam data, no simulation can truly capture all the idiosyncrasies of real data, so images taken real-time with the observatory's 1.2m Auxiliary Telescope (AuxTel) were also passed through the processing pipelines in real-time. With AuxTel's single CCD being of the same type as ComCam's (and the full camera: known as LSSTCam), and with it being sited just a few metres away from the Rubin Observatory, the exposures taken are real-world examples of the type we can expect from ComCam and LSSTCam.

A couple of UK-based astronomers were closely involved with OR4: @Merlin Fisher-Levine – a former-physicist-turned-Rubin-astronomer based in the UK, but funded by US operations - and I, funded by LSST:UK as part of the UK's in-kind contributions. @merMerlin has been heavily involved in all things Rubin for many years now, and has played a major part in getting AuxTel set up and producing data, as well as developing the infrastructure to rapidly analyse and start processing the data when it comes off the camera.

Prior to OR4, Merlin and I worked on developing a database to hold key observation metadata that will be called-upon by various processes downstream. I'm pleased to say that this database held up to being bombarded with data during the rehearsal! Also implemented was the automatic triggering of on-summit Rapid Analysis processing and the streaming to RubinTV – a browser-based system that allows us to ‘eyeball’ images, particularly should they be flagged by Rapid Analysis. Merlin has played a major role in developing both these key pieces of infrastructure.

Something that we discovered during OR4 was how useful it is to have folk on both sides of the Atlantic checking the outputs of the processing pipelines. During the Chilean night, as astronomers on the West Coast of the US were going to bed, those of us in Europe were just waking up, allowing for a quick handover to highlight any issues that needed investigating prior to the next night. This allows for a near 24-hour monitoring of outputs while minimising the number of people having to pull all-nighters. Thankfully, however, the number of issues that were flagged-up during OR4 were minimal - to the extent that one morning I thought that the US and Chilean teams had forgotten to flag problems during the first half of the night! That smooth running is, of course, all credit to the robustness of the processing pipelines and the monumental effort that so many people have made over the years to get us to this stage.
Roll-on ComCam - we're ready for ya!

@James Mullaney

Photometric galaxy clustering with HSC – update

The following is a brief summary and update on an in-kind contribution being carried out by myself (@Thomas Cornish) and PI’d by @David Alonso and @Boris Leistedt. I also gave a talk on this topic last week at NAM 2024, the slides from which are linked below.

The focus of this contribution is to explore existing and novel techniques for mitigating systematics in galaxy clustering analyses, so that we are able to fully exploit the transformational sensitivity and survey area of LSST to achieve unbiased constraints on cosmology. These systematics include any instrumental or observational effect that may cause the observed density of galaxies to fluctuate in ways that are not cosmological in origin. One of the more intuitive examples is how dust from our own Galaxy obscures light from distant galaxies, lowering the observed galaxy density in a way that is dependent on the amount of Galactic dust at a given position on the sky. Such effects introduce biases into galaxy clustering measurements, and can therefore lead to incorrect inferences about the underlying cosmology if not properly accounted for.

As part of the effort to ensure that we are adequately prepared to deal with these systematics for LSST, we are using data from the third data release of the Hyper Suprime-Cam Subaru Strategic Program (HSC-SSP) as a testing ground for various systematics mitigation techniques. With its coverage of ~1260 deg² in the grizy photometric bands, HSC-SSP is the closest real-data analogue of LSST to date, and will therefore provide a strong indication of which techniques will be most applicable to the LSST data. Furthermore these data provide an opportunity to perform our own cosmological measurements through tomographic galaxy clustering (i.e. clustering measurements for samples of galaxies that have been split into bins of photometric redshift).

The next step is to compute angular power spectra – which will tell us how much clustering occurs at a given angular scale in each redshift bin – while testing different methods of removing the contributions from each systematic. However, we recently discovered that there are ‘gaps’ in the data that we downloaded from the HSC-SSP database ­– regions in which there are simply no sources present in the catalogue, despite there being obvious sources at the same positions in the images. This turns out to be due to cuts that were applied when we downloaded the data, where these cuts remove galaxies that are likely to have poor photometry and/or shape measurements, due to their being e.g. close to the edge of the CCD. The removal of these galaxies had not been accounted for when making our survey mask, and therefore risks introducing false signal in our clustering measurements.

We are therefore now working on redefining our survey mask, while investigating which of these cuts are necessary for our analyses. With luck, before long we’ll be moving onto the fun stuff – exploring different techniques of systematics mitigation – so watch this space for updates in the near future!

@Thomas Cornish

Operations Rehearsal for Lasair: Lessons learned

As mentioned above, LSST recently ran a rehearsal of the data pipeline, with simulated data sent from the summit to Santiago to California, with alerts distributed out to the community brokers in real time. Over three Chilean nights, 9.2 million alerts were sent, concerning 1.5 million unique objects. The difference is because most objects were ‘observed’ many times. There was 0.15 terabytes of cutout images sent. The Lasair broker received and processed all the data without failing. The trans-continental data flow and the replication subsystem (Mirrormaker) was able to keep up with a lag of at most a few minutes; however the Ingest subsystem – loading of data into the two Cassandra databases – was not as fast as the Lasair team would have liked. The Sherlock and Filter subsystems kept up well with the speed of the Ingest.

For six years, Lasair has been built and prototyped around the ZTF transient stream – from Palomar rather than Chile – but this Rehearsal was about ten times the rate of alerts, so a signal increase in bandwidth. The team deployed large Kafka and Cassandra clusters, and scaled up some of the instances from the minimal development deployment. We found that the cutout images will be a major storage cost – about 20 terabytes – even though only three months will be kept instead of ten years. However the LSST project will keep the full archive of cut outs, and we expect to be able to fetch from there in the rare cases where older cut outs are wanted.

We found that cut-out images will be a major storage cost, even though only three months of data will be kept

During the Chilean night, the trans-continental networks carrying the alerts will carry an average of 25 megabytes per second, with peak rates of hundreds of megabytes per second. However the internal network connecting the Lasair subsystems will be carrying an even higher load: receiving from Rubin, pushing to both cassandra databases, pushing back to Kafka, pushing to the relational database, pushing out Kafka streams resulting from user filters. In the figure below can be see the rates of the Ingest (green), Sherlock (yellow), and Filter (cyan) subsystems over the last day of the rehearsal. There is a clear change in behaviour; at first the Lasair processing is happening in parallel with the trans-continental data flow, and after 11am the flow has stopped and it is just Lasair processing. There will be further investigations to understand this behaviour.

Lasair uses a NoSQL database called Cassandra. Even more so than relational databases, the key/index structure determines efficiency. For example, if we wish to delete cutout images that are more than three months old, then the timestamp of the cutout must be in the list of keys; but at the same time we want incoming cutouts to be spread between the multiple nodes of the Cassandra cluster, not all going to the same node. The first requirement involves the ‘clustering key’, and the second involves the ‘partition key’, requiring a deep understanding of the database internals.

The Lasair architecture can use multiple VMs to increase speed of any specific subsystem, each VM can have multiple CPUs, and the processing software is built with multiprocessing in mind. The objective is to have the ability to add cloud resources and obtain higher processing speed. Before this Rehearsal, the Sherlock VMs were redeployed with multiple CPU, resulting in higher performance. There is a lot of work ahead profiling and tuning the code and the cloud deployment to be able to reliably handle the enormous data deluge from Rubin.

@Roy Williams

LSST:UK at NAM 2024

We had a pair of two-hour sessions on the first day of NAM 2024. In addition to background information, primarily aimed at the wider NAM audience, talks covered several of the UK’s in-kind contributions to Rubin operations and a variety of preparatory science activities, with the day ending with a project update from Bob Blum, Rubin Director of Operations. Copies of the slides presented are listed below, while recordings of the sessions may be accessed by those who registered for NAM from this webpage.

@Bob Mann and @Stephen Smartt

Elusive brown dwarfs are focus of latest Rubin science release

Recent LSST:UK Science Centre outputs

The LSST:UK Science Centre has recently produced the following technical reports.

@Terry Sloan

Secondary Mirror brought out of storage

Forthcoming meetings of interest

Dates, locations and links… The current list of forthcoming meetings is always available on the Relevant Meetings page. You may also wish to check information held on the LSST organisation website LSST-organised events and the LSST Corporation website.