Table of Contents | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Introduction
The week of 22-26 July saw the first new style Rubin Community Workshop held at SLAC. The change in title, from the Project and Community Workshop of previous years, reflects a change in emphasis as the start of survey approaches: while these annual workshops will continue to include Rubin status update and provide an opportunity for interaction with Observatory staff, the workshops will now focus more on the data rights community and its science. Slides and recordings from sessions will be made available through the workshop website to those who registered to attend.
Closer to home, and a week earlier, NAM 2024 featured a pair of sessions on Preparing for UK involvement in early science with the Rubin LSST, which featured a range of introductory/background material aimed at the wider NAM audience, plus talks on several of the UK’s in-kind contributions and preparatory science work underway in the UK. Links to the presentations are presented below.
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
LSST Solar System Readiness Sprint – sign up now
The Solar System Science Collaboration (SSSC) is having our LSST Solar System Readiness Sprint at the University of Oxford this year on 25-27 September. Interested LSST:UK consortium members can sign up by 7 August.
The goal is to get the SSSC together to spend three dedicated days preparing for the start of Rubin Observatory science operations. After a series of introductory talks and unconference sessions, participants will break into working groups to work on mini-projects ranging from software development to proposal writing.
There is no registration fee. We do have a small amount of funding to help support travel and accommodation for collaboration members who need travel support for early-career researchers and researchers from historically under-represented groups within astronomy and planetary science.
Operations Rehearsal 4 update
With the Vera C. Rubin Observatory's Simonyi Survey Telescope nearing completion, and with the Commissioning Camera (ComCam) due to receive first light in around a month's time, the Data Management team are working hard to make sure that the data processing pipelines are ready to process data in real-time. And despite ComCam only containing 1/20th of the number of CCDs as the full camera, the sheer rate of image capture, combined with the need for real-time processing, makes such preparations a vital part of current operations.
One key way that the project is preparing for first light with ComCam is through a series of Operations Rehearsals. As the name suggests, during these rehearsals the observatory simulates night time operations – opening the dome, pointing the telescope, exposing the CCDs, processing the images etc. While no photons actually pass through the Simonyi telescope (after all, it's not yet fully completed!), telescope operators and observers are present at the observatory and perform a full night's worth of ‘observing’. The idea is that these rehearsals should be a close reproduction of real observation.
The latest Operations Rehearsal took place at the end of June, and was the fourth in a series of rehearsals that have taken place over the previous few years. Importantly, it was the final rehearsal prior to ComCam observations, making it the last opportunity to identify any operational problems or inefficiencies before real data is taken. Two different types of data were passed through the processing pipelines in real-time. The largest of these two datasets were simulated images, created in such a way to faithfully reproduce the depth, dimensions, instrument signatures etc. of real images taken with ComCam. But while the best efforts were made to simulate ComCam data, no simulation can truly capture all the idiosyncrasies of real data, so images taken real-time with the observatory's 1.2m Auxiliary Telescope (AuxTel) were also passed through the processing pipelines in real-time. With AuxTel's single CCD being of the same type as ComCam's (and the full camera: known as LSSTCam), and with it being sited just a few metres away from the Rubin Observatory, the exposures taken are real-world examples of the type we can expect from ComCam and LSSTCam.
A couple of UK-based astronomers were closely involved with OR4: Merlin Fisher-Levine – a former-physicist-turned-Rubin-astronomer based in the UK, but funded by US operations - and I, funded by LSST:UK as part of the UK's in-kind contributions. @merMerlin has been heavily involved in all things Rubin for many years now, and has played a major part in getting AuxTel set up and producing data, as well as developing the infrastructure to rapidly analyse and start processing the data when it comes off the camera.
Prior to OR4, Merlin and I worked on developing a database to hold key observation metadata that will be called-upon by various processes downstream. I'm pleased to say that this database held up to being bombarded with data during the rehearsal! Also implemented was the automatic triggering of on-summit Rapid Analysis processing and the streaming to RubinTV – a browser-based system that allows us to ‘eyeball’ images, particularly should they be flagged by Rapid Analysis. Merlin has played a major role in developing both these key pieces of infrastructure.
Something that we discovered during OR4 was how useful it is to have folk on both sides of the Atlantic checking the outputs of the processing pipelines. During the Chilean night, as astronomers on the West Coast of the US were going to bed, those of us in Europe were just waking up, allowing for a quick handover to highlight any issues that needed investigating prior to the next night. This allows for a near 24-hour monitoring of outputs while minimising the number of people having to pull all-nighters. Thankfully, however, the number of issues that were flagged-up during OR4 were minimal - to the extent that one morning I thought that the US and Chilean teams had forgotten to flag problems during the first half of the night! That smooth running is, of course, all credit to the robustness of the processing pipelines and the monumental effort that so many people have made over the years to get us to this stage.
Roll-on ComCam - we're ready for ya!
Photometric galaxy clustering with HSC – update
The following is a brief summary and update on an in-kind contribution being carried out by myself (Thomas Cornish) and PI’d by David Alonso and Boris Leistedt. I also gave a talk on this topic last week at NAM 2024, the slides from which are linked below.
The focus of this contribution is to explore existing and novel techniques for mitigating systematics in galaxy clustering analyses, so that we are able to fully exploit the transformational sensitivity and survey area of LSST to achieve unbiased constraints on cosmology. These systematics include any instrumental or observational effect that may cause the observed density of galaxies to fluctuate in ways that are not cosmological in origin. One of the more intuitive examples is how dust from our own Galaxy obscures light from distant galaxies, lowering the observed galaxy density in a way that is dependent on the amount of Galactic dust at a given position on the sky. Such effects introduce biases into galaxy clustering measurements, and can therefore lead to incorrect inferences about the underlying cosmology if not properly accounted for.
As part of the effort to ensure that we are adequately prepared to deal with these systematics for LSST, we are using data from the third data release of the Hyper Suprime-Cam Subaru Strategic Program (HSC-SSP) as a testing ground for various systematics mitigation techniques. With its coverage of ~1260 deg2 in the grizy photometric bands, HSC-SSP is the closest real-data analogue of LSST to date, and will therefore provide a strong indication of which techniques will be most applicable to the LSST data. Furthermore these data provide an opportunity to perform our own cosmological measurements through tomographic galaxy clustering (i.e. clustering measurements for samples of galaxies that have been split into bins of photometric redshift).
The next step is to compute angular power spectra – which will tell us how much clustering occurs at a given angular scale in each redshift bin – while testing different methods of removing the contributions from each systematic. However, we recently discovered that there are ‘gaps’ in the data that we downloaded from the HSC-SSP database – regions in which there are simply no sources present in the catalogue, despite there being obvious sources at the same positions in the images. This turns out to be due to cuts that were applied when we downloaded the data, where these cuts remove galaxies that are likely to have poor photometry and/or shape measurements, due to their being e.g. close to the edge of the CCD. The removal of these galaxies had not been accounted for when making our survey mask, and therefore risks introducing false signal in our clustering measurements.
We are therefore now working on redefining our survey mask, while investigating which of these cuts are necessary for our analyses. With luck, before long we’ll be moving onto the fun stuff – exploring different techniques of systematics mitigation – so watch this space for updates in the near future!
Operations Rehearsal for Lasair: Lessons learned
As mentioned above, LSST recently ran a rehearsal of the data pipeline, with simulated data sent from the summit to Santiago to California, with alerts distributed out to the community brokers in real time. Over three Chilean nights, 9.2 million alerts were sent, concerning 1.5 million unique objects. The difference is because most objects were ‘observed’ many times. There was 0.15 terabytes of cutout images sent. The Lasair broker received and processed all the data without failing. The trans-continental data flow and the replication subsystem (Mirrormaker) was able to keep up with a lag of at most a few minutes; however the Ingest subsystem – loading of data into the two Cassandra databases – was not as fast as the Lasair team would have liked. The Sherlock and Filter subsystems kept up well with the speed of the Ingest.
For six years, Lasair has been built and prototyped around the ZTF transient stream – from Palomar rather than Chile – but this Rehearsal was about ten times the rate of alerts, so a signal increase in bandwidth. The team deployed large Kafka and Cassandra clusters, and scaled up some of the instances from the minimal development deployment. We found that the cutout images will be a major storage cost – about 20 terabytes – even though only three months will be kept instead of ten years. However the LSST project will keep the full archive of cut outs, and we expect to be able to fetch from there in the rare cases where older cut outs are wanted.
We found that cut-out images will be a major storage cost, even though only three months of data will be kept
During the Chilean night, the trans-continental networks carrying the alerts will carry an average of 25 megabytes per second, with peak rates of hundreds of megabytes per second. However the internal network connecting the Lasair subsystems will be carrying an even higher load: receiving from Rubin, pushing to both cassandra databases, pushing back to Kafka, pushing to the relational database, pushing out Kafka streams resulting from user filters. In the figure below can be see the rates of the Ingest (green), Sherlock (yellow), and Filter (cyan) subsystems over the last day of the rehearsal. There is a clear change in behaviour; at first the Lasair processing is happening in parallel with the trans-continental data flow, and after 11am the flow has stopped and it is just Lasair processing. There will be further investigations to understand this behaviour.
Lasair uses a NoSQL database called Cassandra. Even more so than relational databases, the key/index structure determines efficiency. For example, if we wish to delete cutout images that are more than three months old, then the timestamp of the cutout must be in the list of keys; but at the same time we want incoming cutouts to be spread between the multiple nodes of the Cassandra cluster, not all going to the same node. The first requirement involves the ‘clustering key’, and the second involves the ‘partition key’, requiring a deep understanding of the database internals.
The Lasair architecture can use multiple VMs to increase speed of any specific subsystem, each VM can have multiple CPUs, and the processing software is built with multiprocessing in mind. The objective is to have the ability to add cloud resources and obtain higher processing speed. Before this Rehearsal, the Sherlock VMs were redeployed with multiple CPU, resulting in higher performance. There is a lot of work ahead profiling and tuning the code and the cloud deployment to be able to reliably handle the enormous data deluge from Rubin.
LSST:UK at NAM2024
We had a pair of two-hour sessions on the first day of NAM 2024. In addition to background information, primarily aimed at the wider NAM audience, talks covered several of the UK’s in-kind contributions to Rubin operations and a variety of preparatory science activities, with the day ending with a project update from Bob Blum, Rubin Director of Operations. Copies of the slides presented are listed below, while recordings of the sessions may be accessed by those who registered for NAM from this webpage.
Session 1:
Introduction and summary of UK involvement in the Rubin LSST (Bob Mann)
LSST survey strategy and early science plans (Stephen Smartt)
Preparations for Rubin Data Release productions (James Mullaney)
Photometric galaxy clustering with HSC DR3: an LSST precursor study (Tom Cornish)
Enabling early Rubin science with robust cross-matches in the crowded LSST sky (Tom Wilson)
Session 2:
Detecting the Galactic interstellar object population with LSST (Matthew Hopkins)
Eliciting astrophysical distributions from imperfect survey data (Dustin Mason)
Probing galaxy clusters with the XMM Cluster Survey and implications for LSST (Paul Giles)
Higher order statistics in Rubin data (Lea Harscouet)
Rubin Observatory update (Bob Blum)
Capturing the faint light of distant brown dwarfs will help scientists better understand the formation of the Milky Way – that’s the message in the most recent Rubin press release.
LSST will detect a population of ancient brown dwarfs that is around 20 times bigger than previously seen.
Sometimes referred to as ‘failed stars’, brown dwarfs are too large to be considered planets but do not have sufficient mass to sustain nuclear fusion, which sets them apart from stars. Despite their perceived shortcomings, brown dwarfs These elusive objects are excellent candidates for understanding helping us to understand the evolution of the Milky Way.
Panel | ||
---|---|---|
| ||
Read the full science release: NSF–DOE Rubin Observatory Will Detect Thousands of Elusive Brown Dwarfs, Unlocking Milky Way Mysteries |
Recent LSST:UK Science Centre outputs
The LSST:UK Science Centre has recently produced the following technical reports.
ID | Title | Author(s) | Description |
---|---|---|---|
D2.1.3: Operational plan for UK Data Facility | G.Beckett, R.Mann | This document contains a high-level, operational plan for the UK Data Facility, which encapsulates significant computational and staff contributions to the Rubin Observatory – focusing on the UK contribution to preparing science-ready data products from telescope observations and of serving those products to the astronomy community via a UK-based data facility. | |
D3.5.3 LSST+VISTA Forced Photometry Module | R.Shirley, M. Banerji | This summarises the work conducted of the LSST and Near IR Data Fusion workpackage in Phase B of the LSST UK project. | |
D2.6.6 Documentation for near-IR and crossmatch data products on UK RSP | G.Beckett, A.Ibsen, M. Read, S.Voutsinas | One of the advantages of LSST:UK operating an Independent Data Access Centre (IDAC) is the opportunity to host additional datasets and tools focused on UK science priorities. In addition to hosting these datasets, the UK IDAC team need to actively publicise and describe them, ensuring that astronomers can grasp the potential benefits of these datasets and confidently integrate them into their research. These datasets are expected to provide a valuable complement to Rubin data products published during the Early Science Program. The deliverable outlines efforts to document two specific UK-provided datasets. The first dataset involves a crossmatch catalogue for the Gaia EDR3 and CatWISE surveys (an early output of the Rubin in-kind contribution, UKD-S9). The second dataset comprises a near-infrared fused dataset that integrates HSC imaging with imaging from several overlapping VISTA surveys (VIKING, VIDEO and VHS). The described work not only enhances the accessibility and usability of the specific datasets but also establishes a methodology for documenting other ancillary datasets and supplementary services. | |
D2.5.3.1: Ingestions of ancillary surveys for crossmatch | M.Read, G.Beckett | This deliverable involves the preparation of input datasets, by the DAC team, to allow the UK In-kind contribution, UKD-S9, to progress experiments with the “Supercrossmatch” algorithm. | |
D1.9.1 LSST:UK Communications Plan | E.O’Kane | This document outlines how we will increase awareness of LSST:UK through external communication and defines our key audiences are. There is great potential for interest in the project beyond the astronomy community and several key audiences are outlined in this Communications Plan. Our plan also covers internal communication activities. A key part of the plan is to equip members of the research team, as ambassadors for LSST:UK, with the skills to talk about the global significance of LSST:UK. | |
D2.5.1.1: Provision of hosting platform for Lasair Version 6 | G.Blow, M.Holliman, G.Beckett | A key responsibility of the DAC team is to provision and support the hosting platform for Lasair. This is expected to be relatively stable, with a requirement for routine maintenance and day-to-day troubleshooting during Phase C, though two significant upgrades are likely to involve reconfiguration of the hosting platform. |
Secondary Mirror brought out of storage
In early July, the summit team brought Rubin’s Secondary Mirror (M2) out of storage to integrate it with the mirror cell. The full M2 assembly consists of the secondary mirror, mirror support system, mirror cell electronics and sensors, thermal control system, and the mirror control system. It is supported by 72 axial actuators and 6 tangential actuators. The M2 assembly will soon be installed on the telescope mount, after which the summit team will re-install the Rubin Commissioning Camera.
Eleanor O'Kane
Forthcoming meetings of interest
Dates, locations and links… The current list of forthcoming meetings is always available on the Relevant Meetings page. You may also wish to check information held on the LSST organisation website LSST-organised events and the LSST Corporation website.
Dates | Meeting Title / Event | Meeting Website/ Contact | Meeting location / venue |
---|---|---|---|
16/Sep/24—20/Sep/24 | LSST@Europe6 | Spain | Island of La Palma | |
25/Sep/24—27/Sep/24 | LSST Solar System Readiness Sprint | Oxford, UK | |
10/Nov/24—14/Nov/24 | Astronomical Data Analysis Software & Systems (ADASS | Malta | Virtual |
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
Members of the Consortium (not in receipt of travel funding through one of the Science Centre grants) may apply for travel support for meetings of this kind via the LSST:UK Pool Travel Fund. Details are available at Forthcoming LSST-related Meetings |
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
If you have significant news or announcements that are directly relevant to LSST:UK and would like to share them in a future newsletter, contact Eleanor O'Kane (email eokane@roe.ac.uk) |
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||