Table of Contents | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Introduction
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse quis elementum magna. Cras varius ex felis, et malesuada nisl molestie quis. In pellentesque nunc vitae ipsum imperdiet consequat. Sed non vestibulum tortor, vitae aliquam lectus. In ac erat eu libero congue lobortis ut id massa.
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
LSST Solar System Readiness Sprint – sign up now
The Solar System Science Collaboration (SSSC) is having our LSST Solar System Readiness Sprint at the University of Oxford this year on 25-27 September. Interested LSST:UK consortium members can sign up by 7 August if interested.
The goal is to get the SSSC together to spend three dedicated days preparing for the start of Rubin Observatory science operations. After a series of introductory talks and unconference sessions, participants will break into working groups to work on mini-projects ranging from software development to proposal writing.
There is no registration fee. We do have a small amount of funding to help support travel and accommodation for collaboration members who need travel support for early-career researchers and researchers from historically under represented groups within astronomy and planetary science
Operations Rehearsal for Lasair: Lessons Learned
The LSST ran a rehearsal of the data pipeline, with simulated data sent from the summit to Santiago to California, with alerts distributed out to the community brokers in real time. Over three Chilean nights, 9.2 million alerts were sent, concerning 1.5 million unique objects. The difference is because most objects were "observed" ‘observed’ many times. There was 0.15 terabytes of cutout images sent. The Lasair broker received and processed all the data without failing. The trans-continental data flow and the replication subsystem (Mirrormaker) was able to keep up with a lag of at most a few minutes; however the Ingest subsystem -- loading of data into the two Cassandra databases -- was not as fast as the Lasair team would have liked. The Sherlock and Filter subsystems kept up well with the speed of the Ingest.
For six years, Lasair has been built and prototyped around the ZTF transient stream -- – from Palomar rather than Chile -- – but this Rehearsal was about ten times the rate of alerts, so a signal increase in bandwidth. The team deployed larged large Kafka and Cassandra clusters, and scaled up some of the instances from the minimal development deployment. We found that the cutout images will be a major storage cost -- – about 20 terabytes -- – even though only three months will be kept instead of ten years. However the LSST project will keep the full archive of cutouts, and e expect to be able to fetch from there in the rare cases where older cutouts are wanted.
We found that cut out images will be a major storage cost, even though only three months of data will be kept
During the Chilean night, the trans-continental networks carrying the alerts will carry an average of 25 megabytes per second, with peak rates of hundreds of megabytes per second. However the internal network connecting the Lasair subsystems will be carrying an even higher load: receiving from Rubin, pushing to both cassandra databases, pushing back to Kafka, pushing to the relational database, pushing out Kafka streams resulting from user filters. In the figure below can be see the rates of the Ingest (green), Sherlock (yellow), and Filter (cyan) subsystems over the last day of the rehearsal. There is a clear change in behaviour; at first the Lasair processsing is happening in parallel with the trans-continental data flow, and after 11am the flow has stopped and it is just Lasair processing. There will be further investigations to understand this behaviour.
Lasair uses a NoSQL database called Cassandra. Even more so than relational databases, the key/index structure determines efficiency. For example, if we wish to delete cutout images that are more than three months old, then the timestamp of the cutout must be in the list of keys; but at the same time we want incoming cutouts to be spread between the multiple nodes of the Casaandra Cassandra cluster, not all going to the same node. The first requirment requirement involves the "clustering key"‘clustering key’, and the second involves the "partition key"‘partition key’, requiring a deep understanding of the database internals.
The Lasair architecture can use multiple VMs to increase speed of any specific subsystem, each VM can have multiple CPUs, and the processing software is built with multiprocessing in mind. The objective is to have the ability to add cloud resources and obtain higher processing speed. Before this ReheasalRehearsal, the Sherlock VMs were redeployed with multiple CPU, resulting in highter preformancehigher performance. There is a lot of work ahead profiling and tuning the code and the cloud deployment to be able to reliably handle the enormous data deluge from Rubin.
Tom Cornish news story - tbc
Recent LSST:UK Science Centre outputs
The LSST:UK Science Centre has recently produced the following technical reports.
ID | Title | Author(s) | Description |
---|---|---|---|
D2.1.3: Operational plan for UK Data Facility | G.Beckett, R.Mann | This document contains a high-level, operational plan for the UK Data Facility, which encapsulates significant computational and staff contributions to the Rubin Observatory – focusing on the UK contribution to preparing science-ready data products from telescope observations and of serving those products to the astronomy community via a UK-based data facility. | |
D3.5.3 LSST+VISTA Forced Photometry Module | R.Shirley, M. Banerji | This summarises the work conducted of the LSST and Near IR Data Fusion workpackage in Phase B of the LSST UK project. | |
D2.6.6 Documentation for near-IR and crossmatch data products on UK RSP | G.Beckett, A.Ibsen, M. Read, S.Voutsinas | One of the advantages of LSST:UK operating an Independent Data Access Centre (IDAC) is the opportunity to host additional datasets and tools focused on UK science priorities. In addition to hosting these datasets, the UK IDAC team need to actively publicise and describe them, ensuring that astronomers can grasp the potential benefits of these datasets and confidently integrate them into their research. These datasets are expected to provide a valuable complement to Rubin data products published during the Early Science Program. The deliverable outlines efforts to document two specific UK-provided datasets. The first dataset involves a crossmatch catalogue for the Gaia EDR3 and CatWISE surveys (an early output of the Rubin in-kind contribution, UKD-S9). The second dataset comprises a near-infrared fused dataset that integrates HSC imaging with imaging from several overlapping VISTA surveys (VIKING, VIDEO and VHS). The described work not only enhances the accessibility and usability of the specific datasets but also establishes a methodology for documenting other ancillary datasets and supplementary services. | |
D2.5.3.1: Ingestions of ancillary surveys for crossmatch | M.Read, G.Beckett | This deliverable involves the preparation of input datasets, by the DAC team, to allow the UK In-kind contribution, UKD-S9, to progress experiments with the “Supercrossmatch” algorithm. | |
D1.9.1 LSST:UK Communications Plan | E.O’Kane | This document outlines how we will increase awareness of LSST:UK through external communication and defines our key audiences are. There is great potential for interest in the project beyond the astronomy community and several key audiences are outlined in this Communications Plan. Our plan also covers internal communication activities. A key part of the plan is to equip members of the research team, as ambassadors for LSST:UK, with the skills to talk about the global significance of LSST:UK. | |
D2.5.1.1: Provision of hosting platform for Lasair Version 6 | G.Blow, M.Holliman, G.Beckett | A key responsibility of the DAC team is to provision and support the hosting platform for Lasair. This is expected to be relatively stable, with a requirement for routine maintenance and day-to-day troubleshooting during Phase C, though two significant upgrades are likely to involve reconfiguration of the hosting platform. |
Forthcoming meetings of interest
Dates, locations and links… The current list of forthcoming meetings is always available on the Relevant Meetings page. You may also wish to check information held on the LSST organisation website LSST-organised events and the LSST Corporation website.
Dates | Meeting Title / Event | Meeting Website/ Contact | Meeting location / venue |
---|---|---|---|
16/Sep/24—20/Sep/24 | LSST@Europe6 | Spain | Island of La Palma |
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
Members of the Consortium (not in receipt of travel funding through one of the Science Centre grants) may apply for travel support for meetings of this kind via the LSST:UK Pool Travel Fund. Details are available at Forthcoming LSST-related Meetings |
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
If you have significant news or announcements that are directly relevant to LSST:UK and would like to share them in a future newsletter, contact Eleanor O'Kane (email eokane@roe.ac.uk) |
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||