| Table of Contents |
|---|
Introduction
Covid-19 continues to have its impact on Rubin Observatory preparations, as it does on all aspects of our lives. At this time of year, the Rubin community would be starting to focus on the annual Project and Community Workshop, but, as explained in this newsletter, that is taking place online this year, so you can participate from the comfort of your own home, rather than in sweltering Arizona. The Covid-induced “rebaselining” of the final stages of Rubin construction and of the pre-operations preparations continues, and Graham Smith , the LSST:UK Commissioning Coordinator explains in a newsletter item, and linked briefing paper, the current state of play and forewarns us that a call for expressions of interest from UK researchers to participate in Commissioning will be issued during August. In a pair of newsletter items, Stephen Smartt explains how scientific guidance is injected into LSST planning - at the Observatory level, though the Science Advisory Committee, and at UK level, through the LSST:UK Science Working Group - and summarises some of the issues currently being discussed by the two groups. Our featured recent output from the STFC-funded LSST:UK Science Centre (LUSC) programme is a report by James Perry on running the current pipeline stack, while our final newsletter item congratulates Alastair Edge , who no sooner stood down from the LSST:UK Executive Group than decided to stand as LSST:UK Consortium Board Deputy Chair, a position to which he has since been elected unopposed.
UK participation in Rubin Observatory Commissioning
The commissioning plan is in the process of being redrawn, as the worst effects of covid-19 are hopefully passing, and the in-kind contribution negotiations are reaching an advanced stage. The UK is well placed to make valuable contributions to commissioning, and has funding in place to support extended visits to Rubin locations in the UK and Chile. It is also clear that the Rubin Project Team are keen for international partners to contribute to some aspects of commissioning, and will be ready to start defining these contributions in the coming months. We therefore plan to publish a call for expressions of interest from people wishing to be considered for a role in Rubin commissioning; this will be emailed to the lusc-announce email list during August. Further details on commissioning are available in this briefing paper. Colleagues interested in commissioning and early science are also encouraged to join the relevant sessions (8-9pm UK time August 11 and 12) at the upcoming online Project and Community Workshop.Please email any questions about commissioning to Graham Smith at gps@star.sr.bham.ac.uk.
Rubin Observatory Science Advisory Committee
The Science Advisory Committee provides advice to the Rubin Observatory project on policy and technical matters. It is compromised of community scientists who are broadly representative of the major science topics that LSST will address. The members do not represent Science Collaborations, nor are they appointed or nominated by the collaborations and the committee is meant to provide independent advice and thoughts to the project, while being knowledgeable about LSST data science. Meg Schwamb and Stephen Smartt sit on the SAC. Some important recent business for the SAC has been
Submission and review of the cadence white papers which advocated different Survey Strategies, and advised the LSST Simulations teams on which survey simulations and strategies to run. You can follow the releases here on LSST Community. A Survey Cadence Optimisation Committee has been formed by the SAC. This important committee will review the new sets of simulations, the metrics and make recommendations through the SAC to the Project for final decisions on the survey cadence. The Charge to the S-C-O-C committee and the membership can be found here.
The LSST Data Policy - this important document was developed by Bob Blum, with advice and input from the SAC. It is essential reading to describe how data rights holders will work with LSST data and we recommend all UK scientists interested in using LSST data to read it. It is linked on the Rubin Observatory’s Key Project Document Page.
The SAC has also advised on Rubin Observatory - Euclid discussions, informing the selection of the Rubin Observatory’s working group to lead on high level discussions on what Derived Data Products would be produced and shared amongst the two projects.
The SAC is working on the “Science Collaboration Federation Charter” which will provide a more formal remit and framework for the science collaborations. An important point, for all LSST scientists, (related to the Charter and the Data Policy) is that Science Collaborations do not hold any proprietary right over any science from LSST. Data will be available to all and while membership of SCs is advisable, it is not mandatory. Each SC will develop its own policies regarding use of software and derived data products as might be expected in any type of astronomical data collaboration.
Finally, two important other areas that the SAC will have input on this year are the Operations and Survey Strategy in Year 1 (and commissioning) and evaluation of the Community Event Broker proposals in 2021.
LSST:UK Science Working Group
The LSST:UK Executive committee have reconstituted the Science Working Group, with a term of office defined as the duration of the Phase B work. The inital SWG was a large body which was charged with raising the awareness of LSST survey data within the UK community an across other projects with major UK interest. We felt that it had achieved its goal, with the UK’s involvement in LSST now widely known and we have become embedded in the LSST Science Collaborations, holding four leadership roles within the LSST Science Collaborations now. The SWG is made up of “Points of Contact” which are related to the Science Collaborations and other UK projects which will jointly exploit LSST data. The roles of the POCs are to actively help review the Phase B work being carried out and commenting if the science requirements are being met or now. They will also provide a link and insight into the ongoing processes within the collaborations. Some have already contributed to early Phase B work reviews and we thank them for their efforts - /wiki/spaces/LUSCSWG/pages/770310145 We encourage contact with these scientists if you are just starting out in LSST related work.
Running the LSST pipeline stack
Over the past few months I have been working to get the LSST software pipeline up and running for processing the images generated by DESC's Data Challenge 2 on UK compute resources. I installed the software on the CSD3 system at Cambridge via the newinstall.sh script and initialised a data repository using calibration data supplied by DESC. I then worked through each stage of the processing pipeline, documenting the process thoroughly in D3.10.5 - Processing DC2 data using the LSST DM Stack on UK Facilities. I was assisted greatly in this by earlier work done by James Mullaney and Darren White.
For ease of installation and use, it is highly desirable to make use of container and workflow technologies. Using container images greatly simplifies the installation of the software and ensures that the correct version of each component is present, while workflow management tools can automate the process of invoking each stage of the pipeline at the right time and on the right data, something which can otherwise be an arduous process when working with large data sets.
A workflow for this purpose is currently under development by Ben Clifford, using the Python-based Parsl workflow management system. This takes care of automatically submitting tasks to a batch system whenever their inputs become available, the software running within a container. Initially this workflow had only been tested on US systems, using Shifter to run container images built by DESC, but I have been working with Ben over the past few weeks to get it running at Cambridge. This necessitated switching to Singularity to run the containers.
The main problem encountered was an MPI error which occurs in many of the pipeline stages on the Cambridge system, believed to be due to an incompatibility between the version of MPI within the container images and the version installed on CSD3. However, the MPI functionality within the LSST stack is not actually required here, since the intention is for all of the parallelism to be handled at a high level by Parsl rather than within the individual jobs. I was able to get the workflow running successfully by replacing the "process pool" module used in the stack with a "fake" version that runs everything serially instead. A better solution, however, is to avoid using the high level "driver scripts" within the stack and instead directly call lower level components that do not use MPI, since this gives a finer granularity of task for Parsl to work with, and this is currently being implemented by Ben Clifford.
2020 Rubin Observatory Project and Community Workshop
The Covid-19 pandemic has necessitated some changes to arrangements for the up-coming Rubin Observatory Project and Community Workshop. This year’s even, which will run during 10th–14th August, will now be run entirely online and this means it is potentially easier for UK-based astronomers to participate in some or all of the sessions.
The Workshop website is now online and includes both the programme for the event and registration instructions. Each day, the programme runs during 5pm--9pm BST. While outside of the usual working day, the timing potentially gives people the option to attend those presentations/ discussions that are particularly interesting to them.
Recent Appointments
We are pleased to announce that Alastair Edge (Durham) has been appointed to the position of Deputy Chair for the LSST:UK Board. Alastair commented “I am looking forward to putting my experience on the Executive Committee to good use”.
Full details of Board Membership and other project appointments is always available on the LSST:UK public website.
The 2020 mid-year call for LSST:UK Junior Associates was open during 23rd June–6th July. Applications are currently being assessed by the LSST:UK Selection Committee and we hope to confirm the outcome to applicants before the end of JulyWith the summit still closed due to Covid restrictions, and, hence, construction still paused, the undoubted highlight of the past month within the Rubin community was the Project and Community Workshop (PCW), which attracted over 700 online participants; as detailed below, material is now available from the workshop sessions, including videos, as well as slide sets. A revised project schedule for the final stages of construction and for pre-operations remains to be announced, but it seems clear that Covid is likely to introduce a delay of something approaching a year to the start of survey operations.
One of the hot topics at the PCW was the potential impact on Rubin science of the constellations of low-Earth orbit (LEO) satellites due to be launched over the next few years by companies like SpaceX and OneWeb. This follows a workshop - SATCON1, the first in an intended series - at the end of June, organised by the AAS and NSF’s NOIRLab and bringing together astronomers and technical staff from the space industry to discuss how to quantify and mitigate the likely impact on ground-based optical and near-infrared astronomy. We present a brief summary below, along with links to the workshop report and presentation materials, which will be of interest to everyone reading this Newsletter.
Despite the delay to the construction schedule, work towards agreement of the international contributions to operations continues apace. With support from STFC/UKRI, LSST:UK is seeking a more substantial involvement in the Legacy Survey of Space and Time than was originally envisaged. Our Letter of Intent was well received, and many people are now working on expanding that into a full proposal by the submission deadline of September 25th. Approval for its contents will be sought from the LSST:UK Consortium Board, which acts as a proxy for the UK community in such matters. The proposal will be reviewed in the US in the autumn, with further iteration on its detailed content to follow in the New Year, with new data rights agreements due to be signed by 31 May 2021.
LSST:UK members are also currently preparing a second Rubin proposal, to host a Community Broker, which would guarantee access to one of the full alert streams to be produced by the near-real-time difference imaging pipeline. The UK’s prototype broker, Lasair, has been ingesting, and supporting analysis of, alerts from a Rubin precursor, the Zwicky Transient Facility (ZTF) for more than two years, and, in an item below, Andy Lawrence summarises some of the capabilities of Lasair and lessons learnt from running it on the ZTF alert stream.
Those with ideas for future newsletter items should contact the LSST:UK Project Managers (George Beckett and Terry Sloan: lusc_pm@mlist.is.ed.ac.uk), while everyone is encouraged to subscribe to the Rubin Observatory Digest for more general news from the US observatory team.
2020 Rubin Observatory Project and Community Workshop
The 2020 Project and Community Workshop (PCW) took place online on August 10-14, with approximately four hours of presentations per day, divided into three main sessions. The table below presents the workshop schedule, with links to relevant page on the PCW website, from which presentation materials - including YouTube recordings of the sessions - can be accessed.
Monday | Tuesday | Wednesday | Thursday | Friday | |||||
|---|---|---|---|---|---|---|---|---|---|
Plenary 1: Construction QA | Plenary 2: Operations QA | Plenary 3: Science Collaborations Report | Plenary 4: Science Keynote: "From disruption, opportunity: the current and future impact of AI on astronomy" (Dr. Brian Nord) | ||||||
Rubin Research Bytes | In-kind proposal workshop | ||||||||
There is a lot of interesting material there, but, obviously, the plenary sessions are a good place to start.
The Rubin staff and Science Collaboration leaders put a lot of thought into how to make a large conference work online. Through the provision of pre-recorded videos and other preparatory materials to study before the sessions, and with dedicated staff carefully monitoring session-specific Slack channels as well as the Zoom chat window, it proved possible to circumvent some of the limitations of the medium, and to get some level of interaction amongst the globally-distributed online participants, although, of course, there is no real online substitute for the chat in the coffee break or in the bar after dinner.
The impact of satellite constellations on the Legacy Survey of Space and Time
Few astronomers can have missed images like that shown on the right of SpaceX Starlink satellites crossing the field of view of the Blanco telescope at CTIO soon after launch last November. Quantitative assessment of the impact of LEO satellite constellations has begun in earnest, with a particular focus on the Rubin Observatory, since the high etendue that make the Simonyi Survey Telescope so powerful for survey astronomy also makes it particularly vulnerable to the effects of LEO constellations, since, crudely, it covers too much of the sky too quickly to make it possible just to dodge the satellites.
First results from these studies were presented at the end of June at a workshop - SATCON1 - organised by the AAS and NSF’s NOIRLab. A report from that workshop has now been published, along with technical appendices, while the workshop website also has copies of the presentations made by members of four working groups, covering observations, simulations, mitigations and metrics. More sophisticated simulations will be needed for detailed assessment of the impact on particular LSST science cases, but it is clear that constellations of LEO satellites will affect astronomy significantly - this initiative focuses on optical/NIR, but there are analogous issues in the radio - requiring coordinated action between the astronomical community and the satellite operators. There is a hierarchy of effects, some of which can be mitigated relatively easily, some of which are harder, and some, especially those relating to twilight observations, seem set to remain, whatever is done. SATCON1 is a good start, but this is, clearly, a topic that will require significant attention from the community in the coming years.
Starlink satellites imaged on 18 Nov 2019 from the Victor M. Blanco 4m Telescope at CTIO.
Image Credit: CTIO/NOIRLab/NSF/AURA/DELVE
Lasair
Lasair-ZTF is now a mature working service, and development of Lasair-LSST is well underway! In this item we will take a quick look at what you can do now, and where we are headed. We have already benefited from considerable user input, but more is always welcome.
The Rubin Observatory is expected to produce (on average) 10 million alerts every night. Getting these to users is a real challenge; the project approach is to feed the stream to third party services known as community brokers. A review process is underway to select a small number of these; the LSST:UK offering is the Lasair broker, being developed by a team in Belfast and Edinburgh. Lasair means "flame" or "flash" in both Irish and Scots Gaelic, which seems appropriate for this Celtic Collaboration. We showcased our efforts so far at a dedicated brokers session at the recent Rubin Observatory "Project and Community Workshop", recently held on line (see the item by Bob Mann in this newsletter).
To prototype Lasair, we built a system to process nightly alerts from the Zwicky Transient Facility (ZTF). However, this has gone well beyond a simple technical prototype - it is a working service being used by astronomers across the world to do science. In April we released Lasair 2.0 (see https://lasair.roe.ac.uk). This is now considered a mature and stable service, which will keep working while in the background we develop and test the next-generation Lasair, for the alerts which will come from the Legacy Survey for Space and Time (LSST) conducted at the Rubin Observatory. For short, we generally refer to the current working service as Lasair-ZTF, and the next generation version as Lasair-LSST.
As the ZTF-alerts stream into Lasair, we add multi-wavelength context, IDs, classification and other added value using our Sherlock software. You can query the accumulating database in various different ways, or produce a filtered stream which is sent on to you. You can also make a "watchlist" of your favourite objects and get alerted when one of them flares up. As well as the web interface, we have a Jupyter interface, which means you can write Python scripts to do all sorts of cunning things. Thanks to the UK IRIS service, there is some pretty high-powered computing facility behind all this. To learn more about what you can do, go the Lasair website (https://lasair.roe.ac.uk), or take a look at the Lasair cookbook (https://lsst-uk.atlassian.net/wiki/spaces/DOC/pages/881360908/Lasair+Cookbook). )
Meanwhile, we have been developing and testing a new and improved version for Rubin/LSST. For Cycles 1 and 2 of our planning process we have been concentrating on technical questions - system architecture, backend database technology, direct Kafka stream handling, and so on. All this was based on the detailed Phase B science requirements. However, as we move into Cycles 3 and 4, our focus will increasingly be on improving the functionality and the user experience. We kicked off this process in June with a kind of focus-group with the core of interested consortium scientists, which was extremely successful and informative. We are also having direct conversations with key projects and collaborations, such as the Zooniverse people, and the TiDES project. However, we are also happy (and indeed keen!) to hear the desires, experiences, and suggestions of any interested users. One of the things we are reviewing right now is how to construct a good helpdesk system - but for now, send your suggestions to lasair-help@lists.roe.ac.uk.
We have a feeling that this process will be about users evolving as well as the project learning - as people get used to the idea of working with streams, as opposed to querying static databases, new ideas will emerge. So we are especially interested in hearing your thoughts on that issue.
Results of JA round
At the end of June, LSST:UK issued a call for applications from potential new Junior Associates. Following a review by the LSST:UK Selection Committee, chaired by Nial Tanvir (Leicester), we are pleased to confirm that 22 new Junior Associates have been approved, plus four existing Junior Associates had their term extended (indicated by *), as follows:
Qasim Afghan (UCL) | Steven Gough-Kelly (Central Lancashire) |
Alba Vega Alonso Tetilla (Southampton) | Noushin Karim (Surrey) |
Marika Asgari (Edinburgh) | Hin Leung (St. Andrews) |
Oliver Bartlett (Hull) | Matt Ratcliffe (Newcastle) |
Matteo Biancoi * (Birmingham) | Agata Rożek (Edinburgh) |
Asa Bluck (Cambridge) | Jaime Ruiz Zapatero (Oxford) |
Umar Burhanudin (Sheffield) | Shubham Srivastav (QUB) |
Matteo Cataneo (Edinburgh) | Paula Stella Teixeira (St. Andrews) |
Cressida Cleland (Birmingham) | Edward Upsdell (sussex) |
Azadeh Fattahi (Durham) | Roy Williams * (Edinburgh) |
Hao Fu * (Southampton) | Tom Wilson (Exeter) |
Carlos Garcia (Oxford) | Bill Wright (QMU) |
Benjamin Giblin * (Edinburgh) | Yirui Zheng (St Andrews) |
The next call, which will be for both Affiliate PIs and Junior Associates will open in October 2020, for appointments from 1st January 2021. The call will be advertised on Confluence (on the https://lsst-uk.atlassian.net/wiki/spaces/HOME ) plus advised to the LUSC Announce mailing list (see LSST:UK Announcements Email List for information on how to subscribe).
Recent LSST:UK outputs
LSST:UK has recently produced the following technical reports.
Title | Author | Description | ||
|---|---|---|---|---|
D3D2.10.5 Processing DC2 data using the LSST DM Stack on UK Facilities | James Perry | This document describes how to process the images generated by DESC's Data Challenge 2 using the LSST software pipeline. Instructions for installing the software, setting up a data repository and running each stage of the pipeline are provided. Possibilities for using container and workflow technologies to improve the process are also discussed5.1 Training resources for LSST:UK DAC users | Bob Mann, Stelios Voutsinas, Roy Williams | This document describes an initial release of documentation for users of current and future services accessed via the UK’s LSST Data Access Centre (DAC). This documentation release is necessarily limited in scope given that the UK DAC is still being developed. It comprises existing documentation for the Lasair alert broker and a very preliminary set of documentation for the LSST Science Platform (LSP). The LSP is the set of data services to be provided by the Rubin Observatory to support analysis of LSST data products. |