Introduction
Rubin-LSST at NAM
Finally, a reminder that registration for the 2021 Project and Community Workshop (PCW) is open, with this year’s event taking place online from August 9-13 and featuring a wealth of programmatic, technical and scientific information related to the Rubin LSST.
Those with ideas for future newsletter items should contact the LSST:UK Project Managers (George Beckett and Terry Sloanlusc_pm@mlist.is.ed.ac.uk), while everyone is encouraged to subscribe to the Rubin Observatory Digest for more general news from the US observatory team.
UK contributions to Rubin commissioning: call for proposals
Colleagues are invited to submit proposals for value added contributions to Rubin commissioning. This opportunity follows acceptance of the commissioning element of the UK’s in-kind proposal, that was in turn based on the Expressions of Interest submitted by UK colleagues in 2020. It is therefore already agreed that the UK will contribute a summit team and a remote team. The focus of the former will include commissioning the wavefront sensor sub-systems, bringing expertise from VISTA. The latter will be a broader and larger team of colleagues contributing a wide range of scientific and operational expertise, much of which will concentrate on analysing commissioning data and giving feedback to the core commissioning team on the performance of the system. This is a great opportunity to get early hands on experience with Rubin data and science platform. Whilst there are strict rules around any scientific output from the commissioning period, experience gained with commissioning data will inevitably help with achieving early science. We are looking to agree the UK team with US colleagues by the end of 2021 with a view to UK team members getting started in early 2022. Travel and subsistence funding to support the UK team is already in place, and modest additional funding of commissioning effort is also available through this call. Full details of this opportunity are described in the call for proposals, which has a submission deadline of September 15, 2021. Please contact Graham Smith if you have any questions or would like to discuss proposal ideas in advance of submission.
Lasair selected as LSST community alert broker
The Rubin Observatory announced on Tuesday 27th July that the UK’s Lasair project has been selected as an official LSST community alert broker and will receive the full Rubin alert stream. Every image that Rubin takes will have a reference sky subtracted and every object that is variable or transient, or moving, will be catalogued and released in a stream of alerts within 60 seconds. The University of Edinburgh and Queen’s University Belfast have been partnering, with STFC Phase B funding, to build a broker that will provide a user friendly and scientifically powerful platform for world-wide users to exploit this information reach data stream. We expect Rubin’s data stream to have a peak production rate of 1 terabyte per night in just the alert data, a data rate that would overwhelm the average science user trying to feed off the complete transient stream. For every visit (i.e. approximately every minute) there will be about 30,000 alerts. Lasair will take the full Rubin alert stream and add scientific value to the data stream, providing computing power, tools and storage for science users to manage selection of objects from the stream. We will manage a large database store of all objets and provide SQL, API and python notebook query functionality. Our added value will be cross-matching to multi-wavelength catalogues, classification, and provision of realtime spectroscopic redshifts and spectral types from the 4MOST and SOXS surveys, photometric redshifts from major catalogues and Rubin itself and a basic lightcurve classifications. We have built the prototype and have had multiple reviews and workshops with the user community in the UK and Europe to shape the design, most recently at the RAS National Astronomy Meeting 2021 hosted by the University of Bath. We welcome user input and feedback at any time. Read more from Roy Williams below on the new concept of streaming data in comparison to database queries, which is core to our vision for science functionality for the LSST data stream. While our Lasair prototype works on ZTF data and is being scientifically exploited by a range of users, we now look forward to developing the Lasair system for LSST data rates.
The Lasair Team : Stephen Smartt Andy Lawrence Roy Williams Ken Smith Gareth Francis Dave Young Dave Morris Stelios Voutsinas
The New World of Streaming Astronomical Data
Many of us are looking for a new home. Even if we are not, we all have a friend or family member who enjoys “property fantasy” – browsing the homes for sale and imagining themselves the buyer. We connect to a property broker, just as astronomers connect to an event broker: not just to find, but to have new results pushed to us, as soon as the property comes on the market (or as soon as the supernova explodes).
The “asking” way is to run the query every so often, and pick through the results to see what is new. There is also the “listening” way to get updates pushed to us: once a query is saved, simply check a box and you will get notified every time a suitable property comes on the market. Your query has been converted to a filter, that takes the stream of properties, and filters out the ones you want. In the screenshot shown here, it says “Create Alert” at top right, which causes an email whenever a property passes the filter.
You can (1) run your query and get all the results, consisting of both new properties and those you have already seen. Or (2) You can get notified whenever a new property comes up. Note that you only get new results, because the server remembers what it gave you before.
The Lasair event broker of LSST:UK has a “query builder” based on SQL syntax, with some extras: users can also query an intelligent crossmatch from catalogues, or upload a list of their favourite sources, or specify a sky area of interest.
In the past, an astronomer would query the database with their morning coffee, and try to find the new ones – method (1) above. Now however, the astronomer can check the “streaming” box on their saved query in Lasair. Alerts will be delivered as in method (2). While email is available for low-frequency streams, the “Kafka” protocol is more suitable for delivering alerts for further processing by machines.
The Kafka server remembers what has been delivered already, through the mechanism of the “groupID”. Each user-made query/filter generates a stream of alerts, that Kafka calls a “topic”. Having yesterday read from a topic with a given groupID, and then read today, you will get only the new alerts that have appeared on that topic since yesterday. You can run a program forever and continuously pull the stream, or you can start the program periodically, and in both cases each alert is received exactly once. Of course, you can get everything in the topic by simply choosing a new groupID, and it will start at the beginning of its cache – generally about a week’s worth of the stream. Details are available at the Lasair website -- see the last notebook at http://lasair-iris.roe.ac.uk/jupyter .
The streaming data has a link to the Lasair page corresponding to that object, showing the lightcurve and sky context, and links that give more information about the object, from the databases of the world. If you have a team, the alerts could be fed into a “marshal” -- so that different people can write comments and vote on whether telescopic follow-up is warranted. Or you could even have automated telescopic follow-up with no human in the loop.
Placeholder title for Catherine
Placeholder title for Dan
Recent LSST:UK outputs
LSST:UK has recently produced the following technical reports.
Title | Author | Description |
|---|---|---|
Forthcoming meetings of interest
The schedule of events is relatively quiet over the Summer months, though we expect new workshops and meetings to be announced in the coming weeks. The most significant event in August is the 2021 Rubin Project and Community Workshop, which will take place online from 9th--13th August. Visit the website to register.