Time and Venue: Wednesday 8th September 2021, via Zoom
Attendees: Andy Lawrence, Roy Williams, Terry Sloan, Cosimo Inserra, Stephen Smartt, Sara Casewell, Dave Young, Micheal Fulton, Julien Peloton, Ken Smith, Jakob Nordin, Gareth Francis, Matt Nicholl, Eric C Bellm, Stelios Voutsinas, Meg Schwamb, George Beckett
Apologies:
Notes from discussion
Session-1: Overviews
Welcomes (AndyL)
Please use the Zoom Chat discussion. This will be saved at the end of meeting.
Code of Conduct reminder
Meeting contacts are AndyL and SaraC
LSST:UK context (Stephen Smartt)
Hope to get funded in Phase C but this will not count towards in-kind. It will however support other UK work that does count towards in-kind.
Spectra information inclusion is increasingly important. UK in a strong position.
The spectrograph of particular note to Lasair is 4MOST, the spectrograph for the VISTA telescope. TiDES is the part of the 4MOST survey focused on transients.
Lasair long term plan (Andy Lawrence)
Today’s meeting primarily concerned with the revised 4 year plan for the remainder of phase B.
Issues from today will noted and fed into the plan.
Overall LSST:UK phases have not changed relative to the Rubin schedule.
Current Cycle 4 of the Lasair plan is taking stock prior to build.
Key change - original plan was ZTF prototype would be frozen at V2 however further ZTF development has continued.
Overall Lasair architecture now believed to be set.
A number of key technology changes including assuming that users will be using Google Colab.
1325: Discussion
Cosimo Q - for Stephen regarding spectroscopic addition to Lasair, confused about the various projects. 4most tides does not do targetted opportunities. So it might be 20 to 100 days after the LSST discovery. Can understand how SOXS can be contribute but not so much 4MOST. So how much can these spectra actually contribute. What spectra information will be available and in what format?
StephenS A - correct that 4MOST pointing cannot be controlled but it is likely to follow the LSST footprints. It is possible to control the fibre pointing within the fields of 4MOST. Anything brighter than magnitude 22 in the field of 4MOST will have a fibre put on it. Spectra of 20K to 30K transients plus 30K host galaxies are expected but cannot control exactly when the spectra will be collected. The challenge is to get objects out of the alert stream and onto the the 4MOST schedule. If fibre can be put on them then when an object is 4MOST observed, then the LSST:UK UK WP3.3 will recognise that the fibre is in place, make estimates and the information will then go back to Lasair as a annotation. All of the data will be public in ESO archives. We will get information back asap into the broker.
Cosimo Q - do you plan the same same thing for SOXS.
Stephen S A - yes, there will be some link from Lasair
Jakob Q - How well defined is the UK programme . Have the spectra discussed what they require in terms of triggering from LSST? To what degree has the UK:LSST community specified their LSST plans, such that these can be sued as Lasair requirements
SJS - yes these discussions are happending at cocneptaual level. Triger are classsified as supernovae by sherlock( not an agn, not a variable star…). Essentially if bright enough for 4MOST to take a spectra then put a fibre on it.
Jakob - sounds like UK will only do supernvoa?
SJS - actually mean extragalactic transients
Question from JulienP - Funding context was highlighted, but the focus was on PI/developers. What about the hardware funding status and plan ?
Stephen S - This will be discussed in afternoon. Top level funding is secured. Lasair is a small part of the UK computing. LSSTUK is a small part of UK IRIS. So overall Lasair are not the dominant partner and hence we believe Lasair is safe. Andy agrees and explained that the plan for UK expected requirements is being put in to UKRI. This is for the 10 years operation and includes Lasair.
Session-2: ZTF prototype experience (chair: Stephen Smartt)
Lasair-ZTF and how it works (Roy Williams)
query, filter and stream are all the same in Lasair
Sherlock - cross-matches location of transient with other catalogs
Q Julien - screen annotation - what is the feedback on this?
Roy - have not built the system yet. Micheal will talk later about one of the classifications. would like to share classifications from other brokers eg FINK if acceptable.
Sara Q - stream annotation - are there plans to create a collaboation so people are not competing on the same follow-ups?
Roy - Mrashall refers to a fgroupo of peole able to submit opinions on transients and make decisions
Andy - Mrashall building byeond scope to solve this. Omn assumption that ouitside world then wnat to know what pepople need to build their own marshalls.
Jakob - annotatins are hard. Decided instead tobe flexible so people can do what they want but they then have the responsibility to maintain and make useful. On 2nd slide, no arrow to/from Rubin Science platform. Have you thought about jhow this ealates tp the DB contens
Andy - grimly aware need to think about this but not yet solved.
Jakob Q to Eric - how frequently can we expect updates and show seriosly to take these
Eric says let discuss during talk
Science experience with ZTF (Matt Nicholl)
Experiments with Lasair-ZTF
Use cases
Nuclear transient research currently migrating to Google Colab notebook
Difficulties are: filtering on colour, streams clogged with old objects, not having full light curve history (this is the biggest problem)
Intend doing citizen science with light curves. Documentation to do this was helpful
Lensed transients
Email alerts could be more human readable
A thumbnail for each object on streaming page would be helpful
Testing spectra in advance would be good.
Emai alerts and web interface are what
Roy Q - light curves are coming so bid probelm will be solved. Like/dislike for an object is the tail of tiger and could mean Lasair building marshalls. do not have push facility yet. Will tel you.
Eric Q - boundaries with marshalls - how do keep track of candidates
Matt - luckly numbers are not big so not a problem yet. Have been using a note in Google collab. In group have started tentative to build own marshall.
Cosimo - Agree color curve would be fantastic. What about using Gaussian processes to interpolate the light curves in order to produce colours (with uncertainties)?
Roy - happy to look at Gaussian code and include in the pipeline
Cosimi - will forward to roy
Jakob - you are all using Google colab. But we have encounterd python limitations
Roy - Ken will talk about this later.
Ken - do not know how it nehaves when it requires C binaries other than numPy. Not yet installed own C code. We should explore this.
Garteth - aware that Nubaldo will be running on RSP next door so if Google Colab not suitable then considering intergration with the RSP.
Differences between ZTF and LSST (Eric Bellm)
(need slides from Eric - have messaged)
ZTF is on telescope that is almost 75 years old
Slide 3 shows physical differences with Rubin. Only advantage for ZTF is in field of view. Total number of exposures per night will be similar on both telescopes.
Major difference is ZTF only processes the data once i.e. as live data comes off the telescope.
Q Julien on chat - Cutouts to be transmitted for LSST are template and difference. Why choosing template over science? The science one is probaby what I look first to judge quality.
A- The reason lost to pre-history. aim was to get 2 out of 3. Aim to go to 3 cut-outs.
Question: (Roy W asked) Can Lasair get awayt with using the LSST cutout service instead of saving them in our own database? How many can we fetch per day?
Answer: Do not think Lasair should use US cutout service. Capacity limits are not clear at moment, but cut-outs won’t become available until after the underlying images, which means a 3-day delay (in line with Government requirements).
Q Jakob - Regarding upper limits on forced photometry, will there always be 12 month limits on alerts with varying limits for forced photometry.?
A: Not built rigourously in pipelines, so remains conceptual. Should have history of every time observe a reason, and provide upper limit on noise estimate where don’t have forced photometry. This can change from night to night.
A: Details of workload management for PPDB are to be confirmed, though it is a concern. Several ways to think about this. Outpcome of Broker W/shop was action on project to offer a database export of PPDB (world-readable) for those who require significant information from DB.
Davy asked to clarify that forced photometry in alerts in one epoch behind actrual allert. Eric confirmed this was case. Getting triggering DSRs, but will. Eric will make a note and hope anomoly can be achieved without significant effort.
Session-3: Architecture and Technology (chair: Andy Lawrence)
Lasair-LSST Architecture (Roy Williams)
Roy will also explain why the architecture is what it is.
Platform maintains ~1-week cache of alerts from ZTF stream
Cassandra and Galera are both scalable. Aim to be able to dynamically scale to deal with changing load
New feature - external annotator - can receive a stream from Lasair and then query DB to draw conclusions. This might be a classifier – e.g., Zooniverse.
Lasair user interface is SQL-based (specifically, simple SELECT), but supports both on-demand and streaming queries
Supports community contributions (queries, classifications, etc.)
Architecture based on scale-up of low-power VMs
1445: Kafka processing (Gareth Francis/Roy Williams)
1455: Hardware implementation (Roy Williams)
1505: Light curve classification (Michael Fulton)
1515: Discussion