...
Experiments with Lasair-ZTF
Use cases
Nuclear transient research currently migrating to Google Colab notebook
Difficulties are: filtering on colour, streams clogged with old objects, not having full light curve history (this is the biggest problem)
Intend doing citizen science with light curves. Documentation to do this was helpful
Lensed transients
Email alerts could be more human readable
A thumbnail for each object on streaming page would be helpful
Testing spectra in advance would be good.
Emai Email alerts and web interface are what
Roy
...
stated that the light curves are coming so
...
the major problem identified will be solved. Like/dislike for an object is the tail of tiger and could mean Lasair in effect building marshalls. As yet do not have the facility to push data into notebooks but will let you know this ready. facility yet
...
.
Eric Q to Matt - Regarding the boundaries with marshalls
...
, how do keep track of candidates?
Matt answer -
...
luckily the numbers are not big so not a problem yet. Have been using a note in Google collab.
...
Also within research group have started tentative steps to
...
building our own marshall.
Cosimo Q to Roy - Agree color curve would be fantastic. What about using Gaussian processes to interpolate the light curves in order to produce colours (with uncertainties)?
Roy - happy to look at Gaussian code and to include this in the Lasair pipeline.
...
Cosimo will forward the code to
...
Roy.
Jakob Q to Roy - you are all using Google colab
...
but we have
...
encountered python limitations
Roy answer - Ken will talk about this later.
Ken - do not know how
...
Google Colab behaves when it requires C binaries other than
...
NumPy. Not yet installed own C code. We should explore this.
...
Gareth answer- aware that Nubaldo will be running on RSP next door so if Google Colab not suitable then
...
can consider integration with the RSP.
Differences between ZTF and LSST (Eric Bellm)
...
ZTF is on telescope that is almost 75 years old
Slide 3 shows physical differences with Rubin. Only advantage for ZTF is in field of view. Total number of exposures per night will be similar on both telescopes.
Major difference is ZTF only processes the data once i.e. as live data comes off the telescope.
Q Julien on chat - Cutouts to be transmitted for LSST are template and difference. Why choosing template over science? The science one is probaby what I look first to judge quality.
A- The reason lost to pre-history. aim was to get 2 out of 3. Aim to go to 3 cut-outs.
Question: (Roy W asked) Can Lasair get awayt away with using the LSST cutout service instead of saving them in our own database? How many can we fetch per day?
Answer: Do not think Lasair should use US cutout service. Capacity limits are not clear at moment, but cut-outs won’t become available until after the underlying images, which means a 3-day delay (in line with Government requirements).
Q Jakob - Regarding upper limits on forced photometry, will there always be 12 month limits on alerts with varying limits for forced photometry.?
A: Not built
rigorously in pipelines, so remains conceptual. Should have history of every time observe a reason, and provide upper limit on noise estimate where don’t have forced photometry. This can change from night to night.
A: Details of workload management for PPDB are to be confirmed, though it is a concern. Several ways to think about this. Outpcome of Broker W/shop was action on project to offer a database export of PPDB (world-readable) for those who require significant information from DB.
Davy asked to clarify that forced photometry in alerts in one epoch behind actrual allertalert.
Eric confirmed this was case. Getting triggering DSRs, but will. Eric will make a note and hope
anomaly can be achieved without significant effort.
Session-3: Architecture and Technology (chair: Andy Lawrence)
Lasair-LSST Architecture (Roy Williams)
...
Julien P asked “Do you make the list of queries public such as others can take inspiration or just play it as is for their own work? Is this something the user that makes the query can choose (public/ private)?
A: Each query has a tick box for making public/ private, as you wish. Public queries can be copied and modified.
Julien P also asked “Is Sherlock doing batch or live processing of the incoming stream?”
A: Sherlock is doing batch processing, as is more efficient, though effectively same outcome as batches are very quick.
Julien noted batch processing may be less similar for LSST, due to data rates. May wish to prioritise processing based on science case. Has Lasair team thought about this?
Dave Y noted that Sherlock consumes whatever it is given. Code is multi-processed and scalable, so believe will be quick enough.
Gareth noted currently running batches of up to 800 alerts (if there are enough). This has been seen to be a reasonable compromise, taking 3--4 minutes according to content.
Andy L asked what was delay in seeing output from an alert
Roy noted around 30 minutes, based on recent sampling
Andy L noted that Rubin aspiration is 60 seconds.
Roy noted some of 30 mins might be within ZTF infra, rather than Lasair. Roy does not see significant use cases for 60-second avalability.
Eric agrees not a lot of science cases need 60-second turn-around. Has been proposed as de-scope opportunity, but has seen pushback from Rubin Directorate for this. Rubin is still aiming to achieve 60 second latency. Once science cases mature, scope to evolve functionality to meet requirements.
Andy noted that, if people need 1-min or 5-min latency, then hopefuly will be achievable somewhere in Rubin landscape – not necessarily in Lasair.
Jakob N noted that idea to have everything evolve around queries and streams is very nice and supports science reproducability. Was intrigued to see Kafka used as a bus, as not something considered by Jakob’s group, to date. Jakob using combination of SQL and No-SQL databases, but for different things. SQL for cut-outs, no-SQL for objects and annotations. Previously proposed to have workshop focusing on database choices. Royt confirmed that this would be interesting. Ken noted that Pit? Google trying to organise workshop on this in late October/ early November (Ken is on Organising Committee). Stephen proposes dedicated session on databases at workshop and encouraged Ken to press for this.
Light curve classification (Michael Fulton)
Building photetric classifier to be integrated into Lasair. Transient detection rate far exceeds rate of classificatoin. Will get worse with LSST.
Plan to run FastFinder on LSST and highlight potential transients very quickly.
ML techniques work better with well-sampled light curves, but often too late for real science. Need to prioritise speed, while not losing reliability, so have chosen parameter fitting (feature selection comparison to parameter space of fast transients).
Template Datastore for lots of different fast transients, from LSST, Pan-STARRS, and …
Features including rise time, peak brightness, etc. extracted and used for classification step.
FastFinder produces list of scores for a candidate, based on different features. high-scoring features are favoured.
Approach is simple, but producing promising results, based on sampling with kilonova class transients, even when peak brightness not observed.
FastFinder in use with PANSTARRS. Found 2021qvw, for example.
FastFinder will act as an external annotator (see previous talk from Roy).
Runs independent of Lasair.
FastFinder outputs could be included in Lasair website, in dedicated section for external annotations.
Eric Q to Michael “can you say more about how you are fitting/estimating the light curves features (peak time, rise and fall times)”.
A. Rise time is the what was peak magnitude, what was the starting magnitude and what was the time between the two. Polynomial fit applied. This depends on the number of points available.
Julien P Q to Michael - Are predictions done using the aggregated light-curve or the last alert information ? What about the evolution of score, do you report the full history of classification or just the last score? Julien asked if prediction is based on light curve
A. Prediction is based on position in parameter space of candidate. This is done for all features and results normalised to create score.
A Lasair will only store the most recent classification from FastFinder, but hope to have option to click through to history of FastFinder classiificatons.
Cosimo asked whether FastFinder needs information from different bands to be on same day or in 3--4 day timeframe?
Features are extracted as they become available, and information is derived as is possible based on available features. So, need to advise how many features have been used to create classification, as indication of relative reliability of classification.
Jacob noted using absolute magniture, which relies on good redshift. how sensitive is classifier. Also why not run on ZTF?
Redshifts have been measured from Sherlock, so errors are attributable to Sherlock. If host nearby does not have redshift then object is ignored. Risk is minimal if have secure association with galaxy of known redshift. Likely dominated by errors in photometry (if spectroscopic). If not, then photometry is likey source of error, so make assumption know distance and filter on that.
Cosimo also asked is there plan to increase the database [of templates] to deal with new discoveries?
Cosimo Q on chat “Do you need a data point in each band or not? What are the plans to increase the data templates.”
StephenS answered on chat . “one band is enough to get a score. The date templates are the most work, and yes we do plan to increase templates “
1515: Discussion