...
(need to get slides)
Will cover topics discovered during Ampel development cycle and what we considered doing.
Until now reproducibility unnecessary in astronomy due to instrument growth but after LSST not sure if that will still be case.
Most alerts are faint and hence junk
Reusable software has not been a priority in science, particularly due to PhD cycle, but with projects running for 10+ years this can no longer be the case.
Reproducibility hard to do due to the black box (isolated) nature of the various steps in studies/pipelines.
Users create an analysis schema that runs across the four different information tiers in Ampel.
Users prefer a UI but the Ampel approach is analysis(code) focussed
Q Roy to Julien - How good are your classifications? How many false +ve/-ve?
Julien answer - difficult question to anwser, using TNS since Nov 2020 about 300 candidates have been reported. Half of these were spectroscopically followed-up. More than 80% of these were true SNs. This is a good result but this not the only thing to look at. Now trying to make reliable detection faster. Currently have to wait 7 days but would prefer 3 days
.Jakob asked Dave - Sherlock is very nice, but how do people see maintaining and updating software over next 15 years.
Dave noted default algorithm now sits with code (individual users can adjust, but default is version controlled). When you run Sherlock, the version of the classifier is stored for reproducability.
Jakob asked whether Dave would be about to update it
Dave hoped so, but believed someone could be trained up to manage it in around two weeks, though noting code documentation needs to be improved.
Dave asked Julein if Fink classifdier is based on light curve data alone.
Yes, for Type 1a Supernovae.
Dave concerned that Type 1a SNe cannot be classified with just two or three data points
Julien agrees, but believes representative training set can help.
In chat, Q to Ken from AndyL, “if Cassandra is widely used in industry, how come we don’t know how to use it? Is it because in Industry its used internally, not by independent users?”
Andy asked Ken about Cassandra and noted concern that not sure how users will interface with Cassandra. Can we learn from Facebook use?
Matter of choice of technology, and past experience of relational databases. Use of Cassandra has been learned from scratch.. Facebook, etc. also use MySQL, etc. Ken has been sampling community use of databases, to see how well supported Cassandra might be. Cassandra just means outsdie of database. Could be file system, or something else.
Andy believes need to consider user requirements in more detail. Ken asked whether Dave had a sense of how many objects could classify per second (the Q in Zoom chat “ do you have a feel about how may transients Sherlock can classify per second?”)
Dave noted recent speed tests, which achieved ~10k per second.
In zoom chat Q Eric to all : “most of the examples we’re discussing here are the explosive transients. Is that reflecting the actual or desired user communities of Lasair, FINK, Ampel? What are the ambitions ( or not) to support variables/AGN/solar system science?”
Eric noted focus on extragalactic transients, which is a mature field, but asked whether there was any ambition/ interest in supporting solar system, variable star, or AGN communities?
Meg noted that this is her interest in Lasair, and ability to exploit Cassandra for Solar System alerts. Ambition to create Sherlock peer to classify solar-system alerts, and intent to pursue funding for this.
AndyL notes same issue for variablew stars, but believes Lasair should be focused on Transents. Mistake to try to reverse engineer for other applications. Integration with DRs and RSP should allow users to undertake variability science, along side Lasair-focused transients. This is an unsolved problem.
Sara stated in Zoom chat “There is interest from the White Dwarf community in using Lasair to look for drop outs/eclipses in WD lightcurves.
Roy noted diffretence magnitudes in ZTF make it difficult to compute proper variables.
Eric noted Rubin would provide more rigourous forced photometry, which would help.
Meg noted less urgent need for turn-around for solar-system science. Next night is good enough.
Answer from Ampel side from Jakob in Zoom chat - “Our current user base mainly involve extragalatic greoups, so AGN modelling is include there. So it is not that we do not want to do it, but we do not have a lot of feedback from eg solar system groups regarding their LSST needs. would be happy to get that, though.”
Stephen noted intent to help solar-system find objects in Lasair, but then link that to the DR olight curve in Rubin, as useful.
JulienP Q in Zoom chat : “ For Fink: the LSST-FR community is, for historical reason highly focussed on explosive transients (incl. SN and MMA). But things change. We now support more and more the SSO science, with a large group of experts joining the team. However, we clearly lack of variable star experts (although we would love working on this as they seem to make most of the stream).”
1620Session-5: User Interface (chair: Dave Young)
Queries, streaming, mining (Roy Williams)
Will cover how the query builder, streaming system, and the light curve mining all work.
Lasair query builder uses very SQL-like syntax
Actions on query depend upon whether or not you are logged in and if it belongs to you or is public.
Checker confirms if the likely run-time will exceed limits
You can ask the Lasair team to promote your query to be public on he Lasair site
Email alert for the query is limited to one per 24 hours.
Soon we will provide annotation capability and if liked can be enabled to allow annotations to be pushed back into Lasair.
Mining light curves system constituent parts will all run on the same cloud.
Julien Q - how do you avoid a SQL injection through the web interface? (Julien Q in Zoom chat : “what about SQL code injection? any internal LIMIT to avoid running full data retrieval? I’m always afraid to put a query builder public and unsupervised …”)
Roy answer - suggests Julien try and hack and report back. Also there is a validation system that checks for keywords.
DaveY also answers - DB is also read-only
Roy anwser in Zoom chat - “There is a LIMIT 1000 added to all queries, and an execution timeout.”
Jakob Q - will you be able to make queries based on other people's annotations?
Roy answer - yes, but there will be delay.
Jakob Q - Is there an internal log system on when queries were run?
Roy answer - currently tracking all queries being run but not sure what to do with that information.
Web interface (Andy Lawrence)
(need slides - messaged him)
Lasair philosphy is to keep it simple. Focus on functionality rather than appearance.
Beginning to define final user interface for start of operations.
Keen to hear comments and guidance from participants.
Functionality
canned queries – e.g., cone search
query builder – custom queries
documentation - user guidance examples on website and a cook book on LSST:UK wiki
helpdesk - email (also GitHub [tickets], though not for regular users)
Embedded image viewer/ interactive plotter for visualising rich information that is available.
Open questions
Is navigation and user flow obvious?
Should we provide general information for public?
Should we have more sophisticated documentation?
Should we aim for more-interactive web pages, forms, etc.?
Sara Q in Zoom chat “I wonder whether some of the information for non-scientists/ interactive stuff could be put together in one of the LSST proposals to the current call”. At NAM there are keen amateurs using Lasair and so UI improvements might be helpful
Meg - the call is only 30K USD, ~ 3mths of post-doc)
Roy helps establish a collaboration
Meg - should you be focussing on Lasair ZTF or is the goal Rubin Lasair? Would the call be better spent elsewhere?
Jacob likes how Lasair information is organised, on a single webpage, very easy to find and scan.
Eric answer - for own science often go to Lasair initially since it is convenient.
George noted “For documentation, I wondered if we could follow a Rubin lead? We should find out what kinds of documentation Rubin will provide. By mirroring their style and tools we can provide something seamless and familiar.”
Eric answer - More documentation is good. Rubin has dedicated team for this.
1710 API and notebooks (Ken Smith)
(need slides)
Lasair team has introduced REST APIs using Django Rest Framework (looks to be defacto standard)
See http://lasair-iris.roe.ac.uk/api for initial notes from Roy
Effectively, these are machine-readable versions of functions provided (interactive) on webpages
/api/cone -
/api/query
…
Python wrapper “lasair” (available via PIP install) to help use API.
Plan to add support for querying Cassandra directly
Also have Jupyter Notebook examples, hosted on Google Colab
Need user account on Lasair to access
1720 Tools interface (Andy Lawrence)
1730: Discussion