...
Cross-match against NED-D is throttled and cached to avoid Sherlock being black-listed by NED-D.
Aim to reduce cross-match to integer-based search fields to speed up queries.
Roy noted that, by end of first year, depth of LSST survey will make most other surveys less relevant. How does this impact role of Sherlock?
Dave Y noted that, for SNe, NED was critical for classification. LSST would help to eliminate ‘background fog'. For QUB, this will be a critical use case.
Roy W noted that Lasair will likely serve other applications, as well as SNe.
Stephen S noted that LSST catalogue (inc. photometric redshift) will be huge help for objects with redshift less than mag. 22, though will not help astronomers to identify those objects for which spectroscopy is interesting.
Data releases will also be critical if choose to separate stellar sources from extragalactic sources.
Bob asked if there was information in the alert regarding stellar sources vs. extragalactic source.
Gareth F noted conflict of tuning for particular applications vs. different use cases. Would it worthwhile to produce confidence information for classification results?
Dave Y noted that Sherlock produces a list of all sources that the transient has probably been associated list. The top-level result, from Sherlock, is the ‘tip of the iceberg’. Also, agrees that providing one-off confidence value would be useful.
Visualisation of Sherlock classification would make it more immediate for users to identify source that has been crossmatched and algorithm that has been most successful.
Dave M asked if there was a case to offer power users, for Sherlock classifier, to individual use cases.
Dave Y suspects users would want to write own algorithms, but that we should simply expose the underlying algorithms to give power users the information they need.
Andy L noted that post-ingestion data mining was a different problem from immediate real-time classification, which was focus for Lasair.
Andy L asked where Sherlock catalogues are stored and processes run.
Sherlock instance and catalogues, for Lasair, is running at ROE.
Instance tracks master version, which runs at QUB.
Matt N noted recent issues with Sherlock, related to offsets not being correctly computed.
Dave Y confirms that this has been fixed (around 12 months ago) though hadn’t been copied to ROE instance.
New transients, since 12th March 2020, should be fine.
Lasair planning process (Andy Lawrence)
George asked whether consideration of collaborating with other alert-broker providers had been considered.
Andy notes some discussion but no hard conclusions to date.
Roy noted three largest projects are Alerces, Antares, and Lasair. We are trying to collaborate with these three, plus may be option to work with French work.
Andy L observed, at Community Broker meeting, that not all parties were interested to provide a full service. Some of these may wish to join one of the big three.
Andy noted open question about how Lasair:ZTF and Lasair:LSST would coexist.
Stephen S believes technology choices need to be made within the next twelve months.
Dave M noted original idea of providing black-box filter to community broker, to be run on their behalf.
Andy L noted idea of providing a containerised platform which others could take were dropped when funding for Phase B was cut.
Roy noted that Antares are planning to support mini-brokers, taking Python code and incorporating into their service.
Dave Y noted online platforms, such as Digital Ocean, who have a community who blog on how to use the platform. Possibly, Lasair could provide blogs/ tutorials/ code snippets, to help others to extend our services.
Roy asks if this is addressed by Jupyter Notebook.
Dave Y noted that Jupiter Notebook is great but limited.
Roy noted that open question how to break out of the notebook.
Roy Williams, Technology Challenges for Lasair
SQL is not a universally understood technology
Semantic differences between static queries and streaming queries--e.g. related to ordering of events/ query results.
How do we provide a consistent filter expression approach for static vs. active queries?
Strasburg team (CRS) has published a web-based form to help people form their query
Meg S not convinced that CRS form is a good way to go. Perhaps better training on how to build good SQL queries.
Bob noted that for WFAU archives, not common to receive poor queries. Plus, was generally easy to flag queries likely to be badly formed.
Mark K noted ZTF database has to support ingest alongside user queries.
Meg S noted that it was possible to input what could be done in a SQL query. Also, noted her comment was related to user experience rather than query performance,
Simply queries unlikely to be enough when dealing with billions of rows of data.
Data mining techniques will be needed
Use of storage approaches – databases vs. blob store, and which data should go where?
Ingestion of new data (LSST alerts) is on the critical path for the service?
Hardware technology choices and implications
What will we do for user data
Service resilience has not up until now been considered seriously.
Andy L noted need to make decision about query-language support.