...
Different interfaces – need to prioritise
Webpage
Scripts (Python, primarily)
Other projects (website)
iDAC/ RSP interface – opportunity in UK, as IDAC is next to Lasair broker, but needs requirement analysis and design work
Topcat – need TAP
Personal storage – e.g., MyDB, VOSpace, …
Eric asked how Kafka authentication is being handled.
Roy noted hope not overwhelmed with requests [for credentials]
Ken noted experience of writing TAP service, which could be useful (Guy Rickson and Thomas Marquat (sp?)).
Jacob asked if annotation is same as a classification
Roy noted one type of annotation, but other classifications were possible.
Julien asked about usage split between REST and Kafka. (Q in zoom chat “how is the usage split between the REST API and Kafka? as an example in Fink, most of users use the API, and very few Kafka.”)
Roy noted that people are very conservative and stick to interfaces they know.
Julien hopes, over time, people will become familiar with new technologies, which will have benefits for users.
Andy notes absence of user tools is a barrier to uptake of Kafka. Generally speaking Kafka and Cassandra are not targetted to end users: they are behind the science, meaning astronomy is unusual.
Dave noted size of data to be handled. Kafka can handle a stream which would cause APIs to fall over.
Ken noted not intended to expose Cassandra to end users. Vision is to have a wraper (web page, for example) to hide Cassandra. PanSTARRS does something similar, for cone-search query.
Jakub asked what Lasair would do if one of Ken, Dave, or Roy moved on.
Stephen answer - there would be a serious delay to replace staff
In Zoom chat George B answered “We are in discussions with Software Sustainability Institute (http://www.software.ac.uk ) on good practice to try and ensure we are less affected by staff turn-over. Open development, code reuse, multi-role appointments/ rotations, all help.”
In Zoom chat Dave Y answered “paired programming has been excellent for developing the SOXS pipeline. 2 developers with shared knowledge of the code. One of us could leave and the project would not lose much knowledge (tacit or explicit).”
Eric Q asked what are the plans in the next cycles regarding scientific integration with Rubin as it starts to come online. What do you need and what can Rubin provide? Data, schemas?
roy - various services that LSST is goimg to offer. It does not need to be good data since already have the dummy data. Would like to have the Kafka stream so can see how fast to read. It would be good to check the plumbing.
AndyL answer - simulated streams and commissioning data in order to exercise the system. Some sort of simulacrum of the DRs in the RSP would help test the integration of the broker and DAC in order to do variability science. do need tyest data.
Eric - all very possible. There are the onging data previews. Is anyine here engaged
Jakob - was under the impression that this is not for the alert streams
Eric - correct but useful for understanding the RSP and interfaces, D0.2 will still be simulated data. Will take as action if can get preferential lane for the community brokers to ensure have easy access to that. On alerts, working on production Kafka stream but do not have timings yet. Definitely expect some of what Roy wants in the next year. some of the other services will be a little harder to mock up.
Jakob - follow-up to simulated alerts. Q to Julien - how to integrate with the data elastic challenge.?
Julien - yes looking at this. Have not heard back though.
Jacokn - interested in the elastic to parse it in order to test throughput
Jlien - agree it is plumbing work
Ken Q - when do Ampel/Fink transition from ZTF alerts? Do you envisage having to run two parallel brokers?
Roy - not paid to do ZTF
Ken - but we are not going to stop ZTF during the transition period.
Jakob - not decided yet. Have designed a system that can run in parallel and ingest both streams but may run separately next to each other
Julien - simpler since not tied to ZTF. Broker part is survey-agnostic but processing part is different. On DB side, need to investigate more. Process live with translation at the beginning
Andly AndyL - what about maintaendance maintenance ? this is where the effort goes ? May keep ZTF running but with no prmises promises about maintenance. If important enough for users, then will need the money to do so. Q to Eric - what about the partial transition.
Eric - ZTF current NSF funding for public stream is to end 2023. That is before Rubin full operations in early 2024. Overlap therefore unlikely depends on funding. ZTF unlikely to be thrown away but long tail of maintenance unlikleyunlikely. On Rubin, it will be a slow ramp-up to full alert volume. even in 1st year of operations.
Roy - perhaps get funding for a multi-survey transient system.
Julien Q to Roy - what is on the screen behind
Roy - the ZTF transient stream
AndyL - need to digest the findings. Many thanks to the externals.
Terry add link to yjhe meeting page.
1730: Discussion