...
Current list of candidate technologies are :
Relevant GitHub projects:
- Phymatopus Dave's GitHub project to experiment with building and running Maria's Kafka experiments.
- Maria's fork of the LSST alert_stream code.The LSST DM alert_stream code.
Topics for discussion:
- Using the transient data from Pan-STARRS and ATLAS that Queens have collected to create a simulation of the firehose data stream we expect to get from LSST.
- Outline some end user use cases for how we think people will want to interact with that stream.
- What software is available that would enable us to build a system that would meet those needs.
- How we can use tools like virtual machines and Docker to enable us to start building small experimental prototypes now, before we have any real idea what the final system will look like.
...
Morning discussion
...
- rare that magnitude changes within one night (apart from flare stars)
- shape measurements
- 'interesting' shapes are likely to be ...
- the more measurements we have the more we can use machine learning
- if it has been seen before - classified as a recurring transient, not interesting to SuperNova group, but may be interesting to others.
- example - originally catalogued as a SuperNova, but re-occurred a few years later, so not a SuperNova
- 10,000 of SuperNova in catalog in Israel
- initially registered as AT Astronomical Transient
- classified as a SuperNova, registration updated to
- cross match alerts with existing catalog
- Queens catalogs
- ROE catalogs
- idea for a SuperCatalog of positions and identifiers from a set of catalogs
- LSST generating 11 million 'real transients' per night
- filter on brightness foisrt
- faint thinsg left to process later
- Guestimate 100 million objects per night ?
- Of which 11 million might be real
- Who does the 'real' filtering ?
- LSST plan to give transients within 15 min of discovery
- single observation within 1min
- combined observation within 15min ?
- Current Queens system
- Pan-STARRS and ATLAS transients are collected in batches (pull)
- Pan-STARRS daily - internal processing is daily batch
- ATLAS twice daily
- LSST will be continuous stream (push)
- Pan-STARRS and ATLAS transients are collected in batches (pull)
- Is there a use case for rapid followup within seconsds ?
- RedFlare - occurs within 15 min
- SuperNova flash breakout - re-observe within ..
- ZTF deep drilling fields, re-visiting several times
Darren EPPC, current work on parallel computing
- Southampton core-collapse
- Starting to look at LSST QServ
...
- spatially distributed data with overlap
- each processor gets a piece of sky
- user interaction is with head node
Use cases
- Astronomer comes up with list of interesting object
- Would subscribe to LSST transients within radius of their objects
- Ken has used install notes from Marcus
- Less interested in it as a live event database - ingest is too slow
- Interested in it for storing trillion object database
- ROE has a history of previous queries to the archives
- Pan-STARRS has an equivalent history
- Process our historical queries to categorise as query patterns ?
- Reference stars from Pan-STARRS data
- Always a spatial search
- Area queries
- Cone - ra,dec,radius
- MOC
- Q3C
- Google S2 (does this cover the poles?)
- Investigation, compare Q3c with HTM, HealPix, Google S2 etc.
- Particularly at the poles
- C++ API is not the same as the C#
- HTMIDs may be different
- Exclude bright stars using HTMID of the known bright stars
- Pan-STARRS 14th mag
- ATLAS 8th mag
- LSST ??
...
Link to Avro schem in Maria's code ..
...
Afternoon white board session
Discussion about who are users are and what use cases they might have:
Discussion about a technology prototype:
Proposed first steps we can take:
