Transient ideas

Ideas about transient and alert processing for the LSST:UK DAC.

The initial choice of technologies is based on the ideas presented by Maria Patterson at the May IVOA interop describing their plan to use Apache Kafka as the main event handling system within LSST.

Current list of candidate technologies are :

Relevant GitHub projects:

Phymatopus Dave's GitHub project to experiment with building and running Maria's Kafka experiments.
Maria's fork of the LSST DM alert_stream code.

Topics for discussion:

Using the transient data from Pan-STARRS and ATLAS that Queens have collected to create a simulation of the firehose data stream we expect to get from LSST.
Outline some end user use cases for how we think people will want to interact with that stream.
What software is available that would enable us to build a system that would meet those needs.
How we can use tools like virtual machines and Docker to enable us to start building small experimental prototypes now, before we have any real idea what the final system will look like.

Morning discussion

Extract the 'observable' events. Why send out events for the 'unobservable' ?

Bright things produce better light curves and spectra. Many of the transient detected by LSST will be too faint to be useful ..

Moving object science want to observe wherever it is in the sky.

Quasars - slow variations over time of known and unknown sources.

Interesting objects within (radius) of my known source.

Events within the galaxy I'm interested in.

Concept of 'expected supernovae' ...

Superluminous supernova - requires a spectrum to distinguish

Bright source at high latitude is more likely to be external to our galaxy.

Variable star community, issues with current LSST cadence.

Near earth asteroids - where the money came from.

Gravitational Wave optical Transient Observatory (GOTO) planning to use LSST stack for data processing.

Pesto takes in events from lots of surveys, cross references across a range of surveys.

LSST events are BIG - lots of parameters

Essential criteria:

ra, dec
Does it have a history ?
- LSST data will include history of an object (only within 1yr of LSST observations)
- Don't know how that association is done - fixed radius ?
- Bright objects saturate detectors - can cause problems to distinguish nearby objects
- Year 1 to year 2 how do we associate the history y1<->y2?
- Dense star field - do we reduce the association radius ?
Cut out images in events
- LSST events will include cut out
- core (30x30 pixels) = small area
- primarily for machine learning - is this PSF like ?
Queens normally get basic
- params, ra dec, history, bright enough ?
- THEN get the stamps
- Gaia doesn't give stamps
- LSST are the only people who can produce stamps because they have the pixels
LSST:UK produce a digest stream of 'interesting' objects
'Transient server
Database grows as we add more and more attributes
Do we send fully annotated alerts ?
If we send simple alerts, then we get a lot of callbacks
If we send everything, then alerts are large
Who are the customers ?
Ourselves - SuperNova community is already involved in Pest
(Roy has list of suggested people)

Different sources will produce different radius of interest
Cross referencing need to bear in mind accuracy of the criteria
- LIGO is a healpix MOC map
- Fermi follow up of LIGO, within 40 arc min of ra,dec
- Neutrion events from ICECUBE, 40 arc min

rare that magnitude changes within one night (apart from flare stars)
shape measurements
'interesting' shapes are likely to be ...
the more measurements we have the more we can use machine learning
if it has been seen before - classified as a recurring transient, not interesting to SuperNova group, but may be interesting to others.
example - originally catalogued as a SuperNova, but re-occurred a few years later, so not a SuperNova
10,000 of SuperNova in catalog in Israel
initially registered as AT Astronomical Transient
classified as a SuperNova, registration updated to
cross match alerts with existing catalog
- Queens catalogs
- ROE catalogs
- idea for a SuperCatalog of positions and identifiers from a set of catalogs
- LSST generating 11 million 'real transients' per night
- filter on brightness foisrt
- faint thinsg left to process later
- Guestimate 100 million objects per night ?
- Of which 11 million might be real
- Who does the 'real' filtering ?
LSST plan to give transients within 15 min of discovery
- single observation within 1min
- combined observation within 15min ?
Current Queens system
- Pan-STARRS and ATLAS transients are collected in batches (pull)
  - Pan-STARRS daily - internal processing is daily batch
  - ATLAS twice daily
- LSST will be continuous stream (push)
Is there a use case for rapid followup within seconsds ?
- RedFlare - occurs within 15 min
- SuperNova flash breakout - re-observe within ..
ZTF deep drilling fields, re-visiting several times

Darren, current work on parallel computing

Southampton core-collapse
Starting to look at LSST QServ

LSST using QServ for L2 data

spatially distributed data with overlap
each processor gets a piece of sky
user interaction is with head node

Use cases

Astronomer comes up with list of interesting object
Would subscribe to LSST transients within radius of their objects
Ken has used install notes from Marcus
Less interested in it as a live event database - ingest is too slow
Interested in it for storing trillion object database
ROE has a history of previous queries to the archives
Pan-STARRS has an equivalent history
Process our historical queries to categorise as query patterns ?
Reference stars from Pan-STARRS data
- Always a spatial search
Area queries
- Cone - ra,dec,radius
- MOC
- Q3C
- Google S2 (does this cover the poles?)
Investigation, compare Q3c with HTM, HealPix, Google S2 etc.
- Particularly at the poles
- C++ API is not the same as the C#
- HTMIDs may be different
Exclude bright stars using HTMID of the known bright stars
- Pan-STARRS 14th mag
- ATLAS 8th mag
- LSST ??

metadata table
created from Pan-STARRS text files
reference links back to the original file
MJD id, exposure id

Note - ATLAS database schema may change in the near future

detection table

Convert Queens data into a LSST event stream.

Need to create a mapping between the ATLAS / Pan-STARRS fields.

Use JOIN queries to denormalize the Queens data.

Better fit for Pan-STARRS data, more tricky to do the conversion for ATLAS.

Python modules for conversion DB row ↔ JSON structure.

Link to Avro schem in Maria's code ..

Afternoon white board session

Discussion about who are users are and what use cases they might have:

Users and use cases

Discussion about a technology prototype:

Technology prototype

Proposed first steps we can take:

Proposed first steps