Page Comparison

A page for ideas about transient and alert processing for the LSST:UK DAC.

The initial choice of technologies is based on the ideas presented by Maria Patterson at the May IVOA interop describing their plan to use Apache Kafka as the main event handling system within LSST.

Current list of candidate technologies are :

• Apache Kafka
• Apache Spark
• Apache Flink
• Ignite

Relevant GitHub projects:

• Phymatopus Dave's GitHub project to experiment with building and running Maria's Kafka experiments.
•  Maria's fork of the LSST alert_stream code.
• The LSST DM alert_stream code.

Topics for discussion:

• Using the transient data from Pan-STARRS and ATLAS that Queens have collected to create a simulation of the firehose data stream we expect to get from LSST.
• Outline some end user use cases for how we think people will want to interact with that stream.
• What software is available that would enable us to build a system that would meet those needs.
• How we can use tools like virtual machines and Docker to enable us to start building small experimental prototypes now, before we have any real idea what the final system will look like.

Bright things produce better light curves and spectra.

Many of the transient detected by LSST will be too faint to be useful ..

Extract the 'observable' events.

Why send out events for the 'unobservable' ?

Moving object science want to observe wherever it is in the sky.

Quasars - slow variations over time of known and unknown sources.

Interesting objects within (radius) of my known source.

Events within the galaxy I'm interested in.

'expected supernovae'

Suplerluminus events - requires spectrum to distinguish

Bright source at high latitude is more likely to be external to our galaxy.

Variable star community, issues with current LSST cadence.

Near earth asteroids - where the money came from.

Pesto takes in events from lots of surveys, cross references across a range of surveys.

LSST events are BIG - lots of parameters

Essential criteria:

...

• LSST data will include history of an object (only within 1yr of LSST observations)
• Don't know how that association is done - fixed radius ?
• Bright objects saturate detectors - can cause problems to distinguish nearby objects
• Year 1 to year 2 how do we associate the history y1<->y2?
• Dense star field - do we reduce the association radius ?

...

• LSST events will include cut out
• core (30x30 pixels) = small area
• primarily for machine learning - is this PSF like ?

...

• params, ra dec, history, bright enough ?
• THEN get the stamps
• Gaia doesn't give stamps
• LSST are the only people who can produce stamps because they have the pixels

...

• Different sources will produce different radius of interest

• Cross referencing need to bear in mind accuracy of the criteria

  • LIGO is a healpix MOC map

  • Fermi follow up of LIGO, within 40 arc min of ra,dec
  • Neutrion events from ICECUBE, 40 arc min

• rare that magnitude changes within one night (apart from flare stars)
• shape measurements
• 'interesting' shapes are likely to be ...
• the more measurements we have the more we can use machine learning
• if it has been seen before - classified as a recurring transient, not interesting to SuperNova group, but may be interesting to others.
• example - originally catalogued as a SuperNova, but re-occurred a few years later, so not a SuperNova
• 10,000 of SuperNova in catalog in Israel
• initially registered as AT Astronomical Transient
• classified as a SuperNova, registration updated to
• cross match alerts with existing catalog
  • Queens catalogs
  • ROE catalogs
  • idea for a SuperCatalog of positions and identifiers from a set of catalogs
  • LSST generating 11 million 'real transients' per night
  • filter on brightness foisrt
  • faint thinsg left to process later
  • Guestimate 100 million objects per night ?
  • Of which 11 million might be real
  • Who does the 'real' filtering ?
• LSST plan to give transients within 15 min of discovery
  • single observation within 1min
  • combined observation within 15min ?
• Current Queens system
  • Pan-STARRS and ATLAS transients are collected in batches (pull)
    • Pan-STARRS daily - internal processing is daily batch
    • ATLAS twice daily
  • LSST will be continuous stream (push)
• Is there a use case for rapid followup within seconsds ?
  • RedFlare - occurs within 15 min
  • SuperNova flash breakout - re-observe within ..
• ZTF deep drilling fields, re-visiting several times

Darren EPPC, parallel computing

• Southampton core-collapse
• Starting to look at LSST QServ

LSST using QServ for L2 data

• spatially distributed data with overlap
• each processor gets a piece of sky
• user interaction is with head node

• Astronomer comes up with list of interesting object
• Would subscribe to LSST transients within radius of their objects
• Ken has used install notes from Marcus
• Less interested in it as a live event database - ingest is too slow
• Interested in it for storing trillion object database
• ROE has a history of previous queries to the archives
• Pan-STARRS has an equivalent history
• Process our historical queries to categorise as query patterns ?
• Reference stars from Pan-STARRS data
  • Always a spatial search
• Area queries
  • Cone - ra,dec,radius
  • MOC
  • Q3C
  • Google S2 (does this cover the poles?)
• Investigation, compare Q3c with HTM, HealPix, Google S2 etc.
  • Particularly at the poles
  • C++ API is not the same as the C#
  • HTMIDs may be different
• Exclude bright stars using HTMID of the known bright stars
  • Pan-STARRS 14th mag
  • ATLAS 8th mag
  • LSST ??

• metadata table
• created from Pan-STARRS text files
• reference links back to the original file
• MJD id, exposure id

Note - ATLAS database schema may change in the near future

• detection table

Re-create Queens data into a LSST event streamThis page has been moved.