LSST Software Tutorial Notes

For the Hands On session, we propose to run through a tutorial that was previously run in Chile last year, by Colin Slater and Mario Juric. This page contains notes from my experiences of installing the necessary software and running through the tutorial exercises.

Processing Raw Data

This part of the tutorial involves processing some "raw" data into a calibrated image and a catalogue of measured sources, using the LSST software pipeline. The required software can be installed directly on Mac OS X and Linux (64-bit only). On Windows it can be installed and run using Docker.

Installing on UNIX platforms (Mac OS X and Linux)

The simplest way to install the LSST software is as a Conda package. This is described here: https://pipelines.lsst.io/install/conda.html. A basic development environment including gcc is required; this is normally present on Linux, but on Mac OS X it may be necessary to install Xcode to obtain it. We recommend using Python 2.7; we have not tested the tutorial with Python 3 and some users have reported difficulties. We also recommend using the Bash shell, as the script used to activate modules installed via Conda does not support other shells.

After installing Conda (either Anaconda or Miniconda), the full set of steps necessary to prepare for the tutorial is as follows:

conda update conda
conda config --add channels http://conda.lsst.codes/stack/0.13.0
conda create --name lsst python=2
source activate lsst
if you get an error like: activate: No such file or directory; this means you are not probably running bash; conda assumes you are running bash.
conda install lsst-distrib
source eups-setups.sh
setup lsst_distrib
pip install sncosmo iminuit
conda install lsst-obs-decam
setup obs_decam

When opening a new shell, the environment needs to be set up before running the LSST software. The steps necessary for this are:

source activate lsst

source eups-setups.sh

setup lsst_distrib

If conda was not added to your PATH during its installation process, you will need to add it before running the other three commands.

Installing on Windows using Docker

On Windows, installing the LSST software via Docker offers a simpler alternative than setting up a Linux VM manually. For versions of Windows up to and included Windows 8.1, you should use the Docker Toolbox, which runs Docker inside a Linux virtual machine. For Windows 10, a native version of Docker CE is also available. These instructions pertain to the Docker Toolbox: those wishing to use Docker CE should consult the included documentation for spawning and configuring containers.

The easiest way to manage your Docker environment on Windows is using the Docker Quickstart Terminal (usually available from the Windows Start menu). This launches a lightweight Linux virtual machine, inside Virtual Box, and opens a terminal console, via which you can manage your containers.

To run a container with the LSST software installed, simply enter:

docker run -ti jamespepcc/lssttutorial:1.0

It will take some time to run this the first time, as the container (around 2 Gigabytes of data) has to be downloaded from a Docker repository: subsequent runs will usually be much faster, as the container is cached locally.

Within the container, run the following in order to set the environment ready for running the tutorial:

. /opt/lsst/software/stack/loadLSST.bash

setup lsst_distrib

setup obs_decam

One issue with using Docker is that, by default, any files created within the container environment are lost when Docker exits. This can be overcome by mapping a local directory to a directory within the container image and moving any files that should be kept into this directory. For example, to map the Windows directory C:\Users\James\Documents\lsstshared to the directory /lsstshared within Docker, use the command:

docker run -v "//c/Users/James/Documents/lsstshared:/lsstshared" -ti jamespepcc/lssttutorial:1.0

Then, after using the LSST software within Docker, copy any generated data that you wish to keep into /lsstshared and it will remain in the corresponding Windows directory even after Docker shuts down.

There are some limitations with the shared directories: for example, actually running the commands described below within /lsstshared will not work as symbolic links are not supported in the shared directories (at least on Windows). Also, sharing directories outside C:\Users with Docker containers may not work due to permission issues.

Processing the data

(These instructions apply to both Docker and Conda installations).

The necessary scripts and documentation for the tutorial can be obtained from: https://github.com/lsst-dmsst/demo-lsst-uk-2017, and an example input image is available at: http://lsst-web.ncsa.illinois.edu/~ctslater/decam_visit.tar.gz. These can be obtained with the following commands:

git clone https://github.com/lsst-dmsst/demo-lsst-uk-2017

cd demo-lsst-uk-2017/ProcessCcd

wget http://lsst-web.ncsa.illinois.edu/~ctslater/decam_visit.tar.gz

(Alternatively, a copy of the GitHub repository is attached to this wiki page as a zip).

Once these have both been downloaded, the data can be "ingested" as follows:

tar xzf decam_visit.tar.gz

mkdir repo

echo lsst.obs.decam.DecamMapper > repo/_mapper

ingestImagesDecam.py repo --filetype instcal decam_data/instcal/*.fits.fz

Please note that the ingest command produces a number of messages, including being unable to find an sqlite3 database file, however these are only warnings and this does not mean it has failed.

Finally, to actually process the data, run:

processCcd.py repo --id visit=303527 ccdnum=10 -C my_config.py --rerun my_reduction

For further details, please see the PDF file in the GitHub repository (a copy is also attached to this wiki page).

Light Curve Analysis

In addition to the raw data processing described above, the tutorial also contains some iPython notebooks that demonstrate basic analysis of light curves. Jupyter (the software needed to run the notebooks) can be installed using Conda:

conda install jupyter

Once installed, the notebook server can be started by running:

jupyter notebook

from within the directory containing the iPython notebooks (demo-lsst-uk-2017/Lightcurves). This will open the notebook viewer in your default web browser.

The notebook that generates the light curves (generate_lightcurves.ipynb) is unlikely to work as it requires a connection to a database in Washington, however the light curves are already present in the same directory as Pickle files, so the other notebooks can be used to analyse these. As well as runnable Python source code, the notebooks also contain instructions and other useful information and should be quite self-explanatory to work through.

If you wish to work through the code snippets in the optional "Debugging" section of the analyze_RR_lyrae.ipynb notebook, some further configuration is required in order to do this without errors. Firstly, the LSST package sims_sed_library needs to be installed. This can be done by running the following commands before starting the notebook server:

conda config --add channels http://conda.lsst.codes/sims

conda install lsst-sims-sed-library

setup sims_sed_library

And secondly, two minor corrections need to be made to the Python code snippets within the notebook. In the first snippet below the "Debugging" heading, replace the function call argmax(total_power) with np.argmax(total_power). Finally, add a new line to the top of the very last snippet:

import re

It should now be possible to run through all three analysis notebooks in their entirety.

Light Curve Analysis using Docker

The iPython notebooks for the light curve analysis can also be run on Windows using Docker. The Docker image described above has Jupyter and the notebooks' dependencies pre-installed, though some configuration is required to make it work. In this case the notebook server itself runs within Docker and is accessed through a web browser on the host Windows system via a forwarded port.

In order to do this, first run the Docker container, set up the environment for LSST and clone the required project from Git as before:

docker run -p 8888:8888 -ti jamespepcc/lssttutorial:1.0

. /opt/lsst/software/stack/loadLSST.bash

setup lsst_distrib

setup obs_decam

git clone https://github.com/lsst-dmsst/demo-lsst-uk-2017

The additional -p 8888:8888 switch forwards TCP port 8888 from the container's internal network interface to an external interface from which it can be accessed by the host operating system and applications.

Because we are running the Jupyter server and the web browser client on different (virtual) machines, it is necessary to set up password authentication for access to the notebook server rather than the default token authentication. It is also necessary to manually specify the IP address when starting the server. To do this, follow the steps below:

mkdir ~/.jupyter

jupyter notebook password

cd demo-lsst-uk-2017/Lightcurves

jupyter notebook --ip 0.0.0.0

The second command will prompt you to enter a password and then confirm it. You can set the password to anything you like, but make sure you remember it - you will need it in order to connect to the notebook server.

The server is now running within the Docker container. To access it, open your standard Windows web browser and visit the address http://192.168.99.100:8888/. If all goes well, you should see the Jupyter notebook login page. Enter the password that you set previously to get to the notebooks.

If you get an error such as "Connection refused", it is possible that the Docker VM's IP address is different on your machine. To find out what it is, open the Docker Quickstart Terminal and make a note of the default IP address printed just below the whale graphic.

Using Docker CE for Mac

It is also possible to run the Docker image on Mac OS using Docker Community Edition for Mac. This avoids having to manually install the LSST software and dependencies as described above, though it also has potential disadvantages (more resource usage due to Docker overheads, less integration with the host system).

Running the Docker image on Mac OS is very similar to running it on Windows. In fact, there are only two differences:

The shared folder switch uses standard Mac path names rather than the non-standard MinGW-style path names used on Windows.
Docker CE for Mac forwards ports to localhost rather than to a different IP address. So in order to access the Jupyter notebook server running in Docker, you just need to visit http://localhost:8888/ in your normal web browser (after forwarding port 8888 as mentioned above).

Other examples

During the hands-on session, several other iPython notebooks were investigated, from https://github.com/RobertLuptonTheGood/notebooks/tree/master/Demos.

A data set for use with these notebooks can be downloaded from: https://datasync.ed.ac.uk/index.php/s/FAhHj7mcpqCktLA (the password for the link is "password").

There are a few things to note when working through these notebooks:

You will need to replace the data path in each notebook (/Volumes/RHLData/hsc-v13_0) with the location where you unzipped the data set.
The Deblender notebook depends on the LSST Analysis software, available from: https://github.com/lsst/analysis. This is not installed as part of a standard LSST installation, but you can clone it from GitHub and then add the python/ directory within to your PYTHONPATH environment variable.
A number of the notebooks can make use of the standalone DS9 software package in order to display their outputs. This can be obtained from: http://ds9.si.edu/site/Home.html - simply add the binary to your PATH. (As it is a GUI application, it will not launch in Docker unless you first set up X11 forwarding or VNC).
A few notebooks are missing import statements. To run the Kron, PSF and Tune Detection notebooks, it is necessary to add "import lsst.daf.base as dafBase" along with the other import statements at the top of the notebook.
Additionally, the final section of PSF.ipynb requires two more imports to be added: "import lsst.afw.geom as afwGeom" and "from math import pi"