DIA template generation how-to

DIA/DIA_How_To_Generate_a_Template_Image.ipynb

@@ -0,0 +1,300 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# DIA: How To Generate a Template Image\n",
+    "<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
+    "<br>Last Verified to Run: **2018-08-24**\n",
+    "<br>Verified Stack Release: **16.0**\n",
+    "<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
+    "\n",
+    "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
+    "\n",
+    "### Learning Objectives:\n",
+    "\n",
+    "After working through this tutorial you should be able to: \n",
+    "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
+    "2. Combine these images into a decorrelated template image.\n",
+    "\n",
+    "### Logistics\n",
+    "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
+    "\n",
+    "## Set-up"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# What version of the Stack am I using?\n",
+    "! echo $HOSTNAME\n",
+    "! eups list -s | grep lsst_distrib"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "! cd .. && python setup.py -q develop --user && cd -"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stackclub import where_is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Background\n",
+    "\n",
+    "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
+    "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
+    "\n",
+    "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
+    ">\n",
+    "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
+    "\n",
+    "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
+    "\n",
+    "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
+    "\n",
+    "* **Standard decorrelated coadds**\n",
+    "\n",
+    "* **Constant-PSF partially-decorrelated coadds**\n",
+    "\n",
+    "* **PSF-matched coadds**\n",
+    "\n",
+    "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The Twinkles_subset Dataset\n",
+    "\n",
+    "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here's what this data repository contains:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!ls -lh TWINKLES_DATA/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!cat TWINKLES_DATA/_mapper"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "All of the relavent images and calibrations have already been ingested into the Butler for this data set."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Making a SkyMap\n",
+    "\n",
+    "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
+    "\n",
+    "# cf the following line from Dominique Fouchez's minicookbook:\n",
+    "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The above command fails with the following error:\n",
+    "```\n",
+    "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
+    "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
+    "```\n",
+    "\n",
+    "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "help(MakeDiscreteSkyMapTask)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "skyMapConfig = MakeDiscreteSkyMapConfig()\n",
+    "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
+    "skyMapTask.run()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Making the Coadd Images"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\"\"\"\n",
+    "#find the tract/patches overlapping visit 182014\n",
+    "./reportPatchesWithImages.py output --visits 182014 --filt r\n",
+    "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
+    "\n",
+    "#make coadd on these patches to create template\n",
+    "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
+    "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
+    "\"\"\";"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

DIA/README.rst

@@ -0,0 +1,29 @@
+Difference Image Analysis
+-------------------------
+
+This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
+To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)
+
+.. list-table::
+   :widths: 10 20 10 10
+   :header-rows: 1
+
+   * - Notebook
+     - Short description
+     - Links
+     - Owner
+
+
+   * - **How_To_Generate_a_Template_Image**
+     - Options for template coadd generation, demonstration on Twinkles images.
+     - `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
+       `rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__
+
+       .. raw:: html
+
+             <a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
+               <img width="72" height="16" src="https://github.com/raw/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
+               </img>
+             </a>
+
+     - `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__

GettingStarted/templates/template_Notebook.ipynb

@@ -42,24 +42,16 @@
     "```\n",
     "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
     "```\n",
-    "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
+    "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
-    "! cd .. && python setup.py -q develop --user && cd -"
+    "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
    ]
   },
   {
@@ -71,14 +63,30 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%load_ext autoreload\n",
     "%autoreload 2"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "During development, it's helpful to have the `where_is` documentation finder loaded in:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from stackclub import where_is"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {

README.md

@@ -16,6 +16,7 @@ the LSST Science Collaborations.
 | Basics           | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics)  |
 | Visualization    | Displaying images and catalogs. | [StackClub/Visualization](Visualization)  |
 | Image Processing | From raw images to `calexp`s and `coadd`s.  | [StackClub/ImageProcessing](ImageProcessing) |
+| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
 | SourceDetection  | Detection of sources in images - including low surface brightness galaxies.  | [StackClub/SourceDetection](SourceDetection)  |
 | Deblending       | Deblending the objects | [StackClub/Deblending](Deblending) |
 | Measurement      | Measuring the objects | [StackClub/Measurement](Measurement) |