Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
783702d
fix coords raises
wx4stg Oct 14, 2024
f9fed05
Merge branch 'master' of https://github.com/deeplycloudy/xlma-python …
wx4stg Jan 11, 2025
ad5dc93
add coordinates documentation
wx4stg Jan 12, 2025
4a77117
add xarray util documentation
wx4stg Jan 12, 2025
612e52b
add flash documentation
wx4stg Jan 14, 2025
7505e99
add dark mode
wx4stg Jan 14, 2025
f55f726
some basic stylization
wx4stg Jan 14, 2025
66b70dc
add glm, lmatools, cf_netcdf documentation
wx4stg Jan 14, 2025
44acbfd
add io.read and lma_intercept_rhi documentation
wx4stg Jan 14, 2025
cc06d48
add gridding documentation
wx4stg Jan 16, 2025
e043be6
add traversal documentation
wx4stg Jan 18, 2025
49c6bec
update svg logo
wx4stg Jan 22, 2025
28fb3c3
standardize LMA dataset wording, add attrs, numpy inventory refs
wx4stg Jan 22, 2025
eefc0fc
add plot documentation
wx4stg Jan 22, 2025
71962a3
add plots markdown references
wx4stg Jan 22, 2025
3967d95
update pyproj link
wx4stg Jan 22, 2025
e6fb39e
add ECEF link/figure
wx4stg Jan 22, 2025
664e8ca
enable inventory/crosslinking
wx4stg Jan 22, 2025
67dc7d9
standardize iterable of syntax
wx4stg Jan 22, 2025
874313f
web beautification
wx4stg Jan 22, 2025
ac4ecea
api in a web link isn't great
wx4stg Jan 22, 2025
04018e9
rename run tests
wx4stg Jan 22, 2025
185eec2
add auto docs build and GH Pages deploy
wx4stg Jan 24, 2025
19cc397
only attempt deployment on pushes to master
wx4stg Jan 24, 2025
14157b1
proj4 is not canon
wx4stg Jan 24, 2025
662cb17
readme modernization, part 1
wx4stg Jan 24, 2025
2481361
Merge branch 'master' of https://github.com/deeplycloudy/xlma-python …
wx4stg Jan 24, 2025
aa37b0d
modernize readme, add install
wx4stg Jan 24, 2025
1091a6d
Add contributing
wx4stg Jan 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions .github/workflows/build-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Build xlma-python documentation.

name: Build Documentation

on: [push, pull_request]

permissions:
id-token: write
contents: read
pages: write

jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: 3.13
- name: Install environment with micromamba
uses: mamba-org/setup-micromamba@v1
with:
environment-file: docs/docs-env.yml
init-shell: bash
- name: Build with mkdocs
run: |
eval "$(micromamba shell hook --shell bash)"
micromamba activate pyxlma-docs
mkdocs build
- name: Upload static site artifact
id: upload
uses: actions/upload-pages-artifact@v3
with:
path: ./site/
deploy:
environment:
name: github-pages
url: ${{ steps.upload.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
steps:
- name: Deploy artifact to GH Pages
uses: actions/deploy-pages@v4
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python

name: Run pytests
name: Run tests

on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
on: [push, pull_request]

jobs:
test:
Expand Down
103 changes: 0 additions & 103 deletions README.md

This file was deleted.

1 change: 1 addition & 0 deletions README.md
21 changes: 21 additions & 0 deletions docs/contributing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Contributor's Guide
---

Discussion of contributions takes place in [GitHub's issues](https://github.com/deeplycloudy/xlma-python/issues) and [pull requests](https://github.com/deeplycloudy/xlma-python/pulls) sections of the xlma-python repository.

To create a "developer install" of xlma-python:

```sh
git clone https://github.com/deeplycloudy/xlma-python.git
cd xlma-python
pip install -e .
```

This will allow you to edit files in the xlma-python directory on your local machine, and the `pip install -e .` command will allow those changes to be applied to your python environment immediately.


xlma-python is built using [setuptools](https://setuptools.pypa.io/en/latest/). The library is entirely pure python, so no additional compliation steps are required for installation.

Automated testing is included to prevent future code contributions from breaking compatibility with previous versions of the library. Tests are stored in the `tests/` directory. pytest is used for the basic test architecture, with the `pytest-mpl` plugin providing image differences. See the `tests/truth/images/` directory for baseline images. `pytest-cov` is used for coverage checking, results are uploaded to [CodeCov](https://codecov.io).

Documentation is built by [mkdocstring-python](https://mkdocstrings.github.io/python/). See the `mkdocs.yml` file for configuration and the `docs/reference/` directory for page layout.
19 changes: 19 additions & 0 deletions docs/docs-env.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: pyxlma-docs
channels:
- conda-forge
dependencies:
- metpy
- mkdocs-material
- mkdocstrings-python
- netcdf4
- numpy
- pandas
- pip
- pyproj
- scikit-learn
- scipy
- xarray
- pip:
- git+https://github.com/deeplycloudy/lmatools
- git+https://github.com/deeplycloudy/glmtools@energy_threshold_util
- -e ../
56 changes: 56 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
<img src="xlma_logo.svg" alt="logo" width="200"/>

# xlma-python
---
*A future, Python-based version of XLMA?*

## Motivation

xlma-python is used for analysis and visualization of [Lightning Mapping Array](https://doi.org/10.1029/1999GL010856) (LMA) data.

XLMA is a venerable IDL GUI program that diplays VHF Lightning Mapping Array data. It was originally written by New Mexico Tech in the late 1990s. This repository represents progress on community-contributed ideas and code to facilitate reimplementing workflows and features from XLMA in Python.

## Features

### Data model and subsetting

xlma-python includes file IO support for the ASCII .dat format used by XLMA and `lma_analysis`. Datasets are read in to the widely-used [xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html#xarray.Dataset) object. VHF sources and network information are added as multidimensional data, with metadata for each variable. Data can be easily saved to and read from the NetCDF format, allowing for conversion from ASCII to `xarray.Dataset` to a NetCDF file.

### Analysis

xlma-python uses a DBSCAN clustering algorithm from [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html) to assign groups of VHF sources recorded by the LMA into discrete lightning flashes, as well as tools to compute properties of the flashes.

Subsetting of all varaibles sharing a dimension and fast indexing are supported. VHF Sources and clustered flashes can be filtered by their properties, and individual flashes/events can be interrogated easily.

### Display

Publication quality plots in the style of XLMA (with a plan view, latitude-altitude and longitude-altitude cross sections, altitude-histogram, and altitude-timeseries plots) are supported. These plots are generated with high level functions, using the popular [matplotlib](https://matplotlib.org/stable/index.html) plotting library.

### Interactive

There are several ongoing, parallel efforts to create an interactive analysis/display tool in python, using different GUI tools, each with their own pros/cons:

- This repository contains an "interactive" module which utilizes matplotlib/jupyter/ipywidgets for interactivity. This was easy to implement with the already existing matplotlib plotting tools, and allows for "good enough" performance in jupyter notebooks, which are widely used in geosciences.

- An older example using [Glue](https://github.com/glue-viz/glue) is also included in the examples directory.

- Other options (for external projects):
- Vispy: OpenGL, so can display plots quickly, but will require more low-level coding. See the SatPy/PyTroll/SIFT efforts.
- ipyvolume: originally written for volume rendering, but has grown to display points, etc. Browser/notebook based.
- HoloViz Suite (holoviews/geoviews plotting libraries with bokeh backend, panel for dashboard layout): browser-based; Javascript and WebGL front end with Python backend.
- HoloViz's Datashader might be useful as a method of data reduction prior to visualization even if we don't use HoloViews/Bokeh.
- Yt: written by the astronomy community in Python ... is it fast enough?
- lonboard: python binding for deck.gl, very fast vector rendering of many points, but limited interactive tools and ONLY allows rendering geospatial data (ie, not cross sections or histograms)
- pydeck: lower-level deck.gl python binding which removes the geospatial data requirement but requires more work on the developer's side.

## Prior art

- [`lmatools`](https://github.com/deeplycloudy/lmatools/)
- Includes readers for LMA and NLDN data (using older methods from 2010)
- Flash sorting and gridding
- Has code for all necessary coordinate transforms.
- [`brawl4d`](https://github.com/deeplycloudy/brawl4d/) A working version of the basic GUI functionality of xlma.
- Based on matplotlib; plots can be drag-repositioned. Slow for large numbers of data points.
- Includes charge analyis that auto-saves to disk
- At one point, could display radar data underneath LMA data
- Built around a data pipeline, with a pool of data at the start, subsetting, projection, and finally display.
46 changes: 46 additions & 0 deletions docs/install.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
## Installation

To install the latest "bleeding edge" commit of xlma-python, you can pip install the git repo directly:
```sh
pip install git+https://github.com/deeplycloudy/xlma-python`
```
Note that this does **NOT** automatically update in a conda environment when running `conda update --all`, you must fully reinstall the library to obtain new changes.

For a development install see [contributing](../contributing/).

## Dependencies
Required:

- xarray (I/O requires the netcdf4 backend)
- pandas
- numpy

Flash clustering:

- scikit-learn
- scipy
- pyproj

Plotting:

- matplotlib
- cartopy
- metpy (optionally, for US county lines)

GLM Plotting:

- glmtools (https://github.com/deeplycloudy/glmtools)

Interactive:

- jupyterlab (or, notebook)
- ipywidgets
- ipympl

Building:

- setuptools
- pytest-cov
- pytest-mpl
- lmatools (https://github.com/deeplycloudy/lmatools)
- ...and all of the above
13 changes: 13 additions & 0 deletions docs/reference/coords/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Coordinates
---
The coords module handles conversions between coordinate systems. The `CoordinateSystem` class represents a generic transform. Subclasses of `CoordinateSystem` are specific coordinate systems that have implemented transformations. These subclasses are documented on the "Transforms" page, see the sidebar.

There are a few useful tools related to coordinate transforms included as well.

::: pyxlma.coords
options:
members:
- centers_to_edges
- centers_to_edges_2d
- semiaxes_to_invflattening
- CoordinateSystem
12 changes: 12 additions & 0 deletions docs/reference/coords/transforms.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Transforms
---
Classes representing implemented coordinate transformations. All classes use the [Earth-Centered, Earth-Fixed (ECEF)](https://en.wikipedia.org/wiki/Earth-centered,_Earth-fixed_coordinate_system) system as a common intermediary system, allowing for conversions between systems. ![Figure demonstrating Earth-Centered, Earth-Fixed](https://upload.wikimedia.org/wikipedia/commons/6/62/Ecef_coordinates.svg)


::: pyxlma.coords
options:
filters:
- "!^centers_to_edges$"
- "!^centers_to_edges_2d$"
- "!^semiaxes_to_invflattening$"
- "!^CoordinateSystem$"
6 changes: 6 additions & 0 deletions docs/reference/lmalib/cf_netcdf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Climate/Forecasting NetCDF

This module is designed to provide a standardized way to name and attribute lightning data in NetCDF4 dataset,
akin to the [Climate and Forecasting](https://cfconventions.org) specification. These tools are primarily used internally for I/O operations, but are provided to the user for convenience.

::: pyxlma.lmalib.io.cf_netcdf
7 changes: 7 additions & 0 deletions docs/reference/lmalib/flash.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Flash processing

::: pyxlma.lmalib.flash.cluster

::: pyxlma.lmalib.flash.properties


3 changes: 3 additions & 0 deletions docs/reference/lmalib/grid.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Gridding

::: pyxlma.lmalib.grid
3 changes: 3 additions & 0 deletions docs/reference/lmalib/lma_intercept_rhi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# LMA with a radar RHI

::: pyxlma.lmalib.lma_intercept_rhi
7 changes: 7 additions & 0 deletions docs/reference/lmalib/read.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Reading data

::: pyxlma.lmalib.io.read
options:
filters:
- "!^to_dataset$"
- "!^lmafile$"
3 changes: 3 additions & 0 deletions docs/reference/lmalib/traversal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Traversal

::: pyxlma.lmalib.traversal
3 changes: 3 additions & 0 deletions docs/reference/plot/base_plot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Base Plot

::: pyxlma.plot.xlma_base_plot
3 changes: 3 additions & 0 deletions docs/reference/plot/interactive.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Jupyter Interactive

::: pyxlma.plot.interactive
Loading
Loading