Update notebooks for 2020

Author: davewhipp

L5/exploring-data-using-pandas.ipynb

@@ -9,9 +9,19 @@
    "source": [
     "# Exploring data using Pandas\n",
     "\n",
+    "<div class=\"alert alert-info\">\n",
+    "\n",
+    "Finnish university students are encouraged to use the CSC Notebooks platform.<br/>\n",
+    "<a href=\"https://notebooks.csc.fi/#/blueprint/7e62ac3bddf74483b7ac7333721630e2\"><img alt=\"CSC badge\" src=\"https://img.shields.io/badge/launch-CSC%20notebook-blue.svg\" style=\"vertical-align:text-bottom\"></a>\n",
+    "\n",
+    "Others can follow the lesson and fill in their student notebooks using Binder.<br/>\n",
+    "<a href=\"https://mybinder.org/v2/gh/geo-python/notebooks/master?urlpath=lab/tree/L4/functions.ipynb\"><img alt=\"Binder badge\" src=\"https://img.shields.io/badge/launch-binder-red.svg\" style=\"vertical-align:text-bottom\"></a>\n",
+    "\n",
+    "</div>\n",
+    "\n",
     "Our first task in this week's lesson is to learn how to **read and explore data files in Python**. We will focus on using [pandas](https://pandas.pydata.org/pandas-docs/stable/) which is an open-source package for data analysis in Python. Pandas is an excellent toolkit for working with **real world data** that often have a tabular structure (rows and columns).\n",
     "\n",
-    "We will first get familiar with **pandas data structures**: *DataFrame* and *Series*:\n",
+    "We will first get familiar with the **pandas data structures**: *DataFrame* and *Series*:\n",
     "\n",
     "![Pandas data structures](img/pandas-structures.png)\n",
     "\n",
@@ -24,34 +34,30 @@
     "\n",
     "As you can see, both DataFrames and Series in pandas have an index that can be used to select values, but they also have column labels to identify columns in DataFrames. In the lesson this week we'll use many of these features to explore real-world data and learn some useful data analysis procedures.\n",
     "\n",
-    "For a comprehensive overview of pandas data structures you can have a look at Chapter 5 in Wes MacKinney's book [Python for Data Analysis (2nd Edition, 2017)](https://geo-python.github.io/site/course-info/resources.html#books) and [Pandas online documentation about data structures](https://pandas.pydata.org/pandas-docs/stable/dsintro.html).\n",
+    "For a comprehensive overview of pandas data structures you can have a look at Chapter 5 in Wes MacKinney's book [Python for Data Analysis (2nd Edition, 2017)](https://geo-python-site.readthedocs.io/en/latest/course-info/resources.html#books) and [Pandas online documentation about data structures](https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html).\n",
     "\n",
     "<div class=\"alert alert-info\">\n",
     "\n",
     "**Note**\n",
-    "    \n",
-    "Pandas is a \"high-level\" package, which means that it makes use of several other packages, such as [NumPy](https://numpy.org/), in the background. There are several ways in which data can be read from a file in Python, and this year we have decided to focus primarily on pandas because it is easy-to-use, efficient and intuitive. If you are curoius about other approaches for interacting with data files, you can find lesson materials from previous years about reading data using [NumPy](https://geo-python.github.io/site/2018/notebooks/L5/numpy/1-Exploring-data-using-numpy.html#Reading-a-data-file-with-NumPy) or [built-in Python functions](https://geo-python.github.io/site/2017/lessons/L5/reading-data-from-file.html). \n",
-    " \n",
-    "</div>\n",
     "\n",
+    "Pandas is a \"high-level\" package, which means that it makes use of several other packages, such as [NumPy](https://numpy.org/), in the background. There are several ways in which data can be read from a file in Python, and this year we have decided to focus primarily on pandas because it is easy-to-use, efficient and intuitive. If you are curoius about other approaches for interacting with data files, you can find lesson materials from previous years about reading data using [NumPy](https://geo-python-site.readthedocs.io/en/2018.1/notebooks/L5/numpy/1-Exploring-data-using-numpy.html#Reading-a-data-file-with-NumPy) or [built-in Python functions](https://geo-python-site.readthedocs.io/en/2017.1/lessons/L5/reading-data-from-file.html).\n",
+    "</div>\n",
     "\n",
     "## Input data: weather statistics\n",
     "\n",
     "Our input data is a text file containing weather observations from Kumpula, Helsinki, Finland retrieved from [NOAA](https://www.ncdc.noaa.gov/)*:\n",
     "\n",
     "- File name: [Kumpula-June-2016-w-metadata.txt](Kumpula-June-2016-w-metadata.txt) (have a look at the file before reading it in using pandas!)\n",
-    "- The file is available in the binder and CSC notebook instances, under L5 folder \n",
+    "- The file is available in the binder and CSC notebook instances, under the L5 folder \n",
     "- The data file contains observed daily mean, minimum, and maximum temperatures from June 2016 recorded from the Kumpula weather observation station in Helsinki.\n",
     "- There are 30 rows of data in this sample data set.\n",
     "- The data has been derived from a data file of daily temperature measurments downloaded from [NOAA](https://www.ncdc.noaa.gov/cdo-web/).\n",
     "\n",
-    "\n",
     "\\*US National Oceanographic and Atmospheric Administration's National Centers for Environmental Information climate database\n",
     "\n",
     "## Reading a data file with Pandas\n",
     "\n",
-    "\n",
-    "Now we're ready to read in our temperature data file. **First, we need to import the Pandas module.** It is customary to import pandas as `pd`:\n"
+    "Now we're ready to read in our temperature data file. **First, we need to import the Pandas module.** It is customary to import pandas as `pd`:"
    ]
   },
   {
@@ -87,29 +93,25 @@
    },
    "outputs": [],
    "source": [
-    "# Read the file using pandas\n",
-    "data = pd.read_csv('Kumpula-June-2016-w-metadata.txt')"
+    "# Read the file using pandas\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "<div class=\"alert alert-info\">\n",
-    "\n",
-    "**Delimiter and other optional parameters**\n",
     "    \n",
-    "Our input file is a comma-delimited file; columns in the data are separted by commas (`,`) on each row. the Pandas `.read_csv()`  -function has comma as the default delimiter so we don't need to specify it separately. In order to make the delimiter visible also in the code, could add the `sep` parameter:\n",
+    "**Delimiter and other optional parameters**\n",
+    "\n",
+    "Our input file is a comma-delimited file; columns in the data are separted by commas (`,`) on each row. the Pandas `.read_csv()` function has the comma as the default delimiter so we don't need to specify it separately. In order to make the delimiter visible also in the code for reading the file, could add the `sep` parameter:\n",
     "    \n",
-    "```   \n",
+    "```python\n",
     "data = pd.read_csv('Kumpula-June-2016-w-metadata.txt', sep=`,`)\n",
     "```\n",
-    "    \n",
-    "The `sep` parameter cam be used to spesify if the input data uses some other character, such as `;` as a delimiter. For a full list of available parameters, please refer to [pandas documentation for pandas.read_csv](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html), or run `help(pd.read_csv)`.\n",
-    "    \n",
-    "</div>\n",
-    "\n",
-    "\n"
+    " \n",
+    "The `sep` parameter cam be used to specify whether the input data uses some other character, such as `;` as a delimiter. For a full list of available parameters, please refer to the [pandas documentation for pandas.read_csv](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html), or run `help(pd.read_csv)`.\n",
+    "</div>"
    ]
   },
   {
@@ -119,12 +121,10 @@
     "<div class=\"alert alert-info\">\n",
     "\n",
     "**Reading different file formats**\n",
-    "    \n",
+    "\n",
     "`pandas.read_csv()` is a general function for reading data files separated by commas, spaces, or other common separators. \n",
     "\n",
-    "    \n",
-    "Pandas has several different functions for parsing input data from different formats. There is, for example, a separate function for reading Excel files `read_excel`. Another useful function is `read_pickle` for reading data stored in the [Python pickle format](https://docs.python.org/3/library/pickle.html). Check out [pandas documentation about input and output functions](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-tools-text-csv-hdf5) and Chapter 6 in [MacKinney (2017): Python for Data Analysis](https://geo-python.github.io/site/course-info/resources.html#books) for more details about reading data.\n",
-    " \n",
+    "Pandas has several different functions for parsing input data from different formats. There is, for example, a separate function for reading Excel files `read_excel`. Another useful function is `read_pickle` for reading data stored in the [Python pickle format](https://docs.python.org/3/library/pickle.html). Check out the [pandas documentation about input and output functions](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-tools-text-csv-hdf5) and Chapter 6 in [MacKinney (2017): Python for Data Analysis](https://geo-python-site.readthedocs.io/en/latest/course-info/resources.html#books) for more details about reading data.\n",
     "</div>"
    ]
   },
@@ -201,7 +201,7 @@
     "editable": true
    },
    "source": [
-    "Fortunately, that's easy to do when reading in data using pandas. We just need to add the `skiprows` parameter when we read the file, listing the number of rows to skip (8 in this case).\n",
+    "Fortunately, skipping over rows is easy to do when reading in data using pandas. We just need to add the `skiprows` parameter when we read the file, listing the number of rows to skip (8 in this case).\n",
     "\n",
     "Let's try reading the datafile again, and this time defining the `skiprows` parameter."
    ]
@@ -251,9 +251,9 @@
     "editable": true
    },
    "source": [
-    "After reading in the data, it is always good to check that everything went well by printing out the data as we did here. However, often it is enough to have a look at the top rows of the data. \n",
+    "After reading in the data, it is always good to check that everything went well by printing out the data as we did here. However, often it is enough to have a look at the top few rows of the data. \n",
     "\n",
-    "We can  use the `.head()` function of the pandas DataFrame object to quickly check the top rows. By default, the `.head()` -function returns 5 first rows of the DataFrame:"
+    "We can use the `.head()` function of the pandas DataFrame object to quickly check the top rows. By default, the `.head()` function returns the first 5 rows of the DataFrame:"
    ]
   },
   {
@@ -334,22 +334,26 @@
    "source": [
     "#### Check your understanding\n",
     "\n",
-    "Read in the file `Kumpula-June-2016-w-metadata.txt` again into a new variable called `temp_data` so that you only read in columns `YEARMODA` and `TEMP`. The new variable `temp_data` should have 30 rows and 2 columns. You can achieve this by using the `usecols` parameter when reading in the file. Check for more help in the [pandas.read_csv documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)."
+    "Read the file `Kumpula-June-2016-w-metadata.txt` in again and store its contents in a new variable called `temp_data`. In this case you should only read in the columns `YEARMODA` and `TEMP`, so the new variable `temp_data` should have 30 rows and 2 columns. You can achieve this using the `usecols` parameter when reading in the file. Feel free to check for more help in the [pandas.read_csv documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "# Read in the data file again here\n"
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "# Check the contents of the first rows of the data file\n"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -368,7 +372,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Let's start by checking the size of our data frame. We can use the `len()` function similarly as with lists to check how many rows we have:"
+    "Let's start by checking the size of our data frame. We can use the `len()` function similar to the use with lists to check how many rows we have:"
    ]
   },
   {
@@ -384,8 +388,7 @@
    },
    "outputs": [],
    "source": [
-    "# Check the number of rows \n",
-    "len(data)"
+    "# Check the number of rows \n"
    ]
   },
   {
@@ -411,8 +414,7 @@
    },
    "outputs": [],
    "source": [
-    "# Check dataframe shape (number of rows, number of columns)\n",
-    "data.shape"
+    "# Check dataframe shape (number of rows, number of columns)\n"
    ]
   },
   {
@@ -422,14 +424,20 @@
     "editable": true
    },
    "source": [
-    "Here we see that our dataset has 30 rows, 4 columns, just as we saw above when printing out the whole DataFrame."
+    "Here we see that our dataset has 30 rows and 4 columns, just as we saw above when printing out the entire DataFrame."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "**Note:** `shape` is one of the several [attributes related to a pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html#attributes-and-underlying-data)."
+    "<div class=\"alert alert-info\">\n",
+    "\n",
+    "**Note**\n",
+    "    \n",
+    "`shape` is one of the several [attributes related to a pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html#attributes-and-underlying-data).\n",
+    "\n",
+    "</div>"
    ]
   },
   {
@@ -452,8 +460,7 @@
    },
    "outputs": [],
    "source": [
-    "#Print column names\n",
-    "data.columns.values"
+    "#Print column names\n"
    ]
   },
   {
@@ -479,8 +486,7 @@
    },
    "outputs": [],
    "source": [
-    "#Print index\n",
-    "data.index"
+    "#Print index\n"
    ]
   },
   {
@@ -490,7 +496,7 @@
     "editable": true
    },
    "source": [
-    "Here we see how the data is indexed, starting at 0, ending at 30, and with an increment of 1 between each value. This is basically the same way in which Python lists are indexed, however, pandas allows also other ways of identifying the rows. DataFrame indices could, for example, be character strings, or date objects. We will learn more about re-setting the index later."
+    "Here we see how the data is indexed, starting at 0, ending at 30, and with an increment of 1 between each value. This is basically the same way in which Python lists are indexed, however, pandas allows also other ways of identifying the rows. DataFrame indices could, for example, be character strings, or date objects. We will learn more about resetting the index later."
    ]
   },
   {
@@ -516,8 +522,7 @@
    },
    "outputs": [],
    "source": [
-    "# Print data types\n",
-    "data.dtypes"
+    "# Print data types\n"
    ]
   },
   {
@@ -534,17 +539,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Check your unerstanding\n",
+    "#### Check your understanding\n",
     "\n",
-    "Print out the number of columns in our DataFrame:"
+    "See if you can find a way to print out the number of columns in our DataFrame."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "# Calculate the number of columns here\n"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -557,7 +564,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We can select only spesific columns based on the column values. The basic syntax is `dataframe[value]`, where value can be a single column name, or a list of column names. Let's start by selecting two columns, `'YEARMODA'` and `'TEMP'`:"
+    "We can select specific columns based on the column values. The basic syntax is `dataframe[value]`, where value can be a single column name, or a list of column names. Let's start by selecting two columns, `'YEARMODA'` and `'TEMP'`:"
    ]
   },
   {
@@ -567,6 +574,13 @@
    "outputs": [],
    "source": []
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -638,7 +652,9 @@
     }
    },
    "outputs": [],
-   "source": []
+   "source": [
+    "# Check datatype of the column\n"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -663,8 +679,10 @@
     "``` \n",
     "data.TEMP\n",
     "```\n",
+    "\n",
     "This syntax works only if the column name is a valid name for a Python variable (e.g. the column name should not contain whitespace).\n",
     "The syntax `data[\"column\"]` works for all kinds of column names, so we recommend using this approach.\n",
+    "\n",
     "</div>"
    ]
   },
@@ -754,19 +772,21 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Check your unerstanding\n",
+    "#### Check your understanding\n",
     "\n",
-    "It doesn't make much sense to print out descriptive statistics for the `YEARMODA` column now that the values are stored as integer values (and not datetime objects that we will learn to handle later on). \n",
+    "It doesn't make much sense to print out descriptive statistics for the `YEARMODA` column now that the values are stored as integer values (and not datetime objects, which we will learn about in later lessons). \n",
     "\n",
-    "Print out descriptive statistics again, this time only for columns `TEMP`, `MAX`, `MIN`:"
+    "See if you can print out the descriptive statistics again, this time only for columns `TEMP`, `MAX`, `MIN`:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "# Get descriptive statistics for selected columns\n"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -779,15 +799,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Visualizing the data is a key part of data exploration. Pandas comes with a handful of plotting methods, which all rely on the plotting library [matplotlib](https://matplotlib.org/). "
+    "Visualizing the data is a key part of data exploration, and Pandas comes with a handful of plotting methods, which all rely on the [matplotlib](https://matplotlib.org/) plotting library. "
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "For very basic plots, we don’t need to import matplotlib separately. \n",
-    "We can already achieve very simple plots using the `DataFrame.plot` -method. \n",
+    "For very basic plots, we don’t need to import matplotlib separately. We can already create very simple plots using the `DataFrame.plot` method. \n",
     "\n",
     "Let's plot all the columns that contain values related to temperatures:"
    ]
@@ -954,16 +973,6 @@
    "source": [
     "Check more details about available paramenters and methods from [the pandas.DataFrame documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas-dataframe)."
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "deletable": true,
-    "editable": true
-   },
-   "source": [
-    "That's it! Next, we will have a look at [basic operations for data analysis in Pandas](processing-data-with-pandas.ipynb)."
-   ]
   }
  ],
  "metadata": {
@@ -983,7 +992,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.6"
+   "version": "3.7.4"
   }
  },
  "nbformat": 4,