ConfiguringNotebookEnvironment.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Notebook Environment Setup\n",
        "\n",
        "This notebook takes you through detailed setup of your settings for\n",
        "Microsoft Sentinel Notebooks and the MSTICPy library. It covers:\n",
        "\n",
        "- Setting up your Python environment for notebooks\n",
        "- Creating and editing your msticpyconfig.yaml file\n",
        "- Understanding and managing you config.json file.\n",
        "\n",
        "If you are\n",
        "using notebooks in the Microsoft Sentinel/Azure ML environment you can skip\n",
        "the first section \"Configuring your Python Environment\" entirely.\n",
        "\n",
        "The main part of this notebook involves setting up your msticpyconfig.yaml.\n",
        "While many of these settings are optional, if you do not configure\n",
        "them correctly you'll experience some loss of functionality. For\n",
        "example, using Threat Intelligence providers usually requires an\n",
        "API key. To save you having to type this in every time you look up\n",
        "an IP Address you should put this in a config file.\n",
        "\n",
        "This section takes you through creating settings for\n",
        "- Microsoft Sentinel workspaces\n",
        "- Threat Intelligence providers\n",
        "- Geo-location providers\n",
        "- Other data providers (e.g. Azure APIs)\n",
        "- Key Vault\n",
        "- Auto-loading options.\n",
        "\n",
        "You'll typically need the first three of these to use most\n",
        "of the notebooks fully.\n",
        "\n",
        "Section 3, \"The config.json file\" can also be ignored if you\n",
        "are happy using `msticpyconfig.yaml`. It is included here\n",
        "for background."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "toc": true
      },
      "source": [
        "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n",
        "<html xmlns=\"http://www.w3.org/1999/xhtml\">\n",
        "\n",
        "<head>\n",
        "    <meta name=\"generator\" content=\"HTML Tidy for Windows (vers 14 February 2006), see www.w3.org\" />\n",
        "    <title></title>\n",
        "</head>\n",
        "\n",
        "<body>\n",
        "    <h2>Contents</h2>\n",
        "    <div class=\"toc\">\n",
        "        <ul class=\"toc-item\">\n",
        "            <li><span><a href=\"#Configuring-your-Python-Environment\">Configuring your Python Environment</a></span>\n",
        "                <ul class=\"toc-item\">\n",
        "                    <li><span><a href=\"#Python-3.6-or-Later\">Python 3.6 or Later</a></span></li>\n",
        "                    <li><span><a href=\"#Creating-a-virtual-environment\">Creating a virtual\n",
        "                                environment</a></span></li>\n",
        "                    <li><span><a href=\"#Using-Requirements.txt\">Using Requirements.txt</a></span></li>\n",
        "                    <li><span><a href=\"#Installing-in-a-Conda-Environment\">Installing in a Conda Environment</a></span></li>\n",
        "                    <li><span><a href=\"#Installing-with---user-option\">Installing with --user option</a></span></li>\n",
        "                    <li><span><a href=\"#Install-Packages-from-this-Notebook\">Install Packages from this Notebook</a></span></li>\n",
        "                </ul>\n",
        "            </li>\n",
        "            <li><span><a href=\"#MSTICPy-Configuration-File---msticpyconfig.yaml\">MSTICPY Configuration File <code>msticpyconfig.yaml</code></a></span>\n",
        "                <ul class=\"toc-item\">\n",
        "                    <li><span><a href=\"#Display-your-existing-msticpyconfig.yaml\">Display your existing msticpyconfig.yaml</a></span></li>\n",
        "                    <li><span><a href=\"#Import-your-Config.json-and-create-a-msticpyconfig.yaml-[Microsoft-Sentinel]\">Import your Config.json and create a msticpyconfig.yaml [Microsoft Sentinel]</a></span></li>\n",
        "                    <li><span><a href=\"#Setting-the-path-to-your-msticpyconfig.yaml\">Setting the path to your msticpyconfig.yaml</a></span></li>\n",
        "                    <li><span><a href=\"#Verify-(or-add)-Azure-Sentinel-Workspace-settings\">Verify (or add) Microsoft Sentinel Workspace settings</a></span></li>\n",
        "                    <li><span><a href=\"#Adding-Threat-Intel-(TI)-Providers\">Adding Threat Intel (TI) Providers</a></span></li>\n",
        "                    <li><span><a href=\"#Adding-GeoIP-Providers\">Adding GeoIP Providers</a></span></li>\n",
        "                    <li><span><a href=\"#Optional-Settings-1---Azure-Data-and-Azure-Sentinel-APIs\">Optional Settings 1 - Azure Data and Microsoft Sentinel APIs</a></span></li>\n",
        "                    <li><span><a href=\"#Optional-Settings-2---Autoload-QueryProviders\">Optional Settings 2 - Autoload QueryProviders</a></span></li>\n",
        "                    <li><span><a href=\"#Optional-Settings-3---Autoloaded-Component\">Optional Settings 3 - Autoloaded Component</a></span></li>\n",
        "                    <li><span><a href=\"#Save-your-file-and-add-the-MSTICPYCONFIG-environment-variable\">Save your file and add the MSTICPYCONFIG environment variable</a></span></li>\n",
        "                    <li><span><a href=\"#Validating-your-msticpyconfig.yaml-settings\">Validating your msticpyconfig.yaml settings</a></span></li>\n",
        "            </li>\n",
        "            <li><span><a href=\"#The-config.json-file\">The <code>config.json</code> file</a></span></li>\n",
        "        </ul>\n",
        "    </div>\n",
        "</body>\n",
        "\n",
        "</html>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "## Configuring your Python Environment\n",
        "### Python 3.6 or Later\n",
        "If you are running in Jupyterhub environment such as Azure Notebooks, Python is already installed. When using any of the sample notebooks or copies of them you only need to ensure that the Python 3.6 (or later) kernel is selected.\n",
        "\n",
        "If you are running the notebooks locally will you need to install Python 3.6 or later. The Ananconda distribution is a good starting point since it comes with many required packages already installed.\n",
        "\n",
        "### Creating a virtual environment\n",
        "If you are running these notebooks locally, it is a good idea to create a clean Python *virtual environment*, before installing any of the packages . This will prevent installed packages conflicting with versions that you may need for other applications.\n",
        "\n",
        "For standard python use the [`venv`](https://docs.python.org/3/library/venv.html?highlight=venv) command. \n",
        "For Conda use the [`conda env`](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) command. \n",
        "In both cases be sure to activate the environment before running jupyter using `venvpath/Scripts/activate` or `conda activate {my_env_name}`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Run this cell to view requirements.txt\n",
        "%pfile requirements.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Installing in a Conda Environment\n",
        "Although you can use pip inside a conda environment it is usually better to try to install conda packages whenever possible.\n",
        "\n",
        "```\n",
        "activate {my_env_name}\n",
        "conda config --append channels conda-forge\n",
        "conda install package1 package2\n",
        "```\n",
        "\n",
        "See [Managing packages](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html) in Anaconda.\n",
        "\n",
        "For packages that are not available as conda packages use pip from with a Conda prompt/shell to install the remaining packages.\n",
        "\n",
        "### Installing with --user option\n",
        "If you are using a shared installation of Python (i.e. one installed by the administrator) you will need to add the `--user` option to your `pip install` commands. E.g.\n",
        "\n",
        "```\n",
        "pip install pkg_name --user --upgrade\n",
        "```\n",
        "\n",
        "This will avoid permission errors by installing into your user folder.\n",
        "\n",
        "> **Note**: the use of the `--user` option is usually not required in a Conda environment \n",
        "> since the Python site packages are normally already installed in a per-user folder."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Install Packages from this Notebook\n",
        "The first time this cell runs for a new Azure ML or Azure Notebooks notebook or other Python environment it will do the following things:\n",
        "1. Check the kernel version to ensure that a Python 3.6 or later kernel is running\n",
        "2. Check the msticpy version - if this is not installed or the version installed is less than the required version (in `REQ_MSTICPY_VER`)\n",
        "   it will attempt to install a new version (you will be prompted whether you want to do this)\n",
        "   The install can take several minutes depending on the versions of packages that you already have installed.\n",
        "3. Once *msticpy* is installed and imported, the `init_notebook` function is run. This:\n",
        "   - imports common modules used in the notebook\n",
        "   - installs additional packages\n",
        "   - sets some global options\n",
        "   \n",
        "> **Note:** In subsequent runs, this cell should run quickly since you will already have the required packages installed.\n",
        "\n",
        "\n",
        "> **Warning:** you may see some warnings about incompatibility with certain packages. This should not affect the functionality of this notebook but you may need to upgrade the packages producing the warnings to a more recent version."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "import os\n",
        "import sys\n",
        "import warnings\n",
        "from IPython.display import display, HTML, Markdown\n",
        "\n",
        "REQ_PYTHON_VER=\"3.8\"\n",
        "REQ_MSTICPY_VER=\"1.5.0\"\n",
        "\n",
        "# If not using Azure Notebooks, install msticpy with\n",
        "# %pip install msticpy\n",
        "from msticpy.nbtools import nbinit\n",
        "nbinit.init_notebook(\n",
        "    namespace=globals(),\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "## MSTICPy Configuration File - `msticpyconfig.yaml`\n",
        "\n",
        "*MSTICPy* is a Python package used in most of the Jupyter notebooks\n",
        "on Azure-Sentinel-Notebooks. It provides a lot of functionality specific\n",
        "to threat hunting and investigations, including:\n",
        "- Data querying against Microsoft Sentinel tables (also MDE, Splunk and other)\n",
        "- Threat Intelligence lookups using multiple TI providers (VirusTotal, AlienVault OTX and others)\n",
        "- Common enrichment functions (GeoIP, IoC extraction, WhoIs, etc.)\n",
        "- Visualization using event timelines, process trees and Geo-mapping\n",
        "- Advanced analysis such as Time Series decomposition, Anomaly detection and clustering.\n",
        "\n",
        "> **Note**: the configuration actions in this section are\n",
        "> an abbreviated version of the\n",
        "> [MPSettingsEditor notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/MPSettingsEditor.ipynb)<br>\n",
        "> Use this notebook for a fuller guide on how to configure your settings.<br>\n",
        "> Also, see these sections in the *MSTICPy* documentation:<br>\n",
        "> [MSTICPy Package Configuration](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)<br>\n",
        "> [MSTICPy Settings Editor](https://msticpy.readthedocs.io/en/latest/getting_started/SettingsEditor.html)\n",
        "\n",
        "\n",
        "\n",
        "`config.json` provides some basic configuration for connecting to your Microsoft Sentinel workspace. \n",
        "However, there are many features that require additional configuration information. Some examples are:\n",
        "- Threat Intelligence Provider connection information\n",
        "- GeoIP connection information\n",
        "- Keyvault configuration for storing secrets remotely\n",
        "- MDE and Azure API connection information.\n",
        "- Connection information for multiple Microsoft Sentinel workspaces.\n",
        "\n",
        "Settings for these are stored in the `msticpyconfig.yaml` file. This file is read from the current directory or you can set an environment variable (`MSTICPYCONFIG`) pointing to its location.\n",
        "Form more information about *msticpy* configuration see [msticpy Package Configuration](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html).\n",
        "\n",
        "The most commonly-used sections are described below.\n",
        "\n",
        "\n",
        "#### Threat Intelligence Provider Setup\n",
        "For more information on the msticpy Threat Intel lookup class see the [documentation here](https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html).\n",
        "\n",
        "Primary providers are used by default. Secondary providers are not run by default but can be invoked by using the `providers` parameter to `lookup_ioc()` or `lookup_iocs()`. Set the `Primary` config setting to `True` or `False` for each provider ID according to how you want to use them. The `providers` parameter should be a list of strings identifying the provider(s) to use. \n",
        "\n",
        "- The provider ID is given by the `Provider:` setting for each of the TI providers - do not alter this value.\n",
        "- Delete or comment out the section for any TI Providers that you do not wish to use.\n",
        "- For most providers you will usually need to supply an authorization (API) key and in some cases a user ID for each provider.\n",
        "- For the Microsoft Sentinel TI provider, you will need the workspace ID and tenant ID and will need to authenticate in order \n",
        "  to access the data (although if you have an existing authenticated connection with the same workspace/tenant, this connection will be re-used).\n",
        "\n",
        "#### GeoIP Providers\n",
        "Like the TI providers these services normally need an API key to access. You can read more about configuration\n",
        "the supported providers here. [msticpy GeoIP Providers](https://msticpy.readthedocs.io/en/latest/data_acquisition/GeoIPLookups.html)\n",
        "\n",
        "#### Browshot Setup\n",
        "The functionality to screenshot a URL in msticpy.sectools.domain_utils relies on a service called BrowShot (https://browshot.com/). An API key is required to use this service and it needs to be defined in the `msticpyconfig` file as well. As this is not a threat intelligence provider it doesn't not fall under the `TIProviders` section of `msticpyconfig` but instead sits alone. See the cell below for example configuration."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Display your existing `msticpyconfig.yaml`\n",
        "\n",
        "We'll be using some of the *MSTICPy* configuration tools:\n",
        "*MPConfigEdit* and *MPConfigFile*, so we'll import these first"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from msticpy import MpConfigFile, MpConfigEdit\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then run MpConfig file to view your current settings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpconfig = MpConfigFile()\n",
        "mpconfig.load_default()\n",
        "mpconfig.view_settings()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### If you see nothing but a pair of curly braces...\n",
        "...in the settings view above it means\n",
        "that you probably need to create up a **msticpyconfig.yaml**\n",
        "\n",
        "If you know that you have configured a msticpyconfig file, \n",
        "you can search for this file using MpConfigFile. Click on **Load file**. \n",
        "Once you've done that go to the [Setting the path to your msticpyconfig.yaml](#Setting-the-path-to-your-msticpyconfig.yaml)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Import your Config.json and create a msticpyconfig.yaml [Microsoft Sentinel]\n",
        "\n",
        "Follow these steps:\n",
        "1. Run MpConfigFile\n",
        "2. Locate your config.json\n",
        "   - click **Load file** button\n",
        "   - Browse - use the controls to navigate to find config.json\n",
        "   - Search - set the starting directory to search and open the **Search** drop-down\n",
        "   - When you see the file click on it and click **Select File** button (below the file browser)\n",
        "   - optionally, click **View Settings** to confirm that this looks right\n",
        "3. Convert to convert to msticpyconfig format\n",
        "   - click **View Settings**\n",
        "4. Save your `msticpyconfig.yaml` file\n",
        "   - type a path into the **Current file** text box\n",
        "   - Click on **Save file**\n",
        "5. You can set this file to always load by assigning the path to an environment variable.\n",
        "   See [Setting the path to your msticpyconfig.yaml](#Setting-the-path-to-your-msticpyconfig.yaml)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Setting the path to your msticpyconfig.yaml\n",
        "\n",
        "This is a good point to set up an environment variable so that\n",
        "you can keep a single configuration file in a known location and always\n",
        "load the same settings. (Of course, you're free to use multiple configs\n",
        "if you need to use different settings for each notebook folder)\n",
        "\n",
        "- decide on a location for your `msticpyconfig.yaml` - this could be in \"~/.msticpyconfig.yaml\" or \"%userprofile%/msticpyconfig.yaml\"\n",
        "- copy the `msticpyconfig.yaml` file that you just created to this location.\n",
        "- set the `MSTICPYCONFIG` environment variable to point to that location:\n",
        "\n",
        "#### Windows\n",
        "<img src=\"https://github.com/Azure/Azure-Sentinel-Notebooks/raw/master/src/images/win_env_var.png\" style=\"width: 500px;\"/>\n",
        "\n",
        "\n",
        "#### Linux\n",
        "In your .bashrc (or somewhere else convenient) add:\n",
        "\n",
        "`export MSTICPYCONFIG=~/.msticpyconfig.yaml`\n",
        "\n",
        "#### Azure ML\n",
        "\n",
        "In Azure ML, you need to decide whether to store your `msticpyconfig.yaml` in\n",
        "the AML file store or on the Compute file system. If you have any secret\n",
        "key material in the file, we recommend storing on the Compute instance, since\n",
        "the AML file store is shared storage, whereas the Compute instance is\n",
        "accessible only by the user who created it.\n",
        "\n",
        "If you are happy to leave the file in the AML file store, you should be set.\n",
        "The init_notebook function run at the start of the notebook\n",
        "will find it there in your root folder and set the MSTICPYCONFIG environment\n",
        "variable to point to it.\n",
        "\n",
        "**Pointing to a path on a compute instance**\n",
        "\n",
        "1. Open a terminal in AML\n",
        "<br>\n",
        "<img src=\"https://github.com/Azure/Azure-Sentinel-Notebooks/raw/master/src/images/aml_terminal.png\" style=\"width: 300px;\"/>\n",
        "\n",
        "2. Verify your msticpyconfig.yaml is accessible\n",
        "\n",
        "   Your current directory should be your AML file store home directory\n",
        "   (this is mounted in the Compute Linux system) and the prompt will look\n",
        "   something like the example below.\n",
        "\n",
        "   If you created a `msticpyconfig.yaml` in the previous step,\n",
        "   this should be visible if you type `ls`.\n",
        "   ```bash\n",
        "   azureuser@ianhelle-azml7:~/cloudfiles/code/Users/ianhelle$ ls msti*\n",
        "   msticpyconfig.yaml\n",
        "   ```\n",
        "\n",
        "\n",
        "3. Move the file to your home folder\n",
        "   \n",
        "   ```bash\n",
        "   mv msticpyconfig.yaml ~\n",
        "   ```\n",
        "\n",
        "\n",
        "4. Add an environment variable\n",
        "   Because the Jupyter server is started before you connect its process\n",
        "   will not inherit and environment variables from you .bashrc\n",
        "   You can set it one of two places:\n",
        "   \n",
        "   - The `kernel.json` file for your Python kernel (there are kernels for\n",
        "     both Python 3.6 and Python 3.8\n",
        "   - Add a Python file `nbuser_settings.py` to the root of your user folder.\n",
        "   \n",
        "   These options are described in the following sections.\n",
        "   \n",
        "**kernel.json**\n",
        "\n",
        "- Python 3.8 location: `/usr/local/share/jupyter/kernels/python38-azureml/kernel.json`\n",
        "- Python 3.6 location: `/usr/local/share/jupyter/kernels/python3-azureml/kernel.json`\n",
        "\n",
        "Make a copy of the file and open the original in an editor (you many need to use sudo to be able to overwrite this file).\n",
        "The file will look something like this\n",
        "```json\n",
        "{\n",
        "    \"argv\": [\n",
        "    \"/anaconda/envs/azureml_py38/bin/python\",\n",
        "    \"-m\",\n",
        "    \"ipykernel_launcher\",\n",
        "    \"-f\",\n",
        "    \"{connection_file}\"\n",
        "    ],\n",
        "    \"display_name\": \"Python 3.8 - AzureML\",\n",
        "    \"language\": \"python\"\n",
        "}\n",
        "```\n",
        "Add the following line after the \"language\" item.\n",
        "```json\n",
        "    \"env\": { \"MSTICPYCONFIG\": \"~/msticpyconfig.yaml\" }\n",
        "```\n",
        "\n",
        "Your file should look like this (remember to add a comma at the end of the\n",
        "`\"language\": \"python\"` line\n",
        "```json\n",
        "{\n",
        "    \"argv\": [\n",
        "    \"/anaconda/envs/azureml_py38/bin/python\",\n",
        "    \"-m\",\n",
        "    \"ipykernel_launcher\",\n",
        "    \"-f\",\n",
        "    \"{connection_file}\"\n",
        "    ],\n",
        "    \"display_name\": \"Python 3.8 - AzureML\",\n",
        "    \"language\": \"python\",\n",
        "    \"env\": { \"MSTICPYCONFIG\": \"~/msticpyconfig.yaml\" }\n",
        "}\n",
        "```\n",
        "\n",
        "If you use both kernels you will need to edit both files.\n",
        "\n",
        "**nbuser_settings.py**\n",
        "\n",
        "Create this file (you can do this from the AML workspace) in the\n",
        "root of your user folder (i.e. inside the folder with your username)\n",
        "and add the following lines\n",
        "```python\n",
        "import os\n",
        "os.environ[\"MSTICPYCONFIG\"] = \"~/msticpyconfig.yaml\"\n",
        "```\n",
        "\n",
        "This file, if it exists, is imported by the `nb_check.check_versions`\n",
        "function at the start of the notebook. It will set the environment\n",
        "variable at the start of each notebook before any configuration is read.\n",
        "This is simpler and less intrusive than editing the kernel.json.\n",
        "However, it only works if you run `check_versions`. If you load\n",
        "a notebook without running this MSTICPy may not be able to find\n",
        "its configuration file.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Verify (or add) Microsoft Sentinel Workspace settings\n",
        "\n",
        "If you loaded a config.json file into your msticpyconfig.yaml, you should see \n",
        "your workspace displayed when you run the following cell. If not,\n",
        "you can add one or more workspaces here. The **Name**, **WorkspaceId** and **TenantId** are\n",
        "mandatory. The other fields are helpful but not essential.\n",
        "\n",
        "Use the Help drop-down panel to find more information about adding workspaces and finding\n",
        "the correct values for your workspace.\n",
        "\n",
        "If this the workspace that you use frequently or all of the time, you may want to set this as the **default.**\n",
        "This creates a duplicate entry named \"Default\". This is used when you connect to AzureSentinel without\n",
        "needing to supply a workspace name. You can override this by specifying a workspace name at connect time,\n",
        "which you need to do if you are working with multiple workspaces.\n",
        "\n",
        "When you've finished, type a file name (usually \"msticpyconfig.yaml\") into the **Conf File** text box\n",
        "and click **Save File**,\n",
        "\n",
        "You can also try the **Validate Settings** button. This should show that you have a few missing\n",
        "sections (we'll fill these in later) but should show nothing under the the \"Type Validation Results\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpedit = MpConfigEdit(settings=mpconfig)\n",
        "mpedit.set_tab(\"AzureSentinel\")\n",
        "mpedit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Adding Threat Intel (TI) Providers\n",
        "\n",
        "You will likely want to do lookups of IP Addresses, URLs and other items to check for any Threat Intelligence reports.\n",
        "To do that you need to add the providers that you want to use. Most TI providers require that you\n",
        "have an account with them and supply an API key or other authentication items when you connect.\n",
        "\n",
        "Most providers have a free use tier (or in cases like AlienVault OTX) are entirely free.\n",
        "Free tiers for paid providers usually impose a certain number of requests that you\n",
        "can make in a given time period.\n",
        "\n",
        "For account creation, each provider does this slightly differently.\n",
        "Use the help links in the editor help to find where to go set each of these up.\n",
        "\n",
        "Assuming that you have done this, we can configure a provider. Be sure to\n",
        "store any authentication keys somewhere safe (and memorable).\n",
        "\n",
        "We are going to use [VirusTotal](https://www.virustotal.com) (VT) as an example TI Provider.\n",
        "For this you will need a VirusTotal API key from the \n",
        "[VirusTotal](https://developers.virustotal.com/v3.0/reference#getting-started) website.<br>\n",
        "We also support a range of other threat intelligence providers - you can read about this here [MSTICPy TIProviders](https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html)\n",
        "<br><br>\n",
        "Taking VirusTotal as our example.\n",
        "- Click on the **TI Providers** tab\n",
        "- Select \"VirusTotal\" from the **New prov** drop-down list\n",
        "- Click **Add**\n",
        "\n",
        "This should show you the values that you need to provide:\n",
        "- a single item **AuthKey** (this is usually referred to as an \"API Key\"\n",
        "\n",
        "You can paste the key into the **Value** field and click the **Save** button.\n",
        "\n",
        "You can opt to store the VT AuthKey as an environment variable. This is a bit more secure than \n",
        "having it laying around in configuration files.\n",
        "Assuming that you have set you VT key as an environment variable\n",
        "```bash\n",
        "set VT_KEY=VGhpcyBzaG91bGQgc2hvdyB5b3UgdGhlIHZhbHVlcyB  (Windows)\n",
        "export VT_KEY=VGhpcyBzaG91bGQgc2hvdyB5b3UgdGhlIHZhbHVlcyB  (Linux/MAC)\n",
        "```\n",
        "Flip the **Storage** radio button to **EnvironmentVar** and type the name of the\n",
        "variable (`VT_KEY` in our example) into the value box.\n",
        "\n",
        "You can also use Azure Key Vault to store secrets like these but we will need to \n",
        "set up the Key Vault settings before this will work.\n",
        "\n",
        "Click the **Save File** button to save your changes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpedit.set_tab(\"TI Providers\")\n",
        "mpedit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Adding GeoIP Providers\n",
        "\n",
        "MSTICPy supports two Geo IP providers - Maxmind GeoIPLite and IP Stack.\n",
        "The main difference between the two is that Maxmind downloads and uses a local\n",
        "database, while IPStack is a purely online solution.\n",
        "\n",
        "For either you need API keys to either download the free database from MaxMind\n",
        "or access the IPStack online lookup\n",
        "\n",
        "We'll use GeoIPLite as our example.\n",
        "You can sign up for a free account and API key at https://www.maxmind.com/en/geolite2/signup.\n",
        "You'll need the API for the following steps.\n",
        "- Select \"GeoIPLite\" from the **New Prov**\n",
        "- Click **Add**\n",
        "- Paste your Maxmind key into the **Value** field\n",
        "\n",
        "Set the maxmind data folder:\n",
        "- This defaults to \"~/.msticpy\"\n",
        "  - On Windows this translates to the foldername `%USERPROFILE%/.msticpy`.\n",
        "  - On Linux/Mac this translates to the folder `.msticpy` in your home folder.\n",
        "- This is where the downloaded GeopIP database will be stored.\n",
        "- Choose another folder name and location if you prefer.\n",
        "\n",
        "> **Note**: as with the TI providers you can opt to store your key\n",
        "> as an environment variable or keep it in Key Vault.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpedit.set_tab(\"GeoIP Providers\")\n",
        "mpedit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<h3 style='color:red'>Important Security Note</h3>\n",
        "\n",
        "> You might not be too comfortable leaving API keys stored in\n",
        "> text files. You can opt to have these settings stored either:\n",
        "> - as Environment Variables\n",
        "> - in Azure Key Vault\n",
        "\n",
        "To see how to do this see these resources\n",
        "- [MPSettingsEditor notebook](https://github.com/microsoft/msticpy/blob/master/docs/notebooks/MPSettingsEditor.ipynb)<br>\n",
        "- [MSTICPy Package Configuration](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)<br>\n",
        "- [MSTICPy Settings Editor](https://msticpy.readthedocs.io/en/latest/getting_started/SettingsEditor.html)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Optional Settings 1 - Azure Data and Microsoft Sentinel APIs\n",
        "\n",
        "### Azure API and Microsoft Sentinel API\n",
        "To access Azure APIs (such as the Sentinel APIs or Azure resource APIs) \n",
        "you need to be able to use Azure Authentication.\n",
        "The setting is named \"AzureCLI\" for historical reasons - don't let that confuse you.\n",
        "We currently support two ways of authenticating:\n",
        "1.  Chained chained authentication (recommended)\n",
        "2.  With a client app ID and secret\n",
        "\n",
        "The former can try up to four methods of authentication:\n",
        "- Using creds set in environment variables\n",
        "- Using creds available in an AzureCLI logon\n",
        "- Using the Managed Service Identity (MSI) credentials of the machine you are\n",
        "  running the notebook kernel on\n",
        "- Interactive browser logon\n",
        "\n",
        "To use chained authentication methods select the methods to want to use and leave\n",
        "the clientId/tenantiId/clientSecret fields empty.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpedit.set_tab(\"Data Providers\")\n",
        "mpedit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Optional Settings 2 - Autoload QueryProviders\n",
        "\n",
        "This section controls which, if any query providers you want to load automatically\n",
        "when you run `nbinit.init_notebook`.\n",
        "\n",
        "This can save a lot of time if you are frequently authoring new notebooks. It also\n",
        "allows the right providers to be loaded before other components that might use them such as\n",
        "- Pivot functions\n",
        "- Notebooklets\n",
        "(more about these in the next section)\n",
        "\n",
        "There are two types of provider support:\n",
        "- Microsoft Sentinel - here you specify both the provider name and the workspace name that you want to connect to.\n",
        "- Other providers - for other query providers, just specify the name of the provider.\n",
        "\n",
        "Available Microsoft Sentinel workspaces are taken from the items you configured in the **Microsoft Sentinel** tab.\n",
        "Other providers are taken from the list of available provider types in *MSTICPy*.\n",
        "\n",
        "There are two options for each of these:\n",
        "- **connect** - if this is True (checked) *MSTICPy* will try to authenticate to the\n",
        "  provider backend immediately after loading. This assumes that you've configured\n",
        "  credentials for the provider in your settings.\n",
        "  Note: if this is not set it defaults to True.\n",
        "- **alias** - when MSTICPy loads a provider it assigns it to a Python variable name.\n",
        "  By default this is \"qry_*workspace_name*\" for Microsoft Sentinel providers and\n",
        "  \"qry_*provider_name*\" for other providers. If you want to use something a bit shorter\n",
        "  and easier to type/remember you can add a *alias*. The variable name created\n",
        "  will be \"qry_*alias*\"\n",
        "  \n",
        "> **Note** if you lose track of which providers have been loaded by\n",
        "> this mechanism they are added to the `current_providers` attribute of\n",
        "> `msticpy`\n",
        "```python\n",
        "   import msticpy\n",
        "   msticpy.current_providers\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpedit.set_tab(\"Autoload QueryProvs\")\n",
        "mpedit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Optional Settings 3 - Autoloaded Component\n",
        "This section controls which, if other components you want to load automatically\n",
        "when you run `nbinit.init_notebook()`.\n",
        "\n",
        "This includes\n",
        "- TILookup - the Threat Intel provider library\n",
        "- GeopIP - the Geo ip provider that you want to use\n",
        "- AzureData - the module used to query details about Azure resources\n",
        "- AzureSentinelAPI - the module used to query the Microsoft Sentinel API\n",
        "- Notebooklets - loads notebooklets from the [msticnb package](https://msticnb.readthedocs.io/en/latest/)\n",
        "- Pivot - pivot functions\n",
        "\n",
        "These are loaded in this order, since the Pivot component needs query and other providers\n",
        "loaded in order to find the pivot functions that it will attach to entities.\n",
        "For more information see [pivot functions](https://msticpy.readthedocs.io/en/latest/data_analysis/PivotFunctions.html)\n",
        "\n",
        "Some components do not require any parameters (e.g. TILookup and Pivot). Others do support or require additional\n",
        "settings:\n",
        "\n",
        "**GeoIpLookup**\n",
        "\n",
        "You must type the name of the GeoIP provider that you want to use - either \"GeoLiteLookup\" or \"IPStack\"\n",
        "\n",
        "**AzureData** and **AzureSentinelAPI**\n",
        "- **auth_methods** - override the default settings for AzureCLI and connect using the selected methods\n",
        "- **connnect** - set to false to load but not connect\n",
        "\n",
        "**Notebooklets**\n",
        "\n",
        "This has a single parameter block **AzureSentinel**. At minumum you\n",
        "should specify the workspace name. This needs to be in the following\n",
        "format:\n",
        "```\n",
        "    workspace:WORKSPACENAME\n",
        "```\n",
        "WORKSPACENAME must be one of the workspaces defined in the Microsoft Sentinel tab.\n",
        "\n",
        "You can also add addition parameters to send to the notebooklets init function:\n",
        "Specify these as addition key:value pairs, separated by newlines.\n",
        "```\n",
        "    workspace:WORKSPACENAME\n",
        "    providers=[\"LocalData\",\"geolitelookup\"]\n",
        "```\n",
        "See the \n",
        "[msticnb `init` documentation](https://msticnb.readthedocs.io/en/latest/msticnb.html#msticnb.data_providers.init)\n",
        "for more details\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpedit.set_tab(\"Autoload Components\")\n",
        "mpedit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Save your file and add the MSTICPYCONFIG environment variable\n",
        "\n",
        "Save your file, and, if you haven't yet done so, create an\n",
        "enviroment variable to point to it. See [Setting the path to your msticpyconfig.yaml](#Setting-the-path-to-your-msticpyconfig.yaml)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "### Validating your `msticpyconfig.yaml` settings\n",
        "\n",
        "MpConfigFile includes a validation function that can help\n",
        "you diagnose setup problems.\n",
        "\n",
        "You can run this interactively or from Python.\n",
        "\n",
        "The examples below assume that you have set `MSTICPYCONFIG` to point\n",
        "to you config file. If not, you will need to use the `load_from_file()`\n",
        "function (or Load File button) to load the file before validating."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpconfig = MpConfigFile()\n",
        "mpconfig.load_default()\n",
        "mpconfig.validate_settings()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To validate interactively:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "mpconfig = MpConfigFile()\n",
        "mpconfig.load_default()\n",
        "mpconfig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "## The `config.json` file\n",
        "When you start a notebook from Microsoft Sentinel for the first time it will create a `config.json` file in\n",
        "your notebooks folder. This should be populated with your workspace and tenant IDs needed to \n",
        "authenticate to Microsoft Sentinel.\n",
        "\n",
        "If you are using notebooks in a different environment you may need to create a `config.json` or `msticpyconfig.yaml` (see below)\n",
        "to supply this information to your notebook.\n",
        "\n",
        "We recommend creating a `msticpyconfig.yaml` since this can hold a wide variety \n",
        "of settings for your notebook, including multiple Microsoft Sentinel workspace settings.\n",
        "The config.json, in contrast, only holds settings for a single Microsoft Sentinel workspace.\n",
        "\n",
        "For more information see this [msticpy Package Configuration](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)\n",
        "\n",
        "---\n",
        "\n",
        "If you need to create or modify your config.json you can run the following cell.\n",
        "\n",
        "You will need the subscription and workspace IDs for your Microsoft Sentinel Workspace. These can be found here in the Microsoft Sentinel portal as shown below.\n",
        "\n",
        "<img src=\"https://github.com/Azure/Azure-Sentinel-Notebooks/raw/master/src/images/az_sentinel_settings1.png\" width=\"600\">\n",
        "\n",
        "<br>Copy the subscription and workspace IDs:<br>\n",
        "<img src=\"https://github.com/Azure/Azure-Sentinel-Notebooks/raw/master/src/images/az_sentinel_settings2.png\" style=\"width: 600px;\"/>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "98d4d8c746fd49c2b3b4ba0b404b2fce",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Text(value='./config.json', description='Path to config.json', layout=Layout(width='95%'), stylâ€¦"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import requests\n",
        "import json\n",
        "import ipywidgets as widgets\n",
        "from pathlib import Path\n",
        "from datetime import datetime\n",
        "\n",
        "config_dict = {}\n",
        "\n",
        "\n",
        "def get_tenant_for_subscription(sub_id):\n",
        "    aad_url = (\n",
        "        f\"https://management.azure.com/subscriptions/{sub_id}?api-version=2016-01-01\"\n",
        "    )\n",
        "\n",
        "    resp = requests.get(aad_url)\n",
        "    if resp.status_code == 401:\n",
        "        hdr_list = resp.headers[\"WWW-Authenticate\"].split(\",\")\n",
        "        hdr_dict = {\n",
        "            item.split(\"=\")[0].strip(): item.split(\"=\")[1].strip() for item in hdr_list\n",
        "        }\n",
        "        return hdr_dict[\"Bearer authorization_uri\"].strip('\"').split(\"/\")[3]\n",
        "    else:\n",
        "        return None\n",
        "\n",
        "\n",
        "def save_config_json(file_path, **kwargs):\n",
        "    if Path(file_path).exists():\n",
        "        bk_file = (\n",
        "            str(Path(file_path))\n",
        "            + \".bak\"\n",
        "            + datetime.now().isoformat(timespec=\"seconds\").replace(\":\", \"-\")\n",
        "        )\n",
        "        print(f\"Exising config found. Saving current config.json to {bk_file}\")\n",
        "        Path(file_path).rename(bk_file)\n",
        "\n",
        "    with open(file_path, \"w\") as fp:\n",
        "        json.dump(kwargs, fp, indent=2)\n",
        "    print(f\"Settings saved config to {file_path}\")\n",
        "\n",
        "\n",
        "def save_config(b):\n",
        "    tenant = input_tenant.value\n",
        "    if not tenant:\n",
        "        tenant = get_tenant_for_subscription(input_wgt[\"tenant\"].value)\n",
        "        print(f\"TenantID found: {tenant_id}\")\n",
        "\n",
        "    save_config_json(\n",
        "        file_path=input_wgt[\"path\"].value,\n",
        "        tenant_id=tenant,\n",
        "        subscription_id=input_wgt[\"sub_id\"].value,\n",
        "        workspace_id=input_wgt[\"ws_id\"].value,\n",
        "        workspace_name=input_wgt[\"workspace\"].value,\n",
        "        resource_group=input_wgt[\"res_grp\"].value,\n",
        "    )\n",
        "\n",
        "\n",
        "DEFAULT_CONFIG = \"./config.json\"\n",
        "WIDGET_DEFAULTS = {\n",
        "    \"layout\": widgets.Layout(width=\"95%\"),\n",
        "    \"style\": {\"description_width\": \"200px\"},\n",
        "}\n",
        "\n",
        "input_wgt = {\n",
        "    \"path\": widgets.Text(\n",
        "        description=\"Path to config.json\", value=DEFAULT_CONFIG, **WIDGET_DEFAULTS\n",
        "    ),\n",
        "    \"workspace\": widgets.Text(\n",
        "        description=\"Workspace name\", placeholder=\"Workspace name\", **WIDGET_DEFAULTS\n",
        "    ),\n",
        "    \"sub_id\": widgets.Text(\n",
        "        description=\"Microsoft Sentinel Subscription ID\",\n",
        "        placeholder=\"for example, ef28a760-8c61-41d7-8167-5c8e5d91268b\",\n",
        "        **WIDGET_DEFAULTS,\n",
        "    ),\n",
        "    \"ws_id\": widgets.Text(\n",
        "        description=\"Microsoft Sentinel Workspace ID\",\n",
        "        placeholder=\"for example, ef28a760-8c61-41d7-8167-5c8e5d91268b\",\n",
        "        **WIDGET_DEFAULTS,\n",
        "    ),\n",
        "    \"res_grp\": widgets.Text(\n",
        "        description=\"Resource group\", placeholder=\"Resource group\", **WIDGET_DEFAULTS\n",
        "    ),\n",
        "    \"tenant\": widgets.Text(\n",
        "        description=\"TenantId\", placeholder=\"Leave blank to look up\", **WIDGET_DEFAULTS\n",
        "    ),\n",
        "}\n",
        "\n",
        "if Path(DEFAULT_CONFIG).exists():\n",
        "    with open(DEFAULT_CONFIG, \"r\") as fp:\n",
        "        config_dict = json.load(fp)\n",
        "    input_wgt[\"path\"].value = DEFAULT_CONFIG\n",
        "    input_wgt[\"sub_id\"].value = config_dict.get(\"subscription_id\", \"\")\n",
        "    input_wgt[\"ws_id\"].value = config_dict.get(\"workspace_id\" \"\")\n",
        "    input_wgt[\"workspace\"].value = config_dict.get(\"workspace_name\" \"\")\n",
        "    input_wgt[\"res_grp\"].value = config_dict.get(\"resource_group\" \"\")\n",
        "    input_wgt[\"tenant\"].value = config_dict.get(\"tenant_id\" \"\")\n",
        "\n",
        "save_button = widgets.Button(description=\"Save config.json file\")\n",
        "save_button.on_click(save_config)\n",
        "\n",
        "display(widgets.VBox([*(input_wgt.values()), save_button]))"
      ]
    }
  ],
  "metadata": {
    "hide_input": false,
    "history": [],
    "kernel_info": {
      "name": "python38-azureml"
    },
    "kernelspec": {
      "display_name": "Python 3.8 - AzureML",
      "language": "python",
      "name": "python38-azureml"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.10"
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": true,
      "title_cell": "Contents",
      "title_sidebar": "Contents",
      "toc_cell": true,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": true
    },
    "uuid": "75a9aa0a-6ec9-4b1a-a5f0-fc14ed6f8fab",
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "state": {},
        "version_major": 2,
        "version_minor": 0
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}