Advertools/Advertools_Check_website_pages_status_code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c5f04051-0d87-43e5-8e32-a938245e3634",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"8%\" alt=\"Advertools.png\" src=\"https://github.com/raw/jupyter-naas/awesome-notebooks/master/.github/assets/logos/Advertools.png\" style=\"border-radius: 15%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6804ca94-e0dc-4072-8850-c131bb1009f1",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# Advertools - Check website pages status code\n",
    "<a href=\"https://bit.ly/3JyWIk6\">Give Feedback</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=Advertools+-+Check+website+pages+status+code:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b9853d8-5344-4973-9f07-8bc49fb05caa",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #advertools #website #status #code #check #pages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af1dcc2d-e7a7-4a0a-a2cc-671f1c2eeb4d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Florent Ravenel](https://www.linkedin.com/in/florent-ravenel)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de7760b-0f78-434f-b29e-0f41daf9c4c2",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Last update:** 2023-08-04 (Created: 2023-08-04)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbd8c55a-275c-445d-965a-8a4c7f87ebb6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Description:** This notebook crawls your website and checks the status code of all pages. It starts from the home page and discovers URLs by following links within the website. It is a useful tool for quickly checking the status of your website and generating a report to take necessary actions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "034bdb3a-9c8a-42b5-a0eb-4c71545f2cd3",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**References:**\n",
    "- [Advertools Documentation](https://advertools.readthedocs.io/en/latest/)\n",
    "- [HTTP Status Codes](https://www.restapitutorial.com/httpstatuscodes.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa56eea7-0ee3-42f0-97dc-d8fc7430b53e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb06c7d5-1b22-42ed-ab60-ac957a3e2e73",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Install libraries\n",
    "If running it on naas, run the code below to uninstall (bug) and install the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ab34592-6dfd-440e-9c5c-cf75ae29eda1",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# !pip uninstall -y scrapy attrs\n",
    "# !pip install advertools adviz pandas==1.5.3 --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c1bbdcb-2239-4fa0-bb8d-b41fbc3e695d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab56e61c-13b1-49b0-9990-c0ac15b50115",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import advertools as adv\n",
    "except ModuleNotFoundError:\n",
    "    !pip install advertools --user\n",
    "    import advertools as adv\n",
    "from datetime import datetime\n",
    "import naas\n",
    "from naas_drivers import emailbuilder, naasauth\n",
    "import plotly.express as px\n",
    "import pandas as pd\n",
    "try:\n",
    "    import adviz\n",
    "except ModuleNotFoundError:\n",
    "    !pip install adviz --user\n",
    "    import adviz\n",
    "import os\n",
    "pd.options.display.max_columns = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a6151cd-258b-4cbf-9cc7-0545d8097ac2",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup variables\n",
    "**Mandatory**\n",
    "\n",
    "- `website_url`: URL of the website page to check\n",
    "- `cron`: We use CRON tasks to schedule notebooks, find the syntax you need to on: https://crontab.guru/\n",
    "- `email_to`: Represents the recipient(s) of the email. By default, your email account on naas will be set.\n",
    "\n",
    "**Optional**\n",
    "\n",
    "- `output_dir`: Represents the output directory for the website crawl.\n",
    "- `timestamp`: Represents the timestamp when the code is executed.\n",
    "- `output_website_crawl`: Represents the output file name for the website crawl.\n",
    "- `output_website_crawl_log`: Represents the output file name for the log file of the website crawl.\n",
    "- `output_status_code_ko`: Represents the output file name for the status code report.\n",
    "- `subject`: Represents the subject line for the email."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be66e7aa-c1cd-4ade-9141-e7877af706bf",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Mandatory\n",
    "website_url = \"https://example.com/\"\n",
    "cron = \"0 0 * * *\" # This notebook will run everyday at 0:00\n",
    "email_to = [naasauth.connect().user.me().get(\"username\")]\n",
    "\n",
    "# Optional\n",
    "output_dir = website_url.split(\"https://\")[-1].split(\"/\")[0]\n",
    "timestamp = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "output_website_crawl = f'{timestamp}_website_crawl.jl'\n",
    "output_website_crawl_log = f'{timestamp}_website_crawl.log'\n",
    "output_status_code_ko = f'{timestamp}_status_code_ko.csv'\n",
    "subject = f\"Status code report: {website_url} as of {datetime.now().strftime('%Y-%m-%d')}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4b92cff-6977-42c9-b700-107fc9b434af",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a716cf9-42b0-442a-bcdc-668f4f3dfc21",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Define output paths\n",
    "Create the output directory and define paths for the output files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4cdfafc-39dc-421e-a516-9298146e37e6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Check if directory exists and create it if not\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "    \n",
    "# Generate outputs files path\n",
    "output_website_crawl_path = os.path.join(output_dir, output_website_crawl)\n",
    "output_website_crawl_log_path  = os.path.join(output_dir, output_website_crawl_log)\n",
    "output_status_code_ko_path  = os.path.join(output_dir, output_status_code_ko)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cf637d2-da47-4613-9c2f-bad22b996ef9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Crawl website"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17c21939-74a6-4964-af14-11ed2a8f1b6c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "crawl_params = dict(\n",
    "    url_list=website_url,\n",
    "    output_file=output_website_crawl_path,  # has to end with .jl\n",
    "    follow_links=True,  # the default is False\n",
    "    allowed_domains=None,\n",
    "    exclude_url_params=None,\n",
    "    include_url_params=None,\n",
    "    exclude_url_regex=None,\n",
    "    include_url_regex=None,\n",
    "    css_selectors=None,\n",
    "    xpath_selectors=None,\n",
    "    custom_settings= {\n",
    "        'LOG_FILE': output_website_crawl_log_path,\n",
    "        'CLOSESPIDER_PAGECOUNT': 0,\n",
    "        'CONCURRENT_REQUESTS_PER_DOMAIN': 8,\n",
    "        'DEFAULT_REQUEST_HEADERS': {},\n",
    "        'DEPTH_LIMIT': 0,\n",
    "        'USER_AGENT': adv.spider.user_agent\n",
    "    }\n",
    ")\n",
    "adv.crawl(**crawl_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f90528f-fefa-4469-9268-922108033eba",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Read crawl DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5416e3b0-c3a3-4c2c-81f2-17ca58d45215",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "crawl_df = pd.read_json(output_website_crawl_path, lines=True)\n",
    "print(\"Row fetched:\", len(crawl_df))\n",
    "crawl_df.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d9c934-3108-4802-b154-311733ea13f4",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Create DataFrame on status code KO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66d4c3fe-7b2b-4ce0-963a-7086315b4248",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_ko = crawl_df[~crawl_df[\"status\"].isin([200, 201, 202, 203])].reset_index(drop=True)\n",
    "print(\"Status code KO:\", len(df_ko))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa7efd9b-4b9b-4564-b368-c71a5df3fa7f",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Create email content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f30883d-429b-4dd3-ae12-9c988a75111b",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "total_urls = len(crawl_df)\n",
    "status_ko = len(df_ko)\n",
    "status_ok = total_urls - status_ko\n",
    "\n",
    "email_content = {\n",
    "    \"text1\": emailbuilder.text(\"Dear Team,\"),\n",
    "    \"text2\": emailbuilder.text(f\"We are sharing the report of status codes for the recent analysis on {website_url}:\"),\n",
    "    \"list\": emailbuilder.list([\n",
    "        f\"Total OK ✅: {status_ok}\",\n",
    "        f\"Total KO ❌: {status_ko}\",\n",
    "    ]),\n",
    "    \"text3\": emailbuilder.text(\"Please find enclosed the detailed of the status code failed.\"),\n",
    "    \"text4\": emailbuilder.text(\"Please take appropriate actions to address these issues.\"),\n",
    "    \"text5\": emailbuilder.text(\"Thank you!\"),\n",
    "    \"text6\": emailbuilder.text(\"Best regards,\"),\n",
    "}\n",
    "\n",
    "content = emailbuilder.generate(\n",
    "    display=\"iframe\",\n",
    "    **email_content\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f97c960-d76d-4912-9721-2e54a7852340",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d90c4742-1bf3-4fa0-b10e-7751fec6758c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Save report in CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40abc3e0-9533-4fe0-b21f-7806a1ba5bcb",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_ko.to_csv(output_status_code_ko_path, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c343400-43bd-4e83-88f4-f0dc68d8ce1c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Send notification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1c56ffe-ba51-4394-9398-7758c6f11e85",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "if len(email_to) > 0:\n",
    "    naas.notification.send(email_to, subject, content, files=[output_status_code_ko_path])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "830d891f-7c82-4954-a2f8-297172ab30ce",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Schedule notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf644383-3fba-455f-944a-b26bd617306c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "naas.scheduler.add(cron=cron)\n",
    "\n",
    "# naas.scheduler.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec5deee2-f478-407f-a60b-3b6b93a4c3dd",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "naas": {
   "notebook_id": "09c6603c1e23892e966a1e1a19a577d94d56c38695e3e2ca640728e81024f848",
   "notebook_path": "Advertools/Advertools_Check_website_pages_status_code.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.4.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}