Observe Azure Collection

This module which streamlines collection from multiple sources within Azure.

It creates three functions responsible for data collection:

event_hub_telemetry_func - captures the data sent to the Event Hub from the Azure resources Diagnostic settings. Event Hub triggers this function and forwards the data to Observe.

timer_resources_func - returns all resources within the location (region) and the corresponding metadata. It runs on an assigned NCRONTAB schedule and set to every 10 minutes by default.

timer_vm_metrics_func - returns virtual machine metrics from the hypervisor. It runs on an assigned NCRONTAB schedule and set to every 5 minutes by default.

Data Collection Module Installation

1. Install Azure's CLI

2. Ensure Azure CLI is properly installed by logging into Azure

   az login
   

   You should receive a token from your browser that looks like:

   [
     {
       "cloudName": "AzureCloud",
       "homeTenantId": "########-####-####-####-############",
       "id": "########-####-####-####-############",
       "isDefault": true,
       "managedByTenants": [],
       "name": "Acme Inc",
       "state": "Enabled",
       "tenantId": "########-####-####-####-############",
       "user": {
         "name": "[email protected]",
         "type": "user"
       }
     }
   ]
   

3. Install Azure's Functions Core Tools

4. Clone Observe's Terraform Collection Module (terraform-azure-collection) repo locally

    git clone [email protected]:observeinc/terraform-azure-collection.git

5. Assign Application Variables

   Inside the root of the terraform-azure-collection create a file named azure.auto.tfvars. The contents of that file will be:

observe_customer = "<OBSERVE_CUSTOMER_ID>"
observe_token = "<DATASTREAM_INGEST_TOKEN>"
observe_domain = "<OBSERVE_DOMAIN(i.e. observe-staging.com)>"
timer_resources_func_schedule = "<TIMER_TRIGGER_FUNCTION_SCHEDULE>" 
timer_vm_metrics_func_schedule = "<TIMER_TRIGGER_FUNCTION_SCHEDULE>"
location = "<AZURE_REGIONAL_NAME>"

Note: Default values are assigned for timer_resources_func_schedule and timer_vm_metrics_func_schedule, both based on NCRONTAB

location's value is Azure's Regional Name and is "eastus" by default

6. Deploy the Application

   Inside the root directory of the terraform-azure-collection module run the following commands:

    terraform init
    terraform apply -auto-approve

Collection should begin shortly

Azure Resource Configuration

To receive logs and metrics for resources please add the appropriate diagnostic settings to each. See "Azure Resource Configuration" section in Observe's Azure Integration page for more info.

Removing Observe's Azure Collection Module

1. Remove the terraform-azure-collection module by running the following in the root directory:

    terraform destroy

Note: You may encounter the following bug in the Azure provider during your destroy:

  Error: Deleting service principal with object ID "########-####-####-####-############", got status 403
  
  ServicePrincipalsClient.BaseClient.Delete(): unexpected status 403 with OData error:
  Authorization_RequestDenied: Insufficient privileges to complete the operation.

If this happens execute simply remove the azuread_service_principal.observe_service_principal from terraform state and continue the destroy.

1. Find the service_principal object name by using the id returned. i.e. (replace with your id)

  terraform state list -id=249783e5-bcfd-480b-b8e8-5f8aaa7452e8

2. Remove the object from state. Make sure to wrap the object in single quotes. i.e. (replace name with that returned in previous step.)

  terraform state rm 'module.collection["eastus"].azuread_service_principal.observe_service_principal' 

3. Re-perform the terraform destroy

  terraform destroy

Requirements

Name	Version
azurerm	>=3.86.0, <4.0.0

Providers

Name	Version
azuread	3.0.1
azurerm	3.116.0

Modules

No modules.

Resources

Name	Type
azuread_application.observe_app_registration	resource
azuread_application_password.observe_password	resource
azuread_service_principal.observe_service_principal	resource
azurerm_eventhub.observe_eventhub	resource
azurerm_eventhub_authorization_rule.observe_eventhub_access_policy	resource
azurerm_eventhub_namespace.observe_eventhub_namespace	resource
azurerm_key_vault.key_vault	resource
azurerm_key_vault_access_policy.app	resource
azurerm_key_vault_access_policy.user	resource
azurerm_key_vault_secret.observe_password	resource
azurerm_key_vault_secret.observe_token	resource
azurerm_linux_function_app.observe_collect_function_app	resource
azurerm_monitor_diagnostic_setting.observe_collect_function_app	resource
azurerm_resource_group.observe_resource_group	resource
azurerm_role_assignment.observe_role_assignment	resource
azurerm_service_plan.observe_service_plan	resource
azurerm_storage_account.observe_storage_account	resource
azuread_client_config.current	data source
azurerm_eventhub_namespace_authorization_rule.root_namespace_access_policy	data source
azurerm_subscription.primary	data source

Inputs

Name	Description	Type	Default	Required
app_settings	Additional app settings	map(string)	{ "FEATURE_FLAGS": "" }	no
func_url	Observe Collect Function source URL zip	string	"https://observeinc.s3.us-west-2.amazonaws.com/azure/azure-collection-functions-0.11.5.zip"	no
function_app_debug_logs	Enables routing of function app logs to eventhub for debugging eventhub & function app	bool	false	no
location	Azure Location to deploy resources	string	"eastus"	no
location_abbreviation	A unique, short abbreviation to use for each location when assiging names to resources	map(string)	{ "asiapacific": "ap", "australia": "as", "australiacentral": "ac", "australiacentral2": "ac2", "australiaeast": "ae", "australiasoutheast": "ase", "brazil": "b", "brazilsouth": "bs", "brazilsoutheast": "bse", "canada": "c", "canadacentral": "cc", "canadaeast": "ce", "centralindia": "ci", "centralus": "cu", "centraluseuap": "cue", "centralusstage": "cus", "eastasia": "ea", "eastasiastage": "eas", "eastus": "eu", "eastus2": "eu2", "eastus2euap": "eu2e", "eastus2stage": "eu2s", "eastusstage": "eus", "eastusstg": "eustg", "europe": "e", "france": "f", "francecentral": "fc", "francesouth": "fs", "germany": "g", "germanynorth": "gn", "germanywestcentral": "gwc", "global": "glob", "india": "i", "japan": "j", "japaneast": "je", "japanwest": "jw", "jioindiacentral": "jic", "jioindiawest": "jiw", "korea": "k", "koreacentral": "kc", "koreasouth": "ks", "northcentralus": "ncu", "northcentralusstage": "ncus", "northeurope": "ne", "norway": "n", "norwayeast": "nwe", "norwaywest": "nww", "qatarcentral": "qc", "singapore": "s", "southafrica": "sa", "southafricanorth": "san", "southafricawest": "saw", "southcentralus": "scu", "southcentralusstage": "scus", "southcentralusstg": "sctg", "southeastasia": "sea", "southeastasiastage": "sas", "southindia": "si", "swedencentral": "sc", "switzerland": "sz", "switzerlandnorth": "sn", "switzerlandwest": "sw", "uae": "uae", "uaecentral": "uc", "uaenorth": "un", "uk": "uk", "uksouth": "us", "ukwest": "uw", "unitedstates": "us", "unitedstateseuap": "use", "westcentralus": "wcu", "westeurope": "we", "westindia": "wi", "westus": "wu", "westus2": "wu2", "westus2stage": "wu2s", "westus3": "wu3", "westusstage": "wus" }	no
observe_customer	Observe customer id	string	n/a	yes
observe_domain	Observe domain	string	"observeinc.com"	no
observe_token	Observe ingest token	string	n/a	yes
prevent_rg_deletion	Prevent resource group deletion if resource group is not empty. Defaults to true.	bool	true	no
timer_resources_func_schedule	Eventhub name to use for resources function	string	"0 */10 * * * *"	no
timer_vm_metrics_func_schedule	Eventhub name to use for vm metrics function	string	"30 */5 * * * *"	no

Outputs

Name	Description
eventhub_name	Eventhub name used for Observe collection.
eventhub_namespace_id	Resource ID of the eventhub namespace used for Observe collection.
function_url	Function URL used for Observe collection.