In case you didn't know, you can set up Alerts in Application Insights. However what you probably also knew is that what you could do with them was very very (seriously, very) limited.

Just to give you an example, here is a list of things you COULD NOT do with the classic Alerts.

  • Base your alerts on automated custom queries that return results.
  • In order to base your alert on a custom metric you had to either send this metric several times to App Insights in order to make it visible in the Metric dropdown or provision the alert with one of the provided ways.
  • Use the "Equals to" condition
  • Customize the template of the email the alert sent is using
  • Set the alert to specific action group
  • Provide custom info about the problem in the email sent to make troubleshooting instant

The biggest problem in my opinion is that unless you ping a custom metric for both success and failure all the time, you cannot use this system as it's not automated on a scheduler.

However this would work for any of the metrics that come out of the box.

Now let's see what's up with the new Alerts they recently released.

By the looks of it Microsoft took all that feedback and delivered.

The new alerts are located right above the old ones. It is easy to miss as their icon is exactly the same as the old one. However the old one has the word "classic" in brackets next to it.

Once you click there and you select "Create new alert", the new workflow in order to create a new alert appears.

It is broken down in 3 sections.

  • Define alert condition
  • Define alert details
  • Define action group

Define alert condition

In this section you need to configure two things. The first thing that needs to be configured is the Application Insights instance that you are targeting. By default this is set to the one you clicked the Alert button from.

The next one is where the magic happens. By clicking "Add criteria" you open the "Configure signal logic" menu.

Under Monitor Service there is a service called "Application Insights (Preview)" which contains a single entry under it (as of 05/04/2018) called "Custom log search".

Let's see what the new view for setting up the new Custom log search looks like

Ok let me explain what's going on from top to bottom here.

The graph

The first thing you can see is a Graph that currently displays no data.
This graph will display more info based on the Query, Period and Frequency. It is a visualization tool based on the provided criteria. I found this very handy because I'm able to see the results of my query without having to run it in App Insight metrics.

Search query

This is the most important part. In there you can paste the query you want app insights to run on an schedule interval and check for results.

This is amazing but it comes with a limitation. You are not able to use things like the app("") function to cross query different app insights. I can live with that.

What this does behind the scenes is that it will append | count to the query provided to get the count of the results that this query returns.
Keep in mind that you should not add any timestamp related querying in the query you provided as this is something that you configure later in this view. In case it's there it will be ignored.

Alert logic

Pretty simple. Based on the number of results that this query returns you can set if the alert will be triggered or not.

Evaluated based on

Here is where you configure the period and the frequency of the check.
The limitation here is that Period cannot be less than Frequency (which makes perfect sense).

If you config configure Period and Frequency to 5 minutes then this query will be run against app insights every 5 minutes and it will check for results in the past 5 minutes.

NOTE: Because the evaluation is done every 5 minutes and metrics can take up to 2 minutes to show on application insights you alert might actually come from 7-10 minutes after the error occurred.

Define alert details

In this part you need to set up things like the name, the description the severity and whether you want this rule to be enabled upon creation. Simple stuff.

Define action group

Here is where you can select and action group or create one.
This essentially is about who will see this alert and has to take some action.
You can set it up to take many actions like: Email, SMS, Webhooks etc.

You can read more about action groups here: https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-action-groups

And that's it. You are up and running.

The alert

What's light years better than the previous alerting is that now the alert email actually includes useful data.

Here's an example.

(A LOT of data underneath as well)

In comparison, that's what the old alerts looked like.

Provisioning with ARM Template

So I exported the ARM template for this new type of alerts and I parameterised it.
You can thank me later. Or don't. :sademoji:

Alert.json

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "alertName": {
      "type": "string"
    },
    "alertDesc": {
      "type": "string"
    },
    "appInsightsName": {
      "type": "string"
    },
    "queryType": {
      "type": "string",
      "defaultValue": "ResultCount"
    },
    "triggerThresholdOperator": {
      "type": "string",
      "defaultValue": "GreaterThan",
      "allowedValues": [ "GreaterThan", "Equal", "LessThan" ]
    },
    "triggerThreshold": {
      "type": "int"
    },
    "frequencyInMinutes": {
      "type": "int",
      "defaultValue": 5
    },
    "timeWindowInMinutes": {
      "type": "int",
      "defaultValue": 5
    },
    "severityLevel": {
      "type": "string",
      "defaultValue": "3"
    },
    "actionGroupName": {
      "type": "string"
    },
    "appInsightsQuery": {
      "type": "string"
    }
  },
  "resources": [
  {
      "name": "[parameters('alertName')]",
      "apiVersion": "2017-09-01-preview",
      "type": "microsoft.insights/scheduledqueryrules",
      "location": "[resourceGroup().location]",
      "tags": {
        "[concat('hidden-link:', resourceId('microsoft.insights/components', parameters('appInsightsName')))]": "Resource"
      },
      "kind": null,
    "properties": {
      "description": "[parameters('alertDesc')]",
      "enabled": "true",
      "skuType": "L1",
      "source": {
        "query": "[parameters('appInsightsQuery')]",
        "authorizedResources": null,
        "resourceId": null,
        "dataSourceId": "[resourceId('microsoft.insights/components', parameters('appInsightsName'))]",
        "queryType": "[parameters('queryType')]"
      },
      "metricName": null,
      "schedule": {
        "frequencyInMinutes": "[parameters('frequencyInMinutes')]",
        "timeWindowInMinutes": "[parameters('timeWindowInMinutes')]"
      },
      "action": {
        "lastFiredTime": null,
        "severity": "[parameters('severityLevel')]",
        "status": "Active",
        "aznsAction": {
          "actionGroup": [
            "[resourceId('microsoft.insights/actionGroups', parameters('actionGroupName'))]"
          ],
          "emailSubject": null,
          "customWebhookPayload": null
        },
        "actionGroup": null,
        "alertStateManagement": null,
        "throttlingInMin": null,
        "trigger": {
          "thresholdOperator": "[parameters('triggerThresholdOperator')]",
          "threshold": "[parameters('triggerThreshold')]",
          "consecutiveBreach": 1,
          "metricTrigger": null
        },
        "odata.type": "Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.Microsoft.AppInsights.Nexus.DataContracts.Resources.ScheduledQueryRules.AlertingAction"
      }
    }
    }
  ]
}

Alert.paramaters.json

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "AlertName": {
      "value": "Provisioned Alert"
    },
    "alertDesc": {
      "value": "This is the description"
    },
    "AppInsightsName": {
      "value": "app-insights-name"
    },
    "TriggerThresholdOperator": {
      "value": "GreaterThan"
    },
    "TriggerThreshold": {
      "value": 5
    },
    "FrequencyInMinutes": {
      "value": 5
    },
    "TimeWindowInMinutes": {
      "value": 5
    },
    "SeverityLevel": {
      "value": "3"
    },
    "actionGroupName": {
      "value": "action-group"
    },
    "AppInsightsQuery": {
      "value": "requests"
    }
  }
}
Common gotchas

Something I noticed and it took me more time than I'd like to admit is what i call the "Inactive Application Insights" state.

You see, if you are provisioning Applications Insights right before you provision this alert then the deployment will fail and the error that the step will give back is that the query you have is invalid. Everything I write is amazing and there is no way I did something wrong (pause for laughter) so I started investigating.

Long story short Application insights has 2 different states.
Right after it's provisioned it has no data so it's inactive until some data is posted to it.

Here is how you can spot the difference.

If it is just provisioned and you try to run a query against it the response looks like this:

Inactive Application Insights

However if there is any sort of data created then the response looks like this:

Active Application Insights

If this is part of your deployment pipeline then i came up with a fix (or a hack depending on how you look at it).

By using Application Insight's REST API I am posting a custom event via powershell to awaken the Application Insights instance and then I add a wait step by using the Start-Sleep -Seconds 30 command which will wait before it provisions the alert. This will give plenty of time to the app insights instance to wake up and start doing it's thing.

Here is the powershell that will add a custom event. You have to provide the Instrumentation Key.

$body = (New-Object PSObject | 
Add-Member -PassThru NoteProperty name 'Microsoft.ApplicationInsights.Event' |  
Add-Member -PassThru NoteProperty time $([System.dateTime]::UtcNow.ToString('o')) |  
Add-Member -PassThru NoteProperty iKey 'InstrumentationKeyHere' |  
Add-Member -PassThru NoteProperty data (New-Object PSObject |  
 Add-Member -PassThru NoteProperty baseType 'EventData' | 
 Add-Member -PassThru NoteProperty baseData (New-Object PSObject | 
 Add-Member -PassThru NoteProperty ver 2 | 
 Add-Member -PassThru NoteProperty name 'Wake up app insights'))) | 
 ConvertTo-JSON -depth 5; 
    Invoke-WebRequest -Uri 'https://dc.services.visualstudio.com/v2/track' -Method 'POST' -UseBasicParsing -body $body

Just wait for 30 seconds after that and you're good to go.

Are you convinced yet?