How to auto-heal your web app so you hopefully get your weekends back

Summary

This document outlines how to set up auto-healing for your web app. At times you may be enjoying your weekend and get a call telling you an app is constantly running at 100% CPU. 
Autoheal is a way to automatically recycle your app pool based on certain triggers.
I have created a POC web app with 3 instance VMs.I have turned on all diagnostic logs and set them to verbose. 
Here is the Event Log before enabling auto-heal in the web.config (there were more records but I removed them as they were all similar):













And here is the new web.config. This will recycle the app pool when we get 20 or more requests in a period of 30 seconds:









So, once this is deployed, I will call the end point 100 times in less than 30 seconds which should cause the trigger to run its action – Recycle
Here is the calling Powershell code (this just simulates a client):

$URL = ‘http://autohealingpoc.azurewebsites.net/api/values’;

$contentType = “application/json”;

for($i=1

    $i -le 100

    $i++){   

    Write-Host ‘I is: ‘ + $i;

   $response = Invoke-RestMethod -Method Get -Uri $URL -Headers $headers -ContentType $contentType;

   $response;

}


After I ran 100 requests in less than 30 seconds you can see in the event log that the instance RD0003FF857039 recycled:












When I really loaded it up, all 3 instance auto-healed:























Thanks
Russ

Advertisements

Monitor your Azure API Management Instance with PowerBI

Monitor your Azure API Management Instance with PowerBI

Summary

This document outlines the steps involved to monitor your Azure API management instance with PowerBI.

Note: I will refer to Azure API management as APIM in this document.

Steps will include:

  • Add a logger, using the APIM REST API, to your APIM instance to send events to an event hub
  • Set up a Stream Analytics job – It consists of one or more input data sources, a query expressing the data transformation, and one or more output targets that results are written to. Together these enable the user to perform data analytics processing for streaming data scenarios
  • Build a PowerBI dashboard to see your APIM data in a format that suits your business requirements.

Adding a logger to APIM

First thing you need to do is add a logger to your APIM instance using the APIM REST API. I will use Postman to do this.

Firstly, in your APIM instance, enable the REST API:















Secondly, go to the bottom of the security page where you enabled the REST API and generate a shared access key:





You will use this later in Postman after we first create an Azure Event hub.


Create an Event Hub

Go into your Azure portal and create an event hub.





Ensure you create to Event Hub shared access policies. 1 for sending to the event hub and 1 for receiving. This is to allow you more granular control over your hub.







































Creating the logger in Postman

Now that you have an Event Hub, the next step is to configure a Logger in your API Management service so that it can log events to the Event Hub.
API Management loggers are configured using the API Management REST API
To create a logger, make an HTTP PUT request using the following URL template.

https://{your service}.management.azure-api.net/loggers/{new logger name}?api-version=2014-02-14-preview

Replace {your service} with the name of your API Management service instance.
Replace {new logger name} with the desired name for your new logger. You will reference this name when you configure the log-to-eventhub policy.


Add the following headers to the request.
Specify the request body using the following template.

{
"type" : "AzureEventHub",
"description" : "Sample logger description",
"credentials" : {
"name" : "Name of the Event Hub from the Azure Classic Portal",
"connectionString" : "Endpoint=Event Hub Sender connection string"
}
}

 Here is mine:












You will see that it returned 201 Created which means we now have a logger.


Now we go into APIM and add a policy on the built in echo API.


This will send event to our event hub when the



ProductUnlimited

APIEcho API

OperationRetrieve resource 


API is hit.


View the Event Hub Events

If you want to view the event hub events for your own sanity check then download Service Bus Explorer and listen to your event hub:





















You will notice that the event hub data shown in the listener is the same data written by our APIM policy.

Create Stream Analytics Job to send data to PowerBI

Next, go to your Azure portal and create a new stream analytics job.

Once it is created then create an input:







































And a PowerBI output: Note that you will be prompted to authorize your PowerBI account.








































Also create a query to transform the data. My query doesnt do anything special:



















Then start your stream analytics job.




















Now go back to your APIM instance and hit the API end point a few times. Make sure it is the API operation 
with the policy on it.

Also, change param1 and param2 on the operation to a few different values so we get somewhat useful data:



















Now look at the trace for that operation in APIM and you will see the log to event hub event has fired:
















Now log into PowerBI on the web and you will see (hopefully) a new streaming dataset:
















Create a dashboard


We will create a PowerBI dashboard to visualise our APIM data using param1 and param2

I will drag a pie chart onto the workspace and set the following: All I did was add param2 as a count. As you can see we get a great visualisation of the number of times param2 was used on the APIM operation.





























So you can see that I set a request with param2=5 a lot more times that I did for other calls.


Obviously you can use your imagination as to what you can use this for.

Here we see a Tree Map, Pie Chart and Funnel displaying data from my APIM. The funnel shows distinct calls from IP address.





thanks
Russ

Deployment considerations in Azure – Cloud Services

Manage Deployments in Azure

Note: this is a work in progress

Staging area is not designed to be a “QA” environment but only a holding-area before production is deployed.
You should open up a new service for Testing environment with its own Prod/Staging. In this case, you will want to maintain multiple configuration file sets, one set per deployment environment (Production, Testing, etc.)
Staging is a temporary deployment slot used mainly for no-downtime upgrades and ability to roll back an upgrade.
Azure provides production and staging environments within which you can create a service deployment. When a service is deployed to either the production or staging environments, a single public IP address, known as a virtual IP address (VIP), is assigned to the service in that environment. The VIP is used for all input endpoints associated with roles in the deployment. Even if the service has no input endpoints specified in the model, the VIP is still allocated and used as the source address assigned to outbound traffic coming from each role.

What happens when a service is promoted from staging to production?

Typically a service is deployed to the staging environment to test it before deploying the service to the production environment. When it is time to promote the service in staging to the production environment, you can do so without redeploying the service. This can be done by swapping the deployments.
The deployments can be swapped by calling the Swap Deployment Service Management API or by swapping the VIPs in the portal, which result in the same underlying operation on the hosted service. For more information on swapping the VIPs, see How to Manage Cloud Services.
Screen shot from Azure Portal – 2015-09-28















When the service is deployed, a VIP is assigned to the environment to which is it is deployed. In the case of the production environment, the service can be accessed by the URL, .cloudapp.net, or by the VIP. When a service is deployed to the staging environment, a VIP is assigned to the staging environment and the service can be accessed by a URL, .cloudapp.net, or by the assigned VIP. The assigned VIPs can be viewed in the portal or by calling the Get Deployment Service Management API.
When the service is promoted to production, the VIP and URL that were assigned to the production environment are assigned to the deployment that is currently in the staging environment, thus “promoting” the service to production. The VIP and URL assigned to the staging environment are assigned to the deployment that was in the production environment.
It is important to remember that neither the production public IP address nor the service URL changes during the promotion.
To examine how this works, we can illustrate a scenario in which there is a Deployment A deployed to the production environment. Additionally, there is a Deployment B deployed to the staging environment. The following table illustrates VIPs after the initial deployment of the services to production and staging:

 

Deployment A
VIP1
.cloudapp.net
Production
Deployment B
VIP2
.cloudapp.net
Staging
Once the Deployment B is promoted to production the VIPs are as follows:
Deployment B
VIP1
.cloudapp.net
Production
Deployment A
VIP2
.cloudapp.net
Staging
When the deployments are swapped, the deployment in the production environment that was associated with the production VIP and URL is now associated with the staging VIP. Likewise, the deployment in the staging environment that was associated with the staging VIP and URL is now associated with the production VIP.

Only new incoming connections are connected to the newly promoted service. Existing connections are not swapped during a deployment swap.

Persistence of VIPs in Windows Azure

Throughout the lifetime of a deployment, the VIP assigned will not change, regardless of the operations on the deployment, including updates, reboots, and reimaging the OS. The VIP for a given deployment will persist until that deployment is deleted. When a customer swaps the VIP between a stage and production deployment in a single hosted service, both deployment VIPs are persisted. A VIP is associated with the deployment and not the hosted service. When a deployment is deleted, the VIP associated with that deployment will return to the pool and be re-assigned accordingly, even if the hosted service is not deleted. Windows Azure currently does not support a customer reserving a VIP outside of the lifetime of a deployment.

Managing ASP.NET machine keys for IIS

Azure automatically manages the ASP.NET machineKey for services deployed using IIS. If you routinely use the VIP Swap deployment strategy, you should manually configure the ASP.NET machine keys. For information on configuring the machine key, see Configuring Machine Keys in IIS 7.












More on machine keys later …

Questions for later:

Is there any way to deploy different instance sizes for test/production












Note that the image above shows multiple cscfg files, but only one csdef file. The cscfg file has the role names, instance counts, configuration values, and so on. The one csdef file is used with whichever configuration you select when you publish. It has a list of all of the configuration settings (but not the values), setup tasks (if applicable), the size of the VM to be used, and so on. The value you want to especially note is the VM size.

Using this methodology of multiple configuration files in one cloud project, you only have one place to set the size of the VM regardless of whether you are publishing to staging or production. You may not want to use the same sizes for staging and production, especially if you are using medium or larger VMs in production and small VMs in staging. In that case, you either have to change this every time you publish, or you have to have another solution.

Note: See the heading “Multiple cloud projects with their own configuration settings”:

http://blogs.msdn.com/b/microsoft_press/archive/2015/03/12/guest-article-microsoft-azure-dev-test-scenario-considerations.aspx

Debugging an custom object using Powershell in the Package Manager Console window.

All,

I was just trying to find out what data my collection of custom objects was getting hydrated with. the context doesn’t matter but for the record, I was hitting a Sitecore index and duplicates were being rendered in my UI.

I started using the immediate window but it had limitations that were preventing me from getting what I needed.

I wanted to see if all the objects that were hydrated that contained the word “test” hence the “test*”.

Anyway, here is what I came up with:

for($i = 0; $i -lt 1000; $i++) // 1000 is a bit high so adjust for your needs
$a = $dte.Debugger.GetExpression(“(()results[$i]).Name”); 

if ($a.Value -match “test*”) 
Write-Host $a.Value 
}

Here are the results:

  • “testcampaignspeed73” 
  • “testcampaignspeed74”
  • “testcampaignspeed2”
  • “testcampaignspeed23”


And here is the PMC window:










I just wanted to put this out there for myself to remember and for anyone else who needs it.

thanks
Russ 

Using Azure Powershell to attach to your azure subscription and create a new website

Hello,

As I haven’t posted for a while I wanted to just quickly post my findings on using Azure Powershell.

My aim here using only powershell is to:

  • Connect to my Azure subscription
  • Create a website in azure
  • Stop the website
  • Remove the website

Ok here goes.

Firstly you will need to install Azure Powershell so go here and work it out:

Then fire it up. It should look like this and you can find it in your windows start menu.


then we need to tell powershell about our azure subscription:


We just type Add-AzureAccount and you will be prompted to log into Azure.
Look here for info on this command:

After that you will see some confirmation that it fould your azure account:


You may need to change the execution policy for you to do things from your local Azure:


So here is my Azure website list before I add a new site:


So next I run New-AzureWebsite -Name MyPowerShellTest:

You can see it looks like it did something and returned info about my new site. If you look in Azure you can see the new site:



Nice!

Ok so for fun, lets stop the site:



So it worked.

Now ill delete the website completely using:
Remove-AzureWebsite -Name MyPowerShellTest

And it’s gone:


Well, you will have to trust me as I cant show you nothing!

thanks
Russ