Azure Cloud Deployment - Log Monitoring

Question

I have deployed my azure cloud service, but some of my instances stays unhealthy and keep recycling. I see 'Recycling (Role has encountered an error and has stopped. Sites were deployed.' in status column for unhealthy instances.

I enabled diagnostics for the role that does not work fine from configuration pane, but I don't know where to find logs of instance. Would you please tell me where to find?

Thanks,

Answer 1

Since this is a very popular issue, I'd like to offer an in-depth explanation here, thus some of this stuff you may already know:

There are a few popular scenarios when instances recycle:

1) Your app is missing dependent DLLs that do not allow IIS to start the app.

2) Your WorkerRole.cs or WebRole.cs contains an unhandled exception

3) Your startup script contains errors

In order to see properly debug this, you need to make sure that:

1) Azure Diagnostics is enabled (this can be done in Visual Studio via a checkbox) on Role's properties. A storage account is required to host the diagnostic data. It can be the one you publish your solution through.

2) Your WebRole.cs and/or WorkerRole.cs files contain proper try/catch with logging to some external storage. Logging to Trace seems simplest since Azure Diagnostics supports transferring the data out to external storage.

3) You have configured Azure Diagnostics to actually transfer the right data to a diagnostic storage account. Best way is to throw in a diagnostics.wadcfg file along with your solution that contains the needed configuration to transfer EventViewer, Infrastructure and Trace logs.

4) Lastly, you need to configure monitoring Trace logs in the app.config of the application so that the Trace infrastructure actually listens to your Trace calls.

This link contains more in-depth info on all of the information above.

Finally, after having done all of this you can actually inspect the data in the diagnostic storage account to see what is wrong. There are tools to help:

1) RedGate/Cerebrata's Management Studio @ http://www.cerebrata.com/products/azure-management-studio/introduction (commercial product)

2) Built-in tools into Visual Studio to help you explore contents of any storage account. (You have to hunt and query the necessary WAD tables and containers manually with it, but it is free)

3) AzureWatch @ http://www.paraleap.com (commercial product, geared toward active monitoring but lets you inspect most of the logs on-demand, disclaimer: I'm affiliated with this product)

Hope this helps

Answer 2

Try using Azure Explorer or the like to explore the content of your blob storage. If diagnostics is enables as you said, then Azure persist the configuration in an Azure blob named wad-control-container.