Azure instances 0 to 3 do not write diagnostic data to WadPerformanceCountersTable

Question

I am trying to request data from Azure WadPerformanceCountersTable.

I am trying to get the last 5 minutes of data.

The problem is that I only get data from nr instances. 4,5 and 6, but not from 0,1,2 and 3.

script I use to pull data:

Microsoft.WindowsAzure.CloudStorageAccount storageAccount = Microsoft.WindowsAzure.CloudStorageAccount.Parse(AppDefs.CloudStorageAccountConnectionString); CloudTableClient cloudTableClient = storageAccount.CreateCloudTableClient(); TableServiceContext serviceContext = cloudTableClient.GetDataServiceContext(); IQueryable<PerformanceCountersEntity> traceLogsTable = serviceContext.CreateQuery<PerformanceCountersEntity>("WADPerformanceCountersTable"); var selection = from row in traceLogsTable where row.PartitionKey.CompareTo("0" + DateTime.UtcNow.AddMinutes(-timespanInMinutes).Ticks) >= 0 && row.DeploymentId == deploymentId && row.CounterName == @"\Processor(_Total)\% Processor Time" select row; CloudTableQuery<PerformanceCountersEntity> query = selection.AsTableServiceQuery<PerformanceCountersEntity>(); IEnumerable<PerformanceCountersEntity> result = query.Execute(); return result; 

My diagnostics.wadcfg file:

 <?xml version="1.0" encoding="utf-8" ?> <DiagnosticMonitorConfiguration xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration" configurationChangePollInterval="PT1M" overallQuotaInMB="4096"> <PerformanceCounters bufferQuotaInMB="0" scheduledTransferPeriod="PT5M"> <PerformanceCounterConfiguration counterSpecifier="\Memory\Available Bytes" sampleRate="PT60S" /> <PerformanceCounterConfiguration counterSpecifier="\Processor(_Total)\% Processor Time" sampleRate="PT60S" /> </PerformanceCounters> </DiagnosticMonitorConfiguration> 

EDIT: Also, I have this code deployed in a test environment in azure mode and it works fine.

EDIT 2 : update to include XML service definitions:

 <ServiceDefinition name="MyApp.Azure" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2012-05.1.7"> <WebRole name="MyApp.Website" vmsize="ExtraSmall"> <Sites> <Site name="Web"> <Bindings> <Binding name="Endpoint1" endpointName="Endpoint1" /> </Bindings> </Site> </Sites> <Endpoints> <InputEndpoint name="Endpoint1" protocol="http" port="80" /> </Endpoints> <Imports> <Import moduleName="Diagnostics" /> </Imports> </WebRole> <WorkerRole name="MyApp.Cache" vmsize="ExtraSmall"> <Imports> <Import moduleName="Diagnostics" /> <Import moduleName="Caching" /> </Imports> <LocalResources> <LocalStorage name="Microsoft.WindowsAzure.Plugins.Caching.FileStore" sizeInMB="1000" cleanOnRoleRecycle="false" /> </LocalResources> </WorkerRole> </ServiceDefinition> 

After I read @Igorek's answer, I turned on the XML configuration of ServiceDefinition.csdef. I still don't know how to configure the LocalResources> LocalStorage configuration. Configuration must be configured for "MyApp.Website".

EDIT 3: I made these changes to the test azure account.

I set this to ServiceDefinitions.csdef

 <LocalResources> <LocalStorage name="DiagnosticStore" sizeInMB="4096" cleanOnRoleRecycle="false"/> </LocalResources> 

And I downgraded TotalQuota and BufferQuota in diagnosing .wadcfg In the end, in the WAD-control container, I have this configuration for each instance: http://pastebin.com/aUywLUfE

I will need to put this in an active account in order to see the results.

FINAL EDIT: Apparently, the overall quota was a problem, although I cannot guarantee this.

In the end, after a new post, I noticed this:

• the role instance had an XML configuration in wad-control-container with an overall quota 1024 MB and a BufferQuotaInMB of 1024 MB →, that was correct,
• 2 more role instances had a total quota of 4080 MB and BufferQuotaInMB 500 MB →, it was wrong, they were not written to the WADPerformanceCounters table.
• both XML configuration files (which were in wad-control-container ) that belonged to each instance of the role were deleted before the new publication.
• diagnostics.wadcfg configuration file is configured correctly: 1024 MB each

So, I think there is a problem with their publisher.

Two solutions were tested:

• I removed 1 invalid XML from "wad-control-container" and rebooted the machine. XML was rewritten and the role instance began to write in WADPerfCountTable.

• I used the script below on another invalid instance, and the incorrect role instance started to write in WADPerfCountTable.

    var storageAccount = CloudStorageAccount.Parse(AppDefs.CloudStorageAccountConnectionString); DeploymentDiagnosticManager diagManager = new DeploymentDiagnosticManager(storageAccount, deploymentId); IEnumerable<RoleInstanceDiagnosticManager> instanceManagers = diagManager.GetRoleInstanceDiagnosticManagersForRole(roleName); foreach (var roleInstance in instanceManagers) { DiagnosticMonitorConfiguration currentConfiguration = roleInstance.GetCurrentConfiguration(); TimeSpan configurationChangePollInterval = TimeSpan.FromSeconds(60); if (!IsCurrentConfigurationCorrect(currentConfiguration, overallQuotaInMb, TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1))) { // Add a performance counter for processor time. PerformanceCounterConfiguration pccCPU = new PerformanceCounterConfiguration(); pccCPU.CounterSpecifier = @"\Processor(_Total)\% Processor Time"; pccCPU.SampleRate = TimeSpan.FromSeconds(60); // Add a performance counter for available memory. PerformanceCounterConfiguration pccMemory = new PerformanceCounterConfiguration(); pccMemory.CounterSpecifier = @"\Memory\Available Bytes"; pccMemory.SampleRate = TimeSpan.FromSeconds(60); currentConfiguration.ConfigurationChangePollInterval = TimeSpan.FromSeconds(60); currentConfiguration.OverallQuotaInMB = overallQuotaInMb; currentConfiguration.PerformanceCounters.BufferQuotaInMB = overallQuotaInMb; currentConfiguration.PerformanceCounters.DataSources.Add(pccCPU); currentConfiguration.PerformanceCounters.DataSources.Add(pccMemory); roleInstance.SetCurrentConfiguration(currentConfiguration); } } 

Also, from time to time I get this error. The configuration file is missing a diagnostic connection string for one or more roles .

In the end, I will select the current answer as the answer because I found the problem. Unfortunately, I did not find the cause of the problem. Each time I publish, I risk getting a modified XML interface.

Answer 1

Seeing how your first instances do not transfer data to the diagnostics, and later instances, one of the possible reasons is this:

The local diagnostic storage on your servers is populated with diagnostic data, and Azure can no longer transfer data from your local storage to storage. Make sure that the space allocated for DiagnosticStore in the role configuration (in local storage) is greater than the amount of buffer quota allocated in diagnostics .wadcfg

Detailed explanation: I experienced it first hand with a number of clients, so my next interpretation is based on Microsoft support comments. The Azure Diagnostics API does not clear the local storage according to BufferQuota until this quota is exceeded. The diagnostic store in the cloud project is by default the same size as the BufferQuota used in all examples (4096). What happens is that your BufferQuota is terribly close to 4096megs, but not equal to the limit, and your diagnostic API does not start during the cleanup process. At the same time, your capture of diagnostic data no longer works properly, because the local storage is almost full, and the Azure server stops the application from writing to DiagnosticStore.

Your other servers should stop recording diagnostic data as soon as their local storage is also full.

Hope this makes sense.

Editing my answer to pinpoint the changes for those reading later:

The simplest approach is to reduce the need for CommonQuotaInMb, indicated in diagnostics.wadcfg, to be something like 4000 (make sure that all other combined buffers do not exceed this number)

Alternatively, or in addition, you can manually specify the space allocated to the diagnostic store in the virtual machine using the LocalStorage parameter in the .CSDEF file. This link shows how: http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.diagnostics.diagnosticmonitorconfiguration.overallquotainmb.aspx