Mysterious Disappearing Azure Storage Assets

Question

We are building a site supported by Azure Storage. One working role has several files that it loads from the blob when it starts. Files never change after they are stored, we just pull them out and use them.

Sometimes, when trying to download these files from the development repository, the Storage Emulator service returns 500 errors. We can list the files in the blob and get metadata, but not download the file itself. The only solution we found was to remove blob and reupload.

Anyone else come across this?

Update : 1.7 SDK

Answer 1

Possible solution (work around)

• Exit Storage Emulator;
• Open administrative Sql Server Management Studio 2012;
• Attach the file C:\Users\<username>\DevelopmentStorageDb201206.mdf (where <username> is the name of the affected user);
• If it does not allow you to attach, copy the mdf and log files to another disk and then join, otherwise you can access LocalDB ( (localdb)\v11.0 );
• Find the stored procedure CommitBlockList ;
• Change SET UncommittedBlockIdLength = NULL to SET UncommittedBlockIdLength = 0 ;
• Execute it;
• Close Management Studio;
• Copy these edited mdf files, log files to the original location;
• Run the storage emulator;

How did i get there

I found that approximately every seven days, BLOBs were deleted.
Creating these blobs again for testing purposes was painful, while in the middle of development / testing.
I tried to find the source code of the storage emulator, but could not find it.
Enabling the log when C:\Users\<username>\AppData\Local\DevelopmentStorage adding the following to DevelopmentStorage.201206.config

 <LogPath>C:\Users\<username>\AppData\Local\DevelopmentStorage\Logs</LogPath> <LoggingEnabled>true</LoggingEnabled> 

After a painful wait, the following in the magazines:

DefragmentBlobFiles Name BlobInfo 40f5e12f-65a5-4a3a-ae46-41c71c8514c0 / file1.txt, ContainerName storage1, Directory c: \ users \ username \ appdata \ local \ developmentstorage \ ldb \ blockblobroot \ 1 \ 12735b4b-f9f9b-f9f9b-f9f1bfb-f9 ROFile, RWFile c: \ users \ username \ appdata \ local \ developmentstorage \ ldb \ blockblobroot \ 1 \ 12735b4b-f9ed-481b-a091-78387facf05b \ 1, Size5

I don’t think that there were problems with defragmentation.
Another log found:

BlockBlob: loading interval failed. IsGC: True, Exception in System.Number.ParseDouble (String value, NumberStyles, NumberFormatInfo numfmt options) in Microsoft.WindowsAzure.DevelopmentStorage.Store.BlockBlobGarbageCollector.GetTimerIntervalOrDefault (Boolean isGC)

So, for BlockBlobs, Unblocked blocks are collected using this BlockBlobGarbageCollector block. Nowhere could I find how often these uncommitted blocks collect garbage. I do not think that even this causes a problem.

Another magazine:

BlockBlob: check the directory C: \ Users \ username \ AppData \ Local \ DevelopmentStorage \ LDB \ BlockBlobRoot \ 1 \ 0477877c-4cb3-4ddb-a035-14a5cf52d86f in the list of valid directories
BlockBlob: delete the directory C: \ Users \ username \ AppData \ Local \ DevelopmentStorage \ LDB \ BlockBlobRoot \ 1 \ 0477877c-4cb3-4ddb-a035-14a5cf52d86f

THIS HIGHER MODE COMPLETES THE PROBLEM. The emulator must determine the correct blockblob directories.

Proven DevelopmentStorageDb201206 database schema. Multiple columns found, for example IsCommitted and UncommittedBlockIdLength . Found ClearUncommittedBlocks set UncommittedBlockIdLength to null . Any Blob objects that were not deleted were set to UncommittedBlockIdLength 0. So we checked the CommitBlockList stored procedure and changed UncommittedBlockIdLength to 0 instead of null . I think that the emulator in the previous version should check IsCommitted and UncommittedBlockIdLength both to determine the correct blockblob directories, whereas in this version it can only check UncommittedBlockIdLength as null and delete all these block block files.

As I said, it takes about seven days to find out if this fixes a decision. I have 4 more days to check it out.

If this is a workaround that works ... Microsoft owes me 6 hours;)

Answer 2

I ran into the same problem. I suspect that something is wrong with the Azure storage emulator when it runs on top of LocalDB. That's why:

• when you get 500 errors, open the Blob table using Management Studio or Server Explorer and find the value of the DirectoryPath field for the blob in question (it will be something like c: \ users \ username \ appdata \ local \ developmentstorage \ AML \ blockblobroot \ 1 \ 305469d0-7b68-4b1e-a41a-a429c21b6b9d

• navigate to this path using explorer and note that this directory is empty

• now reload the file and navigate to the new directory that it downloaded and note that there is your file

So now the question is why are the blob files disappearing?

Answer 3

I have no solution, but I can add that this also happens when the backup storage repository of the storage emulator is SQL Server 2012. We have seen this many times. Everything is fine for a while, then the drops disappear. Our experience is that all or almost all blobs disappear from the file system, while database links are preserved. I don’t know why this is happening - there is no obvious event accelerating the process.

Answer 4

I also saw that the storage emulator sometimes returns for valid requests. But usually try again, it will work fine. It is always recommended that you retry a failed request if the response does not indicate that the request is invalid (in this case, retrying does not work at all). Even if you use cloud storage, sometimes it may still encounter a problem, such as a temporary network, not available. In fact, a retry policy is recommended for all network-related operations.

In addition, if you use the .NET client repository library, you can use the built-in retry policy: http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/03/overview-of-retry-policies-in -the-windows-azure-storage-client-library.aspx .

Best wishes,

Ming Xu.

Answer 5

Perhaps an easier way to solve this problem is to upgrade to Azure SDK 1.8 - since I upgraded from 1.7 (from 2012-11-23, about two weeks ago), my drops no longer disappear.

You can use the new storage emulator, even if you decide to use the old libraries - in fact, the libraries are installed side by side, only emulators are updated (that is, new versions replace the old ones), but they still remain compatible with old versions. For example, now I am using the new repository emulator, but I still refer to the 1.7 libraries in my .NET projects.

Update 2013-08-30 18:00 UTC I again had a problem with Azure SDK 2.0 - I write drops in a few weeks and after a few minutes I get HTTP 500 when I access them, I wonder if this is caused by the behavior of some the characteristics of my storage emulator (for example, too many blobs and / or containers). In this regard, I want to add that I updated it from Storage Emulator 1.8, renaming the database files from the schema (compared to Visual Studio) seemed to be the same, but maybe I missed something important.

Now I am evaluating whether I need to upgrade to Azure SDK 2.1 and hope that this problem is fixed or switch to using real storage accounts for development.

Update 2013-09-02 18:00 UTC I decided to use a real account for storage for the following reasons:

• the ability to save stored data through the Azure SDK updates - often the storage emulator changes the layout or location of the database without a note in the release note, forcing manual migration;
• the ability to use all the features of Azure Storage without fear of a disparity between the production and storage emulator (as Richard Astbury said in his comment);
• Avoid performance and maintenance issues specific to SQL Server implementation (for me, this is a random complexity).