Azure File System - Can I watch or just a poll?

Question

I am an experienced Windows C # developer, but new to the Azure world and therefore try to find “best practice” when I implement one or more Azure Cloud services.

I have several (external and external elements) that can save files to a folder (or perhaps a set of folders). In the current state of my Windows system, I have a FileSystemWatcher configured to monitor a folder and create an event when a file appears there.

In the Azure world, what is the equivalent way to do this? Or is there?

I know that I can create a timer (or sleep) to take some time (say, 30 seconds) and poll the folder, but I'm just not sure if this is the “best” way in the cloud.

It is important to note that I do not control the input data - in other words, files are saved by an external device that I do not control; therefore, I can’t, for example, push a message into the queue while saving the file and reply to this message ...

Although, in the end, this goal ... Therefore, I intend to have the "Observer" service, which (through events or polls) detects the presence of one or more files and clicks on the appropriate queue for the next step in my workflow to respond.

It should be noted that I use VS2015 and the latest Azure SDK materials, so I am not limited to any legacy.

What I still basically do is (a snippet of a larger code base):

storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString")); // Create a CloudFileClient object for credentialed access to File storage. fileClient = storageAccount.CreateCloudFileClient(); // Obtain the file share name from the config file string sharenameString = CloudConfigurationManager.GetSetting("NLRB.Scanning.FileSharename"); // Get a reference to the file share. share = fileClient.GetShareReference(sharenameString); // Ensure that the share exists. if (share.Exists()) { Trace.WriteLine("Share exists."); // Get a reference to the root directory for the share. rootDir = share.GetRootDirectoryReference(); //Here is where I want to start watching the folder represented by rootDir... } 

Thanks in advance.

Answer 1

If you use an attached drive (or a local disk with scratches), the behavior will be similar to any other Windows machine, so you just set up the file watcher using FileSystemWatcher and handle callbacks as you usually would.

There, the Azure File Service, which is SMB as a service, will support any action you could take on a regular SMB volume on your local network.

There is Azure blob storage. They cannot be viewed. You will need to poll the changes in, say, the blob container.

Answer 2

You can create a loop that periodically checks the root directory using CloudFileDirectory.ListFilesAndDirectories . https://msdn.microsoft.com/en-us/library/dn723299.aspx

You can also write a small recursive method to call this in subdirectories.

To detect differences, you can create a hash map in the memory of all files and directories. If you want something like a persistent distributed cache, you can use ie. Redis save this list of files / directories. Each time you conduct a survey, if a file or directory is not on your list, you find a new file / directory as root.

You can share the responsibility of discovery and business logic, i.e. the worker role continues to poll the directory and writes new files to the queue, and the consumer finishes another worker role / web service that processes this information.

Answer 3

Azure Blob Storage transfers events through the Azure Event Grid. Blob storage has two types of events: Microsoft.Storage.BlobCreated and Microsoft.Storage.BlobDeleted. Thus, instead of a lengthy survey, you can simply respond to the generated event.

See this link for more information:

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview

Answer 4

I had a very similar requirement. I used the BOX app. It has a Webhook function for events occurring in files or folders: such as Add, Move, Delete, etc.

There are also several new alternatives with Azure Autromation.

Answer 5

I'm also quite new to Azure, and I'm actually learning a topic like a watch file. Because of this, I am considering something that is related to Azure features, which looks like a way to run some code when creating or updating a blog. You can also specify a template there: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob