Azure Table storage access time - insert / read from

Question

I am creating a program that stores and reads from Azure tables some that are stored in CSV files. I have CSV files that can have a different number of columns and between 3k and 50k rows. I need to load this data into an Azure table. So far, I have managed to upload and download data. I use the REST API, and for loading I create a batch XML request with 100 lines per request. Now it works fine, except that it takes too much time to load, for example. for 3k lines it takes about 30 seconds. Is there any way to speed this up? I noticed that it takes longer to respond to the execution (for the ReadToEnd() command ReadToEnd() . I read somewhere that setting the proxy to zero may help, but in my case it does not.

I also found that you could load the whole XML request into blob and then execute it from there, but I could not find any example for this.

  using (Stream requestStream = request.GetRequestStream()) { requestStream.Write(content, 0, content.Length); } using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { Stream dataStream = response.GetResponseStream(); using (var reader = new StreamReader(dataStream)) { String responseFromServer = reader.ReadToEnd(); } } 

As for obtaining data from azure tables, I managed to get 1000 objects per request. As for this, it takes me about 9 seconds for CS with 3k lines. It also takes longer to read from the stream. When I call this part of the code ( ReadToEnd() again):

  response = request.GetResponse() as HttpWebResponse; using (StreamReader reader = new StreamReader(response.GetResponseStream())) { string result = reader.ReadToEnd(); } 

Any tips?

Answer 1

As you mentioned, you use the REST API, you may have to write additional code and depend on your own methods to improve performance, and then use the client library. In your case, using the library of clients of the warehouse will be better, since you can use the already created functions to speed up insertion, upsert, etc., as described here .

However, if you use the Storage Client Library and ADO.NET, you can use the following article written by the Windows Azure Table team as a supported way to improve Azure Access performance:

.NET and ADO.NET Data Service Performance Recommendations for Windows Azure Tables