Download multiple files from one large file and unzip through a socket

I need my client to upload files with a cost of 30 MB.

Below is the setting.

  • They consist of 3,000 small files.
  • They boot via the tcp bsd socket.
  • They are stored in the client database as they are downloaded.
  • The server can store all the necessary files in memory (without access to files on the server side)

I have not seen many cases where a client uploads such a large number of files that I suspect due to access to the file on the server side.
I am also concerned that the multiplexer (select / epoll) will be overloaded with excessive processing of network requests. (Do I need to worry about this?)

With the above suspicions, I have archived 3,000 files to 30 files. (total size doesn't change much because files are already compressed (png))

The test shows Uploading 3000 files is 25% faster than 30 files downloading and unpacking.
I suspect because the client device cannot load when unpacking and pasting into the database, I am testing on handheld devices .. iPhone ..
(I cut the expansion and unlocking of the database separately from the network code, but working with the database seemed to intercept the entire system. I was a little profiled, and unzipping takes a little time, inserting DB does. On the server side, files with zip files are stored in memory in advance .)

I am considering returning to downloading 3000 files because it is faster for clients.

Interestingly, other experienced network users will say two strategies,
1. a lot of small data
2. A small amount of big data and unpacking.

EDIT

For experienced iphone developers, I perform a threading DB operation using NSOperationQueue.
Does NSOperationQueue really work fine?
I am very suspicious of its performance.
- I tried the posix thread, no significant difference.

+6
source share
1 answer

I answer my question.

It turned out that inserting many images into the SQL database immediately in the client takes a lot of time, as a result, the network packet in the path is not delivered to the client quickly enough.

http://www.sqlite.org/faq.html#q19

After I accepted the offer in faq to speed up the "many insert", it actually surpasses the "many files that load an individual strategy."

+1
source

Source: https://habr.com/ru/post/899749/


All Articles