I tested loading objects in swift using a multi-threaded application that made separate requests for creating objects. With 20 threads, I averaged ~ 6 objects per second per thread. Mathematics has shown that it will take quite a while. I switched to bulk operations and now has a multi-threaded application that downloads tar.gz files that include files in their respective containers. It works, but it is slower than what I had with individual object queries. I start 10 threads, each of which loads tar.gz with 4000 objects. These 10 threads operate at a speed close to ~ 2 objects per second per stream. It seems that I should do something wrong.
It seems that swift receives files in 5-10 seconds, but spends from 300 to 1600 seconds, expanding and placing objects in its containers. I'm not sure about this, it is based on viewing network traffic on a bootable machine quickly.
Thinking of factors that can affect performance:
- objects created in one container, against each object, another container
- number of objects per volumetric operation
- number of parallel operations with mass
- bulk tar file type, gz compression level (full or nothing)
What are the best methods for this kind of operation?
source share