How can I compete with xcopy speed?

Question

Is there an open source project or best practice guide that shows the fastest way to copy files around a local machine, lan, san and wan, which can compete with the built-in xcopy speed for windows7 (or 8) or a copy of Windows Explorer?

To be blunt, not all IO files are created equal. Some protocols and methods have various overheads. Some libraries do not take advantage of asynchronous operations or use linear hardware speed.

I take an inventory of the big data that we use and evaluate the performance of our client and third-party applications. Some server applications are the worst offenders (java-based is the worst of the worst).

I limit the scope of these studies to SMB 2 and 3 (cifs on windows 7 and 8).

• Is there a lack of speed in using POSIX libraries? (fread, fopen, fseek, etc.)
• Is there any advantage when using win32 calls (CopyFile2, ReadFileEx)

Answer 1

xcopy is actually not the fastest way to copy files, especially to disks or through a local network. There, a commercial product called TeraCopy is much faster. This is a closed source, so I don’t know completely how it works, but one of the main differences is that instead of using one cycle to read a piece of data into a memory buffer and then write this buffer to a new location, it uses two streams and queues of producers / consumers.

The manufacturer reads the fragments of the source file and puts them in the queue. The consumer reads from the queue and writes to the target. The advantage here is that reading and writing can be done simultaneously. You should be careful, although you have a manufacturer to monitor the size of the queue, and not make too many queues to use too much memory - usually reading will be faster than writing, but it also depends on the source and destination of the places.