See this discussion , in particular this answer .
The solution seems to be to run rsync --sparse followed by rsync --inplace .
First, --sparse , a call, also use --ignore-existing to prevent overwriting already migrated sparse files, and -z to save network resources.
The second call --inplace should update only the changed fragments . Here, compression is optional.
Also see this post .
Update
I believe that the above suggestions will not help solve your problem. I also think rsync not suitable for this task. You should look for other tools that will provide you with a good balance between network and disk I / O performance.
rsync was designed to leverage a single resource, the network. It is assumed that reading and writing to the network is much more expensive than reading and writing source and target files.
We assume that both machines are connected using a two-band communication line with high bandwidth and low bandwidth. Rsync algorithm, abstract .
Algorithm summarized in four steps.
- The receiving side β sends checksums of blocks of size S of the target file B.
- The sending side α identifies the blocks that correspond to the source file A , at any offset.
- α sends β a list of instructions made from verbatim, non-compliant, data or corresponding block references.
- β restores the entire file from these instructions.
Note that rsync usually restores file B as a temporary file T , then replaces B with T. In this case, it should write the entire file.
--inplace does not free rsync from writing blocks mapped to α , as one might imagine. They can coincide with different offsets. Scanning B a second time to receive new data checksums is excessive in terms of performance. A block that matches the same offset that was read in the first step may be skipped, but rsync does not. In the case of a sparse file, block zero B will correspond for each block zero A and must be overwritten.
--inplace just makes rsync write directly to B instead of T. It will overwrite the entire file.