I need to move 90 million files from the NFS folder to the second NFS folder, both connections to the NFS folder use the same eth0, which is 1 Gb / s for NFS servers, Sync is not required, just move (rewrite if it exists). I think my main problem is the number of files, not the total size. The best way should be a way with fewer system calls for each file in the NFS folders.
I tried cp, rsync and finally http://moo.nac.uci.edu/~hjm/parsync/ parsync first took 10 hours to generate a 12 GB gzip file list, after it took 40 hours and no one file was copied, it worked up to 10 threads, until I canceled it and started debugging, I found that it again calls (stat?) in each file (from the list) with the -vvv parameter (it uses rsync):
[sender] make_file(accounts/hostingfacil/snap.2017-01-07.041721/hostingfacil/homedir/public_html/members/vendor/composer/62ebc48e/vendor/whmcs/whmcs-foundation/lib/Domains/DomainLookup/Provider.php,*,0)*
parsync command:
time parsync --rsyncopts="-v -v -v" --reusecache --NP=10 --startdir=/nfsbackup/folder1/subfolder2 thefolder /nfsbackup2/folder1/subfolder2
Each rsync has the following form:
rsync --bwlimit=1000000 -v -v -v -a --files-from=/root/.parsync/kds-chunk-9 /nfsbackup/folder1/subfolder2 /nfsbackup2/folder1/subfolder2
NFS folders installed:
server:/export/folder/folder /nfsbackup2 nfs auto,noexec,noatime,nolock,bg,intr,tcp,actimeo=1800,nfsvers=3,vers=3 0 0
Any idea how to get rsync to copy files already in the list from nfs to the nfs2 folder? Or any way to make this copy efficiently (one system call per file?)