Using RSync to Copy a Serial Range of Files

Question

Sorry if this does not make sense, but I will try to provide all the necessary information!

I would like to use rsync to copy a series of sequentially numbered files from one folder to another.

I am archiving DCDM (his film) and it contains about 600,000 individually numbered consecutive .tif image files (~ 10 mb.).

I need to split this for proper archiving on LTO6 tapes. And I would like to use rsync to prepare folders so that my simple bash.sh file can automate various folders and files that I want to create on tape.

I usually use the rsync command:

sudo rsync -rvhW --progress --size only <src> <dest> 

If necessary, I use sudo , and I always check the result with --dry-run

The only way Ive can work (without errors) is to use the wildcard character * . However, these are only files with a given pattern (for example, 01* will only move files from the range 010000 - 019999 ), and I would have to repeat for 02 , 03 , 04 , etc.

I looked on the Internet and am struggling to find an answer that works.

This may not be possible, and with 600,000 .tif files, I cannot write an exception for each of them.

Any thoughts on how (if at all) this can be done?

Owen.

Answer 1

Globing is a shell property to expand a wildcard into a list of matching file names. You have already used it in your question.

In the following explanations, I assume that we are in a directory with the following files:

 $ ls -l 

 -rw-r----- 1 5gon12eder staff 0 Sep 8 17:26 file.txt -rw-r----- 1 5gon12eder staff 0 Sep 8 17:26 funny_cat.jpg -rw-r----- 1 5gon12eder staff 0 Sep 8 17:26 report_2013-1.pdf -rw-r----- 1 5gon12eder staff 0 Sep 8 17:26 report_2013-2.pdf -rw-r----- 1 5gon12eder staff 0 Sep 8 17:26 report_2013-3.pdf -rw-r----- 1 5gon12eder staff 0 Sep 8 17:26 report_2013-4.pdf -rw-r----- 1 5gon12eder staff 0 Sep 8 17:26 report_2014-1.pdf -rw-r----- 1 5gon12eder staff 0 Sep 8 17:26 report_2014-2.pdf 

The simplest case is to map all files. The following is an example for a poor person ls .

 $ echo * 

 file.txt funny_cat.jpg report_2013-1.pdf report_2013-2.pdf report_2013-3.pdf report_2013-4.pdf report_2014-1.pdf report_2014-2.pdf 

If we want to compare all reports from 2013, we can narrow the correspondence:

 $ echo report_2013-*.pdf 

 report_2013-1.pdf report_2013-2.pdf report_2013-3.pdf report_2013-4.pdf 

We could, for example, leave the .pdf part, but I like to be as specific as possible.

You already have a decision to use it to select a range of numbered files. For example, we can match reports by quater:

 $ for q in 1 2 3 4; do echo "$q. quater: " report_*-$q.pdf; done 

 1. quater: report_2013-1.pdf report_2014-1.pdf 2. quater: report_2013-2.pdf report_2014-2.pdf 3. quater: report_2013-3.pdf 4. quater: report_2013-4.pdf 

If we are lazy to enter 1 2 3 4 , we could use $(seq 4) . This calls the seq program with argument 4 and replaces its output ( 1 2 3 4 in this case).

Now back to your problem: if you need block sizes that have a capacity of 10, you should expand this example to suit your needs.

Answer 2

You can check the file name starting with a digit using pattern matching :

 for file in [0-9]*; do # do something to $file name that starts with digit done 

Or you can enable the extglob option and extglob over all file names containing only numbers. This can eliminate any potential unwanted files that start with a number but contain numbers without the first character.

 shopt -s extglob for file in +([0-9]); do # do something to $file name that contains only digits done 

• +([0-9]) expands to one or more digits

Update:

Based on the file name template in your last comment:

 shopt -s extglob for file in legendary_dcdm_3d+([0-9]).tif; do # do something to $file done 

Answer 3

an old question that I know, but someone might find this helpful. the above range extension examples also work with rsync . for example, to copy files starting with a, b and c, but not d and e from dir /tmp/from_here to dir /tmp/to_here :

 $ rsync -avv /tmp/from_here/[ac]* /tmp/to_here sending incremental file list delta-transmission disabled for local transfer or --whole-file alice/ bob/ cedric/ total: matches=0 hash_hits=0 false_alarms=0 data=0 sent 89 bytes received 24 bytes 226.00 bytes/sec total size is 0 speedup is 0.00 

Answer 4

If you are writing LTO6 tapes, you should include the "--inplace" command in the command. Inplace is designed to write to linear file systems such as LTO