Script to distribute a large number of files into smaller groups

Question

I have folders containing a large number of files (for example, 1000+) of different sizes, which I want to move into smaller groups, say, from 100 files per folder.

I wrote an Apple Script that counted the files, created a numbered subfolder, and then moved 100 new files to a new folder (the number of files could be specified) that looped until there was less than the specified number of files that he moved to the last created folder.

The problem was that she walked terribly slowly. I am looking for either an Apple Script or a Script wrapper. I can run on my MacBook and / or Linux box, which will efficiently move files into smaller groups.

How files are grouped is not particularly important, I just want fewer files in each folder.

Answer 1

This should help you:

DIR=$1
BATCH_SIZE=$2
SUBFOLDER_NAME=$3
COUNTER=1

while [ `find $DIR -maxdepth 1 -type f| wc -l` -gt $BATCH_SIZE ] ; do
  NEW_DIR=$DIR/${SUBFOLDER_NAME}${COUNTER}
  mkdir $NEW_DIR
  find $DIR -maxdepth 1 -type f | head -n $BATCH_SIZE | xargs -I {} mv {} $NEW_DIR
  let COUNTER++
if [ `find $DIR -maxdepth 1 -type f| wc -l` -le $BATCH_SIZE ] ; then
  mkdir $NEW_DIR
  find $DIR -maxdepth 1 -type f | head -n $BATCH_SIZE | xargs -I {} mv {} $NEW_DIR
fi
done

The nested if statement retrieves the last remaining files. You can add some additional checks, as you see, after you change for your use.

Answer 2

This is a huge shred, but it should not be too terrible:

rm /tmp/counter*
touch /tmp/counter1
find /source/dir -type f -print0 | 
    xargs -0 -n 100 \
        sh -c 'n=$(echo /tmp/counter*); \
               n=${n#/tmp/counter}; \
               counter="/tmp/counter$n"; \
               mv "$counter" "/tmp/counter$((n+1))"; \
               mkdir "/dest/dir/$n"; \
               mv "$@" "/dest/dir/$n"' _

This is completely inaudible as to which files go there.

Answer 3

. :

:

aardvark
apple
architect
...
zebra
zork

:

a/aardvark
a/apple
a/architect
b/...
...
z/zebra
z/zork

, :

a/aa/aardvark
a/ap/apple
a/ar/architect
...
z/ze/zebra
z/zo/zork

This should work pretty fast, because the move command that your script executes can use the simple glob extension to select all the files to move, ala mv aa* a/aa, as opposed to having to individually run the move command for each file (which would be my first guess as to why the original script was slow)