Add wait between concurrent processes in bash

Question

I have a bash script to upload data to a site. I was getting slow download speeds, so I started running it in parallel, 5 at the same time, using xargs and -N1.

However, the problem is that the server asks me to solve the captcha if I run it 5 at a time, while it works fine with 1 at a time.

I believe that this is due to the fact that all processes start exactly at the same time, I get a checkbox.

In any case, here's the question: is there a way to add a wait (say 1 second) between the initial processes in the xargs / gnu parallel?

The only thing I could think of was to use a pgrep script | wc -1 to count instances of the script and hibernate in this number of seconds.

However, this is really not optimal, are there any better ways to do this?

Answer 1

If the download takes a certain amount of time, you just need the first 5 to start with a delay of 1-5 seconds:

cat list | parallel -j5 [ {#} -lt 6 ] \&\& sleep {#}\; upload {} 

Answer 2

Instead of using xargs, I think you just need a loop, as in

 for i in {1..5}; do sleep 5; your-command & done 

This drops commands every 5 seconds. To increase the delay (if necessary):

 for i in {1..5}; do ((w=i*5)); sleep $w; your-command & done 

Another alternative:

 files="a.txt b.txt c.txt" for i in $files; do upload-command $i& sleep 5; done 

Answer 3

This may work for you (uses GNU parallel):

  find . -type f -name "*.txt" -print | parallel 'script {} & sleep 1' 

Here's a terminal session showing an example:

 for x in {a..c};do for y in {1..3};do echo $x >>$x;done;done ls abc cat a a a a cat /tmp/job #!/bin/bash sed -i -e '1e date' -e 's/./\U&/' $1 sleep 5 sed -i '${p;s,.*,date,e}' $1 find . -type f -name "?" -print | parallel '/tmp/job {} & sleep 1' cat ? Sat Mar 10 20:25:10 GMT-1 2012 A A A Sat Mar 10 20:25:15 GMT-1 2012 Sat Mar 10 20:25:09 GMT-1 2012 B B B Sat Mar 10 20:25:14 GMT-1 2012 Sat Mar 10 20:25:08 GMT-1 2012 C C C Sat Mar 10 20:25:13 GMT-1 2012 

As you can see, each task runs on the second side, that is, file c starts from 08 ends at 13, file b 09-14 and writes from 10 to 15.

Answer 4

You can pause script execution after each process using

 read -p "Press [Enter] key to continue..". 

Now you can decide, at your discretion, when to start the next process.

I agree that this is due to manual intervention. But since in this particular case you need to start only 5 processes, this should work out in order.

EDIT . Since read stops automation, you can use

 sleep 5 

who will sleep for 5 seconds.