Run commands with a loop variable in parallel, no more than N at the same time

Question

I have three thousand files on the server. I can get one at a time using a REST API call. I wrote a command to extract these files. It works fine, but for my login time after about 200 downloads.

I would like to download all these files in parallel, and not one at a time. Ideally, I would like to get 1-200 files at a time, 200-400 at a time, 400-600 at the same time .... etc.

So my attempt:

FOR /L %i in (0,1,200) do wget --no-check-certificate --content-disposition  --load-cookies cookies.txt \ -p https://username:[e-mail]/download/%i

How can I convert this to the parallel call I want to create?

Thank.

Answer 1

Cygwin GNU Parallel 3000 200 , , :

seq 3000 | parallel -j 200 wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:[e-mail]/download/{}

Answer 2

, , , , .. %i %%i.

, :

FOR /L %%i IN (0,1,200) DO ( 
    start wget --no-check-certificate --content-disposition --load-cookies cookies.txt -p "https://username:[e-mail]/download/%%i"
)

200 , ( !) . , , . , .

: .bat, , .

Answer 3

; Windows UNIX . PowerShell.

200 , :

invoke-command -asjob -scriptblock {$files = @(1..200);$files | foreach-object{ & wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:[e-mail]/download/$_}};
invoke-command -asjob -scriptblock {$files = @(201..400);$files | foreach-object{ & wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:[e-mail]/download/$_}};
invoke-command -asjob -scriptblock {$files = @(601..400);$files | foreach-object{ & wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:[e-mail]/download/$_}};

Invoke-Parallel :

$filenames = @(1..600);    
invoke-parallel -InputObject $servers -throttle 200 -runspaceTimeout 30 -ScriptBlock { & wget --no-check-certificate --content-disposition  --load-cookies cookies.txt -p https://username:[e-mail]/download/$_}

(, , ) invoke-webrequest, , cookie .

: , Windows URL.

Answer 4

An alternative method for the GNU methodparallel is a good ol ' xargswith the option -P:

$ seq 3000 | xargs -i '{}' -n 1 -P 200 wget <url_start>{}<url_end>