How to twist several resources into one team?

Question

Let's say I'm trying to download a set of 50 lectures. These notes are in the prof subdirectory of the university website. The 45th lecture is located inside the lect45 subdirectory in PDF format called lect45.pdf . I get my first pdf as follows:

 curl -O http://www.university.edu/~prof/lect1/lect1.pdf 

How to get all my 50 notes efficiently using cURL and bash? I am trying to do this from the command line and not through the Python / Ruby / Perl script. I know that something like below will generate a lot of 404s:

 curl -O http://www.university.edu/~prof/lect{1..50}/lect{1..50}.pdf 

so what will work better? I would prefer an elegant single line line above the outline.

Answer 1

Do this in several processes:

 for i in {1..50} do curl -O http://www.university.edu/~prof/lect$i/lect$i.pdf & done 

or as single line (just different formatting):

 for i in {1..50}; do curl -O http://www.university.edu/~prof/lect$i/lect$i.pdf & done 

& executes all processes in parallel.

Do not be afraid of going out; the shell tells you that 50 processes have been started, which is a lot of spam. He will later tell you about each of them that they have stopped. Lots of results again.

You probably don't want to run all 50 in parallel; -)

EDIT:

Your example using {1..50} twice makes a matrix of numbers. See for example echo {1..3}/{1..3} to understand what I mean. And I guess that way you create a lot of 404s.

Answer 2

See the parallel shell tool.

So, for this particular case, it will look like

 seq 50 | parallel curl -O http://www.university.edu/~prof/lect{}/lect{}.pdf 

As for curl - it does not have its own parallel mechanism, and what should it really be for? And your example with shell extension {1..50} seems valid for me.