Bash curl and variable in the middle of the url

Question

I would need to read certain data using curl. I mainly read keywords from a file

while read line do curl 'https://gdata.youtube.com/feeds/api/users/'"${line}"'/subscriptions?v=2&alt=json' \ > '/home/user/archive/'"$line" done < textfile.txt 

In any case, I did not find a way to form a curl url for it to work. I tried as all kinds of options with one and two quotes. I tried mostly:

 '...'"$line"'...' "..."${line}"..." '...'$line'...' 

etc. Just name it, and I'm sure I tried.

When I print out the URL at best, it will be formed as:

  /subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE 

or something similar. If you know what might be causing this, I would appreciate information. Thanks!

Answer 1

These are not quotation marks. The problem is that your keyword file is in DOS format, that is, each line ends with a carriage return and line feed (\ r \ n), and not just a line feed (\ n). The carriage return is read into the line variable and included in the URL. The downside is that when you echo it, it seems to print:

 /subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE" 

but it does print:

 https://gdata.youtube.com/feeds/api/users/KEYWORD FROM FILE /subscriptions?v=2&alt=json 

... with a carriage return between them, so the second one overwrites the first.

And what can you do about it? Here's a pretty simple way to trim cr at the end of a line:

 cr=$'\r' while read line do line="${line%$cr}" curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \ > "/home/user/archive/$line" done < textfile.txt 

Answer 2

Your current version should work, I think. More elegant is the use of a single pair of double quotes throughout the URL with a variable in ${} :

 "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" 

Answer 3

Just use it like this should be enough:

 curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" > "/home/user/archive/${line}" 

If your shell gives you problems with & , just put \& , but it works fine for me without it.

Answer 4

If the data from the file may contain spaces, and you have no objection to spaces in the file name in the /home/user/archive directory, then what you have should be fine.

Given the contents of the rest of the URL, you can simply write:

 while read line do curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \ > "/home/user/archive/${line}" done < textfile.txt 

where strictly ${line} can be just $line in both places. This works because strings are fixed and do not contain shell metacharacters.

Since your code is close to this, but you claim to see keywords from a file elsewhere, there may be a slight rewrite to facilitate debugging in order:

 while read line do url="https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" file="/home/user/archive/${line}" curl "$url" > "$file" done < textfile.txt 

Since strings can contain spaces, it seems (do you need to expand spaces to + in the URL?), It is strongly recommended to use quotes around variables. Now you can run the script with sh -x (or add the set -x line to the script) and see what the shell thinks, what it does, how it does it.