Run the curl command on each line of the file and extract the data from the result

Question

Suppose I have a file containing a list of web page links.

www.xyz.com/asdd
www.wer.com/asdas
www.asdas.com/asd
www.asd.com/asdas

I know that execution curl www.xyz.com/asddwill bring me the html of this webpage. I want to get some data from this web page.

So, the script uses curl to hit all the links in the file each time and extract some data from the web page and store it in another place. Any ideas or suggestions.

Answer 1

As stated in the comments, this will go through your_fileand curlevery line:

while IFS= read -r line
do
   curl "$line"
done < your_file

<title> , grep - :

grep -iPo '(?<=<title>).*(?=</title>)' file

,

while IFS= read -r line
do
   curl -s "$line" | grep -Po '(?<=<title>).*(?=</title>)'
done < your_file

curl -s . . google:

$ curl -s http://www.google.com | grep -Po '(?<=<title>).*(?=</title>)'
302 Moved