Problems with piping through sed

Question

I'm having trouble connecting through sed. Once I get the output in sed, I cannot execute the output of sed elsewhere.

wget -r -nv http://127.0.0.1:3000/test.html 

Outputs:

 2010-03-12 04:41:48 URL:http://127.0.0.1:3000/test.html [99/99] -> "127.0.0.1:3000/test.html" [1] 2010-03-12 04:41:48 URL:http://127.0.0.1:3000/robots.txt [83/83] -> "127.0.0.1:3000/robots.txt" [1] 2010-03-12 04:41:48 URL:http://127.0.0.1:3000/shop [22818/22818] -> "127.0.0.1:3000/shop.29" [1] 

I process the output via sed to get a clean list of URLs:

 wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g' 

Outputs:

 http://127.0.0.1:3000/test.html http://127.0.0.1:3000/robots.txt http://127.0.0.1:3000/shop 

I would then like to upload the output to a file, so I do this:

 wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g' > /tmp/DUMP_FILE 

I interrupt the process after a few seconds and check the file, but it is empty.

Interestingly, the following result gives no output (the same as above, but sed pipelined output via cat):

 wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g' | cat 

Why can't I pass sed output to another program like cat?

Answer 1

When sed writes to another process or to a file, it will buffer the data.

Try adding the --unbuffered options to sed.

Answer 2

you can also use awk. since your url appears in box 3, you can use $ 3 and you can also remove grep.

 awk '!/ERROR/{sub("URL:","",$3);print $3}' file