Why do these simple shell commands fail when used in the sed replacement part

Question

While trying to find the answer to this sed question , I came up with strange behavior that I could not understand.

Say I have a file called data

 $> cat data foo.png abCd.png bar.png baZ.png 

The challenge is to use sed in a string to replace all strings with uppercase ASCII letters. Thus, the output should be:

 $> cat data foo.png abCd.png bar.png baZ.png 

The solution should work with non-gnu sed as well as sed on Mac

I tried to include this built-in awk in the sed replacement part:

 sed -E 's/[^ ]*[AZ][^ ]*.png/'$(echo \&|awk '{printf("<%s>[%s]",$0, tolower($0))}')'/' data 

Oddly, this outputs this:

 foo.png <abCd.png>[abCd.png] bar.png <baZ.png>[baZ.png] 

As you can see, sed collects the correct strings with uppercase alphabets and also reaches awk, but the awk tolower() function does not work and creates the same text as the input.

Can the shell expert explain this strange behavior.

Answer 1

 sed 'y/ABCDEFGHIJKLMNOPQRSYUVWXYZ/abcdefghijklmnopqrstuvwxyz/' 

Answer 2

Your awk command runs before the sed command, and not as a subprocess of the sed command, so awk only gets the input ampersand as its input, as a result of which it outputs

 <&>[&] 

This line is then inserted into the line that sed receives as its argument, from which it should be fairly obvious why sed produces the output it draws.

Sequence of events

• The shell sees this command line

   sed -E 's/[^ ]*[AZ][^ ]*.png/'$(echo \&|awk '{printf("<%s>[%s]",$0, tolower($0))}')'/' data 

• It handles command substitution (in which awk turns & into <&>[&] ) to create an intermediate command line

   sed -E 's/[^ ]*[AZ][^ ]*.png/'<&>[&]'/' data 

• The shell then performs sed with the s/[^ ]*[AZ][^ ]*.png/<&>[&]/

Answer 3

Perhaps tr is what you are really looking for?

 tr AZ az file 

sed equivalent:

 sed -e 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' 

It doesn't seem like you can use character range notation ( AZ and / or [AZ] ), which is unsuccessful and annoying.

Answer 4

I am 99% sure that you cannot do this directly on Mac / BSD sed without much ugliness ( sed -es/A/a/g -es/B/b/g ... ), so until no sed solution will be found, here is an awk that does this inline:

 awk '{print tolower($0) >FILENAME}' data 

Answer 5

If you really need to lowercase to regular sed , this is possible, but rather ugly:

 sed -es/A/a/g -es/B/b/g -es/C/c/g -es/D/d/g -es/E/e/g -es/F/f/g -es/G/g/g -es/H/h/g -es/I/i/g -es/J/j/g -es/K/k/g -es/L/l/g -es/M/m/g -es/N/n/g -es/O/o/g -es/P/p/g -es/Q/q/g -es/R/r/g -es/S/s/g -es/T/t/g -es/U/u/g -es/V/v/g -es/W/w/g -es/X/x/g -es/Y/y/g -es/Z/z/g 

EDIT: Ignore @Bruce Barnett's solution is better

Answer 6

Are you really sure you cannot use Perl?

 perl -pi.bak -e 's/([^ ]*[AZ][^ ]*\.png)/\l\1/' file 

This backslash to indicate a lowercase, backslash to repeat the first comparable group.

Answer 7

Based on some good answers, we offer a solution :

 sed -i.bak 'y/'$(awk 'BEGIN {for(i=65; i<=90; i++) printf("%c", i); printf("/"); for(i=97; i<=122; i++) printf("%c", i)}')'/' data