Bash arbitrary glob pattern (with spaces) for loop

Question

Is there a way to reliably use an arbitrary globe template that is stored in a variable? I am having difficulty if the template contains both spaces and metacharacters. Here is what I mean. If I have a template stored in a variable without spaces, everything looks fine:

<prompt> touch aa.{1,2,3} "ab".{1,2,3} <prompt> p="aa.?" <prompt> for f in ${p} ; do echo "|$f|" ; done |aa.1| |aa.2| |aa.3| <prompt> declare -a A=($p) ; for f in "${A[@]}" ; do echo "|$f|" ; done |aa.1| |aa.2| |aa.3| 

However, as soon as I throw a space in the template, everything becomes untenable:

 <prompt> p="a b.?" <prompt> for f in ${p} ; do echo "|$f|" ; done |a| |b.?| <prompt> declare -a A=($p) ; for f in "${A[@]}" ; do echo "|$f|" ; done |a| |b.?| <prompt> for f in "${p}" ; do echo "|$f|" ; done |a b.?| <prompt> for f in $(printf "%q" "$p") ; do echo "|$f|" ; done |a\| |b.\?| 

Obviously, if I know the template in advance, I can save it manually:

 <prompt> for f in a\ b.* ; do echo "|$f|" ; done |a b.1| |a b.2| |a b.3| 

The problem is that I am writing a script where I do not know the template in advance. Is there a way to reliably make bash treat the contents of a variable as a global pattern without resorting to some kind of eval trick?

Answer 1

You need to disable word splitting. Recall that this does not work:

 $ p="a b.?" $ for f in ${p} ; do echo "|$f|" ; done |a| |b.?| 

This, however, does:

 $ ( IFS=; for f in ${p} ; do echo "|$f|" ; done ) |a b.1| |a b.2| |a b.3| 

IFS is an "Internal Field Separator" wrapper. It is usually set to a space, a tab, and a new line character. It is used to break words after expanding a variable. Setting IFS to start stopping word splitting and thus allows you to work with glob.

Array Example

The same applies to array examples:

 $ declare -a A=($p) ; for f in "${A[@]}" ; do echo "|$f|" ; done |a| |b.?| $ ( IFS=; declare -a A=($p) ; for f in "${A[@]}" ; do echo "|$f|" ; done ) |a b.1| |a b.2| |a b.3| 

Make sure that the IFS returns to its normal value

In the examples above, I put the IFS assignment inside a subshell. Although not necessary, the advantage of this is that IFS automatically reverts to its previous value as soon as the subshell ends. If subshells are not suitable for your application, here is another approach:

 $ oldIFS=$IFS; IFS=; for f in ${p} ; do echo "|$f|" ; done; IFS=$oldIFS |a b.1| |a b.2| |a b.3| 

Matching Shell Symbols

Suppose we have files that have a literal * in their names:

 $ touch ab.{1,2,3} 'a*b'.{1,2,3} $ ls a*b.1 ab.1 a*b.2 ab.2 a*b.3 ab.3 

And suppose we want to match this star. Since we want our heads to be addressed literally, we must avoid this:

 $ p='a\*b.?' $ ( IFS=; for f in ${p} ; do echo "|$f|" ; done ) |a*b.1| |a*b.2| |a*b.3| 

Since ? not escaped, it is considered as a wildcard. Since * escaped, it matches only a literal * .

Answer 2

Pattern used in

 p="a b.?" 

incorrect, which is clear if you use it directly:

 A=( a b.? ) 

As stated in the question, the correct template

 a\ b.? 

therefore the correct assignment of a variable

 p='a\ b.?' 

Where do the templates come from? Can they be fixed at the source? For instance, if the template is created by adding '.?' to the base, you can use 'printf' to do the required citation:

 base='ab' printf -vp '%q.?' "$base" 

'set' then shows:

 p='a\ b.?' 

Unfortunately, word splitting still fails if you try to execute

 A=( $p ) 

'set' shows:

 A=([0]="a\\" [1]="b.?") 

One way to solve the problem is to use "eval":

 eval "A=( $p )" 

'set' then shows:

 A=([0]="a b.1" [1]="a b.2" [2]="a b.3") 

This is a “hoax”, but it’s clearly no worse than the IFS hoax previously described. In addition, cheating IFS will not help if the files contain metacharacters in their names. What do you do for an instance if you have files created using

 touch ab.{1,2,3} 'a*b'.{1,2,3} 

and you want to combine those whose base is "a * b"? No amount of IFS trick will allow you to correctly match the variable p if it is set using

 p="a*b.?" 

After

 base='a*b' printf -vp '%q.?' "$base" 

'set' shows:

 p='a\*b.?' 

and after

 eval "A=( $p )" 

He shows:

 A=([0]="a*b.1" [1]="a*b.2" [2]="a*b.3") 

I consider using "eval" to be the last, but in this case I cannot think of a better option, and it is completely safe.