Using Perl memory with a map and file

Question

Does calling map { function($_) } <FILEHANDLE>; load whole file into memory when using perl?

Answer 1

Yes - or at least the way I interpret this result.

 $ perl -e "map {0} <>" big_data_file Out of memory! $ perl -e "map {0} 1 .. 1000000000" Out of memory! 

Can one ask if the memory is running out because Perl is trying to save the map output. However, I understand that Perl is optimized to avoid this work whenever map is called in the void context. For a specific example, see the discussion in this question .

Perhaps the best example is :

 $ perl -e "sub nothing {} map nothing(), <>" big_data_file Out of memory! 

Based on the comments, it seems that the question is motivated by the desire for compact syntax when processing big data.

 open(my $handle, '<', 'big_data_file') or die $!; # An ordinary while loop to process a data file. while (my $line = <$handle>){ foo($line); } # Here Perl assigns each line to $_. while (<$handle>){ foo($_); } # And here we do the same thing on one line. foo($_) while <$handle>; 

Answer 2

Yes, the operands for map , the foreach loop and subselects are computed before map , the foreach loop or subhead even starts.

One exception:

 for my $i (EXPR_X..EXPR_Y) 

(with or without my $i ) is optimized in the counting loop, something along the lines

 my $x = EXPR_X; my $y = EXPR_Y; for (my $i = $x; $i <= $y; ++$i) 

Perl6 will have built-in support for lazy lists.

Answer 3

The question you ask is, I assume the following: does the map function support the file before it starts to process, or uses line by line.

Allows quick comparison of processing lists:

 while (<FILEHANDLE>) { ... } 

This case obviously works line by line. Each iteration, a new value for $_ is selected.

 for my $line (<FILEHANDLE>) { ... } 

In this case, the LIST expands before the start of the loop. In http://perldoc.perl.org/functions/map.html the map link is like a foreach , and I believe that LISTs extended before passing the function.