What is the best strategy to delete a huge folder using Perl?

Question

I need to delete all contents (files and folders) in this folder. Problems in a folder contain millions of files and folders inside it. Therefore, I do not want to download all the file names at one time .

The logic should be like this:

• iterate a folder without loading everything
• get file or folder
• delete it (a detailed description of the fact that the file or folder "X" has been deleted)
• go to next

I am trying something like this:

sub main(){ my ($rc, $help, $debug, $root) = (); $rc = GetOptions ( "HELP" => \$help, "DEBUG" => \$debug, "ROOT=s" => \$root); die "Bad command line options\n$usage\n" unless ($rc); if ($help) { print $usage; exit (0); } if ($debug) { warn "\nProceeding to execution with following parameters: \n"; warn "===============================================================\n"; warn "ROOT = $root\n"; } # write debug information to STDERR print "\n Starting to delete...\n"; die "usage: $0 dir ..\n" unless $root; *name = *File::Find::name; find \&verbose, @ARGV; } sub verbose { if (!-l && -d _) { print "rmdir $name\n"; } else { print "unlink $name\n"; } } main(); 

It works fine, but whenever they "find" it reads a huge folder, the application gets stuck, and I see the system memory for Perl expanding to a timeout. What for? Is he trying to download all the files in one go?

Thanks for your help.

Answer 1

perlfaq indicates that File::Find does the hard work of moving the directory, but the work is not so difficult (provided that your directory tree does not have named pipes, device blocks, etc.):

 sub traverse_directory { my $dir = shift; opendir my $dh, $dir; while (my $file = readdir($dh)) { next if $file eq "." || $file eq ".."; if (-d "$dir/$file") { &traverse_directory("$dir/$file"); } elsif (-f "$dir/$file") { # $dir/$file is a regular file # Do something with it, for example: print "Removing $dir/$file\n"; unlink "$dir/$file" or warn "unlink $dir/$file failed: $!\n"; } else { warn "$dir/$file is not a directory or regular file. Ignoring ...\n"; } } closedir $dh; # $dir might be empty at this point. If you want to delete it: if (rmdir $dir) { print "Removed $dir/\n"; } else { warn "rmdir $dir failed: $!\n"; } } 

Substitute your own code to do something with the file or (possibly) an empty directory and call this function once in the root of the tree that you want to process. Find opendir/closedir , readdir , -d and -f if you haven’t seen them before.

Answer 2

The remove_tree function from File :: Path can portablely and verbosely remove the directory hierarchy, preserving the top directory if necessary.

 use strict; use warnings; use File::Path qw(remove_tree); my $dir = '/tmp/dir'; remove_tree($dir, {verbose => 1, keep_root => 1}); 

Pre-5.10, use the rmtree function from File :: Path . If you still need the top directory, you can just mkdir it again.

 use File::Path; my $dir = '/tmp/dir'; rmtree($dir, 1); # 1 means verbose mkdir $dir; 

Answer 3

What happened with:

 `rm -rf $folder`; // ?? 

Answer 4

You can use File::Find to systematically navigate the directory and delete files and directories under it.

Answer 5

OK, I gave up and used the built-in Perl, but you should use File :: Path :: rmtree , which I completely forgot about:

 #!/usr/bin/perl use strict; use warnings; use Cwd; use File::Find; my ($clean) = @ARGV; die "specify directory to clean\n" unless defined $clean; my $current_dir = getcwd; chdir $clean or die "Cannot chdir to '$clean': $!\n"; finddepth(\&wanted => '.'); chdir $current_dir or die "Cannot chdir back to '$current_dir':$!\n"; sub wanted { return if /^[.][.]?\z/; warn "$File::Find::name\n"; if ( -f ) { unlink or die "Cannot delete '$File::Find::name': $!\n"; } elsif ( -d _ ) { rmdir or die "Cannot remove directory '$File::Find::name': $!\n"; } return; } 

Answer 6

Download the unix tools for windows , and then you can do rm -rv or something else.

Perl is a great tool for many purposes, but it is best done with a specialized tool.

Answer 7

Here's a cheap cross-platform method:

 use Carp qw<carp croak>; use English qw<$OS_NAME>; use File::Spec; my %deltree_op = ( nix => 'rm -rf %s', win => 'rmdir /S %s' ); my %group_for = ( ( map { $_ => 'nix' } qw<linux UNIX SunOS> ) , ( map { $_ => 'win' } qw<MSWin32 WinNT> ) ); my $group_name = $group_for{$OS_NAME}; sub chop_tree { my $full_path = shift; carp( "No directory $full_path exists! We're done." ) unless -e $full_path; croak( "No implementation for $OS_NAME!" ) unless $group_name; my $format = $deltree_op{$group_name}; croak( "Could not find command format for group $group_name" ) unless $format; my $command = sprintf( $format, File::Spec->canonpath( $full_path )); qx{$command}; }