Linux shell script to delete old files from ftp

Question

There was a problem - you need to back up the database to FTP. FTP should not have more than 10 backups, i.e. After you have added the backup to FTP, you must delete the oldest files so that the total number of files does not exceed 10 pieces. How can we implement such removal from ftp? I am trying to write a script but delete does not work:

x=1 ftp -vn $FTP_SERVER<<! user $FTP_LOGIN $FTP_PASSWORD binary put $DUMP_FILE_NAME for i in `ls -t` do if [ $x -le $keep ] then ((x++)) continue fi delete $i done bye EOF </i> 

Answer 1

This is a script that I wrote to delete any files on a remote ftp site older than 7 days. It works by retrieving a list of directories, analyzing the changed date, and then reconnecting to delete files older than ndays.

I suspect that the numbers encoded in the loop (item date) may vary depending on your system setup. The returned formatting for the ls command depends on the local system settings.

Assuming you have your backups every day, setting ndays to 10 may solve your problem.

 #!/bin/bash # get a list of files and dates from ftp and remove files older than ndays ftpsite="ftp.yourserver.com" ftpuser="loginusername" ftppass="password" putdir="/public_ftp/admin/logs" ndays=7 # work out our cutoff date MM=`date --date="$ndays days ago" +%b` DD=`date --date="$ndays days ago" +%d` echo removing files older than $MM $DD # get directory listing from remote source listing=`ftp -i -n $ftpsite <<EOMYF user $ftpuser $ftppass binary cd $putdir ls quit EOMYF ` lista=( $listing ) # loop over our files for ((FNO=0; FNO<${#lista[@]}; FNO+=9));do # month (element 5), day (element 6) and filename (element 8) #echo Date ${lista[`expr $FNO+5`]} ${lista[`expr $FNO+6`]} File: ${lista[`expr $FNO+8`]} # check the date stamp if [ ${lista[`expr $FNO+5`]}=$MM ]; then if [[ ${lista[`expr $FNO+6`]} -lt $DD ]]; then # Remove this file echo "Removing ${lista[`expr $FNO+8`]}" ftp -i -n $ftpsite <<EOMYF2 user $ftpuser $ftppass binary cd $putdir delete ${lista[`expr $FNO+8`]} quit EOMYF2 fi fi done 

Answer 2

This question is related to sftp and has the correct date check since the script from @Graeme only works for one month:

 #!/bin/bash # get a list of files and dates from ftp and remove files older than ndays ftpsite="sftp -b- -oIdentityFile=<KEYFILE> -oPort=<PORT> <USER>@<HOST>" putdir="/data" ndays=19 # work out our cutoff date MM=`date --date="$ndays days ago" +%b` DD=`date --date="$ndays days ago" +%d` TT=`date --date="$ndays days ago" +%s` echo removing files older than $MM $DD # get directory listing from remote source echo " cd $putdir ls -l "|$ftpsite >dirlist # skip first three and last line, ftp command echo listing="`tail -n+4 dirlist|head -n-1`" lista=( $listing ) # loop over our files for ((FNO=0; FNO<${#lista[@]}; FNO+=9));do # month (element 5), day (element 6) and filename (element 8) # echo Date ${lista[`expr $FNO+5`]} ${lista[`expr $FNO+6`]} File: ${lista[`expr $FNO+8`]} fdate="${lista[`expr $FNO+5`]} ${lista[`expr $FNO+6`]} ${lista[`expr $FNO+7`]}" sdate=`date --date="$fdate" +%s` # check the date stamp if [ $sdate -lt $TT ] then # Remove this file echo "$MM $DD: Removing ${lista[`expr $FNO+5`]} / ${lista[`expr $FNO+6`]} / ${lista[`expr $FNO+8`]}" $ftpsite <<EOMYF2 cd $putdir delete ${lista[`expr $FNO+8`]} quit EOMYF2 fi done 

Answer 3

Here's a shorter solution using lftp , which will also work with subdirectories:

 #!/bin/bash # get a list of files and dates from ftp and remove files older than ndays ftpsite="ftpback-xxx.ovh.net" ftpuser="user" ftppass="pass" #remote folder in which you want to delete files putdir="/" nullfolder="/tmp/null" ndays=19 mkdir -p nullfolder MM=`date --date="$ndays days ago" +%b` DD=`date --date="$ndays days ago" +%d` echo removing files older than $MM $DD #The no-ssl is there because it on a local network and remote doesn't support ftp lftp -e "set ftp:ssl-allow no; mirror $putdir $nullfolder --older-than=now-${ndays}days --Remove-source-files; exit" -u $ftpuser,$ftppass $ftpsite rm $nullfolder/* -Rf 

It has two drawbacks:

• Download these old files first (before deleting them later), so it uses even more bandwidth
• These old files take up some space on the local disk before deletion. You can use nullfs to bind to this

Answer 4

Do not use FTP (for many reasons). However, if you do this in ssh, you get tty, the environment, and everything else. In addition, if you want to delete a file in 10 days (and not one of 10 calendar days), a simple find -mtime +10 -delete will replace this with a loop. So:

 ssh user@host 'find $DIR -mtime +10 -delete' 

If you want FTP, look at NcFTP, if I remember correctly, there were some pretty decent scripting capabilities.