Sort results from find and grab subset (bash)

StackOverflow https://stackoverflow.com/questions/2300387

  •  21-09-2019
  •  | 
  •  

Question

I have a pair of bash scripts, 1 that dumps mysql dbs, and the second to purge old backups.

I have always relied on date, so any files older than 7 days are purged, and new backups created daily. The result was a set of backups 7 days back.

Well now that I have 20+ dbs I have separated that daily job into a daily, and weekly job. If I continue to use the find and date method I would lose the past week.

QUESTION

How can I sort the output of a find by date, and then purge all but the most recent 7. (so even if the most recent is 2 months old, it remains)

This is as far as I got, but it is not as expected. Perhaps using ls might be the better choice 9find is a remnant from using date as the sole criteria for purging)

OLDFILES=`find -regex .*sql.*`

set -- $OLDFILES

if [ -z $1 ]
then
    printf "\tNo files matching purge criteria\n" | tee -a $LOGFILE
else
    printf "\tSQL Files being Delete from $HERE\n" | tee -a $LOGFILE
    printf "\t\t%s\n" $OLDFILES  | tee -a $LOGFILE
fi
# $RETAIN is the # of recent to keep
# so I want to grab a subset from 0 to (total - retain), and purge them
TOBURN=$(( $# - $RETAIN ))
printf " grab first %s files. %s  - %s" $TOBURN $# $RETAIN
if [ $TOBURN -gt 0 ]
then
    TOPURGE=("$@:0:$TOBURN")
    printf "\n\nREMOVING..\n\t\t%s\n" $TOPURGE
fi

Existing files

            ./webbmaster_bellarose_joomla_01-09-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-25-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-10-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-04-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-25-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-29-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-30-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-28-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-06-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-13-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-24-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-21-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-24-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-27-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-26-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-17-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-03-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-21-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-20-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-16-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-31-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-11-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-05-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-14-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-22-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-12-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-17-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-18-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-20-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-08-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-22-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-27-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-23-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-07-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-26-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-19-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-29-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-15-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-28-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-01-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-18-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-02-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-23-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-19-2009.sql.gz

Targets for purging (which I want to set in variable for simple rm -f $var)

            ./webbmaster_bellarose_joomla_01-09-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-25-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-10-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-04-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-30-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-28-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-06-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-13-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-21-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-24-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-27-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-17-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-03-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-21-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-20-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-16-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-31-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-11-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-05-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-14-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-22-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-12-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-17-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-18-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-20-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-08-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-22-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-23-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-07-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-26-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-19-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-29-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-15-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-01-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-18-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-02-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-19-2009.sql.gz

(dates 1/23 - 1/29 are left intact)

Was it helpful?

Solution

Please do yourself (and everyone else) a big favour and start writing dates in ISO-8601 format (i.e. YYYY-MM-DD).

E.g. instead of

./webbmaster_bellarose_joomla_12-29-2009.sql.gz

you would have

./webbmaster_bellarose_joomla_2009-12-29.sql.gz

Doing so have many benefits, most importantly here that chronological and alphabetical order becomes identical. No need for find, just list the files, reverse order, delete the first 7 lines and delete any remaining files:

ls *sql* | tac | sed 1,7d | tr '\012' '\000' | xargs -0 --no-run-if-empty rm

Update: What do you mean by "moving forward"? As fas as I can tell my solution solves your problem 100%:

/tmp/so>cat existing_files
./webbmaster_bellarose_joomla-2009-12-17.sql.gz
./webbmaster_bellarose_joomla-2009-12-18.sql.gz
./webbmaster_bellarose_joomla-2009-12-19.sql.gz
./webbmaster_bellarose_joomla-2009-12-20.sql.gz
./webbmaster_bellarose_joomla-2009-12-21.sql.gz
./webbmaster_bellarose_joomla-2009-12-22.sql.gz
./webbmaster_bellarose_joomla-2009-12-23.sql.gz
./webbmaster_bellarose_joomla-2009-12-24.sql.gz
./webbmaster_bellarose_joomla-2009-12-25.sql.gz
./webbmaster_bellarose_joomla-2009-12-26.sql.gz
./webbmaster_bellarose_joomla-2009-12-27.sql.gz
./webbmaster_bellarose_joomla-2009-12-28.sql.gz
./webbmaster_bellarose_joomla-2009-12-29.sql.gz
./webbmaster_bellarose_joomla-2009-12-30.sql.gz
./webbmaster_bellarose_joomla-2009-12-31.sql.gz
./webbmaster_bellarose_joomla-2010-01-01.sql.gz
./webbmaster_bellarose_joomla-2010-01-02.sql.gz
./webbmaster_bellarose_joomla-2010-01-03.sql.gz
./webbmaster_bellarose_joomla-2010-01-04.sql.gz
./webbmaster_bellarose_joomla-2010-01-05.sql.gz
./webbmaster_bellarose_joomla-2010-01-06.sql.gz
./webbmaster_bellarose_joomla-2010-01-07.sql.gz
./webbmaster_bellarose_joomla-2010-01-08.sql.gz
./webbmaster_bellarose_joomla-2010-01-09.sql.gz
./webbmaster_bellarose_joomla-2010-01-10.sql.gz
./webbmaster_bellarose_joomla-2010-01-11.sql.gz
./webbmaster_bellarose_joomla-2010-01-12.sql.gz
./webbmaster_bellarose_joomla-2010-01-13.sql.gz
./webbmaster_bellarose_joomla-2010-01-14.sql.gz
./webbmaster_bellarose_joomla-2010-01-15.sql.gz
./webbmaster_bellarose_joomla-2010-01-16.sql.gz
./webbmaster_bellarose_joomla-2010-01-17.sql.gz
./webbmaster_bellarose_joomla-2010-01-18.sql.gz
./webbmaster_bellarose_joomla-2010-01-19.sql.gz
./webbmaster_bellarose_joomla-2010-01-20.sql.gz
./webbmaster_bellarose_joomla-2010-01-21.sql.gz
./webbmaster_bellarose_joomla-2010-01-22.sql.gz
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so>cat targets_for_purging
./webbmaster_bellarose_joomla-2009-12-17.sql.gz
./webbmaster_bellarose_joomla-2009-12-18.sql.gz
./webbmaster_bellarose_joomla-2009-12-19.sql.gz
./webbmaster_bellarose_joomla-2009-12-20.sql.gz
./webbmaster_bellarose_joomla-2009-12-21.sql.gz
./webbmaster_bellarose_joomla-2009-12-22.sql.gz
./webbmaster_bellarose_joomla-2009-12-23.sql.gz
./webbmaster_bellarose_joomla-2009-12-24.sql.gz
./webbmaster_bellarose_joomla-2009-12-25.sql.gz
./webbmaster_bellarose_joomla-2009-12-26.sql.gz
./webbmaster_bellarose_joomla-2009-12-27.sql.gz
./webbmaster_bellarose_joomla-2009-12-28.sql.gz
./webbmaster_bellarose_joomla-2009-12-29.sql.gz
./webbmaster_bellarose_joomla-2009-12-30.sql.gz
./webbmaster_bellarose_joomla-2009-12-31.sql.gz
./webbmaster_bellarose_joomla-2010-01-01.sql.gz
./webbmaster_bellarose_joomla-2010-01-02.sql.gz
./webbmaster_bellarose_joomla-2010-01-03.sql.gz
./webbmaster_bellarose_joomla-2010-01-04.sql.gz
./webbmaster_bellarose_joomla-2010-01-05.sql.gz
./webbmaster_bellarose_joomla-2010-01-06.sql.gz
./webbmaster_bellarose_joomla-2010-01-07.sql.gz
./webbmaster_bellarose_joomla-2010-01-08.sql.gz
./webbmaster_bellarose_joomla-2010-01-09.sql.gz
./webbmaster_bellarose_joomla-2010-01-10.sql.gz
./webbmaster_bellarose_joomla-2010-01-11.sql.gz
./webbmaster_bellarose_joomla-2010-01-12.sql.gz
./webbmaster_bellarose_joomla-2010-01-13.sql.gz
./webbmaster_bellarose_joomla-2010-01-14.sql.gz
./webbmaster_bellarose_joomla-2010-01-15.sql.gz
./webbmaster_bellarose_joomla-2010-01-16.sql.gz
./webbmaster_bellarose_joomla-2010-01-17.sql.gz
./webbmaster_bellarose_joomla-2010-01-18.sql.gz
./webbmaster_bellarose_joomla-2010-01-19.sql.gz
./webbmaster_bellarose_joomla-2010-01-20.sql.gz
./webbmaster_bellarose_joomla-2010-01-21.sql.gz
./webbmaster_bellarose_joomla-2010-01-22.sql.gz
/tmp/so>sed 's/\(.\{29\}\)_\(..-..\)-\(....\)/\1-\3-\2/' existing_files | sort > existing_files.8601
/tmp/so>sed 's/\(.\{29\}\)_\(..-..\)-\(....\)/\1-\3-\2/' targets_for_purging | sort > targets_for_purging.8601
/tmp/so>mkdir files
/tmp/so>cd files
/tmp/so/files>xargs touch < ../existing_files.8601
/tmp/so/files>ls -1
webbmaster_bellarose_joomla-2009-12-17.sql.gz
webbmaster_bellarose_joomla-2009-12-18.sql.gz
webbmaster_bellarose_joomla-2009-12-19.sql.gz
webbmaster_bellarose_joomla-2009-12-20.sql.gz
webbmaster_bellarose_joomla-2009-12-21.sql.gz
webbmaster_bellarose_joomla-2009-12-22.sql.gz
webbmaster_bellarose_joomla-2009-12-23.sql.gz
webbmaster_bellarose_joomla-2009-12-24.sql.gz
webbmaster_bellarose_joomla-2009-12-25.sql.gz
webbmaster_bellarose_joomla-2009-12-26.sql.gz
webbmaster_bellarose_joomla-2009-12-27.sql.gz
webbmaster_bellarose_joomla-2009-12-28.sql.gz
webbmaster_bellarose_joomla-2009-12-29.sql.gz
webbmaster_bellarose_joomla-2009-12-30.sql.gz
webbmaster_bellarose_joomla-2009-12-31.sql.gz
webbmaster_bellarose_joomla-2010-01-01.sql.gz
webbmaster_bellarose_joomla-2010-01-02.sql.gz
webbmaster_bellarose_joomla-2010-01-03.sql.gz
webbmaster_bellarose_joomla-2010-01-04.sql.gz
webbmaster_bellarose_joomla-2010-01-05.sql.gz
webbmaster_bellarose_joomla-2010-01-06.sql.gz
webbmaster_bellarose_joomla-2010-01-07.sql.gz
webbmaster_bellarose_joomla-2010-01-08.sql.gz
webbmaster_bellarose_joomla-2010-01-09.sql.gz
webbmaster_bellarose_joomla-2010-01-10.sql.gz
webbmaster_bellarose_joomla-2010-01-11.sql.gz
webbmaster_bellarose_joomla-2010-01-12.sql.gz
webbmaster_bellarose_joomla-2010-01-13.sql.gz
webbmaster_bellarose_joomla-2010-01-14.sql.gz
webbmaster_bellarose_joomla-2010-01-15.sql.gz
webbmaster_bellarose_joomla-2010-01-16.sql.gz
webbmaster_bellarose_joomla-2010-01-17.sql.gz
webbmaster_bellarose_joomla-2010-01-18.sql.gz
webbmaster_bellarose_joomla-2010-01-19.sql.gz
webbmaster_bellarose_joomla-2010-01-20.sql.gz
webbmaster_bellarose_joomla-2010-01-21.sql.gz
webbmaster_bellarose_joomla-2010-01-22.sql.gz
webbmaster_bellarose_joomla-2010-01-23.sql.gz
webbmaster_bellarose_joomla-2010-01-24.sql.gz
webbmaster_bellarose_joomla-2010-01-25.sql.gz
webbmaster_bellarose_joomla-2010-01-26.sql.gz
webbmaster_bellarose_joomla-2010-01-27.sql.gz
webbmaster_bellarose_joomla-2010-01-28.sql.gz
webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>ls *sql* | tac | sed 1,7d | tr '\012' '\000' | xargs -0 --no-run-if-empty rm
/tmp/so/files>ls -1 ./*
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>comm -3 ../existing_files.8601 ../targets_for_purging.8601
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>

OTHER TIPS

If the files are all in one directory (i.e. you do not need to use find to recurse though subdirectories), you can use the -t argument to ls(1) to order the file list by time and then remove the first 7 entries with tail(1).

rm -f $(ls -t *sql* | tail -n +8)

[Edit: I added -f in case the substitution results in an empty list]

or in a loop...

ls -t *sql* | tail -n +8 | while read file
    echo "Purging: $file"
    rm "$file"
done

The argument to tail is +8 because you want it to start tailing from the 8th line onwards (i.e. drop the first 7).

After removing the files, you can just burn what's left.

OLDIFS="$IFS"
IFS=$'\n'
topurge=($(find -mindepth 1 -printf "%T@ %P\n" | sort -n -r | cut -d' ' -f 2- | tail -n +7))
IFS="$OLDIFS"

Here is one thought:

oldest_to_keep=`find . -name \*.sql | xargs \ls -1rc | tail -7 | head -1`
find . -name \*.sql -not -samefile $oldest_to_keep -not -newer $oldest_to_keep

Where the second find ought to have -exec rm \{\} ; if you've convinced yourself this is sane. :)

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top