Delete all but the most recent X files in bash

Question

Is there a simple way, in a pretty standard UNIX environment with bash, to run a command to delete all but the most recent X files from a directory?

To give a bit more of a concrete example, imagine some cron job writing out a file (say, a log file or a tar-ed up backup) to a directory every hour. I'd like a way to have another cron job running which would remove the oldest files in that directory until there are less than, say, 5.

And just to be clear, there's only one file present, it should never be deleted.

Answer 1

The problems with the existing answers:

• inability to handle filenames with embedded spaces or newlines.
  • in the case of solutions that invoke rm directly on an unquoted command substitution (rm `...`), there's an added risk of unintended globbing.
• inability to distinguish between files and directories (i.e., if directories happened to be among the 5 most recently modified filesystem items, you'd effectively retain fewer than 5 files, and applying rm to directories will fail).

wnoise's answer addresses these issues, but the solution is GNU-specific (and quite complex).

Here's a pragmatic, POSIX-compliant solution that comes with only one caveat: it cannot handle filenames with embedded newlines - but I don't consider that a real-world concern for most people.

^{For the record, here's the explanation for why it's generally not a good idea to parse ls output: http://mywiki.wooledge.org/ParsingLs}

ls -tp | grep -v '/$' | tail -n +6 | xargs -I {} rm -- {}

The above is inefficient, because xargs has to invoke rm once for each filename.
Your platform's xargs may allow you to solve this problem:

If you have GNU xargs, use -d '\n', which makes xargs consider each input line a separate argument, yet passes as many arguments as will fit on a command line at once:

ls -tp | grep -v '/$' | tail -n +6 | xargs -d '\n' -r rm --

^{-r (--no-run-if-empty) ensures that rm is not invoked if there's no input.}

If you have BSD xargs (including on OS X), you can use -0 to handle NUL-separated input, after first translating newlines to NUL (0x0) chars., which also passes (typically) all filenames at once (will also work with GNU xargs):

ls -tp | grep -v '/$' | tail -n +6 | tr '\n' '\0' | xargs -0 rm --

Explanation:

• ls -tp prints the names of filesystem items sorted by how recently they were modified , in descending order (most recently modified items first) (-t), with directories printed with a trailing / to mark them as such (-p).
• grep -v '/$' then weeds out directories from the resulting listing, by omitting (-v) lines that have a trailing / (/$).
  • Caveat: Since a symlink that points to a directory is technically not itself a directory, such symlinks will not be excluded.
• tail -n +6 skips the first 5 entries in the listing, in effect returning all but the 5 most recently modified files, if any.
  Note that in order to exclude N files, N+1 must be passed to tail -n +.
• xargs -I {} rm -- {} (and its variations) then invokes on rm on all these files; if there are no matches at all, xargs won't do anything.
  • xargs -I {} rm -- {} defines placeholder {} that represents each input line as a whole, so rm is then invoked once for each input line, but with filenames with embedded spaces handled correctly.
  • -- in all cases ensures that any filenames that happen to start with - aren't mistaken for options by rm.

---

A variation on the original problem, in case the matching files need to be processed individually or collected in a shell array:

# One by one, in a shell loop (POSIX-compliant):
ls -tp | grep -v '/$' | tail -n +6 | while IFS= read -r f; do echo "$f"; done

# One by one, but using a Bash process substitution (<(...), 
# so that the variables inside the `while` loop remain in scope:
while IFS= read -r f; do echo "$f"; done < <(ls -tp | grep -v '/$' | tail -n +6)

# Collecting the matches in a Bash *array*:
IFS=$'\n' read -d '' -ra files  < <(ls -tp | grep -v '/$' | tail -n +6)
printf '%s\n' "${files[@]}" # print array elements

Answer 2

Remove all but 5 (or whatever number) of the most recent files in a directory.

rm `ls -t | awk 'NR>5'`

Answer 3

(ls -t|head -n 5;ls)|sort|uniq -u|xargs rm

This version supports names with spaces:

(ls -t|head -n 5;ls)|sort|uniq -u|sed -e 's,.*,"&",g'|xargs rm

Answer 4

Simpler variant of thelsdj's answer:

ls -tr | head -n -5 | xargs --no-run-if-empty rm 

ls -tr displays all the files, oldest first (-t newest first, -r reverse).

head -n -5 displays all but the 5 last lines (ie the 5 newest files).

xargs rm calls rm for each selected file.

Answer 5

find . -maxdepth 1 -type f -printf '%T@ %p\0' | sort -r -z -n | awk 'BEGIN { RS="\0"; ORS="\0"; FS="" } NR > 5 { sub("^[0-9]*(.[0-9]*)? ", ""); print }' | xargs -0 rm -f

Requires GNU find for -printf, and GNU sort for -z, and GNU awk for "\0", and GNU xargs for -0, but handles files with embedded newlines or spaces.

Answer 6

All these answers fail when there are directories in the current directory. Here's something that works:

find . -maxdepth 1 -type f | xargs -x ls -t | awk 'NR>5' | xargs -L1 rm

This:

1. works when there are directories in the current directory

2. tries to remove each file even if the previous one couldn't be removed (due to permissions, etc.)

3. fails safe when the number of files in the current directory is excessive and xargs would normally screw you over (the -x)

4. doesn't cater for spaces in filenames (perhaps you're using the wrong OS?)

Answer 7

ls -tQ | tail -n+4 | xargs rm

List filenames by modification time, quoting each filename. Exclude first 3 (3 most recent). Remove remaining.

EDIT after helpful comment from mklement0 (thanks!): corrected -n+3 argument, and note this will not work as expected if filenames contain newlines and/or the directory contains subdirectories.

Answer 8

Ignoring newlines is ignoring security and good coding. wnoise had the only good answer. Here is a variation on his that puts the filenames in an array $x

while IFS= read -rd ''; do 
    x+=("${REPLY#* }"); 
done < <(find . -maxdepth 1 -printf '%T@ %p\0' | sort -r -z -n )

Answer 9

If the filenames don't have spaces, this will work:

ls -C1 -t| awk 'NR>5'|xargs rm

If the filenames do have spaces, something like

ls -C1 -t | awk 'NR>5' | sed -e "s/^/rm '/" -e "s/$/'/" | sh

Basic logic:

• get a listing of the files in time order, one column
• get all but the first 5 (n=5 for this example)
• first version: send those to rm
• second version: gen a script that will remove them properly

Answer 10

With zsh

Assuming you don't care about present directories and you will not have more than 999 files (choose a bigger number if you want, or create a while loop).

[ 6 -le `ls *(.)|wc -l` ] && rm *(.om[6,999])

In *(.om[6,999]), the . means files, the o means sort order up, the m means by date of modification (put a for access time or c for inode change), the [6,999] chooses a range of file, so doesn't rm the 5 first.

Answer 11

I realize this is an old thread, but maybe someone will benefit from this. This command will find files in the current directory :

for F in $(find . -maxdepth 1 -type f -name "*_srv_logs_*.tar.gz" -printf '%T@ %p\n' | sort -r -z -n | tail -n+5 | awk '{ print $2; }'); do rm $F; done

This is a little more robust than some of the previous answers as it allows to limit your search domain to files matching expressions. First, find files matching whatever conditions you want. Print those files with the timestamps next to them.

find . -maxdepth 1 -type f -name "*_srv_logs_*.tar.gz" -printf '%T@ %p\n'

Next, sort them by the timestamps:

sort -r -z -n

Then, knock off the 4 most recent files from the list:

tail -n+5

Grab the 2nd column (the filename, not the timestamp):

awk '{ print $2; }'

And then wrap that whole thing up into a for statement:

for F in $(); do rm $F; done

This may be a more verbose command, but I had much better luck being able to target conditional files and execute more complex commands against them.

Answer 12

found interesting cmd in Sed-Onliners - Delete last 3 lines - fnd it perfect for another way to skin the cat (okay not) but idea:

 #!/bin/bash
 # sed cmd chng #2 to value file wish to retain

 cd /opt/depot 

 ls -1 MyMintFiles*.zip > BigList
 sed -n -e :a -e '1,2!{P;N;D;};N;ba' BigList > DeList

 for i in `cat DeList` 
 do 
 echo "Deleted $i" 
 rm -f $i  
 #echo "File(s) gonzo " 
 #read junk 
 done 
 exit 0

Answer 13

Removes all but the 10 latest (most recents) files

ls -t1 | head -n $(echo $(ls -1 | wc -l) - 10 | bc) | xargs rm

If less than 10 files no file is removed and you will have : error head: illegal line count -- 0

To count files with bash

Answer 14

I needed an elegant solution for the busybox (router), all xargs or array solutions were useless to me - no such command available there. find and mtime is not the proper answer as we are talking about 10 items and not necessarily 10 days. Espo's answer was the shortest and cleanest and likely the most unversal one.

Error with spaces and when no files are to be deleted are both simply solved the standard way:

rm "$(ls -td *.tar | awk 'NR>7')" 2>&-

Bit more educational version: We can do it all if we use awk differently. Normally, I use this method to pass (return) variables from the awk to the sh. As we read all the time that can not be done, I beg to differ: here is the method.

Example for .tar files with no problem regarding the spaces in the filename. To test, replace "rm" with the "ls".

eval $(ls -td *.tar | awk 'NR>7 { print "rm \"" $0 "\""}')

Explanation:

ls -td *.tar lists all .tar files sorted by the time. To apply to all the files in the current folder, remove the "d *.tar" part

awk 'NR>7... skips the first 7 lines

print "rm \"" $0 "\"" constructs a line: rm "file name"

eval executes it

Since we are using rm, I would not use the above command in a script! Wiser usage is:

(cd /FolderToDeleteWithin && eval $(ls -td *.tar | awk 'NR>7 { print "rm \"" $0 "\""}'))

In the case of using ls -t command will not do any harm on such silly examples as: touch 'foo " bar' and touch 'hello * world'. Not that we ever create files with such names in real life!

Sidenote. If we wanted to pass a variable to the sh this way, we would simply modify the print (simple form, no spaces tolerated):

print "VarName="$1

to set the variable VarName to the value of $1. Multiple variables can be created in one go. This VarName becomes a normal sh variable and can be normally used in a script or shell afterwards. So, to create variables with awk and give them back to the shell:

eval $(ls -td *.tar | awk 'NR>7 { print "VarName=\""$1"\""  }'); echo "$VarName"

Answer 15

leaveCount=5
fileCount=$(ls -1 *.log | wc -l)
tailCount=$((fileCount - leaveCount))

# avoid negative tail argument
[[ $tailCount < 0 ]] && tailCount=0

ls -t *.log | tail -$tailCount | xargs rm -f

Answer 16

I made this into a bash shell script. Usage: keep NUM DIR where NUM is the number of files to keep and DIR is the directory to scrub.

#!/bin/bash
# Keep last N files by date.
# Usage: keep NUMBER DIRECTORY
echo ""
if [ $# -lt 2 ]; then
    echo "Usage: $0 NUMFILES DIR"
    echo "Keep last N newest files."
    exit 1
fi
if [ ! -e $2 ]; then
    echo "ERROR: directory '$1' does not exist"
    exit 1
fi
if [ ! -d $2 ]; then
    echo "ERROR: '$1' is not a directory"
    exit 1
fi
pushd $2 > /dev/null
ls -tp | grep -v '/' | tail -n +"$1" | xargs -I {} rm -- {}
popd > /dev/null
echo "Done. Kept $1 most recent files in $2."
ls $2|wc -l

Answer 17

Running on Debian (assume its the same on other distros I get: rm: cannot remove directory `..'

which is quite annoying..

Anyway I tweaked the above and also added grep to the command. In my case I have 6 backup files in a directory e.g. file1.tar file2.tar file3.tar etc and I want to delete only the oldest file (remove the first file in my case)

The script I ran to delete the oldest file was:

ls -C1 -t| grep file | awk 'NR>5'|xargs rm

This (as above) deletes the first of my files e.g. file1.tar this also leaves be with file2 file3 file4 file5 and file6