Removing unwanted files from history including all refs with filter-branch

Question

I recently have cloned an SVN repository which used to have a few binaries in it, which are not needed any longer. Unfortunately, I have already pushed it to Github with the binaries inlcuded. I now want to remove these using 'git filter-branch' but I am facing some problems when it comes to tags and branches.

Basically, I have created a simple shell script to remove a list of files which have been determined by the following command:

git rev-list --objects --all | grep .jar > files.txt

The script for removal looks like the following:

#!/bin/sh
while read file_hash file_to_remove
do
    echo "Removing "$file_to_remove;
    git filter-branch --index-filter "git rm --cached --ignore-unmatch $file_to_remove"
    rm -rf .git/refs/original/;
    git reflog expire --all --expire-unreachable=0;
    git repack -A -d;
    git prune
done < $1

I have a few tags (all listed in .git/packed-refs), one .git/refs/remotes/origin (pointing to the Github repo). The removal of the files using the above script does not have the wanted effect ('du -cm' remains to output the same size; 'git rev-list' still listing the files) until I manually remove all references from .git/packed-refs and the .git/refs/remotes/origin directory.

Naturally, I am losing all tags as well as the possibility to push my local changes back to Github with this approach. Is there anything I have missed or is there an alternative way for removing files from all branches/tags without destroying my history?

Many thanks in advance, Matthes

Answer 1

I ended up using the BFG Repo Cleaner on a bare cloned repository (git clone --mirror repo-url). It goes through every branch/tag, leaving each working and it is even much faster than filter-branch. Hope this helps other people having similar issues.

Here is my wrapper script:

#!/bin/bash
#usage: ./remove_files.sh file_list.txt bare-repo-dir
while read file_hash file_to_remove
do
    echo "Removing "$file_to_remove;
    lastFile=`echo $file_to_remove | awk -F/ '{print $NF}'`;
    java -jar bfg.jar --delete-files $lastFile $2;
done < $1

cd $2;
git gc --prune=now --aggressive;
cd ..;