Git issue - out of memory, malloc failed. Need to modify repo with filter-branch or similar

Question

I use a privately hosted git repo on a raspberry pi to store all my school work. I did this to help me familiarize myself with Git, and running a linux server in general. All was working fine, until last night I get an error that says the following:

remote: Counting objects: 2688, done.
remote: Compressing objects: 100% (1784/1784), done.
remote: fatal: Out of memory, malloc failed (tried to allocate 243315665 bytes)
error: git upload-pack: git-pack-objects died with error.rRemote: aborting due t
fatal: git upload-pack: aborting due to possible repository corruption on the re
mote side.ly EOF:  72% (1937/2688), 42.41 MiB | 293.00 KiB/s
fatal: index-pack failed

I'm assuming the actual problem here is that the server is simply running out of memory. I checked the size of my repo, and it was over 300mb. This is because as a new user, I didn't realize that uploading things such as Visual Studio, Eclipse and Netbeans temp user files was a Bad Thing. I know how to remove these files for current and future commits - but I've been having a very hard time trying to remove these from the repository completely. All the filter-branch methods Google has helped me dig up seem to be only for a couple files. I need to do a batch remove of many files, and I need it to not complain at me when it so much as can't find a single file in a given directory.

So my question is, is there a reasonable way to do what I am asking? Or would it be easier in my case to just lose my commit message history, and start a new repo with all .gitignore files safely in place from day 1?

Answer 1

I fixed the problem, but not using any of the methods I would have thought. Turns out I should have just trusted the error message, because my repository was actually corrupted. I manually copied the two commits I had made locally to another machine holding that repo, manually copied the commit messages out by hand and deleted the corrupted repo from my pi. I then re initialized a bare repo on the pi and pushed my local manually updated repo to the server which worked perfectly. I can now pull and push to it with no problems. There never was a memory issue, I checked the size of another repo I had on it and because there's a few HD videos in there, it was over 800MB in size, working perfectly.

Thank you to those who attempted to answer my question for taking the time. And to the creator of BFG - thanks for posting. I'll definitely be using your tool to clean out my repo now that I have it working again.

Answer 2

https://www.kernel.org/pub/software/scm/git/docs/git-filter-branch.html

The first example looks relevant for you.

Basically:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD

You can substitute a script for 'git rm --cached --ignore-unmatch filename' that will remove any and all the files you don't want. As long as it returns "success" for every outcome it should work fine.

I haven't tried it! Make a backup copy of your repo before you mess with it (whether this solution or any other).

Answer 3

I'm pretty sure you don't need filter-branch to complete the push, though it looks like a good idea for other reasons you already know. There are pack size and compression options available to keep pack-objects from going nuts like this. See the git config docs starting with pack.window, the ones you need are the various memory and size caps. Remember pack compression is multithreaded, each core will use a full compression window.

But for your filtering, the BFG Repo-Cleaner might be what you're after. I've never used it but it seems to be well-regarded.

Answer 4

Use The BFG to remove big files, not git-filter-branch...

The BFG gives a foolproof method of getting rid of large files, much easier than using git filter-branch, see http://rtyley.github.io/bfg-repo-cleaner/ :

$ bfg  --strip-blobs-bigger-than 100M  my-repo.git

Or if you want to target specific folders:

$ bfg  --delete-folders "{VisualStudio,Eclipse,NetBeans}"  my-repo.git

Full disclosure: I'm the author of the BFG Repo-Cleaner.