fatal: early EOF fatal: index-pack failed

StackOverflow https://stackoverflow.com/questions/21277806

  •  01-10-2022
  •  | 
  •  

Question

I have googled and found many solutions but none work for me.

I am trying to clone from one machine by connecting to the remote server which is in the LAN network.
Running this command from another machine cause error.
But running the SAME clone command using git://192.168.8.5 ... at the server it's okay and successful.

Any ideas ?

user@USER ~
$ git clone  -v git://192.168.8.5/butterfly025.git
Cloning into 'butterfly025'...
remote: Counting objects: 4846, done.
remote: Compressing objects: 100% (3256/3256), done.
fatal: read error: Invalid argument, 255.05 MiB | 1.35 MiB/s
fatal: early EOF
fatal: index-pack failed

I have added this config in .gitconfig but no help also.
Using the git version 1.8.5.2.msysgit.0

[core]
    compression = -1
Was it helpful?

Solution

First, turn off compression: 

git config --global core.compression 0

Next, let's do a partial clone to truncate the amount of info coming down: 

git clone --depth 1 <repo_URI>

When that works, go into the new directory and retrieve the rest of the clone: 

git fetch --unshallow

or, alternately, 

git fetch --depth=2147483647

Now, do a regular pull: 

git pull --all

I think there is a glitch with msysgit in the 1.8.x versions that exacerbates these symptoms, so another option is to try with an earlier version of git (<= 1.8.3, I think).

OTHER TIPS

This error may occur for memory needs of git. You can add these lines to your global git configuration file, which is .gitconfig in $USER_HOME, in order to fix that problem.

[core] 
packedGitLimit = 512m 
packedGitWindowSize = 512m 
[pack] 
deltaCacheSize = 2047m 
packSizeLimit = 2047m 
windowMemory = 2047m

finally solved by git config --global core.compression 9

From a BitBucket issue thread:

I tried almost five times, and it still happen.

Then I tried to use better compression and it worked!

git config --global core.compression 9

From the Git Documentation:

core.compression
An integer -1..9, indicating a default compression level. -1 is the zlib default.
0 means no compression, and 1..9 are various speed/size tradeoffs, 9 being slowest.
If set, this provides a default to other compression variables, such as core.looseCompression and pack.compression.

As @ingyhere said:

Shallow Clone

First, turn off compression:

git config --global core.compression 0

Next, let's do a partial clone to truncate the amount of info coming down:

git clone --depth 1 <repo_URI>

When that works, go into the new directory and retrieve the rest of the clone:

git fetch --unshallow

or, alternately,

git fetch --depth=2147483647

Now, do a pull:

git pull --all

Then to solve the problem of your local branch only tracking master

open your git config file (.git/config) in the editor of your choice

where it says:

[remote "origin"]
    url=<git repo url>
    fetch = +refs/heads/master:refs/remotes/origin/master

change the line 

fetch = +refs/heads/master:refs/remotes/origin/master

to

fetch = +refs/heads/*:refs/remotes/origin/*

Do a git fetch and git will pull all your remote branches now

In my case this was quite helpful:

git clone --depth 1 --branch $BRANCH $URL

This will limit the checkout to mentioned branch only, hence will speed up the process.

Hope this will help.

I tried all of that commands and none works for me, but what works was change the git_url to http instead ssh

if is clone command do :

git clone <your_http_or_https_repo_url>

else if you are pulling on existing repo, do it with

git remote set-url origin <your_http_or_https_repo_url>

hope this help someone!

I was getting the same error, on my side i resolved by running this command, In windows it has some memory issue.

git config --global pack.windowsMemory 256m

I faced this problem with macOS Big Sur M1 Chip and none of the solutions worked for me.

I solved it by increasing ulimits below.

ulimit -f 2097152
ulimit -c 2097152
ulimit -n 2097152

Running the commands above, will be valid for only current terminal session, so first run this and then clone the repository.

I got this error when git ran out of memory.

Freeing up some memory (in this case: letting a compile job finish) and trying again worked for me.

In my case it was a connection problem. I was connected to an internal wifi network, in which I had limited access to ressources. That was letting git do the fetch but at a certain time it crashed. This means it can be a network-connection problem. Check if everything is running properly: Antivirus, Firewall, etc.

The answer of elin3t is therefore important because ssh improves the performance of the downloading so that network problems can be avoided

Setting below's config doesn't work for me.

[core] 
packedGitLimit = 512m 
packedGitWindowSize = 512m 
[pack] 
deltaCacheSize = 2047m 
packSizeLimit = 2047m 
windowMemory = 2047m

As previous comment, it might the memory issue from git. Thus, I try to reduce working threads(from 32 to 8). So that it won't get much data from server at the same time. Then I also add "-f " to force to sync other projects.

-f: Proceed with syncing other projects even if a project fails to sync.

Then it works fine now.

repo sync -f -j8

A previous answer recommends setting to 512m. I'd say there are reasons to think that's counterproductive on a 64bit architecture. The documentation for core.packedGitLimit says:

Default is 256 MiB on 32 bit platforms and 32 TiB (effectively unlimited) on 64 bit platforms. This should be reasonable for all users/operating systems, except on the largest projects. You probably do not need to adjust this value.

If you want to try it out check if you have it set and then remove the setting:

git config --show-origin core.packedGitLimit
git config --unset --global core.packedGitLimit

In my case nothing worked when the protocol was https, then I switched to ssh, and ensured, I pulled the repo from last commit and not entire history, and also specific branch. This helped me:

git clone --depth 1 "ssh:.git" --branch “specific_branch”

Note that Git 2.13.x/2.14 (Q3 2017) does raise the default core.packedGitLimit which influences git fetch:
The default packed-git limit value has been raised on larger platforms (from 8 GiB to 32 GiB) to save "git fetch" from a (recoverable) failure while "gc" is running in parallel.

See commit be4ca29 (20 Apr 2017) by David Turner (csusbdt).
Helped-by: Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit d97141b, 16 May 2017)

Increase core.packedGitLimit

When core.packedGitLimit is exceeded, git will close packs.
If there is a repack operation going on in parallel with a fetch, the fetch might open a pack, and then be forced to close it due to packedGitLimit being hit.
The repack could then delete the pack out from under the fetch, causing the fetch to fail.

Increase core.packedGitLimit's default value to prevent this.

On current 64-bit x86_64 machines, 48 bits of address space are available.
It appears that 64-bit ARM machines have no standard amount of address space (that is, it varies by manufacturer), and IA64 and POWER machines have the full 64 bits.
So 48 bits is the only limit that we can reasonably care about. We reserve a few bits of the 48-bit address space for the kernel's use (this is not strictly necessary, but it's better to be safe), and use up to the remaining 45.
No git repository will be anywhere near this large any time soon, so this should prevent the failure.

It's confusing because Git logs may suggest any connection or ssh authorization errors, eg: ssh_dispatch_run_fatal: Connection to x.x.x.x port yy: message authentication code incorrect, the remote end hung up unexpectedly, early EOF.

Server-side solution

Let's optimize git repository on the server side:

  1. Enter to my server's git bare repository.
  2. Call git gc.
  3. Call git repack -A

Eg:

ssh admin@my_server_url.com
sudo su git
cd /home/git/my_repo_name # where my server's bare repository exists.
git gc
git repack -A

Now I am able clone this repository without errors, e.g. on the client side:

git clone git@my_server_url.com:my_repo_name

The command git gc may be called at the git client side to avoid similar git push problem.

If you are an administrator of Gitlab service - trigger Housekeeping manually. It calls internally git gc or git repack.

Client-side solution

Other (hack, client-side only) solution is downloading last master without history:

git clone --single-branch --depth=1 git@my_server_url.com:my_repo_name

There is a chance that buffer overflow will not occur.

Tried almost all the answers here but no luck.. Finally got it worked by using the Github desktop app, https://desktop.github.com/

Macbook with M1 chip/Monterey not sure if it mattered.

I have the same problem. Following the first step above i was able to clone, but I cannot do anything else. Can't fetch, pull or checkout old branches.

Each command runs much slower than usual, then dies after compressing the objects.

I:\dev [master +0 ~6 -0]> git fetch --unshallow
remote: Counting objects: 645483, done.
remote: Compressing objects: 100% (136865/136865), done.

error: RPC failed; result=18, HTTP code = 20082 MiB | 6.26 MiB/s

fatal: early EOF

fatal: The remote end hung up unexpectedly

fatal: index-pack failed

This also happens when your ref's are using too much memory. Pruning the memory fixed this for me. Just add a limit to what you fetching like so -> 

git fetch --depth=100

This will fetch the files but with the last 100 edits in their histories. After this, you can do any command just fine and at normal speed.

Tried most of the answers here, I got the error with the PUTTY SSH Client with all possible constellations.

Once I switched to OpenSSH the error was gone (remove the Environment Variable GIT_SSH and restart the git bash).

I was using a new machine and newest git versions. On many other/older machines (AWS as well) it did work as expected with PUTTY as well without any git configuration.

None of the solutions above worked for me.

The solution that finally worked for me was switching SSH client. GIT_SSH environment variable was set to the OpenSSH provided by Windows Server 2019. Version 7.7.2.1

C:\Windows\System32\OpenSSH\ssh.exe

I simply installed putty, 0.72

choco install putty

And changed GIT_SSH to

C:\ProgramData\chocolatey\lib\putty.portable\tools\PLINK.EXE

Using @cmpickle answer, I built a script to simplify the clone process.

It is hosted here: https://gist.github.com/gianlucaparadise/10286e0b1c5409bd1049d67640fb7c03

You can run it using the following line:

curl -sL https://git.io/JvtZ5 | sh -s repo_uri repo_folder

Tangentially related and only useful in case you have no root access and manually extract Git from an RPM (with rpm2cpio) or other package (.deb, ..) into a subfolder. Typical use case: you try to use a newer version of Git over the outdated one on a corporate server.

If git clone fails with fatal: index-pack failed without early EOF mention but instead a help message about usage: git index-pack, there is a version mismatch and you need to run git with the --exec-path parameter:

git --exec-path=path/to/subfoldered/git/usr/bin/git clone <repo>

In order to have this happen automatically, specify in your ~/.bashrc:

export GIT_EXEC_PATH=path/to/subfoldered/git/usr/libexec

I had the same problem, I even tried to download the project directly from the website as a zip file but the download got interrupted at the exact same percent.

This single line fixed my problem like a charm

git config --global core.compression 0

I know other answers have mentioned this but, no one here mentioned that this line alone can fix the problem.

Hope it helps.

In my case the problem was none of the git configuration parameters but the fact that my repository had one file exceeding the maximum file size allowed on my system. I was able to check it trying to download a large file and getting an "File Size Limit Exceeded" on Debian.

After that I edited my /etc/security/limits.conf file adding et the end of it the following lines:

  • hard fsize 1000000
  • soft fsize 1000000

To actually "apply" the new limit values you need to re-login

I have tried for several times after I set git buffer, as I mentioned in the question, it seems work now.

So if you met this error, run this command:

git config --global http.postBuffer 2M

and then try again for some times.

Reference:

git push error: RPC failed; result=56, HTTP code = 0

Network quality matters, try to switch to a different network. What helped me was changing my Internet connection from Virgin Media high speed land-based broadband to a hotspot on my phone.

Before that I tried the accepted answer to limit clone size, tried switching between 64 and 32 bit versions, tried disabling the git file cache, none of them helped.

Then I switched to the connection via my mobile, and the first step (git clone --depth 1 <repo_URI>) succeeded. Switched back to my broadband, but the next step (git fetch --unshallow) also failed. So I deleted the code cloned so far, switched to the mobile network tried again the default way (git clone <repo_URI>) and it succeeded without any issues.

For me it worked when I changed the compression to git config --global core.compression 9

This works

From a git clone, I was getting:

error: inflate: data stream error (unknown compression method)
fatal: serious inflate inconsistency
fatal: index-pack failed

After rebooting my machine, I was able to clone the repo fine.

I turned off all the downloads I was doing in the meantime, which freed some space probably and cleared up/down bandwidth

The git-daemon issue seems to have been resolved in v2.17.0 (verified with a non working v2.16.2.1). I.e. workaround of selecting text in console to "lock output buffer" should no longer be required.

From https://github.com/git/git/blob/v2.17.0/Documentation/RelNotes/2.17.0.txt:

  • Assorted fixes to "git daemon". (merge ed15e58efe jk/daemon-fixes later to maint).

I've experience the same problem. The REPO was too big to be downloaded via SSH. Just like @elin3t recommended, I've cloned over HTTP/HTTPS and change the REMOTE URL in .git/config to use the SSH REPO.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top