How do I clone a subdirectory only of a Git repository?

https://stackoverflow.com/questions/600079

03-07-2019
|

Question

I have my Git repository which, at the root, has two sub directories:

/finisht
/static

When this was in SVN, /finisht was checked out in one place, while /static was checked out elsewhere, like so:

svn co svn+ssh://admin@domain.com/home/admin/repos/finisht/static static

Is there a way to do this with Git?

Solution

EDIT: As of Git 2.19, this is finally possible, as can be seen in this answer: https://stackoverflow.com/a/52269934/2988.

Consider upvoting that answer.

Note: in Git 2.19, only client-side support is implemented, server-side support is still missing, so it only works when cloning local repositories. Also note that large Git hosters, e.g. GitHub, don't actually use the Git server, they use their own implementation, so even if support shows up in the Git server, it does not automatically mean that it works on Git hosters. (OTOH, since they don't use the Git server, they could implement it faster in their own implementations before it shows up in Git server.)

No, that's not possible in Git.

Implementing something like this in Git would be a substantial effort and it would mean that the integrity of the clientside repository could no longer be guaranteed. If you are interested, search for discussions on "sparse clone" and "sparse fetch" on the git mailinglist.

In general, the consensus in the Git community is that if you have several directories that are always checked out independently, then these are really two different projects and should live in two different repositories. You can glue them back together using Git Submodules.

OTHER TIPS

What you are trying to do is called a sparse checkout, and that feature was added in git 1.7.0 (Feb. 2012). The steps to do a sparse clone are as follows:

mkdir <repo>
cd <repo>
git init
git remote add -f origin <url>

This creates an empty repository with your remote, and fetches all objects but doesn't check them out. Then do:

git config core.sparseCheckout true

Now you need to define which files/folders you want to actually check out. This is done by listing them in .git/info/sparse-checkout, eg:

echo "some/dir/" >> .git/info/sparse-checkout
echo "another/sub/tree" >> .git/info/sparse-checkout

Last but not least, update your empty repo with the state from the remote:

git pull origin master

You will now have files "checked out" for some/dir and another/sub/tree on your file system (with those paths still), and no other paths present.

You might want to have a look at the extended tutorial and you should probably read the official documentation for sparse checkout.

As a function:

function git_sparse_clone() (
  rurl="$1" localdir="$2" && shift 2

  mkdir -p "$localdir"
  cd "$localdir"

  git init
  git remote add -f origin "$rurl"

  git config core.sparseCheckout true

  # Loops over remaining args
  for i; do
    echo "$i" >> .git/info/sparse-checkout
  done

  git pull origin master
)

Usage:

git_sparse_clone "http://github.com/tj/n" "./local/location" "/bin"

Note that this will still download the whole repository from the server – only the checkout is reduced in size. At the moment it is not possible to clone only a single directory. But if you don't need the history of the repository, you can at least save on bandwidth by creating a shallow clone. See udondan's answer below for information on how to combine shallow clone and sparse checkout.

You can combine the sparse checkout and the shallow clone features. The shallow clone cuts off the history and the sparse checkout only pulls the files matching your patterns.

git init <repo>
cd <repo>
git remote add origin <url>
git config core.sparsecheckout true
echo "finisht/*" >> .git/info/sparse-checkout
git pull --depth=1 origin master

You'll need minimum git 1.9 for this to work. Tested it myself only with 2.2.0 and 2.2.2.

This way you'll be still able to push, which is not possible with git archive.

git clone --filter from Git 2.19

This option will actually skip fetching unneeded objects from the server:

git clone --depth 1 --no-checkout --filter=blob:none \
  "file://$(pwd)/server_repo" local_repo
cd local_repo
git checkout master -- mydir/

The server should be configured with:

git config --local uploadpack.allowfilter 1
git config --local uploadpack.allowanysha1inwant 1

An extension was made to the Git remote protocol to support this feature in v2.19.0, but there is no server support at that time. But it can already be locally tested.

TODO: --filter=blob:none skips all blobs, but still fetches all tree objects. But on a normal repo, this should be tiny compared to the files themselves, so this is already good enough. Asked at: https://www.spinics.net/lists/git/msg342006.html Devs replied a --filter=tree:0 is in the works to do that. It was added in 2.20.

Remember that --depth 1 already implies --single-branch, see also: How do I clone a single branch in Git?

file://$(path) is required to overcome git clone protocol shenanigans: How to shallow clone a local git repository with a relative path?

The format of --filter is documented on man git-rev-list.

Docs on Git tree:

Test it out

#!/usr/bin/env bash
set -eu

list-objects() (
  git rev-list --all --objects
  echo "master commit SHA: $(git log -1 --format="%H")"
  echo "mybranch commit SHA: $(git log -1 --format="%H")"
  git ls-tree master
  git ls-tree mybranch | grep mybranch
  git ls-tree master~ | grep root
)

# Reproducibility.
export GIT_COMMITTER_NAME='a'
export GIT_COMMITTER_EMAIL='a'
export GIT_AUTHOR_NAME='a'
export GIT_AUTHOR_EMAIL='a'
export GIT_COMMITTER_DATE='2000-01-01T00:00:00+0000'
export GIT_AUTHOR_DATE='2000-01-01T00:00:00+0000'

rm -rf server_repo local_repo
mkdir server_repo
cd server_repo

# Create repo.
git init --quiet
git config --local uploadpack.allowfilter 1
git config --local uploadpack.allowanysha1inwant 1

# First commit.
# Directories present in all branches.
mkdir d1 d2
printf 'd1/a' > ./d1/a
printf 'd1/b' > ./d1/b
printf 'd2/a' > ./d2/a
printf 'd2/b' > ./d2/b
# Present only in root.
mkdir 'root'
printf 'root' > ./root/root
git add .
git commit -m 'root' --quiet

# Second commit only on master.
git rm --quiet -r ./root
mkdir 'master'
printf 'master' > ./master/master
git add .
git commit -m 'master commit' --quiet

# Second commit only on mybranch.
git checkout -b mybranch --quiet master~
git rm --quiet -r ./root
mkdir 'mybranch'
printf 'mybranch' > ./mybranch/mybranch
git add .
git commit -m 'mybranch commit' --quiet

echo "# List and identify all objects"
list-objects
echo

# Restore master.
git checkout --quiet master
cd ..

# Clone. Don't checkout for now, only .git/ dir.
git clone --depth 1 --quiet --no-checkout --filter=blob:none "file://$(pwd)/server_repo" local_repo
cd local_repo

# List missing objects from master.
echo "# Missing objects after --no-checkout"
git rev-list --all --quiet --objects --missing=print
echo

echo "# Git checkout fails without internet"
mv ../server_repo ../server_repo.off
! git checkout master
echo

echo "# Git checkout fetches the missing directory from internet"
mv ../server_repo.off ../server_repo
git checkout master -- d1/
echo

echo "# Missing objects after checking out d1"
git rev-list --all --quiet --objects --missing=print

GitHub upstream.

Output in Git v2.19.0:

# List and identify all objects
c6fcdfaf2b1462f809aecdad83a186eeec00f9c1
fc5e97944480982cfc180a6d6634699921ee63ec
7251a83be9a03161acde7b71a8fda9be19f47128
62d67bce3c672fe2b9065f372726a11e57bade7e
b64bf435a3e54c5208a1b70b7bcb0fc627463a75 d1
308150e8fddde043f3dbbb8573abb6af1df96e63 d1/a
f70a17f51b7b30fec48a32e4f19ac15e261fd1a4 d1/b
84de03c312dc741d0f2a66df7b2f168d823e122a d2
0975df9b39e23c15f63db194df7f45c76528bccb d2/a
41484c13520fcbb6e7243a26fdb1fc9405c08520 d2/b
7d5230379e4652f1b1da7ed1e78e0b8253e03ba3 master
8b25206ff90e9432f6f1a8600f87a7bd695a24af master/master
ef29f15c9a7c5417944cc09711b6a9ee51b01d89
19f7a4ca4a038aff89d803f017f76d2b66063043 mybranch
1b671b190e293aa091239b8b5e8c149411d00523 mybranch/mybranch
c3760bb1a0ece87cdbaf9a563c77a45e30a4e30e
a0234da53ec608b54813b4271fbf00ba5318b99f root
93ca1422a8da0a9effc465eccbcb17e23015542d root/root
master commit SHA: fc5e97944480982cfc180a6d6634699921ee63ec
mybranch commit SHA: fc5e97944480982cfc180a6d6634699921ee63ec
040000 tree b64bf435a3e54c5208a1b70b7bcb0fc627463a75    d1
040000 tree 84de03c312dc741d0f2a66df7b2f168d823e122a    d2
040000 tree 7d5230379e4652f1b1da7ed1e78e0b8253e03ba3    master
040000 tree 19f7a4ca4a038aff89d803f017f76d2b66063043    mybranch
040000 tree a0234da53ec608b54813b4271fbf00ba5318b99f    root

# Missing objects after --no-checkout
?f70a17f51b7b30fec48a32e4f19ac15e261fd1a4
?8b25206ff90e9432f6f1a8600f87a7bd695a24af
?41484c13520fcbb6e7243a26fdb1fc9405c08520
?0975df9b39e23c15f63db194df7f45c76528bccb
?308150e8fddde043f3dbbb8573abb6af1df96e63

# Git checkout fails without internet
fatal: '/home/ciro/bak/git/test-git-web-interface/other-test-repos/partial-clone.tmp/server_repo' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

# Git checkout fetches the missing directory from internet
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 45 bytes | 45.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 45 bytes | 45.00 KiB/s, done.

# Missing objects after checking out d1
?8b25206ff90e9432f6f1a8600f87a7bd695a24af
?41484c13520fcbb6e7243a26fdb1fc9405c08520
?0975df9b39e23c15f63db194df7f45c76528bccb

Conclusions: all blobs from outside of d1/ are missing. E.g. 0975df9b39e23c15f63db194df7f45c76528bccb, which is d2/b is not there after checking out d1/a.

Note that root/root and mybranch/mybranch are also missing, but --depth 1 hides that from the list of missing files. If you remove --depth 1, then they show on the list of missing files.

I have a dream

This feature could revolutionize Git.

Imagine having all the code base of your enterprise in a single repo without ugly third-party tools like repo.

Imagine storing huge blobs directly in the repo without any ugly third party extensions.

Imagine if GitHub would allow per file / directory metadata like stars and permissions, so you can store all your personal stuff under a single repo.

Imagine if submodules were treated exactly like regular directories: just request a tree SHA, and a DNS-like mechanism resolves your request, first looking on your local ~/.git, then first to closer servers (your enterprise's mirror / cache) and ending up on GitHub.

For other users who just want to download a file/folder from github, simply use:

svn export <repo>/trunk/<folder>

e.g.

svn export https://github.com/lodash/lodash.com/trunk/docs

(yes, that's svn here. apparently in 2016 you still need svn to simply download some github files)

Courtesy: Download a single folder or directory from a GitHub repo

Important - Make sure you update the github URL and replace /tree/master/ with '/trunk/'.

As bash script:

git-download(){
    folder=${@/tree\/master/trunk}
    folder=${folder/blob\/master/trunk}
    svn export $folder
}

Note This method downloads a folder, does not clone/checkout it. You can't push changes back to the repository. On the other hand - this results in smaller download compared to sparse checkout or shallow checkout.

If you never plan to interact with the repository from which you cloned, you can do a full git clone and rewrite your repository using git filter-branch --subdirectory-filter. This way, at least the history will be preserved.

Git 1.7.0 has “sparse checkouts”. See “core.sparseCheckout” in the git config manpage, “Sparse checkout” in the git read-tree manpage, and “Skip-worktree bit” in the git update-index manpage.

The interface is not as convenient as SVN’s (e.g. there is no way to make a sparse checkout at the time of an initial clone), but the base functionality upon which simpler interfaces could be built is now available.

This looks far simpler:

git archive --remote=<repo_url> <branch> <path> | tar xvf -

It's not possible to clone subdirectory only with Git, but below are few workarounds.

Filter branch

You may want to rewrite the repository to look as if trunk/public_html/ had been its project root, and discard all other history (using filter-branch), try on already checkout branch:

git filter-branch --subdirectory-filter trunk/public_html -- --all

^{Notes: The -- that separates filter-branch options from revision options, and the --all to rewrite all branches and tags. All information including original commit times or merge information will be preserved. This command honors .git/info/grafts file and refs in the refs/replace/ namespace, so if you have any grafts or replacement refs defined, running this command will make them permanent.}

Warning! The rewritten history will have different object names for all the objects and will not converge with the original branch. You will not be able to easily push and distribute the rewritten branch on top of the original branch. Please do not use this command if you do not know the full implications, and avoid using it anyway, if a simple single commit would suffice to fix your problem.

Sparse checkout

Here are simple steps with sparse checkout approach which will populate the working directory sparsely, so you can tell Git which folder(s) or file(s) in the working directory are worth checking out.

Clone repository as usual (--no-checkout is optional):
```
git clone --no-checkout git@foo/bar.git
cd bar
```
^{You may skip this step, if you've your repository already cloned.}

Hint: For large repos, consider shallow clone (--depth 1) to checkout only latest revision or/and --single-branch only.
Enable sparseCheckout option:
```
git config core.sparseCheckout true
```
Specify folder(s) for sparse checkout (without space at the end):
```
echo "trunk/public_html/*"> .git/info/sparse-checkout
```
or edit .git/info/sparse-checkout.
Checkout the branch (e.g. master):
```
git checkout master
```

Now you should have selected folders in your current directory.

You may consider symbolic links if you've too many levels of directories or filtering branch instead.

I just wrote a script for GitHub.

Usage:

python get_git_sub_dir.py path/to/sub/dir <RECURSIVE>

Here's a shell script I wrote for the use case of a single subdirectory sparse checkout

coSubDir.sh

localRepo=$1
remoteRepo=$2
subDir=$3


# Create local repository for subdirectory checkout, make it hidden to avoid having to drill down to the subfolder
mkdir ./.$localRepo
cd ./.$localRepo
git init
git remote add -f origin $remoteRepo
git config core.sparseCheckout true

# Add the subdirectory of interest to the sparse checkout.
echo $subDir >> .git/info/sparse-checkout

git pull origin master

# Create convenience symlink to the subdirectory of interest
cd ..
ln -s ./.$localRepo$subDir $localRepo

This will clone a specific folder and remove all history not related to it.

git clone --single-branch -b {branch} git@github.com:{user}/{repo}.git
git filter-branch --subdirectory-filter {path/to/folder} HEAD
git remote remove origin
git remote add origin git@github.com:{user}/{new-repo}.git
git push -u origin master

Licensed under: CC-BY-SA with attribution

Not affiliated with StackOverflow