31

For example, I want to download PCL 3d_rec_framework.

This is the git repository of PCL: https://github.com/PointCloudLibrary/pcl.git

How can I download this directory?

https://github.com/PointCloudLibrary/pcl/tree/master/apps

I tried this, but it didn't work:

sam@sam:~/code/pcl_standalone$ git clone https://github.com/PointCloudLibrary/pcl/tree/master/apps/3d_rec_framework
Cloning into '3d_rec_framework'...
error: The requested URL returned error: 403 while accessing https://github.com/PointCloudLibrary/pcl/tree/master/apps/3d_rec_framework/info/refs
fatal: HTTP request failed
sam@sam:~/code/pcl_standalone$ 

I don't want to download git of PCL and remove all other directories that I don't want.

How do I download just a single directory?

8

You cannot. With git, you clone the entire repository, and the full history of the repository.

There are some workaround solutions to be able to get a single file out of a git archive, listed on a Stack Exchange answer for the same question, but you will still have to download the entire repository to get that single file or directory you want.

| improve this answer | |
  • 7
    Simply not true: askubuntu.com/a/729798/384425 – CelticParser Feb 4 '16 at 23:31
  • 3
    @CelticParser So you claim my answer is not true, and then proceed to point at an answer which requires downloading every file from a git repository, to get a single file? – dobey Feb 4 '16 at 23:34
  • 2
    or askubuntu.com/a/645276/384425 – CelticParser Feb 4 '16 at 23:41
  • > "I don't want to download git of PCL and remove all other directories that I don't want." That is open-ended. I read it as @sam does not want to manually remove the directories. – CelticParser Feb 4 '16 at 23:44
  • Can you update your answer for 2019, with indicated Git version(s)? See e.g. skukx's answer and other places ("Git 1.7.10 (April 2012) actually allows you to clone only one branch"). – Peter Mortensen Dec 16 '19 at 15:31
44

dobey's answer is no longer the case since git v1.7. You can now checkout certain folders from a repository. The full instructions are found here.

git init <repo>
cd <repo>
git remote add -f origin <url>

git config core.sparseCheckout true

echo "some/dir/" >> .git/info/sparse-checkout
echo "another/sub/tree" >> .git/info/sparse-checkout

This tells git which directories you want to checkout. Then you can pull just those directories

git pull origin master
| improve this answer | |
  • 3
    This implies all Ubuntu versions have 1.7 available. You should check that to be the case and comment on your answer here as to which individual versions will actually work. PowerShell is also not Ubuntu and therefore should not be included, in my opinion. – Thomas Ward Jul 6 '15 at 22:07
  • 2
    @ThomasW. All currently supported versions of Ubuntu do include at least git 1.7, and most are 2.x now. – dobey Feb 4 '16 at 23:38
  • 4
    Still this will clone the whole repository and then do that sparse checkout. – Clerenz Jun 7 '18 at 12:46
  • @dobey, Seriously you removed useful information that people finding this question with Google might very much be looking for?! If I was forced to use powershell I would definitely like to see the pipe details, they are not obvious! echo "some/dir/" | Out-File -Encoding ascii .git/info/sparse-checkout echo "another/sub/tree/" | Out-File -Append -Encoding ascii .git/info/sparse-checkout – Samuel Åslund Dec 19 '18 at 7:53
  • I am so glad that it worked, but how do I include .gitignore file? I tried echo '.gitignore' >> .git/info/sparse-checkout and echo './.gitignore' >> .git/info/sparse-checkout, neither worked. Thanks! – zyy Feb 19 at 4:27
10

First, do:

git clone --depth 1 [repo root] [name of destination directory]

Then:

cd [name of destination directory]

...And lastly:

git filter-branch --prune-empty --subdirectory-filter [path to sub-dir] HEAD

It's that easy. Git will rewrite the repo so that only the desired sub-dir is included. This works even if the sub-dir is several layers deep. Just name the destination directory the name of the sub-dir. Then in the "git filter-branch" command put the relative path to the sub-dir. Oh, the --depth 1 tells git to only download the top of the head (essentially removing the history).

| improve this answer | |
  • This allows you to download a single sub directory, but the question pertains to multiple directories.. is that possible this way? I have to say I don't see how this works, looking at the documentation. – Joeppie Jan 11 '18 at 10:25
  • 1
    Is there an easy way to refresh that directory from time to time? – Clerenz Jun 7 '18 at 13:11
5

git clone --filter from Git 2.19

This option will actually skip fetching most unneeded objects from the server:

git clone --depth 1 --no-checkout --filter=blob:none \
  "file://$(pwd)/server_repo" local_repo
cd local_repo
git checkout master -- mydir/myfile

The server should be configured with:

git config --local uploadpack.allowfilter 1
git config --local uploadpack.allowanysha1inwant 1

There is no server support as of v2.19.0, but it can already be locally tested.

TODO: --filter=blob:none skips all blobs, but still fetches all tree objects. But on a normal repo, this should be tiny compared to the files themselves, so this is already good enough. Asked at: https://www.spinics.net/lists/git/msg342006.html Devs replied a --filter=tree:0 is in the works to do that.

Remember that --depth 1 already implies --single-branch, see also: https://stackoverflow.com/questions/1778088/how-to-clone-a-single-branch-in-git

file://$(path) is required to overcome git clone protocol shenanigans: https://stackoverflow.com/questions/47307578/how-to-shallow-clone-a-local-git-repository-with-a-relative-path

The format of --filter is documented on man git-rev-list.

An extension was made to the Git remote protocol to support this feature.

Docs on Git tree:

See also: https://stackoverflow.com/questions/2466735/how-to-checkout-only-one-file-from-git-repository-sparse-checkout

Test it out

#!/usr/bin/env bash
set -eu

list-objects() (
  git rev-list --all --objects
  echo "master commit SHA: $(git log -1 --format="%H")"
  echo "mybranch commit SHA: $(git log -1 --format="%H")"
  git ls-tree master
  git ls-tree mybranch | grep mybranch
  git ls-tree master~ | grep root
)

# Reproducibility.
export GIT_COMMITTER_NAME='a'
export GIT_COMMITTER_EMAIL='a'
export GIT_AUTHOR_NAME='a'
export GIT_AUTHOR_EMAIL='a'
export GIT_COMMITTER_DATE='2000-01-01T00:00:00+0000'
export GIT_AUTHOR_DATE='2000-01-01T00:00:00+0000'

rm -rf server_repo local_repo
mkdir server_repo
cd server_repo

# Create repo.
git init --quiet
git config --local uploadpack.allowfilter 1
git config --local uploadpack.allowanysha1inwant 1

# First commit.
# Directories present in all branches.
mkdir d1 d2
printf 'd1/a' > ./d1/a
printf 'd1/b' > ./d1/b
printf 'd2/a' > ./d2/a
printf 'd2/b' > ./d2/b
# Present only in root.
mkdir 'root'
printf 'root' > ./root/root
git add .
git commit -m 'root' --quiet

# Second commit only on master.
git rm --quiet -r ./root
mkdir 'master'
printf 'master' > ./master/master
git add .
git commit -m 'master commit' --quiet

# Second commit only on mybranch.
git checkout -b mybranch --quiet master~
git rm --quiet -r ./root
mkdir 'mybranch'
printf 'mybranch' > ./mybranch/mybranch
git add .
git commit -m 'mybranch commit' --quiet

echo "# List and identify all objects"
list-objects
echo

# Restore master.
git checkout --quiet master
cd ..

# Clone. Don't checkout for now, only .git/ dir.
git clone --depth 1 --quiet --no-checkout --filter=blob:none "file://$(pwd)/server_repo" local_repo
cd local_repo

# List missing objects from master.
echo "# Missing objects after --no-checkout"
git rev-list --all --quiet --objects --missing=print
echo

echo "# Git checkout fails without internet"
mv ../server_repo ../server_repo.off
! git checkout master
echo

echo "# Git checkout fetches the missing file from internet"
mv ../server_repo.off ../server_repo
git checkout master -- d1/a
echo

echo "# Missing objects after checking out d1/a"
git rev-list --all --quiet --objects --missing=print

GitHub upstream.

Output in Git v2.19.0:

# List and identify all objects
c6fcdfaf2b1462f809aecdad83a186eeec00f9c1
fc5e97944480982cfc180a6d6634699921ee63ec
7251a83be9a03161acde7b71a8fda9be19f47128
62d67bce3c672fe2b9065f372726a11e57bade7e
b64bf435a3e54c5208a1b70b7bcb0fc627463a75 d1
308150e8fddde043f3dbbb8573abb6af1df96e63 d1/a
f70a17f51b7b30fec48a32e4f19ac15e261fd1a4 d1/b
84de03c312dc741d0f2a66df7b2f168d823e122a d2
0975df9b39e23c15f63db194df7f45c76528bccb d2/a
41484c13520fcbb6e7243a26fdb1fc9405c08520 d2/b
7d5230379e4652f1b1da7ed1e78e0b8253e03ba3 master
8b25206ff90e9432f6f1a8600f87a7bd695a24af master/master
ef29f15c9a7c5417944cc09711b6a9ee51b01d89
19f7a4ca4a038aff89d803f017f76d2b66063043 mybranch
1b671b190e293aa091239b8b5e8c149411d00523 mybranch/mybranch
c3760bb1a0ece87cdbaf9a563c77a45e30a4e30e
a0234da53ec608b54813b4271fbf00ba5318b99f root
93ca1422a8da0a9effc465eccbcb17e23015542d root/root
master commit SHA: fc5e97944480982cfc180a6d6634699921ee63ec
mybranch commit SHA: fc5e97944480982cfc180a6d6634699921ee63ec
040000 tree b64bf435a3e54c5208a1b70b7bcb0fc627463a75    d1
040000 tree 84de03c312dc741d0f2a66df7b2f168d823e122a    d2
040000 tree 7d5230379e4652f1b1da7ed1e78e0b8253e03ba3    master
040000 tree 19f7a4ca4a038aff89d803f017f76d2b66063043    mybranch
040000 tree a0234da53ec608b54813b4271fbf00ba5318b99f    root

# Missing objects after --no-checkout
?f70a17f51b7b30fec48a32e4f19ac15e261fd1a4
?8b25206ff90e9432f6f1a8600f87a7bd695a24af
?41484c13520fcbb6e7243a26fdb1fc9405c08520
?0975df9b39e23c15f63db194df7f45c76528bccb
?308150e8fddde043f3dbbb8573abb6af1df96e63

# Git checkout fails without internet
fatal: '/home/ciro/bak/git/test-git-web-interface/other-test-repos/partial-clone.tmp/server_repo' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

# Git checkout fetches the missing directory from internet
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 45 bytes | 45.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 45 bytes | 45.00 KiB/s, done.

# Missing objects after checking out d1
?f70a17f51b7b30fec48a32e4f19ac15e261fd1a4
?8b25206ff90e9432f6f1a8600f87a7bd695a24af
?41484c13520fcbb6e7243a26fdb1fc9405c08520
?0975df9b39e23c15f63db194df7f45c76528bccb

Conclusions: all blobs except d1/a are missing. E.g. f70a17f51b7b30fec48a32e4f19ac15e261fd1a4, which is d1/b, is not there after checking out d1/.

Note that root/root and mybranch/mybranch are also missing, but --depth 1 hides that from the list of missing files. If you remove --depth 1, then they show on the list of missing files.

| improve this answer | |
1

For GitHub repos, you can clone any sub-directories of any GitHub repository (at any reference) using https://github.com/HR/github-clone

| improve this answer | |
1

If the url of the repository is this

https://github.com/blah/blah2.git

and from there you want the folder images which you see through this url

https://github.com/blah/blah2/tree/master/images

Then do

# Install subversion to use svn
!apt-get install subversion

# Get what you want by adding "/trunk" to the repo url and the folder you want
#!svn checkout REPO_URL/trunk/DIRECTORY
!svn checkout https://github.com/blah/blah2.git/trunk/images

This works inside Google Colab

| improve this answer | |
-1

CHANGE tree/master/ to trunk/

SIMPLE :

svn export https://github.com/REPONAME/examples/trunk/lite/examples/

Like for your

svn export https://github.com/PointCloudLibrary/pcl/trunk/apps

To Install svn UBUNTU:https://linuxtechlab.com/simple-guide-to-install-svn-on-ubuntu/

Windows: https://tortoisesvn.net/downloads.html

enter image description here

| improve this answer | |
  • please add comments before doing a downvote. And have you tried this before downvoting? I myself was searching for this after I got the solution after reading hell lot everywhere and failed attempts I got this solution so I posted it here. – Shubham Kumar Gupta Ggps Jul 28 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.