How to clone all repositions at once from GitHub?

Question

How to clone all repositions at once from GitHub?

I have a GitHub account, and I want to back up all the repositories inside, given everything new that can be created for automation. I was hoping for something like this:

git clone git@github.com:company/*.git

or something like that would work, but it doesn't seem like a template.

Is there a way in Git to clone and then pull everything if you have the appropriate permissions?

+79

git github git-clone

numb3rs1x Oct 24 '13 at 9:12

source share

26 answers

On Windows and on all UNIX / LINUX systems, using Git Bash or any other terminal , replace YOURUSERNAME with your username and use:

 CNTX={users|orgs}; NAME={username|orgname}; PAGE=1 curl "https://api.github.com/$CNTX/$NAME/repos?page=$PAGE&per_page=100" | grep -e 'git_url*' | cut -d \" -f 4 | xargs -L1 git clone

Set CNTX=users and NAME=yourusername to download all your repositories. Set CNTX = orgs and NAME = yourorgname to download all the repositories of your organization.

The maximum page size is 100, so you need to call this number several times with the correct page number to get all your repositories (set PAGE to the desired page number that you want to download).

Here is the shell script that does the above: https://gist.github.com/erdincay/4f1d2e092c50e78ae1ffa39d13fa404e

+95

Erdinc Ay Sep 28 '15 at 23:36

source share

Storage organization

To clone all repositories from your organization, try the following single-line shell:

 GHORG=company; curl "https://api.github.com/orgs/$GHORG/repos?per_page=1000" | grep -o 'git@[^"]*' | xargs -L1 git clone

User repositories

Cloning everyone using the Git repository URL:

 GHUSER=CHANGEME; curl "https://api.github.com/users/$GHUSER/repos?per_page=1000" | grep -o 'git@[^"]*' | xargs -L1 git clone

Clone everyone using the Clone URL:

 GHUSER=CHANGEME; curl "https://api.github.com/users/$GHUSER/repos?per_page=1000" | grep -w clone_url | grep -o '[^"]\+://.\+.git' | xargs -L1 git clone

Here is a useful shell function that you can add to custom launch files (using curl + jq ):

 # Usage: gh-clone-user (user) gh-clone-user() { curl -sL "https://api.github.com/users/$1/repos?per_page=1000" | jq -r '.[]|.clone_url' | xargs -L1 git clone }

Private repositories

If you need to clone private repositories, you can add an authorization token in your header, for example:

 -H 'Authorization: token <token>'

or pass it in the parameter ( ?access_token=TOKEN ), for example:

 curl -s "https://api.github.com/users/$GHUSER/repos?access_token=$GITHUB_API_TOKEN&per_page=1000" | grep -w clone_url | grep -o '[^"]\+://.\+.git' | xargs -L1 git clone

Notes:

To get only personal repositories, add type=private to the query string.
Another way is to use hub after setting up the API key.

See also:

GitHub REST API v3 - a list of your repositories
How to download GitHub Release from a private repository using the command line .
How to get a list of all human github repositories? ,

^{Tips :}
^{- To increase the speed, set the number of parallel processes by specifying the -P parameter for -P4 ( -P4 = 4 processes).}
^{- If you need to increase the limits of GitHub, try to authenticate by specifying your API key.}
^{- Add --recursive to enter registered submodules and update all nested submodules inside.}

+28

kenorb Sep 26 '15 at 23:23

source share

This essence performs the task on one line on the command line:

 curl -s https://api.github.com/orgs/[your_org]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Replace [your_org] name of your organization. And set your per_page if necessary.

UPDATE:

As mentioned in ATutorMe, the maximum page size is 100, according to the GitHub documentation .

If you have more than 100 repositories, you need to add the page parameter to your URL, and you can run a command for each page.

 curl -s "https://api.github.com/orgs/[your_org]/repos?page=2&per_page=100" | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Note. The per_page parameter defaults to per_page 30 .

+19

seancdavis Jan 06 '15 at 18:18

source share

Go to Account Settings → Application and create an API key
Then paste the API key, github instance url and organization name into the script below

 #!/bin/bash # Substitute variables here ORG_NAME="<ORG NAME>" ACCESS_TOKEN="<API KEY>" GITHUB_INSTANCE="<GITHUB INSTANCE> URL="https://${GITHUB_INSTANCE}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN}" curl ${URL} | ruby -rjson -e 'JSON.load(STDIN.read).each {|repo| %x[git clone #{repo["ssh_url"]} ]}'

Save this in a file, chmod u+x file, then run it.

_{Thanks Arno for the ruby code.}

+5

Cory Klein Jun 26 '14 at 17:37

source share

So, I will add my answer too. :) (I found it simple)

Extract List (I used magento):

 curl -si https://api.github.com/users/magento/repos | grep ssh_url | cut -d '"' -f4

Use clone_url instead of ssh_url to use HTTP access.

So, let them clone them all! :)

 curl -si https://api.github.com/users/magento/repos | \ grep ssh_url | cut -d '"' -f4 | xargs -i git clone {}

If you want to get a private repo just add the parameter GET ?access_token=YOURTOKEN

+4

Kirby Nov 03 '16 at 23:36

source share

I found a comment on gist @seancdavis, which was very useful, especially because, like the original poster, I wanted to sync all repositories for quick access, however the vast majority of them were private.

 curl -u [[USERNAME]] -s https://api.github.com/orgs/[[ORGANIZATION]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Replace [[USERNAME]] with your github username and [[ORGANIZATION]] in your Github organization. The output (JSON repo metadata) will be passed in a simple ruby script:

 # bring in the Ruby json library require "json" # read from STDIN, parse into ruby Hash and iterate over each repo JSON.load(STDIN.read).each do |repo| # run a system command (re: "%x") of the style "git clone <ssh_url>" %x[git clone #{repo["ssh_url"]} ] end

+3

Garren S Feb 26 '15 at 10:50

source share

I created a script with Python3 and Github APIv3

https://github.com/muhasturk/gitim

Just run

 ./gitim

+2

muhasturk 04 Oct '15 at 11:19

source share

 curl -s https://api.github.com/orgs/[GITHUBORG_NAME]/repos | grep clone_url | awk -F '":' '{ print $2 }' | sed 's/\"//g' | sed 's/,//' | while read line; do git clone "$line"; done

+2

julianzhang Apr 23 '18 at 11:07

source share

I tried some of the above commands and tools, but decided that they were too troublesome, so I wrote another command line tool for this, called github-dl .

To use it (assuming you have nodejs installed)

 npx github-dl -d /tmp/test wires

This will allow you to get a list of all repo with wires and write information to the test directory using the authorization data (user / pass) that you provide in the CLI.

Detail this

Requests authorization (supports 2FA)
Gets a list of repositories for a user / organization through the Github API
Is there a pagination for this, so more than 100 repos are supported

In fact, it does not clone repositories, but instead writes a .txt file that you can transfer to xargs for cloning, for example:

 cd /tmp/test cat wires-repo-urls.txt | xargs -n2 git clone # or to pull cat /tmp/test/wires-repo-urls.txt | xargs -n2 git pull

Maybe this is good for you; these are just a few lines of JS, so it should be easy to adapt to your needs

+2

wires Jun 20 '19 at 12:04 on

source share

So, in practice, if you want to clone all the repositories from the FOO organization that correspond to the BAR , you can use the single line below, which requires jq and common cli utilities

 curl 'https://api.github.com/orgs/FOO/repos?access_token=SECRET' | jq '.[] | .ssh_url' | awk '/BAR/ {print "git clone " $0 " & "}' | sh

+1

konr Apr 15 '14 at 16:13

source share

This one line python will do what you need. It:

checks github for available repositories

for everyone, makes a git clone system call

 python -c "import json, urllib, os; [os.system('git clone ' + r['ssh_url']) for r in json.load(urllib.urlopen('https://api.github.com/orgs/<<ORG_NAME>>/repos?per_page=200'))]"

+1

Evren Ozkan Aug 11 '15 at 9:10

source share

The npm module is also very useful for this. It can not only clone, but also pull (update the data that you already have).

You simply create the configuration as follows:

 [{ "username": "BoyCook", "dir": "/Users/boycook/code/boycook", "protocol": "ssh" }]

and do gitall clone , for example. Or gitall pull

+1

Vladislav Rastrusny Nov 06 '15 at 14:27

source share

In case someone is looking for a solution for Windows, here is a small function in PowerShell to do the trick (maybe oneliner / alias, if not the fact that I need it to work with or without a proxy server )

 function Unj-GitCloneAllBy($User, $Proxy = $null) { (curl -Proxy $Proxy "https://api.github.com/users/$User/repos?page=1&per_page=100").Content | ConvertFrom-Json | %{ $_.clone_url } # workaround git printing to stderr by @wekempf aka William Kempf # https://github.com/dahlbyk/posh-git/issues/109#issuecomment-21638678 | %{ & git clone $_ 2>&1 } | % { $_.ToString() } }

+1

Andrzej Lichnerowicz Oct 06 '16 at 11:34

source share

I created a pip module for this. Runs on Windows, Linux, and OSX.

https://github.com/zeusofjuice/starclone

You can clone the repo with:

 starclone <user>

There are several flags that you can specify from the help file or from the README file.

+1

samdixon Aug 27 '18 at 20:21

source share

A simple solution:

 NUM_REPOS=1000 DW_FOLDER="Github_${NUM_REPOS}_repos" mkdir ${DW_FOLDER} cd ${DW_FOLDER} for REPO in $(curl https://api.github.com/users/${GITHUB_USER}/repos?per_page=${NUM_REPOS} | awk '/ssh_url/{print $2}' | sed 's/^"//g' | sed 's/",$//g') ; do git clone ${REPO} ; done

+1

Antonio Feitosa Feb 16 '19 at 15:11

source share

You can get the list of repositories using curl , and then iterate over the specified list using the bash loop:

 GIT_REPOS=`curl -s curl https://${GITHUB_BASE_URL}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN} | grep ssh_url | awk -F': ' '{print $2}' | sed -e 's/",//g' | sed -e 's/"//g'` for REPO in $GIT_REPOS; do git clone $REPO done

0

Sriram Mageswaran Sep 20 '16 at 2:30

source share

You can use the open source tool to clone a bunch of github repositories: https://github.com/artiomn/git_cloner

Example:

git_cloner --type github --owner octocat --login user --password user https://my_bitbucket

Use the JSON API from api.github.com . You can see the sample code in the github documentation: https://developer.github.com/v3/

Or there:

https://github.com/artiomn/git_cloner/blob/master/src/git_cloner/github.py

0

AN Aug 20 '17 at 11:49 on

source share

To clone only private repositories with a given access key and installed Python 3 module and queries:

 ORG=company; ACCESS_KEY=0000000000000000000000000000000000000000; for i in $(python -c "import requests; print(' '.join([x['ssh_url'] for x in list(filter(lambda x: x['private'] ,requests.get('https://api.github.com/orgs/$ORG/repos?per_page=1000&access_token=$ACCESS_KEY').json()))]))"); do git clone $i; done;

0

ubugnu Mar 21 '18 at 15:52

source share

Python3 solution that includes full pagination through Link Header.

Prerequisites:

Github API "Personal Access Token"
pip3 install links-from-link-header
hub

 import json import requests from requests.auth import HTTPBasicAuth import links_from_header respget = lambda url: requests.get(url, auth=HTTPBasicAuth('githubusername', 'githubtoken')) myorgname = 'abc' nexturl = f"https://api.github.com/orgs/{myorgname}/repos?per_page=100" while nexturl: print(nexturl) resp = respget(nexturl) linkheads = resp.headers.get('Link', None) if linkheads: linkheads_parsed = links_from_header.extract(linkheads) nexturl = linkheads_parsed.get('next', None) else: nexturl = None respcon = json.loads(resp.content) with open('repolist', 'a') as fh: fh.writelines([f'{respconi["full_name"]}\n' for respconi in respcon])

Then you can use xargs or in parallel and: cat repolist | parallel -I% hub clone % cat repolist | parallel -I% hub clone % cat repolist | parallel -I% hub clone % cat repolist | parallel -I% hub clone %

0

Jose Alban Jun 08 '18 at 16:00

source share

If you have a list of repositories in this list, then this shell script works:

 user="https://github.com/user/" declare -a arr=("repo1", "repo2") for i in "${arr[@]}" do echo $user"$i" git clone $user"$i" done

0

Kasthuri Shravankumar Jul 16 '18 at 13:44

source share

If your organization has 100 repos, then sometimes you want to clone at the command level for speed. This tool only supports repository cloning: https://github.com/steinfletcher/github-org-clone

0

Stein Jan 17 '19 at 13:22

source share

I created a sample batch script. You can download all private / public repositories from github.com. After downloading the repository, it is automatically converted to a ZIP file.

 @echo off setlocal EnableDelayedExpansion SET "username=olyanren" SET "password=G....." set "mypath=%cd%\" SET "url=https://%username%:%password%@github.com/%username%/" FOR /F "tokens=* delims=" %%i in (files.txt) do ( SET repo=%%i rmdir /s /q !repo! git clone "!url!!repo!.git" cd !repo! echo !mypath! git archive --format=zip -o "!mypath!!repo!.zip" HEAD cd .. )

Note: the files.txt file should only contain repository names, such as:

 repository1 repository2

0

olyanren Jan 25 '19 at 12:44

source share

May 19th update

use this bash command to organize (including private repo)

 curl -u "{username}" "https://api.github.com/orgs/{org}/repos?page=1&per_page=100" | grep -o 'git@[^"]*' | xargs -L1 git clone

0

Tommy May 04 '19 at 9:03

source share

The predominant answers here do not take into account that the Github API will return a maximum of 100 repositories, despite the fact that you can specify in per_page . If you are cloning a Github organization with more than 100 repositories, you will need to follow the page links in the API response.

I wrote a CLI tool for this :

 clone-github-org -o myorg

This clones all repositories in the myorg organization into the current working directory.

0

Chad Aug 29 '19 at 14:22

source share

Another shell script with comments that clones all repositories (public and private) from the user:

 #!/bin/bash USERNAME=INSERT_USERNAME_HERE PASSWORD=INSERT_PASSWORD_HERE # Generate auth header AUTH=$(echo -n $USERNAME:$PASSWORD | base64) # Get repository URLs curl -iH "Authorization: Basic "$AUTH https://api.github.com/user/repos | grep -w clone_url > repos.txt # Clean URLs (remove " and ,) and print only the second column cat repos.txt | tr -d \"\, | awk '{print $2}' > repos_clean.txt # Insert username:password after protocol:// to generate clone URLs cat repos_clean.txt | sed "s/:\/\/git/:\/\/$USERNAME\:$PASSWORD\@git/g" > repos_clone.txt while read FILE; do git clone $FILE done <repos_clone.txt rm repos.txt & rm repos_clone.txt

0

hlorand Sep 01 '19 at 17:04 on

source share

Thomas Kelley · Accepted Answer · 2013-10-24 21:40

I do not think it can be done like this. It is best to find and scroll through the list of Organization repositories using the API.

Try the following:

Create an API token by going to Account Settings → Applications
Call: http://${GITHUB_BASE_URL}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN}
The response will be an array of JSON objects. Each object will contain information about one of the repositories within this Organization. I think that in your case you will look for exactly the ssh_url property.
Then git clone each of these ssh_url s.

This is a bit of extra work, but for GitHub you need to have proper authentication.

How to clone all repositions at once from GitHub?

Storage organization

User repositories

Private repositories

More articles: