How to obtain code of all projects in gitlab in batch

background

Now more and more companies use gitlab to manage code. There is a problem with gitlab. The free version does not support global code search, which is troublesome. If you clone all the code to the local, you can easily carry out a variety of searches. However, gitlab does not provide the functions of all git clone projects.

The company is getting bigger and bigger, and there are more and more projects. What should we do? Write a script to batch git clone.

thinking

gitlab provides api to get the list of projecct, so you can traverse the list to do git clone

See: https://docs.gitlab.com/ee/api/projects.html "list all projects"

Script

Note: the api of gitlab can only get 100 projecct information at a time. I provide this script with paging function, which can support more than 100 projects.

 
# Pass the test in Python 3.0
# You need to create an AccessToken in gitlab
# You need to set an environment variable on the local machine, such as git? SSH? Command = SSH - I C: / users / WSQ / key / wsq.key
 
from urllib.request import urlopen
import json
import subprocess, shlex
import time
import os
 
gitlabAddr  = '[gitlabAddr like: 192.168.1.200:5678]'
gitlabToken = '[your access token created from gitlab]'
 
for index in range(10):
    url     = "http://%s/api/v4/projects?private_token=%s&per_page=100&page=%d&order_by=name" % (gitlabAddr, gitlabToken, index)
    print(url)
    allProjects     = urlopen(url)
    allProjectsDict = json.loads(allProjects.read().decode())
    if len(allProjectsDict) == 0:
        break
    for thisProject in allProjectsDict: 
        try:
            thisProjectURL  = thisProject['ssh_url_to_repo']
            thisProjectPath = thisProject['path_with_namespace']
            print(thisProjectURL + ' ' + thisProjectPath)
            
            if os.path.exists(thisProjectPath):
                command     = shlex.split('git -C "%s" pull' % (thisProjectPath))
            else:
                command     = shlex.split('git clone %s %s' % (thisProjectURL, thisProjectPath))
            
            resultCode  = subprocess.Popen(command)
            time.sleep(1)
        except Exception as e:
            print("Error on %s: %s" % (thisProjectURL, e.strerror))


Script usage
This is a python script, which can be saved as gitall.py for example. Then install Python on your computer. Another batch file, updategit.bat, contains a command: python gitall.py. Then double-click updategit.bat to use it.
 

For those who have a strong focus on team code.

Tags: GitLab git Python ssh

Posted on Fri, 10 Jan 2020 09:20:07 -0500 by JCF22Lyoko