WeChat Public Number: Operations and Maintenance Development Story, Author: Huazi
GitHub address: https://github.com/sunsharing-note/harbor
background
During a recent inspection tour, harbor storage usage has reached 80%. So, look at the number of mirror labels under each item. There are hundreds of mirrored labels found under a particular item. When asked, you know that the project is currently in the debugging phase, debugging many times a day. Now that you don't have much storage space, go to harbor and delete the previous mirror labels. Just keep the last few. During the manual deletion process, hundreds were found, and only ten were displayed on each page. I have to sort by push time first, then delete page by page. Just think about it once, and don't have another one.
Later, when you think about it carefully, it is also not easy to control. Every time a patrol finds it, it is too troublesome to delete it manually. So you're going to write a script that deletes the mirrored labels every time, just keep the last few. Just happened to be learning golang recently, just use it to write.
Embarrassingly, I finished the script and after testing it, I found that the new version of harbor was ready to set retention policies on the UI. To comfort yourself, consider it an exercise and a try!
target
- The command line allows you to query all current items, whether or not they are public, the number of warehouses
- The command line allows you to query warehouse and mirror names, pull times under a project
- Ability to specify labels and number of reserved labels on the command line to remove mirrored labels
- Number of labels that can get mirrors
- Immediate garbage cleaning is not supported after removal, please do it manually (consider that you cannot push and pull a mirror during cleaning)
statement
- The script is written purely as a personal exercise and does not make any suggestions
- Beginning golang, just to achieve the goal, the code quality is poor, please understand
- The harbor used this time is v2.3.1
- Please move all code to github
Realization
- Get all the items in harbor, the API is available through harbor's swagger
//Define the data structure to get based on the results of the harbor swagger test
type MetaData struct {
Public string `json:"public"`
}
type ProjectData struct {
MetaData MetaData `json:"metadata"`
ProjectId int `json:"project_id"`
Name string `json:"name"`
RepoCount int `json:"repo_count"`
}
type PData []ProjectData
// Provide harbor Address to get project
func GetProject(url string) []map[string]string {
//Define url
url = url + "/api/v2.0/projects"
//url = url + "/api/projects"
// Construct Request
request, _ := http.NewRequest(http.MethodGet, url,nil)
//Cancel Validation
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
//Define Client
client := &http.Client{Timeout: 10 * time.Second, Transport: tr}
//client := &http.Client{Timeout: 10 * time.Second}
request.Header.Set("accept", "application/json")
//Set User and Password
request.SetBasicAuth("admin", "Harbor12345")
response, err := client.Do(request)
if err != nil {
fmt.Println("excute failed")
fmt.Println(err)
}
// Get body
body, _ := ioutil.ReadAll(response.Body)
defer response.Body.Close()
ret := PData{}
json.Unmarshal([]byte(string(body)), &ret)
var ps = []map[string]string{}
// Get the returned data
for i := 0; i < len(ret); i++ {
RData := make(map[string]string)
RData["name"] = (ret[i].Name)
RData["project_id"] = strconv.Itoa(ret[i].ProjectId)
RData["repo_count"] =strconv.Itoa(ret[i].RepoCount)
RData["public"] = ret[i].MetaData.Public
ps = append(ps, RData)
}
return ps
}
- Get repo under Project
// Define the data structure to get
type ReposiData struct {
Id int `json:"id"`
Name string `json:"name"`
ProjectId int `json:"project_id"`
PullCount int `json:"pull_count"`
}
type RepoData []ReposiData
//Get repo under project by providing harbor Address and corresponding project
func GetRepoData(url string, proj string) []map[string]string {
// /api/v2.0/projects/goharbor/repositories
url = url + "/api/v2.0/projects/" + proj + "/repositories"
//Construct Request
request, _ := http.NewRequest(http.MethodGet, url,nil)
//Ignore authentication
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Timeout: 10 * time.Second, Transport: tr}
request.Header.Set("accept", "application/json")
//Set User Name and Password
request.SetBasicAuth("admin", "Harbor12345")
response, err := client.Do(request)
if err != nil {
fmt.Println("excute failed")
fmt.Println(err)
}
// Get body
body, _ := ioutil.ReadAll(response.Body)
defer response.Body.Close()
ret := RepoData{}
json.Unmarshal([]byte(string(body)), &ret)
var ps = []map[string]string{}
// Get the returned data
for i := 0; i < len(ret); i++ {
RData := make(map[string]string)
RData["name"] = (ret[i].Name)
pId := strconv.Itoa(ret[i].ProjectId)
RData["project_id"] = pId
RData["id"] =(strconv.Itoa(ret[i].Id))
RData["pullCount"] = (strconv.Itoa(ret[i].PullCount))
ps = append(ps, RData)
}
return ps
}
- Mirror tag operation
//Define tag data structure to get
type Tag struct {
ArtifactId int `json:"artifact_id"`
Id int `json:"id"`
Name string `json:"name"`
RepositoryId int `json:"repository_id"`
PushTimte string `json:"push_time"`
}
type Tag2 struct {
ArtifactId string `json:"artifact_id"`
Id string `json:"id"`
Name string `json:"name"`
RepositoryId string `json:"repository_id"`
PushTimte string `json:"push_time"`
}
type Tag2s []Tag2
// delete tag by specified count, where count first gets the list of tags to delete
func DeleTagsByCount(tags []map[string]string ,count int) []string {
var re []string
tt := tags[0]["tags"]
ss := Tag2s{}
json.Unmarshal([]byte(tt), &ss)
// have a sort
for i := 0; i < len(ss); i++ {
for j := i + 1; j < len(ss); j++ {
//Sort by pushtime
if ss[i].PushTimte > ss[j].PushTimte {
ss[i], ss[j] = ss[j], ss[i]
}
}
}
// get all tags
for i := 0; i < len(ss); i++ {
re = append(re, ss[i].Name)
}
// Returns count tag s that will be deleted,
return re[0:count]
}
// delete tag by specified tag Deletes the specified tag
func DelTags(url string, project string, repo string, tag string) (int, map[string]interface{}) {
url = url + "/api/v2.0/projects/" + project + "/repositories/" + repo + "/artifacts/" + tag + "/tags/" + tag
request, _ := http.NewRequest(http.MethodDelete, url,nil)
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Timeout: 10 * time.Second, Transport: tr}
request.Header.Set("accept", "application/json")
request.SetBasicAuth("admin", "Pwd123456")
// Execute Delete tag
response,_ := client.Do(request)
defer response.Body.Close()
var result map[string]interface{}
bd, err := ioutil.ReadAll(response.Body)
if err == nil {
err = json.Unmarshal(bd, &result)
}
return response.StatusCode,result
}
//Define tag data structure to get
type ArtiData struct {
Id int `json:"id"`
ProjectId int `json:"project_id"`
RepositoryId int `json:"repository_id"`
//Digest string `json:"digest"`
Tags []Tag `json:"tags"`
}
type AData []ArtiData
// Get tag data from harbor address, project, and repo
func GetTags(url string, project string, repo string) []map[string]string {
url = url + "/api/v2.0/projects/" + project + "/repositories/" + repo + "/artifacts"
request, _ := http.NewRequest(http.MethodGet, url,nil)
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Timeout: 10 * time.Second, Transport: tr}
request.Header.Set("accept", "application/json")
request.Header.Set("X-Accept-Vulnerabilities", "application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0")
request.SetBasicAuth("admin", "Harbor12345")
// Get tag
response, err := client.Do(request)
if err != nil {
fmt.Println("excute failed")
fmt.Println(err)
}
body, _ := ioutil.ReadAll(response.Body)
defer response.Body.Close()
ret := AData{}
json.Unmarshal([]byte(string(body)),&ret)
var ps = []map[string]string{}
sum := 0
RData := make(map[string]string)
RData["name"] = repo
// Get the returned data
for i := 0; i < len(ret); i++ {
RData["id"] = (strconv.Itoa(ret[i].Id))
RData["project_id"] = (strconv.Itoa(ret[i].ProjectId))
RData["repository_id"] =(strconv.Itoa(ret[i].RepositoryId))
//RData["digest"] = ret[i].Digest
var tdata = []map[string]string{}
sum = len((ret[i].Tags))
// Get tag
for j := 0; j < len((ret[i].Tags)); j++ {
TagData := make(map[string]string)
TagData["artifact_id"] = strconv.Itoa((ret[i].Tags)[j].ArtifactId)
TagData["id"] = strconv.Itoa((ret[i].Tags)[j].Id)
TagData["name"] = (ret[i].Tags)[j].Name
TagData["repository_id"] = strconv.Itoa((ret[i].Tags)[j].RepositoryId)
TagData["push_time"] = (ret[i].Tags)[j].PushTimte
tdata = append(tdata, TagData)
}
RData["count"] = strconv.Itoa(sum)
ss, err := json.Marshal(tdata)
if err != nil {
fmt.Println("failed")
os.Exit(2)
}
RData["tags"] = string(ss)
ps = append(ps, RData)
}
return ps
}
- Get user command line input, list all items in harbor
// Define command actions related to getting project s in harbor
var projectCmd = &cobra.Command{
Use: "project",
Short: "to operator project",
Run: func(cmd *cobra.Command, args []string) {
output, err := ExecuteCommand("harbor","project", args...)
if err != nil {
Error(cmd,args, err)
}
fmt.Fprint(os.Stdout, output)
},
}
// project list
var projectLsCmd = &cobra.Command{
Use: "ls",
Short: "list all project",
Run: func(cmd *cobra.Command, args []string) {
url, _ := cmd.Flags().GetString("url")
if len(url) == 0 {
fmt.Println("url is null,please specified the harbor url first !!!!")
os.Exit(2)
}
// Get all project s
output := harbor.GetProject(url)
fmt.Println("Project Name Access Level Warehouse Quantity")
for i := 0; i < len(output); i++ {
fmt.Println(output[i]["name"], output[i]["public"], output[i]["repo_count"])
}
},
}
// init
func init() {
// ./harbor project ls -u https://
rootCmd.AddCommand(projectCmd)
projectCmd.AddCommand(projectLsCmd)
projectLsCmd.Flags().StringP("url", "u", "","defaults: [https://127.0.0.1]")
}
- Get the repo list
// repo command
var repoCmd = &cobra.Command{
Use: "repo",
Short: "to operator repository",
Run: func(cmd *cobra.Command, args []string) {
output, err := ExecuteCommand("harbor","repo", args...)
if err != nil {
Error(cmd,args, err)
}
fmt.Fprint(os.Stdout, output)
},
}
// repo list
var repoLsCmd = &cobra.Command{
Use: "ls",
Short: "list project's repository",
Run: func(cmd *cobra.Command, args []string) {
url, _ := cmd.Flags().GetString("url")
project, _ := cmd.Flags().GetString("project")
if len(project) == 0 {
fmt.Println("sorry, you must specified the project which you want to show repository !!!")
os.Exit(2)
}
// get all repo
output := harbor.GetRepoData(url, project)
// Show data
fmt.Println("Warehouse name----------Number of pulls")
for i := 0; i < len(output); i++ {
fmt.Println(output[i]["name"],output[i]["pullCount"])
}
},
}
func init() {
// ./harbor repo ls -u https:// -p xxx
rootCmd.AddCommand(repoCmd)
repoCmd.AddCommand(repoLsCmd)
repoLsCmd.Flags().StringP("url", "u", "","defaults: [https://127.0.0.1]")
repoLsCmd.Flags().StringP("project", "p","", "the project")
}
- tag operation
// tag command
var tagCmd = &cobra.Command{
Use: "tag",
Short: "to operator image",
Run: func(cmd *cobra.Command, args []string) {
output, err := ExecuteCommand("harbor","tag", args...)
if err != nil {
Error(cmd,args, err)
}
fmt.Fprint(os.Stdout, output)
},
}
// tag ls
var tagLsCmd = &cobra.Command{
Use: "ls",
Short: "list all tags of the repository you have specified which you should specified project at the same time",
Run: func(cmd *cobra.Command, args []string) {
url, _ := cmd.Flags().GetString("url")
project, _ := cmd.Flags().GetString("project")
repo, _ := cmd.Flags().GetString("repo")
// get all tags
ss := harbor.GetTags(url, project, repo)
for i := 0; i < len(ss); i++ {
count, _ := strconv.Atoi((ss[i])["count"])
fmt.Printf("the repo %s has %d images\n", repo, count)
}
},
}
// tag del by tag or the number of image you want to save
var tagDelCmd = &cobra.Command{
Use: "del",
Short: "delete the tags of the repository you have specified which you should specified project at the same time",
Run: func(cmd *cobra.Command, args []string) {
// Get user input and format
url, _ := cmd.Flags().GetString("url")
project, _ := cmd.Flags().GetString("project")
repo, _ := cmd.Flags().GetString("repo")
tag,_ := cmd.Flags().GetString("tag")
count,_ := cmd.Flags().GetString("count")
ret,_ := strconv.Atoi(count)
if len(tag) != 0 && ret != 0 {
fmt.Println("You can't choose both between count and tag")
os.Exit(2)
} else if len(tag) == 0 && ret != 0 {
// get all tags
retu := harbor.GetTags(url, project, repo)
//delete tag by you hsve specied the number of the images you want to save
rTagCount, _ := strconv.Atoi((retu[0])["count"])
// Decide whether to delete tags by comparing the count entered by the user with the actual number of tags
if ret == rTagCount {
fmt.Printf("the repository %s of the project %s only have %d tags, so you can't delete tags and we will do nothing!!\n", repo, project,ret)
} else if ret > rTagCount {
fmt.Printf("the repository %s of the project %s only have %d tags, but you want to delete %d tags, so we suggest you to have a rest and we will do nothing!!\n", repo, project,rTagCount, ret)
} else {
// Delete tag can be performed
fmt.Printf("we will save the latest %d tags and delete other %d tags !!!\n", ret, (rTagCount - ret))
tags := harbor.GetTags(url, project, repo)
retu := harbor.DeleTagsByCount(tags, (rTagCount - ret))
for i := 0 ; i < len(retu); i++ {
// to delete tag
code, msg := harbor.DelTags(url, project, repo, retu[i])
fmt.Printf("the tag %s is deleted,status code is %d, msg is %s\n", retu[i], code, msg)
}
}
} else {
// delete tag by you specied tag
code, msg := harbor.DelTags(url, project, repo, tag)
fmt.Println(code, msg["errors"])
}
},
}
func init() {
// ./harbor tag ls -u -p -r
rootCmd.AddCommand(tagCmd)
tagCmd.AddCommand(tagLsCmd)
tagLsCmd.Flags().StringP("url", "u", "","defaults: [https://127.0.0.1]")
tagLsCmd.Flags().StringP("project", "p", "","the project")
tagLsCmd.Flags().StringP("repo", "r", "","the repository")
// ./harbor tag del -u -p -r [-t | -c]
tagCmd.AddCommand(tagDelCmd)
tagDelCmd.Flags().StringP("url", "u", "","defaults: [https://127.0.0.1]")
tagDelCmd.Flags().StringP("project", "p", "","the project which you should specified if you want to delete the tag of any repository ")
tagDelCmd.Flags().StringP("repo", "r", "","the repository which you should specified if you want to delete the tag")
tagDelCmd.Flags().StringP("tag", "t", "","the tag, You can't choose it with tag together")
tagDelCmd.Flags().StringP("count", "c", "","the total number you want to save.for example: you set --count=10, we will save the 10 latest tags by use push_time to sort,can't choose it with tag together")
}
test
//get help harbor % ./harbor -h https://harbor.zaizai.com Usage: harbor [flags] harbor [command] Available Commands: completion generate the autocompletion script for the specified shell help Help about any command project to operator project repo to operator repository tag to operator image Flags: -h, --help help for harbor Use "harbor [command] --help" for more information about a command. //List all project s harbor % ./harbor project ls -u https://harbor.zaizai.com Project Name Access Level Warehouse Quantity goharbor false 3 library true 0 public true 1 //List all repo s harbor % ./harbor repo ls -u https://harbor.zaizai.com -p goharbor Warehouse name----------Number of pulls goharbor/harbor-portal 0 goharbor/harbor-db 1 goharbor/prepare 0 //List tags harbor%. /harbor tags ls-u https://harbor.zaizai.com -p goharbor-r harbor-db the repo harbor-db has 9 images // Delete tag s by keeping the last 20 mirrors harbor % ./harbor tag del -u https://harbor.zaizai.com -p goharbor -r harbor-db -c 20 the repository harbor-db of the project goharbor only have 9 tags, but you want to delete 20 tags, so we suggest you to have a rest and we will do nothing!! // Delete tag s by retaining the last 10 mirrors harbor % ./harbor tag del -u https://harbor.zaizai.com -p goharbor -r harbor-db -c 10 the repository harbor-db of the project goharbor only have 9 tags, but you want to delete 10 tags, so we suggest you to have a rest and we will do nothing!! // Delete tag s by preserving the last five mirrors harbor % ./harbor tag del -u https://harbor.zaizai.com -p goharbor -r harbor-db -c 5 we will save the latest 5 tags and delete other 4 tags !!! the tag v2.3.9 is deleted,status code is 200, msg is map[] the tag v2.3.10 is deleted,status code is 200, msg is map[] the tag v2.3.8 is deleted,status code is 200, msg is map[] the tag v2.3.7 is deleted,status code is 200, msg is map[] //Specify tag for deletion caicloud@MacBook-Pro-2 harbor % ./harbor tag del -u https://harbor.zaizai.com -p goharbor -r harbor-db -t v2.3.6 200 <nil> !!!! Finally, you need to do it manually harbor UI Garbage collection on!!!
Reference resources
- https://github.com/spf13/cobra
- harbor swagger