[Docker] the registry built by docker login Harbor reports 503 errors

The production and test environments maintain two sets of Harbor warehouses and are ready to synchronize images. To achieve such a function, first, the environment networks on both sides must be connected and released through temporary network policies. Then, at least both sides dock login with each other. The other's registry should also be successful, but there is a strange problem.

The registry of the production environment docker login test environment is OK, but the reverse is not, which means that image synchronization cannot be performed.

At first, it was suspected that some special ports were not connected (only port 80 was opened). Contact the operation and maintenance students and open all ip ports to ip, but it still failed. The packet capture found that the production environment would actively disconnect, and then returned to 503.

Then check the log. First, check how the log reports when the test environment logs in to the production environment successfully.

Nov 17 17:54:27 172.18.0.x proxy[22141]: production environment  ip - "GET /v2/ HTTP/1.1" 401 76 "-" "docker/19.03.11 go/go1.13.10 git-commit/42e35e61f3 kernel/5.4.107-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.11 \x5C(linux\x5C))" 0.003 0.003 .
Nov 17 17:54:27 172.18.0.x proxy[22141]: production environment  ip - "GET /service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry HTTP/1.1" 200 890 "-" "docker/19.03.11 go/go1.13.10 git-commit/42e35e61f3 kernel/5.4.107-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.11 \x5C(linux\x5C))" 0.028 0.028 .
Nov 17 17:54:27 172.18.0.x proxy[22141]: production environment  ip - "GET /v2/ HTTP/1.1" 200 2 "-" "docker/19.03.11 go/go1.13.10 git-commit/42e35e61f3 kernel/5.4.107-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.11 \x5C(linux\x5C))" 0.008 0.008 .

From the status code, there will be a 401 first, and then go to get the token. After getting the docker, go to login and return 200 successfully. This is the normal process and is also in line with our impression of the docker client logging in to the registry. You can see the official picture.

Then look at the log of login failure, that is, log in to the test environment from the production environment. The log is as follows.

Nov 17 17:56:30 172.18.0.x proxy[26421]: testing environment ip - "GET /v2/ HTTP/1.1" 401 76 "-" "docker/19.03.11 go/go1.13.10 git-commit/42e35e61f3 kernel/5.4.107-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.11 \x5C(linux\x5C))" 0.003 0.002 .

I was surprised to find that only one log was recorded during one login, that is, the one with 401 status code. From the official figure, we can probably judge that there should be a problem with the links of 3 and 4.

This means that the docker in the formal environment will go to auth service to get the token, and then look at the docker log.

Nov 18 00:27:08 szglbd dockerd[22141]: time="2021-11-18T00:27:08.146062335+08:00" level=error msg="Handler for POST /v1.40/auth returned error: Get http://Formal environment ip/v2/: received unexpected HTTP status: 503 Service Unavailable“
Nov 18 00:27:15 szglbd dockerd[22141]: time="2021-11-18T00:27:15.894732084+08:00" level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://harbor.com/v2/: dial tcp test environment IP: 443: Connect: connection rejected“

It is obvious that the dockerd of the formal environment accesses auth service 503. So I checked docker info and agent information. Sure enough, I added an agent! Cause the request to be HTTP_PROXY forwards the token, so the local docker cannot normally request the token, so the verification fails. 503 is the status code returned by the proxy server. Finally, remove the proxy, and the whole process will be normal.

Simply verify that the request is forwarded by the proxy. You can find that the request for token is forwarded by capturing the packet.

Posted on Fri, 03 Dec 2021 21:08:21 -0500 by rh-penguin