[Bug] Docker Registry Example with non-root path results in Default Backend 404

[gaby]

NGINX Ingress controller version: Latest

What happened:
When deploying the docker-registry using the example provided on this repository and using a non-root path docker is unable to push images.

What you expected to happen:
Be able to push images to non-root paths

How to reproduce it (as minimally and precisely as possible):

I just followed the docker-registry example and added a non-root path using REGISTRY_HTTP_PREFIX in the deployment.

Anything else we need to know:

Deployment.yml

apiVersion: v1
kind: Namespace
metadata:
  name: docker-registry

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: docker-registry
  namespace: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
        - name: docker-registry
          image: registry:2.6.2
          env:
            - name: REGISTRY_HTTP_ADDR
              value: ":5000"
            - name: REGISTRY_HTTP_PREFIX
              value: "/registry/"
            - name: REGISTRY_HTTP_HOST
              value: "<mydomain>"
            - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
              value: "/var/lib/registry"
          ports:
          - name: http
            containerPort: 5000
          volumeMounts:
          - name: image-store
            mountPath: "/var/lib/registry"
      volumes:
        - name: image-store
          emptyDir: {}

---

kind: Service
apiVersion: v1
metadata:
  name: docker-registry
  namespace: docker-registry
  labels:
    app: docker-registry
spec:
  selector:
    app: docker-registry
  ports:
  - name: http
    port: 5000
    targetPort: 5000

Ingress Rule

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/rewrite-target: "/registry/"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    kubernetes.io/tls-acme: 'true'
  name: docker-registry
  namespace: docker-registry
spec:
  tls:
  - hosts:
    - <mydomain>
    secretName: registry-tls
  rules:
  - host: <mydomain>
    http:
      paths:
      - backend:
          serviceName: docker-registry
          servicePort: 5000
        path: /registry/ 

Note:

Visiting https://mydomain.com/registry/v2/_catalog from a browser works fine and I get a respond back from the registry.

Ex.

docker pull alpine:latest
docker tag alpine:latest mydomain.com/registry/alpine:latest
docker push mydomain.com/registry/alpine:latest

When doing a push I get default backend 404.

I'm not sure if this is an issue with Docker, Registry, NGINX Ingress, or what.

[aledbf]

First, remove nginx.ingress.kubernetes.io/rewrite-target: "/registry/"
Then add an additional ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /registry/v2$1
  name: docker-registry-v2
  namespace: docker-registry
spec:
  rules:
  - host: <mydomain>
    http:
      paths:
      - backend:
          serviceName: docker-registry
          servicePort: 5000
        path: /v2(.*)

Why? Your docker client uses the /v2 path to get information about the registry. Without this will not work.

[gaby]

I'm still getting the same error when doing a Docker push.

Error parsing HTTP 404 response body: invalid character 'd' looking for beginning of value: "default backend - 404"

The only way I can get a registry to work with nginx-ingress is to use subdomains or root-path "/". Any other options results in default backend 404.

I also try using "/registry/v2(.*)" for the path, and that didn't work.

The registry should be accesible via:

• https://mydomain.com/registry/

Images tag should be:

• mydomain.com/registry/imageName:tag

[aledbf]

@gabrielcalderon this is working for me. Please check your definitions

apiVersion: v1
kind: Namespace
metadata:
  name: docker-registry

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: docker-registry
  namespace: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
        - name: docker-registry
          image: registry:2.6.2
          env:
            - name: REGISTRY_HTTP_ADDR
              value: ":5000"
            - name: REGISTRY_HTTP_PREFIX
              value: "/registry"
            - name: REGISTRY_HTTP_HOST
              value: "<mydomain>"
            - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
              value: "/var/lib/registry"
          ports:
          - name: http
            containerPort: 5000
          volumeMounts:
          - name: image-store
            mountPath: "/var/lib/registry"
      volumes:
        - name: image-store
          emptyDir: {}

---

kind: Service
apiVersion: v1
metadata:
  name: docker-registry
  namespace: docker-registry
  labels:
    app: docker-registry
spec:
  selector:
    app: docker-registry
  ports:
  - name: http
    port: 5000
    targetPort: 5000

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    kubernetes.io/tls-acme: 'true'
  name: docker-registry
  namespace: docker-registry
spec:
  tls:
  - hosts:
    - <mydomain>
    secretName: registry-tls
  rules:
  - host: <mydomain>
    http:
      paths:
      - backend:
          serviceName: docker-registry
          servicePort: 5000
        path: /registry

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /registry/v2$1
  name: docker-registry-v2
  namespace: docker-registry
spec:
  rules:
  - host: <mydomain>
    http:
      paths:
      - backend:
          serviceName: docker-registry
          servicePort: 5000
        path: /v2(.*)

mydomain.com/registry/imageName:tag

Yes. That's correct.

Also, please check you are using the latest version of the ingress controller 0.24.1

[aledbf]

Error parsing HTTP 404 response body: invalid character 'd' looking for beginning of value: "default backend - 404"

That means you have some issue with the ingress definition and are being directed to the default backend

[gaby]

I just redeployed the registry/ingress using the provided YML. Although I no longer get the default backend 404, I get a different error.

Using the exact yml that you posted I am unable to docker push. It can find the registry and then I get the following error:

docker push <domain>/registry/alpine:latest
The push refers to repository [<domain>/registry/alpine]
a464c54f93a9: Retrying in 1 second
"blob upload unknown"

I just keeps retrying, retrying to post the image.

Looking at the logs for NGINX-Ingress I can see the following:

HTTP 301: POST /v2/registry/alpine/blobls/uploads/ 0
HTTP 301:  GET /registry/v2/registry/alpine/blobs/uploads/registry/alpine/blobs/uploads 108
HTTP 404:  GET /registry/v2/registry/alpine/blobs/uploads/registry/alpine/blobs/uploads/ 108

Somehow the /registry/v2$1 is duplicating the path when the GET happens.

[aledbf]

@gabrielcalderon please check that only the yaml files I posted exist in the cluster.

docker push docker-registry-v2.rocket-science.io/registry/alpine:0
The push refers to repository [docker-registry-v2.rocket-science.io/registry/alpine]
cd7100a72410: Pushed 
0: digest: sha256:8c03bb07a531c53ad7d0f6e7041b64d81f99c6e493cb39abba56d956b40eacbc size: 528
$ docker pull docker-registry-v2.rocket-science.io/registry/alpine:0
0: Pulling from registry/alpine
Digest: sha256:8c03bb07a531c53ad7d0f6e7041b64d81f99c6e493cb39abba56d956b40eacbc
Status: Image is up to date for docker-registry-v2.rocket-science.io/registry/alpine:0

[gaby]

Thanks! We decided to deploy the registry using a sub-domain, we couldn't get the /registry to work at all.

[lpanichi]

NGINX Ingress controller version: Latest

What happened:
When deploying the docker-registry using the example provided on this repository and using a non-root path docker is unable to push images.

What you expected to happen:
Be able to push images to non-root paths

How to reproduce it (as minimally and precisely as possible):

I just followed the docker-registry example and added a non-root path using REGISTRY_HTTP_PREFIX in the deployment.

Anything else we need to know:

Deployment.yml

apiVersion: v1
kind: Namespace
metadata:
  name: docker-registry

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: docker-registry
  namespace: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
        - name: docker-registry
          image: registry:2.6.2
          env:
            - name: REGISTRY_HTTP_ADDR
              value: ":5000"
            - name: REGISTRY_HTTP_PREFIX
              value: "/registry/"
            - name: REGISTRY_HTTP_HOST
              value: "<mydomain>"
            - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
              value: "/var/lib/registry"
          ports:
          - name: http
            containerPort: 5000
          volumeMounts:
          - name: image-store
            mountPath: "/var/lib/registry"
      volumes:
        - name: image-store
          emptyDir: {}

---

kind: Service
apiVersion: v1
metadata:
  name: docker-registry
  namespace: docker-registry
  labels:
    app: docker-registry
spec:
  selector:
    app: docker-registry
  ports:
  - name: http
    port: 5000
    targetPort: 5000

Ingress Rule

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/rewrite-target: "/registry/"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    kubernetes.io/tls-acme: 'true'
  name: docker-registry
  namespace: docker-registry
spec:
  tls:
  - hosts:
    - <mydomain>
    secretName: registry-tls
  rules:
  - host: <mydomain>
    http:
      paths:
      - backend:
          serviceName: docker-registry
          servicePort: 5000
        path: /registry/ 

Note:

Visiting https://mydomain.com/registry/v2/_catalog from a browser works fine and I get a respond back from the registry.

Ex.

docker pull alpine:latest
docker tag alpine:latest mydomain.com/registry/alpine:latest
docker push mydomain.com/registry/alpine:latest

When doing a push I get default backend 404.

I'm not sure if this is an issue with Docker, Registry, NGINX Ingress, or what.

Well actually it's a bit tricky because when I use your two Ingress, it seems to work until I try to push an image.
Doing so, the push keeps retrying to POST layers and I see in docker registry pod logs stuff like this :

10.2.0.6 - - [17/Feb/2020:10:56:02 +0000] "POST /docker/v2/docker/v2/springapp/blobs/uploads/ HTTP/1.1" 202 0 "" "pointless stuff"
time="2020-02-17T10:56:02.869435323Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=nope http.request.id=9c2790d5-51b8-4580-9a1d-8e27093d3287 http.request.method=POST http.request.remoteaddr=nope http.request.uri="/docker/v2/docker/v2/springapp/blobs/uploads/" "stuff" http.response.duration=22.025435ms http.response.status=202 http.response.written=0 
10.2.0.6 - - [17/Feb/2020:10:56:02 +0000] "POST /docker/v2/docker/v2/springapp/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.6 go/go1.12.16 git-commit/stuff"

The docker/v2/docker/v2 is strange to me.
Anyway, I think it's best to use a sub domain.