run-buildset-registry: run a dual registry

The docker registry daemon can either act as a private registry,
or as a pull-through proxy, but not both.  Yet we need to be able
to serve private (speculative buildset) images as well as plain
upstream images.  Our registry is used as a mirror and requires
authentication, therefore docker's normal behavior of falling back
on docker.io won't work because it will attempt to use our
credentials.

However, the registry daemon stores all of its state in the
filesystem, therefore we can run two instances of the registry
service, both pointing at the same data store.  The first acts
as a pull-through proxy and will serve whatever files are already
in the local storage, or will fetch them from docker.io.  The second
can be used to upload images into the local storage.

To make a long story short, whenever we push into the buildset
registry, we will use the second endpoint.  Whenever the docker
daemon pulls from the buildset registry, it will use the first.

Change-Id: I296029068b5ef28ee56543741fe8c8deeefb5dfa

roles/run-buildset-registry/README.rst

@@ -2,7 +2,10 @@ Runs a docker registry for the use of this buildset.
 
 This may be used for a single job running on a single node, or it may
 be used at the root of a job graph so that multiple jobs running for a
-single change can share the registry.
+single change can share the registry.  Two registry endpoints are
+provided -- one is a read-only endpoint which acts as a pull-through
+proxy and serves upstream images as well as those which are pushed to
+the registry.  The second is intended only for pushing images.
 
 **Role Variables**
 
@@ -25,6 +28,14 @@ single change can share the registry.
 
       The port on which the registry is listening.
 
+   .. zuul:rolevar:: push_host
+
+      The host (IP address) to use when pushing images to the registry.
+
+   .. zuul:rolevar:: push_port
+
+      The port to use when pushing images to the registry.
+
    .. zuul:rolevar:: username
 
       The username used to access the registry via HTTP basic auth.

roles/run-buildset-registry/tasks/main.yaml

@@ -59,9 +59,9 @@
 - name: Decode TLS certificate
   set_fact:
     certificate: "{{ certificate.content | b64decode }}"
-- name: Start a docker registry
+- name: Start a docker proxy
   docker_container:
-    name: buildset_registry
+    name: buildset_proxy
     image: registry:2
     state: started
     restart_policy: always
@@ -80,11 +80,31 @@
       - "{{ buildset_registry_root}}/data:/var/lib/registry"
       - "{{ buildset_registry_root}}/certs:/certs"
       - "{{ buildset_registry_root}}/auth:/auth"
+- name: Start a docker registry
+  docker_container:
+    name: buildset_registry
+    image: registry:2
+    state: started
+    restart_policy: always
+    ports:
+     - "5001:5000"
+    env:
+      REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
+      REGISTRY_HTTP_TLS_KEY: /certs/domain.key
+      REGISTRY_AUTH: htpasswd
+      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
+      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
+    volumes:
+      - "{{ buildset_registry_root}}/data:/var/lib/registry"
+      - "{{ buildset_registry_root}}/certs:/certs"
+      - "{{ buildset_registry_root}}/auth:/auth"
 - name: Set registry information fact
   set_fact:
     buildset_registry:
       host: "{{ ansible_host }}"
       port: 5000
+      push_host: "{{ ansible_host }}"
+      push_port: 5001
       username: zuul
       password: "{{ registry_password }}"
       cert: "{{ certificate }}"