Commit Graph

32 Commits

Author SHA1 Message Date
James E. Blair e69d879cae buildset registry: run socat in new session
We spawn socat processes in the background in buildset registry
related roles.  Ansible 5 is much better at killing all processes
in its session when the task is complete.  Work around that by
starting socat within a new session with setsid.

Change-Id: Iaab17f5d4068be6b08e3d89d2abe80199f0cd654
2022-05-24 10:46:42 -07:00
Ian Wienand 72d4e506d5 intermediate-registry: handle socat warning out
Zuul switched to a new base image, and it seems the new socat puts out
a warning (something like

 ...  socat[489590] W ioctl(5, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...): Inappropriate ioctl for device

for reference).

Grep the output so we only get the line about what port it is
listening on.

Change-Id: I74fb86a9158b45e6601ee1fbc199ba80cd4991fe
2021-10-14 16:04:46 +11:00
James E. Blair 8bdb78476d Fix default value for zuul_artifacts
The case where this isn't set isn't exercised by the tests, so we missed
this.  We need to supply an empty list of artifacts to iterate over if
there are no zuul artifacts.

Change-Id: I082e3546ddc0ff57386063a4f697ae6584db9f90
2021-06-24 10:49:23 -07:00
Clark Boylan 4c40b92950 Prevent leaks of buildset registry credentials
Because buildset registries may be used by jobs that finish before other
jobs are finished using the buildset registry we must be careful not to
expose the registry credentials in the jobs that finish sooner.
Otherwise logs for the earlier job runs could potentially be used to
poison the registry for later jobs.

This is likely currently incomplete. Other Zuulians should look over it
carefully to ensure we're covering all the bases here.

The cases I've identified so far are:

* Setting facts that include passwords
* Reading and writing to files that include passwords (as content may be
  logged)
* Calling modules with passwords passed as arguments (the module
  invocation is logged)

I've also set no_log on zuul_return that passes up credentials because
while the logging for zuul_return is minimal today, I don't want to
count on it remaining that way.

We also use the yet to be merged secret_data attribute on zuul_return to
ensure that zuul_return itself does not expose anything unwanted.

Finally it would be great if others could check over the use of
buildset_registry variables to make sure there aren't any that got
missed. One thing I'm not sure of is whether or not when conditionals
get logged and if we need to be careful about their use too.

Temporarily remove some buildset-regitry jobs which are in a catch-22.

Change-Id: I2dea683e27f00b99a7766bf830981bf91b925265
2021-06-24 09:56:19 -07:00
Sorin Sbarnea 90d263941a More E208
Change-Id: I8c3d224682d7d6780d6ad7823e390c47114e702b
2020-11-05 08:33:28 +00:00
Monty Taylor 5f31625a94 Add --all to skopeo copy from insecure registry
skopeo needs to be told to copy all instances of a given image,
otherwise it just grabs one of them.

https://github.com/containers/skopeo/pull/741

Change-Id: If78ad50602e745ef7747d983b987cf964ff6e67f
2020-05-08 15:25:12 -05:00
Clark Boylan 4ed66807a0 Use unique loop vars to avoid conflicts
We have to be careful about avoiding outer loop loop_var conflicts in
ansible. Because the zuul-jobs roles are meant to be reconsumed
elsewhere we should not use 'item' loopvars and instead set them to
something a bit more unique.

We use a zj_ prefix to try and be unique to this repo and document this
convention.

Change-Id: I20b9327a914890e9eafcb2b36dc8c23fb472bc8f
2020-02-04 12:23:36 -08:00
James E. Blair d48667a3c5 use-buildset-registry: Add podman support
Change-Id: I41718073962c8e7eb3d8810276e550fb84bd6e99
2019-12-05 11:44:15 -08:00
James E. Blair 2f2d6ce3f7 Add a registry test job
This exercises the intermediate and buildset registry roles.

Change-Id: Ic0da2d6b48a7b9f9b616033c70db471ba98181b4
2019-06-03 09:03:35 -07:00
Monty Taylor 454b5349b0 Add retries to skopeo copy operations
We think that sometimes the network derps so the whole copy goes
bad. Add retries in to the mix since a copy command is idempotent.

Change-Id: I2d2891f2ebe3ca6a99874d9cf028addea888c3b7
2019-04-25 17:22:30 +00:00
James E. Blair bd5542e5cc Don't log setting up creds for the intermediate registry
This could expose credential information.

Change-Id: I7c134289b627f90f7972e0d59ec21a7f35bd2c01
2019-04-22 15:54:31 -07:00
James E. Blair aae4df5024 Fix localhost cert location for push-to-intermediate-registry
We're using 127.0.0.1 everywhere rather than localhost; the cert
directory and auth information need to match.

Change-Id: Id72332625c234519ce4c819e88c184035eac8203
2019-04-22 14:43:29 -07:00
Clark Boylan 23826e05a6 Only alias docker registries on test nodes
Skopeo has problems with ipv6 address literals just like docker as they
use the same underlying checks for url validity. However, we think we
can fix that by using a port forward from the executor to the buildset
registry so that skopeo can connect via ipv4

Go back to aliases the registries on test nodes via /etc/hosts.

Change-Id: I5f9316ffe84de06cb2fb2b65a7e1c31d9f8b0e35
Co-Authored-By: James E. Blair <jeblair@redhat.com>
2019-04-22 10:37:33 -07:00
Clark Boylan ba4b2c68b9 Revert "Only alias registries on build nodes"
This reverts commit 05f20a5396.

Apparently skopeo is properly cloud native too and doesn't support ipv6
either. I think it is pulling the same docker
distribution/reference/regexp.go lib in and using docker's regex.

The error we get from skopeo:

 time="2019-04-10T15:15:48Z" level=fatal msg="Invalid source name docker://[2607:ff68:100:54:f816:3eff:fef2:fc69]:5000/zuul/nodepool:latest: invalid reference format"

Change-Id: I6f916574c9f46e8fdd2464465e2b36ecf8719b16
2019-04-10 08:22:04 -07:00
Clark Boylan 05f20a5396 Only alias registries on build nodes
We only need to alias registries on the build nodes when running docker.
We cannot alias them in /etc/hosts in roles that are expected to run on
localhost beacuse /etc/hosts is bindmounted read only on localhost. This
assumes that skopeo handles ipv6 properly (which has not been tested).

If skopeo does not handle ipv6 properly then we'll need additional
fixing.

Change-Id: I40e5b1bac5aeaf2d42aa05a72b9ced72b7d222c0
2019-04-09 14:45:19 -07:00
Clark Boylan 3a29935f32 Revert "Use /etc/hosts with docker registries to handle ipv6"
This reverts commit a307259776.

We need to additionally handle this on localhost (the zuul executor)
where we cannot edit /etc/hosts. At least I don't think we can. We also
need to handle the case where buildset_regsitry is not yet defined.

Change-Id: I4928f7fcf58e88cf360de253f01b16546220aace
2019-04-09 10:10:03 -07:00
Clark Boylan a307259776 Use /etc/hosts with docker registries to handle ipv6
Docker doesn't appear to understand properly escaped ipv6 addrs in its
"urls". Address this by adding /etc/hosts entries for any
buildset_registry that is specified by an ip address (v4 or v6). This
allows us to use a named alias instead of the ipv6 address.

An example failure for posterity:

  "[2607:ff68:100:54:f816:3eff:fe7c:e98a]:5000/zuul/nodepool:latest" is not a valid repository/tag: invalid reference format

Change-Id: Id865dc7d3382174b61f9eaa76e29b637a85f5142
2019-04-08 13:40:58 -07:00
James E. Blair 325b666b6d Use ipwrap filter in registry roles
We need to escape some ipv6 addresses in URLs.

Change-Id: Ica5fe73dd65e138cdc4817de3903cdd271402941
2019-04-05 16:15:02 -07:00
James E. Blair 0509a390ac Fix error in pull-from-intermediate-registry
A recent change to make this safer is causing the error:

The conditional check 'metadata in item and item.metadata.type | default('')
== 'container_image'' failed. The error was: error while evaluating conditional
(metadata in item and item.metadata.type | default('') == 'container_image'):
'metadata' is undefined

Change-Id: Ib709996dc950466a3673c422cd288ba874187f5c
2019-04-02 16:21:33 -07:00
Clark Boylan d2aaa5529c Handle no metadata in zuul artifact items
When pulling from intermediate registries we check zuul for artifacts of
a certain type. Unfortunately we do so blindly without checking that the
metadata field exists for the artifact. These leads to errors like:

  "msg": "The conditional check 'item.metadata.type | default('') == 'container_image'' failed.
  The error was: error while evaluating conditional (item.metadata.type | default('') == 'container_image'): 'dict object' has no attribute 'metadata'

  http://logs.openstack.org/12/643712/1/gate/opendev-buildset-registry/1016e6e/job-output.txt.gz#_2019-03-18_19_28_39_060210

Address this by checking the metadata field prior to accessing it.

Change-Id: I02bbeddccdda836fc313eccce09e4cb0beb6262a
2019-03-18 14:20:55 -07:00
James E. Blair 05d6187825 buildset registry: fix proxy_port error in push/pull roles
This was copy/pasta from the use_buildset_registry role; the
intermediate registry has no proxy_port, and the push/pull operations
don't need to use the proxy buildset registry, so remove both
entries from the docker config.

Change-Id: I7c2d57d027e457f4f093497938574624cd5a444c
2019-03-18 10:15:02 -07:00
James E. Blair f4db0f0979 buildset registry: don't put skopeo creds on command line
Use the docker user config file rather than the skopeo command line
when performing skopeo push/pull operations.  This should allow
us to log the command.

Change-Id: If6b1f3ab34461d77e619b188f48c5d209df7afce
2019-03-18 09:13:38 -07:00
James E. Blair 0d3b5278d1 Add no_log entries to skopeo copy commands
Now that these work, we should disable logging because the password
appears in the command.

Change-Id: I0c5d0be9bffb48bb576f61b7ff19f4451353b7e2
2019-03-11 14:29:04 -07:00
James E. Blair 9c0d25f349 Fix buildset registry
The approach of having the proxy serve the local data as well as
the remote wasn't working -- it seems that the proxy would always
check upstream and prefer that data even if it had been pushed
locally.

To correct this, separate the data stores of the two registries,
and add both of them to the registry_mirror setting for the
docker daemon.  Now we will pull from our buildset registry first,
and fall back on the proxy to talk to upstream if an image is not
found locally.

The proxy is still required in order to mask out the username and
password which dockerd will otherwise use when talking to upstream.

Change-Id: Iab11954a4b5431d3b1a4d4753f519b6b71f64094
2019-03-01 15:52:01 -08:00
James E. Blair a358f21736 Use buildset registry push endpoint
When copying images from the intermediate registry to the buildset
registry, use the new push endpoint of the dual-registry system.

Also, use the push endpoint after a docker build to push the
new image to the buildset registry.

Change-Id: I3a11036bb9fb7cb3457a3d744fa83647c1b1b085
2019-02-21 14:15:45 -08:00
James E. Blair c58e2a51dc Load buildset_registry in intermediate pull
The intermediate registry pull role is designed to be used in
the same playbook as the run-buildset-registry role, which sets
the buildset_registry fact.  However, that fact is set on the
host where the registry runs, not localhost.  Theoretically we
should be able to delegate setting that fact to all hosts in
the inventory, plus localhost, however, that doesn't seem to
work in local testing.

Work around this by, once again, loading the buildset_registry
fact from the zuul_return file.

Change-Id: Ia16b3af8782c875e64ad5eeeeb5f107482a3e30a
2019-02-18 10:46:26 -08:00
James E. Blair 71b7cb0ae5 Update docker image roles
* In the build-image role, push to the buildset registry if it is defined.
* In the intermediate registry push and pull roles, ensure that the
  buildset registry TLS cert is in place.  This is a self-signed cert,
  and so needs to be written for each run.  This happens inside
  bubblewrap where we have permission to write to /etc, which is an
  ephemeral volume.

Change-Id: I47781d8a7adb93817dfe9266e2f4ad5fd829385c
2019-02-16 10:02:48 -08:00
James E. Blair ee1b1ea2e4 Intermediate registry: correct arg order
The --insecure-policy argument must appear before copy.

Change-Id: If182405b57c026ced237bf152d323bdea6ff54b2
2019-02-15 17:46:42 -08:00
James E. Blair c7818fe499 Intermediate registry: use --insecure-policy
There is no policy file available, and we aren't concerned with
verifying image signatures at this point.  Add this option to
tell skopeo to proceed regardless.

Change-Id: I15a4978ec0fb338bc05c974b0ec6a21f680c853e
2019-02-15 16:13:42 -08:00
James E. Blair 2a3d69c8be Fix undefined attrs in registry push/pull roles
The attribute zuul.artifacts is only present if there are artifacts.
Use the empty list as default.

The default for image.tags should be 'latest' to match the rest
of the docker roles.

Change-Id: Iff6863043e3a0311cb1c8c2ef4cd3d37ff79cce5
2019-02-14 15:03:11 -08:00
James E. Blair 582e95db53 Enable logging on registry/push/pull jobs
This may expose credentials, but these don't work right now.

Change-Id: I8b55734d17329ecc6ac9d9a9ad4189482b6c07ee
2019-02-14 12:59:29 -08:00
James E. Blair 8efc1cf1af Add intermediate registry push/pull roles
Change-Id: Ie2554005f924d2736d1f1fd1c51cfd5ca9e09199
Depends-On: https://review.openstack.org/634825
2019-02-11 17:41:30 -08:00