This role uses skopeo to perform image operations.
Also update the container roles docs to add missing documentation
for the already existing upload-container-image role. Clarify
some ambiguity about the registry and repository attributes of
the container images data structure.
Change-Id: Ib66c85daf0edacf0dd797ab34b0d629f99c7111b
Co-Authored-By: James E. Blair <jim@acmegating.com>
The build we are doing here goes only into the temporary registry, and
is then pulled back locally. So this image only needs the tag that it
will get pulled with, which is done just below -- we can remove the
"tag_prefix" and buildset-registry tags that won't be used.
Change-Id: Ia9a3d3476e7971ee5a094f74628a6f6575b535a9
This currently tags the image for the temporary container with
change_change_12345 (note extra change_). I think this was just a
typo introduced with I8eb7d2baa24905e7aac51fce0b2f9b1f24f037f9, which
was updating the tags for periodic pipelines.
Change-Id: I0ce0e475e325907f9ecd46b86c7cc4a655f4a1d3
I noticed this by accident when I ran ansible-lint over this repo from
an outside context; it didn't use the .yamllint in here and started
compalining about eof whitespace.
After scratching my head for a bit as to why this didn't fail here, I
realised we've allowed various newlines since the initial commit
I936fe2c997597972d884c5fc62655d28e8aaf8c5.
Remove this and just use the default eof rules, and fixup the
whitespace as required. This is fairly unimportant, but is nice for
consistency.
Change-Id: Idb46a1f39ba798b0bf70eaa27b4c6b4758ce3d26
If we build an image with `docker_registry` set, it will fail to
tag the image for the buildset registry since we do not prefix the
image with `docker_registry` when retagging it (if set).
This patch resolves this by adding that prefix if it is set so
it will refer to an image that exists.
Change-Id: Iba29161156c8e8ff4f79a92771456cf105d780fe
It seems like BuildKit is the next generation, but not likely to be
enabled by default soon (https://github.com/moby/moby/issues/40379).
Add a flag so people who want to use its features can easily opt-in.
Change-Id: I862819959c77a557199f64b4d42109bc7915959c
Because buildset registries may be used by jobs that finish before other
jobs are finished using the buildset registry we must be careful not to
expose the registry credentials in the jobs that finish sooner.
Otherwise logs for the earlier job runs could potentially be used to
poison the registry for later jobs.
This is likely currently incomplete. Other Zuulians should look over it
carefully to ensure we're covering all the bases here.
The cases I've identified so far are:
* Setting facts that include passwords
* Reading and writing to files that include passwords (as content may be
logged)
* Calling modules with passwords passed as arguments (the module
invocation is logged)
I've also set no_log on zuul_return that passes up credentials because
while the logging for zuul_return is minimal today, I don't want to
count on it remaining that way.
We also use the yet to be merged secret_data attribute on zuul_return to
ensure that zuul_return itself does not expose anything unwanted.
Finally it would be great if others could check over the use of
buildset_registry variables to make sure there aren't any that got
missed. One thing I'm not sure of is whether or not when conditionals
get logged and if we need to be careful about their use too.
Temporarily remove some buildset-regitry jobs which are in a catch-22.
Change-Id: I2dea683e27f00b99a7766bf830981bf91b925265
Use '{{ zuul.pipeline }}' tag prefix in *-docker-image instead of
'change_{{ zuul.change }}' one when zuul.change is not provided, that is
the case with periodic jobs. This allows to build, upload and promote images
using periodic jobs e.g:
- project:
periodic:
- project-buildset-registry
- project-build-image1:
dependencies:
- name: project-buildset-registry
- project-build-image2:
dependencies:
- name: project-buildset-registry
# pulls from buildset registry and tests both images
- project-test:
dependencies:
- name: project-build-image1
- name: project-build-image2
# pre-pulls images from buildset registry for fast build
- project-upload-image1:
dependencies:
- name: project-test
- project-upload-image2:
dependencies:
- name: project-test
- project-promote:
dependencies:
- name: project-upload-image1
- name: project-upload-image2
This fuctionality will allow to keep latest images up to date for the
case when image incorporates continuously updating code from multiple
repositories.
Using true ternary for tag evaluation because ternary filter requires
all passed to it variables be defined or defaulted [0].
[0] https://github.com/ansible/ansible/issues/51276
Change-Id: I8eb7d2baa24905e7aac51fce0b2f9b1f24f037f9
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
When using docker buildx to build a container image, use a temporary
registry to receive the built image instead of requiring a buildset
registry.
A multi-arch test is also added with a publication registry
using the same task list to reduce duplication.
Change-Id: Ib20d1c97f6cb63e0ff9d8888ea792d1941cd8690
Co-Authored-By: James E. Blair <jeblair@redhat.com>
This corrects the following error when upload-docker-image is
used in a release pipeline:
The task includes an option with an undefined variable.
The error was: 'dict object' has no attribute 'change'
Change-Id: I70cd566ddd0b931d635b20107d652f3591a133b6
Add a job that tests the build-container-image role as it would be
used in a tag-based release pipeline (as opposed to check or
gate+promote).
Also, correct an issue in the role where we assumed zuul.change
existed.
Change-Id: If2566764a52726ce45fff9b5e96ce9a42d513d8d
When building normally we tag the images in docker, this lets
upload push them. But in the buildx case, we tag them for the
buildset registry but they never end up with change-specific
tags on the docker host itself, so they can't be found by
the upload playbook.
Change-Id: I4f51df3ac67602fd2d48f66639bb0715e7b06cd3
When building multi-arch it's done in parallel which can result
in the same layer being pushed at the same time, which is bad
for the registries.
Instead, build everything in paralle, then push each arch independently,
then push all the arches again to cause the manifest to be written
and pushed properly.
Change-Id: I8036a9b4d4c515c20a05994741540b999e7cbcae
We seem to have flaky issues with buildx builds (because of course
we do). Try retrying the build to see if it's an eventually
consistent utility.
Change-Id: I6bd625ad7ffcf0c629c85017b1d5d3727e27b9d9
The buildx patch unfortunately changed the logic associated with
siblings to set up siblings in a loop one time, rather than to
do a loop of "set up siblings, build, cleanup siblings". This causes
builds to fail when they're using siblings with an error about
siblings dir not having been cleaned up.
Change-Id: I3c45bfa77ec9f2609689e04044c18f066adc9741
It's a generally useful file for people using buildkitd. It's also
more appropriate to write it in use-buildset-registry and then
just have build-docker-image copy it.
Do the same thing with writing the cert - we don't need to know
which path on the host use-buildset-registry wrote the file to,
we can just write the content from the dir into the container,
and then it's just a consistent command to apply it.
Change-Id: Iaa485c2e8628900dccbed1f4b0773b6d1b5f7983
We need ca_dir to copy the certs in, but when we run in multi-node
cases the use-buildset-registry invocation is in a separate place
so we dont' leak the var in.
This will get deleted in the next patch, but that patch is not
working and we need this to unbreak the multi-arch functionality
for nodepool.
Change-Id: I4f92f0415fb471f304fdd0a1e576812c8d67ab24
We currently are pulling from br/repo/tag and then re-tagging to
just repo/tag. But we have buildset registry in the mirrors list
for docker, so we should just be able to pull directly from repo/tag
to prime the local image cache and have everything just work.
Change-Id: I4d73f10acfc84d94772b13e3be16790e661c7047
Docker has experimental support for building multi-arch
container images with a buildx command. Currently it only
supports pushing to a registry after running and the images
don't end up in the local docker images list. To work around
that, push to the buildset registry then pull back. This
is the inverse of the normal case where we build, then
retag, then push. The end result should be the same.
Change-Id: I6a4c4f9e262add909d2d5c2efa33ec69b9d9364a
We can attach some metadata to images we build indicating where
the image was built from. We can also allow users to add additional
labels, should they choose, although honestly for users doing it in
the Dockerfile makes more sense.
Change-Id: I01de15279a69026a09633eb488dac62910b324f5
This adds a custom ansible-lint rule at .rules/ZuulJobsNamespaceLoopVar.py
that enforces the loop var policy described at:
https://zuul-ci.org/docs/zuul-jobs/policy.html#ansible-loops-in-roles
It also updates existing roles to follow the policy.
Change-Id: I92b2ff56a1c2702542fc07b316f1809087a4c92f
Don't try to create change tag if zuul.change is not defined that is the
case for periodic builds.
Change-Id: I5a7e02caf0f502e90e0477daeee868bf06dfbb5c
There are a number of issues with this. Firstly, it needs to copy the
parent directories to make a heirarchy in the .zuul-siblings
directory. The current "cp -r" was only copying the final directory.
Switch into the source directory and use "--parent" to do this.
Also, it should be copying into the context dir. Add the
{{ item.context }} to the path where appropriate.
Make new testing image that copies in files from the siblings.
Because COPY will fail if the sources aren't there, this is like an
assert that we copied it correctly.
Change-Id: I9f3b0a1f71d20cf7511f224648dd2fa51a039015
When you build from a Dockerfile, it runs in a given "context"; that
is the directory the Dockerfile is in and the directories below it.
It can not access anything outside that context during the build.
When building a container for a project in the gate, you may wish to
install sibling projects that Zuul has checked-out into your container
(i.e. so that Depends-On works). As mentioned, because
/home/zuul/src/<project> is not in the context of the current project,
you will not be able to access this source code during the container
build.
So to help facilitate dependencies, add a siblings: tag which can copy
some or all of the required-projects already specified for the job
into a special sub-directory of the current source.
Because all the code is now in the same context, this will allow build
scripts to be written that look for directories in .zuul-siblings and
can install the source code from there. To further help the scripts,
the ZUUL_SIBLINGS arg is set for the docker build giving the copied
paths.
The test is updated with some paths to test the copy.
Change-Id: I079d823e7194e15b1b496aea0f53f70f6b563f02
We recently added several tasks to the build-docker-image role to
work around docker bugs when using the buildset registry; but this
role doesn't require it so they should be wrapped in a conditional.
Change-Id: Id6e85d07fe34aeb272d7388c778455d5d2a402dd
Skopeo has problems with ipv6 address literals just like docker as they
use the same underlying checks for url validity. However, we think we
can fix that by using a port forward from the executor to the buildset
registry so that skopeo can connect via ipv4
Go back to aliases the registries on test nodes via /etc/hosts.
Change-Id: I5f9316ffe84de06cb2fb2b65a7e1c31d9f8b0e35
Co-Authored-By: James E. Blair <jeblair@redhat.com>
This reverts commit 05f20a5396.
Apparently skopeo is properly cloud native too and doesn't support ipv6
either. I think it is pulling the same docker
distribution/reference/regexp.go lib in and using docker's regex.
The error we get from skopeo:
time="2019-04-10T15:15:48Z" level=fatal msg="Invalid source name docker://[2607:ff68:100:54:f816:3eff:fef2:fc69]:5000/zuul/nodepool:latest: invalid reference format"
Change-Id: I6f916574c9f46e8fdd2464465e2b36ecf8719b16
We only need to alias registries on the build nodes when running docker.
We cannot alias them in /etc/hosts in roles that are expected to run on
localhost beacuse /etc/hosts is bindmounted read only on localhost. This
assumes that skopeo handles ipv6 properly (which has not been tested).
If skopeo does not handle ipv6 properly then we'll need additional
fixing.
Change-Id: I40e5b1bac5aeaf2d42aa05a72b9ced72b7d222c0
This reverts commit a307259776.
We need to additionally handle this on localhost (the zuul executor)
where we cannot edit /etc/hosts. At least I don't think we can. We also
need to handle the case where buildset_regsitry is not yet defined.
Change-Id: I4928f7fcf58e88cf360de253f01b16546220aace
Docker doesn't appear to understand properly escaped ipv6 addrs in its
"urls". Address this by adding /etc/hosts entries for any
buildset_registry that is specified by an ip address (v4 or v6). This
allows us to use a named alias instead of the ipv6 address.
An example failure for posterity:
"[2607:ff68:100:54:f816:3eff:fe7c:e98a]:5000/zuul/nodepool:latest" is not a valid repository/tag: invalid reference format
Change-Id: Id865dc7d3382174b61f9eaa76e29b637a85f5142
The approach of having the proxy serve the local data as well as
the remote wasn't working -- it seems that the proxy would always
check upstream and prefer that data even if it had been pushed
locally.
To correct this, separate the data stores of the two registries,
and add both of them to the registry_mirror setting for the
docker daemon. Now we will pull from our buildset registry first,
and fall back on the proxy to talk to upstream if an image is not
found locally.
The proxy is still required in order to mask out the username and
password which dockerd will otherwise use when talking to upstream.
Change-Id: Iab11954a4b5431d3b1a4d4753f519b6b71f64094