Commit Graph

20 Commits

Author SHA1 Message Date
Clark Boylan 4c40b92950 Prevent leaks of buildset registry credentials
Because buildset registries may be used by jobs that finish before other
jobs are finished using the buildset registry we must be careful not to
expose the registry credentials in the jobs that finish sooner.
Otherwise logs for the earlier job runs could potentially be used to
poison the registry for later jobs.

This is likely currently incomplete. Other Zuulians should look over it
carefully to ensure we're covering all the bases here.

The cases I've identified so far are:

* Setting facts that include passwords
* Reading and writing to files that include passwords (as content may be
  logged)
* Calling modules with passwords passed as arguments (the module
  invocation is logged)

I've also set no_log on zuul_return that passes up credentials because
while the logging for zuul_return is minimal today, I don't want to
count on it remaining that way.

We also use the yet to be merged secret_data attribute on zuul_return to
ensure that zuul_return itself does not expose anything unwanted.

Finally it would be great if others could check over the use of
buildset_registry variables to make sure there aren't any that got
missed. One thing I'm not sure of is whether or not when conditionals
get logged and if we need to be careful about their use too.

Temporarily remove some buildset-regitry jobs which are in a catch-22.

Change-Id: I2dea683e27f00b99a7766bf830981bf91b925265
2021-06-24 09:56:19 -07:00
Albin Vass 06e35b2125 Test multiarch release builds and use temp registry with buildx
When using docker buildx to build a container image, use a temporary
registry to receive the built image instead of requiring a buildset
registry.

A multi-arch test is also added with a publication registry
using the same task list to reduce duplication.

Change-Id: Ib20d1c97f6cb63e0ff9d8888ea792d1941cd8690
Co-Authored-By: James E. Blair <jeblair@redhat.com>
2020-07-01 10:29:31 -07:00
Sorin Sbarnea 4af438d136 Made sequence indent consistent
Change-Id: I5ea4232ca4fd6e03d5b5f72eb6704bee84d04ea5
2020-05-07 14:13:55 +01:00
Monty Taylor f449ae20fb Set up siblings dirs for each build in the loop
The buildx patch unfortunately changed the logic associated with
siblings to set up siblings in a loop one time, rather than to
do a loop of "set up siblings, build, cleanup siblings". This causes
builds to fail when they're using siblings with an error about
siblings dir not having been cleaned up.

Change-Id: I3c45bfa77ec9f2609689e04044c18f066adc9741
2020-05-05 17:52:50 -05:00
Albin Vass bee0c6ae2f ansible-lint: use matchplay instead of matchtask
For some reason matchtask doesn't match includes, matchplay does so use
that instead.

Change-Id: I040f7f3394503e92d06c05e8ff671a43b14baebc
2020-05-05 20:42:38 +02:00
Zuul 9d7551e4ea Merge "Revert "Revert "Do not set buildset_fact if it's not present in results.json""" 2020-05-05 14:35:24 +00:00
Monty Taylor 63bd307e63 Support multi-arch image builds with docker buildx
Docker has experimental support for building multi-arch
container images with a buildx command. Currently it only
supports pushing to a registry after running and the images
don't end up in the local docker images list. To work around
that, push to the buildset registry then pull back. This
is the inverse of the normal case where we build, then
retag, then push. The end result should be the same.

Change-Id: I6a4c4f9e262add909d2d5c2efa33ec69b9d9364a
2020-05-04 14:02:13 -05:00
Albin Vass a5f5091897 Revert "Revert "Do not set buildset_fact if it's not present in results.json""
This reverts commit b5e87e62f1.

Change-Id: I839c0fb9811ce68e2949402f12aab1387d1a812e
2020-04-28 21:26:16 +02:00
Albin Vass b5e87e62f1 Revert "Do not set buildset_fact if it's not present in results.json"
This reverts commit bc0ca442a1.

Change-Id: If0885877a1c976ed430d6e3a7a462e06042874fb
2020-04-28 18:35:10 +00:00
Albin Vass bc0ca442a1 Do not set buildset_fact if it's not present in results.json
Change-Id: I9a8790e6f3628924f796b4f3eff052b5e36eff03
2020-04-27 19:53:47 +02:00
Ian Wienand 0b0cb18a60 build-docker-image: add option to install siblings
When you build from a Dockerfile, it runs in a given "context"; that
is the directory the Dockerfile is in and the directories below it.
It can not access anything outside that context during the build.

When building a container for a project in the gate, you may wish to
install sibling projects that Zuul has checked-out into your container
(i.e. so that Depends-On works).  As mentioned, because
/home/zuul/src/<project> is not in the context of the current project,
you will not be able to access this source code during the container
build.

So to help facilitate dependencies, add a siblings: tag which can copy
some or all of the required-projects already specified for the job
into a special sub-directory of the current source.

Because all the code is now in the same context, this will allow build
scripts to be written that look for directories in .zuul-siblings and
can install the source code from there.  To further help the scripts,
the ZUUL_SIBLINGS arg is set for the docker build giving the copied
paths.

The test is updated with some paths to test the copy.

Change-Id: I079d823e7194e15b1b496aea0f53f70f6b563f02
2019-12-03 20:59:45 +11:00
James E. Blair bbfcfe7d20 Add missing conditionals to build-docker-image
We recently added several tasks to the build-docker-image role to
work around docker bugs when using the buildset registry; but this
role doesn't require it so they should be wrapped in a conditional.

Change-Id: Id6e85d07fe34aeb272d7388c778455d5d2a402dd
2019-04-28 13:39:02 -06:00
James E. Blair 6e7797adec Don't repeat the etc/alias setup for buildset registry pushes
This is currently run for each image that is built; we only need
to run it once.

Change-Id: I45cdf6caaaa0e3a5ebc6bc7f0fcb96f3ea47b9a8
2019-04-25 14:33:13 -07:00
James E. Blair 885f02e217 Handle multiple docker images with the same repository
So that users can specify two docker image builds for the same
repository, but with different tags, ensure that the temporary
change_ tag attached to the image also includes the final tag
name.

This allows this configuration to work:

docker_images:
  - repository: foo/image
    context: opensuse
    tags:
      - opensuse-latest
  - repository: foo/image
    context: ubuntu
    tags:
      - ubuntu-latest

Change-Id: I917dcf8a74fc864ea06dc70bdb3e212dc170eb48
2019-02-27 11:08:44 -08:00
James E. Blair 71b7cb0ae5 Update docker image roles
* In the build-image role, push to the buildset registry if it is defined.
* In the intermediate registry push and pull roles, ensure that the
  buildset registry TLS cert is in place.  This is a self-signed cert,
  and so needs to be written for each run.  This happens inside
  bubblewrap where we have permission to write to /etc, which is an
  ephemeral volume.

Change-Id: I47781d8a7adb93817dfe9266e2f4ad5fd829385c
2019-02-16 10:02:48 -08:00
Jean-Philippe Evrard 0e50a2bb89 Allow different filenames for Dockerfiles
In certain build projects, multiple Dockerfiles exist (for example,
one per distro) to simplify reading. However, this role is hardwired
to use dockerfiles only named "Dockerfile".

This is a problem, as you can't override the filename neither per
image, or globally.

This should fix the problem, allowing certain images to be build
by providing the dockerfile argument in docker_images, but also
have a globally overridable flag if you are using a different
convention (for example Dockerfile.distro_minordistroversion).

Change-Id: I075c365bc9f4f85f9ada832d22d1f1e213e68e21
2019-01-26 08:26:06 +00:00
Monty Taylor 0ac452ea28
Apply requested tags locally for docker build
When doing the local build, go ahead and apply the tags to the
local image, so that one can use the role for building local
images for testing that will eventually be published with the given
tag.

Change-Id: I0249ddc4f9a8a2e17466f96a5711672282ce025c
2019-01-23 18:06:47 +00:00
Monty Taylor 2b27f720a2
Pull target from item in build image
In the image building loop, the loop variable is 'item' - which is
where target is located.

Change-Id: I4c9e612f58c1fd92a32da1888af195f17cb821eb
2019-01-22 18:59:50 +00:00
James E. Blair 1c827e4761 docker: prefix role vars
These probably should have been prefixed to start with.  The roles
are brand new, not publicised, and likely not widely used.  I think
we can merge this without announcement or deprecation.

Change-Id: I7825ef6fee1325b6d4fcc179032652eb5530d016
2019-01-17 11:28:37 -08:00
James E. Blair 8640466183 Add docker image promotion roles
This adds three roles which can be used to build a docker image
promotion system.

Change-Id: Iefd9278cdb90bbbaab93a4d23c055e9289fde5ba
2019-01-15 14:03:00 -08:00