Commit Graph

26 Commits

Author SHA1 Message Date
Ian Wienand 65aa2bdc1f use-buildset-registry: support microk8s
This enables microk8s/containerd to pull through the intermediate zuul
registry.  This is tested with the new
zuul-jobs-test-registry-buildset-registry-k8s-microk8s job.

Change-Id: I5a6c0d63a6ba0acf94ab9f0ef94777fab58fec6e
2023-01-09 10:37:45 -08:00
Ian Wienand b2ae863f50
Add names to include tasks
Latest ansible-lint is finding this.  It seems reasonable enough to
ensure the task is named; it's always nice to have context about what
is happening as you read the file.

Change-Id: Ia7e490aaba99da9694a6f3fdb1bca9838221b30a
2022-11-07 10:37:53 +11:00
Clark Boylan 4c40b92950 Prevent leaks of buildset registry credentials
Because buildset registries may be used by jobs that finish before other
jobs are finished using the buildset registry we must be careful not to
expose the registry credentials in the jobs that finish sooner.
Otherwise logs for the earlier job runs could potentially be used to
poison the registry for later jobs.

This is likely currently incomplete. Other Zuulians should look over it
carefully to ensure we're covering all the bases here.

The cases I've identified so far are:

* Setting facts that include passwords
* Reading and writing to files that include passwords (as content may be
  logged)
* Calling modules with passwords passed as arguments (the module
  invocation is logged)

I've also set no_log on zuul_return that passes up credentials because
while the logging for zuul_return is minimal today, I don't want to
count on it remaining that way.

We also use the yet to be merged secret_data attribute on zuul_return to
ensure that zuul_return itself does not expose anything unwanted.

Finally it would be great if others could check over the use of
buildset_registry variables to make sure there aren't any that got
missed. One thing I'm not sure of is whether or not when conditionals
get logged and if we need to be careful about their use too.

Temporarily remove some buildset-regitry jobs which are in a catch-22.

Change-Id: I2dea683e27f00b99a7766bf830981bf91b925265
2021-06-24 09:56:19 -07:00
Sorin Sbarnea 0eaa5cf59a Upgrade ansible-lint to 5.0
- bumps ansible-lint to 5.0
- updates our custom rules to make them compatible with 5.0
- replace custom module mocking with native ansible-lint ones
- remove custom call of ansible-playbook --syntax-check as now this
  is done by ansible-lint
- assured molecule vars are hosted under a vars/ folder in order to
  avoid confusing linter detection.
- replaced custom rule for loop var names in role as now this this an
  optional core feature of the linter (see config)
- replaced custom rule no-same-owner with opt-in one (see config)

Change-Id: I233fae8c9036d295968a97ee80e07fde8846c633
2021-03-17 15:44:43 +00:00
Sorin Sbarnea 856866fdde More E208 mode fixes
Change-Id: I8157ec1f31b8c5a064b63002e8311b91ef9ce9ab
See: https://ansible-lint.readthedocs.io/en/latest/default_rules.html#file-permissions-not-mentioned
2020-10-31 07:34:03 +00:00
Andrii Ostapenko fd2dc3bc91
Fix certificate issue with use buildset registry
Use buildset_registry_alias for certificate file name to address
'unknown certificate authority' issue on consumer side when using standalone
use-buildset-registry role.

Issue is not visible with opendev parent jobs because certificate is handled
in pull-from-intermediate-registry role there.

Change-Id: Ieaf43be0dd4ccb92a8240a493ee6636a23c9d484
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-29 10:24:23 -05:00
Zuul c8bf4fc5e8 Merge "use-buildset-registry: do not update ca when not necessary" 2020-05-05 20:22:57 +00:00
Monty Taylor 70db82198c Write buildkitd.toml in use-buildset-registry
It's a generally useful file for people using buildkitd. It's also
more appropriate to write it in use-buildset-registry and then
just have build-docker-image copy it.

Do the same thing with writing the cert - we don't need to know
which path on the host use-buildset-registry wrote the file to,
we can just write the content from the dir into the container,
and then it's just a consistent command to apply it.

Change-Id: Iaa485c2e8628900dccbed1f4b0773b6d1b5f7983
2020-05-05 14:47:19 -05:00
Tristan Cacqueray e16e89758c use-buildset-registry: do not update ca when not necessary
This change skip the update ca task when the certificate
is not modified.

Change-Id: I77d6d8771197d027774223ff5c81ab7d737672fb
2020-05-01 13:40:38 +00:00
Albin Vass 9062289151 Check for loop_control in with_ type loops
Change-Id: I191265df7709a6262b44a428d78fe28ffaeb4b75
2020-05-01 13:45:34 +02:00
Tristan Cacqueray 7413c0577a use-buildset-registry: wait for kubernetes service to be available
When kubernetes services exists, this change waits for the api to be
available.

Change-Id: Id615c4d1cc296d73b884104443d9439cc608b84a
2020-04-13 19:27:27 +00:00
James E. Blair b6cb191295 Add cri-o support to use-buildset registry
This lets use-buildset-registry notify cri-o about the new
registries.conf file if it is being used as the container backend
for k8s.

Change-Id: Ia1805519ab4b6bb5f79df0492f702effc6a3e024
2020-01-07 07:35:28 -08:00
James E. Blair 6a9aa82fcc openshift speculative containers
Change-Id: I81f7be6065a74f45386ea735a5291d48b7e042ce
2019-12-06 13:50:25 -08:00
James E. Blair d48667a3c5 use-buildset-registry: Add podman support
Change-Id: I41718073962c8e7eb3d8810276e550fb84bd6e99
2019-12-05 11:44:15 -08:00
James E. Blair 8766890012 Use zuul-registry as buildset registry
The proxy functionality is no longer needed so it is removed.

Change-Id: I29ff75d331b433ea4ad3b66ed723eee14a90b404
Depends-On: https://review.opendev.org/689829
2019-10-28 15:42:26 -07:00
Jeremy Stanley 2711d10186 Be consistent about spaces before and after vars
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.

Adjust the new violations accordingly and remove the rule
206 exclusion.

Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
2019-08-07 20:17:41 +01:00
James E. Blair f855033f45 Don't use underscore in buildset_registry_alias
Docker won't recognize this as a hostname in that case.

Change-Id: Ibbaef5d5ca145f956e3d546e9f3774ddff1fface
2019-04-22 11:39:42 -07:00
Clark Boylan 23826e05a6 Only alias docker registries on test nodes
Skopeo has problems with ipv6 address literals just like docker as they
use the same underlying checks for url validity. However, we think we
can fix that by using a port forward from the executor to the buildset
registry so that skopeo can connect via ipv4

Go back to aliases the registries on test nodes via /etc/hosts.

Change-Id: I5f9316ffe84de06cb2fb2b65a7e1c31d9f8b0e35
Co-Authored-By: James E. Blair <jeblair@redhat.com>
2019-04-22 10:37:33 -07:00
Clark Boylan ba4b2c68b9 Revert "Only alias registries on build nodes"
This reverts commit 05f20a5396.

Apparently skopeo is properly cloud native too and doesn't support ipv6
either. I think it is pulling the same docker
distribution/reference/regexp.go lib in and using docker's regex.

The error we get from skopeo:

 time="2019-04-10T15:15:48Z" level=fatal msg="Invalid source name docker://[2607:ff68:100:54:f816:3eff:fef2:fc69]:5000/zuul/nodepool:latest: invalid reference format"

Change-Id: I6f916574c9f46e8fdd2464465e2b36ecf8719b16
2019-04-10 08:22:04 -07:00
Clark Boylan 05f20a5396 Only alias registries on build nodes
We only need to alias registries on the build nodes when running docker.
We cannot alias them in /etc/hosts in roles that are expected to run on
localhost beacuse /etc/hosts is bindmounted read only on localhost. This
assumes that skopeo handles ipv6 properly (which has not been tested).

If skopeo does not handle ipv6 properly then we'll need additional
fixing.

Change-Id: I40e5b1bac5aeaf2d42aa05a72b9ced72b7d222c0
2019-04-09 14:45:19 -07:00
Clark Boylan 3a29935f32 Revert "Use /etc/hosts with docker registries to handle ipv6"
This reverts commit a307259776.

We need to additionally handle this on localhost (the zuul executor)
where we cannot edit /etc/hosts. At least I don't think we can. We also
need to handle the case where buildset_regsitry is not yet defined.

Change-Id: I4928f7fcf58e88cf360de253f01b16546220aace
2019-04-09 10:10:03 -07:00
Clark Boylan a307259776 Use /etc/hosts with docker registries to handle ipv6
Docker doesn't appear to understand properly escaped ipv6 addrs in its
"urls". Address this by adding /etc/hosts entries for any
buildset_registry that is specified by an ip address (v4 or v6). This
allows us to use a named alias instead of the ipv6 address.

An example failure for posterity:

  "[2607:ff68:100:54:f816:3eff:fe7c:e98a]:5000/zuul/nodepool:latest" is not a valid repository/tag: invalid reference format

Change-Id: Id865dc7d3382174b61f9eaa76e29b637a85f5142
2019-04-08 13:40:58 -07:00
James E. Blair 325b666b6d Use ipwrap filter in registry roles
We need to escape some ipv6 addresses in URLs.

Change-Id: Ica5fe73dd65e138cdc4817de3903cdd271402941
2019-04-05 16:15:02 -07:00
James E. Blair 9c0d25f349 Fix buildset registry
The approach of having the proxy serve the local data as well as
the remote wasn't working -- it seems that the proxy would always
check upstream and prefer that data even if it had been pushed
locally.

To correct this, separate the data stores of the two registries,
and add both of them to the registry_mirror setting for the
docker daemon.  Now we will pull from our buildset registry first,
and fall back on the proxy to talk to upstream if an image is not
found locally.

The proxy is still required in order to mask out the username and
password which dockerd will otherwise use when talking to upstream.

Change-Id: Iab11954a4b5431d3b1a4d4753f519b6b71f64094
2019-03-01 15:52:01 -08:00
James E. Blair 42df455705 use-buildset-registry: support running before docker installed
To accomodate running in a production-simulation environment,
make it safe to run this role on a host before docker is installed.

This also adds support for the new dual-registry configuration
that run-buildset-registry uses.

This removes the region-local proxy from the registry-mirrors
configuration.  Because the buildset registry acts as a pull-through
proxy, the region-local proxy won't be used even if we did include it.
Instead, we should update the run-buildset-registry role to proxy
to the region-local proxy if present.

Change-Id: I21011a3708f17ee61afd0034d90d75e8dc885575
2019-02-21 14:07:33 -08:00
James E. Blair 0f6796d12b Add role to use buildset registry
Change-Id: Ieacbd033c49afd0bac11cdb17891386c9e1d6a4b
2019-02-01 13:26:03 -08:00