New versions of docker are no longer compatible with old versions
of skopeo. To correct this, build a new version of skopeo for
the container images. We need 1.14+ which is not available in
debian yet, so we build 1.15 (the latest tagged release) from
source.
Change-Id: I5a5c351e90b06d3acdd02f3117aa29eafb72445e
This upgrades our base container image from bullseye to bookworm.
It also removes some backported packages that were only needed on
bullseye.
Change-Id: I08c083bf3432531072278de52a06136f6e24e974
Newer bwrap has added the ability to disable additional nested user
namespace creation from with the bwrap execution context. Take advantage
of this feature in Zuul if we are able to in order to fortify Zuul's
security position.
In particular we need two conditions to take advantage of this. 1) bwrap
must be new enough to support the feature (>=0.8.0) and 2) we must be
running with user namespaces enabled. We explicitly check for both
conditions and add the appropriate invocation flags to bwrap when the
conditions are met.
Change-Id: Idf933a0847cb8570b551892186ca9c0057be127f
This allows the use of the PBR_VERSION environment variable when
building container imasges. This facilitates custom version numbers
with builds.
Change-Id: Ib0156836285a798ebe184691d109301bdf751efb
2.0 has breaking api changes:
https://www.sqlalchemy.org/blog/2023/01/26/sqlalchemy-2.0.0-released/
Revert "Update git"
This reverts commit 944b9852c9.
Upstream Debian has updated git to 2.30.2-1+deb11u1 which patches git
for the issues we manually patched to cover. We don't need the manual
patch anymore and can switch to the distro hence this revert.
Reviewers should double check that the image build process installs the
expected 2.30.2-1+deb11u1 version.
Change-Id: I02c4746a0f6651dfcd04ea88ccfd7d62e481d490
Co-Authored-By: Clark Boylan <cboylan@sapwetik.org>
skopeo has started failing with
unsupported MIME type for compression: application/vnd.in-toto+json
since the buildx v.10.0 release [1]. The version in bullseye is a
long way behind upstream, and unfortunately there's no practical way
to backport the package (it would require also pulling in a lot of
exploded out go container dependencies).
Thus we take the alternative solution of just building it ourselves
for the executor image. I couldn't figure out how to build this with
the Debian-packaged go (I think it has something to do with the
aforementioned libraries) but it works with the upstream monolith. It
produces a binary that is linked to a few local libraries, which are
installed here.
[1] https://github.com/containers/skopeo/issues/1874
Change-Id: Iab667a92a5b6e6f8591db2aa435a782913d9d34f
This updates git to address CVE-2022-23521.
Change-Id: Ib08ff1fc7b3c8623fa6b927f3010af72e1b946cf
Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
This updates the openshift client install to use the latest stable
release. Hashes of the oc and kubectl command remain the same which
should continue to allow us to avoid copying both files.
Note we don't fetch the client from the stable-4.11/ path because the
versions of the client under this path are updated when the stable
version updates. Instad we fetch it from the permanent location for the
current stable release (4.11.20/).
Change-Id: Ie78ecd9108f8d6d100479910aa524f867020774f
These binaries are about 115MB each and we copy both of them.
Fortunately they are identical according to hashing routines so we can
save space by copying them once and using a symlink. We choose to make
`oc` the canonical file as these binaries come from openshift.
Change-Id: I3a34acf4ee20db935a471c4fa9ca5e2f7d297d39
This adds python 3.11 testing and drops python3.10 in order to keep
testing only the bounds of what Zuul supports. Note that currently the
python 3.11 available for jammy is based on an RC release. This should
be fine as we do functional testing with a released python 3.11 and that
is what people will consume via the docker images.
Change-Id: Ic5ecf2e23b250d3dbf592983b17ec715d6e9722e
This adds python3.10 testing on Jammy and switches the docker images to
python3.10 from 3.8.
We run sudo for postgres with -Hi to avoid non fatal errors when
postres' client attempts to write command history to Zuul's homedir (it
is running as the postgres user which can't write to zuul's homedir). We
also need to update the libffi package version for jammy to 8 in
bindep.txt. Finally, python_version values need to be quoted as "3.10"
is different than 3.10 which is equivalent to 3.1 when serialized by
yaml as a float.
Force setuptools to use stdlib (shipped by the distro) distutils to
avoid problems with virtualenvs not actually being virtualenvs.
Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy
as the default python there is 3.10.
Change-Id: I97b90bb7a23c90f108f23dda9fdd0e89f9f4dbca
We had been using version 14 which is the previous LTS. Now there are
problems running:
npx browserslist@latest --update-db
running out of memory. Update to the current nodejs LTS version to
ensure we are running an up to date runtime that hopefully performs more
consistently with the browserslist command.
Change-Id: Ib20c1090ea0f30b7dac2780b6ed963dd6e4b6f77
Debian bullseye includes skopeo, so we can drop the kubic repository
(which as I write this is having synchronization errors) in favor of
just using the version from the underlying OS.
Change-Id: Ifde147c8c690dd6a421b0133dbabcff7dd9b9649
Bullseye is out and Buster is old news :) Bump up the base image that we
build Zuul on to Bullseye from Buster. The python version remains the
same but this gives us a more up to date userland. In particular git
goes from 2.20 to 2.30.
Change-Id: I2decbe805a4d0d1daa96de24e317339cee318850
We can use more than one builder image. Use the node image for
the javascript build so we can avoid spending time installing
node in the python builder image.
Change-Id: I19b18dd419b98119f75a8c3a9f3978f37f329e58
Git versions between 2.18 and 2.26 have version 2 protocol support
but it's not enabled by default. Starting in 2.26, it is the
default. V2 is more efficient at negotiating refs and can reduce
the time spent updating a repo by 50% on large repos.
This adds a setting to the container images to specify that v2
should be used. The images are currently built with git 2.20, which
is in the range where this is needed.
Change-Id: Ifc471c5fde49c65137bc34e49734b3ac3d33e3e1
This breaks ensure-twine in opendev which is currently incompatible
with py38.
This reverts commit 429d73e93d.
Change-Id: I983b871d6b42b6d190bc055b57ba0a5f1a4618c0
After dropping support of Ansible 2.7 which has compatibility issues
with python 3.8 we now can finally upgrade to Python 3.8 which has
improvements regarding performance and memory usage.
Change-Id: I346c2fe22f8409c600a3ee2a229369778e3cabd6
There was a bug in opendevorg/python-builder that prevented extras from
being installed properly. Zuul worked around this with an explicit step
to install those extras. The depends on of this change fixes the
python-builder bug so we can remove the workaround from Zuul's
dockerfile.
Depends-On: https://review.opendev.org/744531
Change-Id: I28a783ca0582669c7fe9045aa912908f4229fd29
The js content tarball creation was broken. Instead of leaving the
symlink to a non-existing directory which gets created during python
setup we should just remove that symlink as well and create the
symlink and the static dir during the python setup. This way nothing
will be in the way of the javascript content generation.
This reverts commit eb7b18b38e.
Change-Id: I5f8bfa62cd2d4d9823b86dbcda14885230847a82
Fixed error observed locally when trying to reproduce CI command failure:
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
Configuring openafs-client
--------------------------
AFS filespace is organized into cells or administrative domains. Each
workstation belongs to one cell. Usually the cell is the DNS domain name of
the site.
AFS cell this workstation belongs to: ^C
Change-Id: I5e818d1fcdac360b13947c750c536ed5d0811cb9
This has been incorporated into the upstream image now.
Depends-On: https://review.opendev.org/738204
Change-Id: I9761bdf04c6e7b559e82ffd0d2c1accd62d86e46
We added openafs-krb5 but we need krb5-user for kinit.
Add DEBIAN_FRONTEND=noninteractive to the Dockerfile
to prevent the krb5-user package from asking for our
default realm.
Change-Id: Ifbef43887e541a3edc259ffaf9a75d7343c97dca
virtualenv 20.0.24 creates ~/.local/share/virtualenv with the
seed packages needed for making virtualenvs per-python version.
Creating empty virtualenvs is quick, so run those in sequence
to avoid race possibilities. Then, we can still run the
installs into the virtualenvs in parallel.
We also fix a bug in the console stream functional jobs and install pip
with the use of ensure-pip. This is necessary because the virtualenv
fix runs the stream functional jobs and the update to the stream
functional jobs relies on working docker images.
Change-Id: I3dec251d19dd7b3807848a54e6a20a8e89d30a4e
We are consuming OpenDev's default python-builder and python-base images
which happen to be python3.7 today. Zuul specifically wants python3.7 so
we should explicitly use OpenDev's 3.7 tags of these images. This way if
OpenDev updates the default to 3.8 zuul can continue to assert its
dependency on 3.7.
Change-Id: I1f344eccfddff9bf1050c571604038371eec1a3f
Ansible 2.7 is known to be broken with python 3.8 so switch back to
python 3.7 for now until we dropped support for Ansible 2.7.
Change-Id: Ibfe1b226630c8db202a294590c6435666afcb62d
We suspect a memory leak in python 3.7. BMW saw one and reverted
to 3.6 and has been better. OpenDev is seeing one which seems to
be improved by sending SIGUSR2 signals to the process, which makes
us think there may be an issue in the GC.
Try updating to 3.8. If it fixes it, we can just shrug and say
"3.7 lol". If it doesn't, we can follow BMW's lead and try 3.6.
Change-Id: Iaa53fa2b3125dd3a2c79ba25191b2b44ed473200
The container roles in zuul jobs use skopeo to copy to and from
the intermediate registry. It is therefore helpful to have
skopeo in the images.
Change-Id: I1550b2eeca6cb1755976dd884e668bddf960f296
python-base is buster now, so this is not necessary. We can
just install bubblewrap and socat from buster directly.
Change-Id: If65f3c2d6367a7c79cf9d6d8f788021ba72cccd4
When we get a pod from nodepool, this starts a kubectl port-forward
on the pod so that zuul-console and the normal method of streaming
command output will work.
Change-Id: Iae85347c3d8e0a74e330a7b62b513c7b41641383
Story: 2007321
Task: 38832
Depends-On: https://review.opendev.org/709259
The current options stated as 'do not run as a daemon' is actually
'run in debug mode in foreground'. When running in container we
actually want an option for running normally in foreground. Thus add a
new option -f for foreground operations and change the docker imaged
to use this accordingly.
Change-Id: I16173a73dbfb79dc2c2b05c2002ac41e20a48225
The binaries were only installed on the builder image. This change
installs them on the executor image.
Change-Id: I86f7102218a77908c02b7da8a913fd20d417794d
Rather than relying on the setup hook to build the javascript
on demand, we always want to build the javascript in the container
context, so just run yarn explicitly.
Change-Id: Iaf456a78b5f798d3b4d05aa3925beb920dc56258
This installs oc and kubectl (really oc masquerading as kubectl)
into the zuul-executor container image, so that Ansible kubectl
connections work as expected.
Change-Id: Ib40cacea6751674346154457db4187e8600788d5
Ara and openstacksdk are installed into the ansible virtualenvs now so
there is no need to pull them as executor specific dependencies.
Change-Id: I3aeac278b8e1f4bb796e0f3a3f475b3c97208a85
As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.
The actual usage of multiple ansible versions will be done in
follow-ups.
For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.
Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
Depends-On: https://review.openstack.org/623927
Most configuration options depend on storing state data inside
the /var/lib/zuul path which does not exist by default, causing
a lot of configurations to not work out of the box.
This patch creates the folder in the zuul base image, which will
reduce the number of options to be moved around to store state.
Change-Id: I22d05ca26324ffc02cd0b8d2a3059247a21ffdd9
The Service Workers seem to be consistently causing issues for people
that are strange, meaning many of our deployers are disabling them.
Since they aren't super necessary for the Zuul use case, change the
default behavior to be to disable them instead of enable them.
Change-Id: Iea8348a3b007badaae74fc1837b55bb0b076ac65
James E. Blair