Commit Graph

229 Commits

Author SHA1 Message Date
Zuul ad1ef8e862 Merge "Add python 3.12 testing to Zuul" 2024-03-25 17:37:19 +00:00
James E. Blair dff9bba490 Add container image sanity checks
Since we're building skopeo ourselves in the container image, and
we don't have any functional testing of that, perform a sanity check.

Change-Id: I362e97d1120f3ce6c6be85ccf6948ced763b6cc1
2024-03-22 06:32:56 -07:00
Clark Boylan 20657c13bd Add python 3.12 testing to Zuul
We use pyenv to install python3.12 on ubuntu jammy (which doesn't have
native packages for this version of python). We don't update the
container images just yet as we want to build confidence through testing
before end users see the new python version in production.

Change-Id: I20a57f6d29699886867ce912e9943c59aa9963f4
2024-03-20 09:17:19 -07:00
James E. Blair 5a8e373c3b Replace Ansible 6 with Ansible 9
Ansible 6 is EOL and Ansible 9 is available.  Remove 6 and add 9.

This is usually done in two changes, but this time it's in one
since we can just rotate the 6 around to make it a 9.

command.py has been updated for ansible 9.

Change-Id: I537667f66ba321d057b6637aa4885e48c8b96f04
2024-02-15 16:20:45 -08:00
James E. Blair 6ecfe5912c Allow circular dependencies in the Zuul project
Since we're currently merging significant changes related to
circular dependencies, the Zuul community might want to experiment
with behavior on a live production system.  Enable them for the
"zuul" shared queue for testing purposes only.

The Zuul maintainers should still not approve changes that would
impede our ability to continuously deploy Zuul and its components.

Change-Id: I65bc34fb1040a4a3083a9941e9006fbe24d8a93b
2024-02-07 14:57:03 -08:00
Zuul bf889081d8 Merge "Use bookworm container images" 2023-09-06 16:28:12 +00:00
James E. Blair 12f9eef44a Add failure-output option to zuul-nox-py311 job
This uses the new regex-based early detection feature to mark
the nox-py311 job as failing while it's still running.

Change-Id: I3fb23d51ccc02e0db4b288e9f343c6e282af2cb6
2023-08-29 16:08:01 -07:00
James E. Blair a7f1aba3b2 Use bookworm container images
This upgrades our base container image from bullseye to bookworm.

It also removes some backported packages that were only needed on
bullseye.

Change-Id: I08c083bf3432531072278de52a06136f6e24e974
2023-08-28 10:13:53 -07:00
James E. Blair ce37097e11 Use Python 3.11 when publishing docs
This is required due to the python classifier change, and matches
the check/gate jobs.

Change-Id: I856683a7cde290d36c81cff0755f5f250ea1449e
2023-08-16 11:08:36 -07:00
James E. Blair ae544efe21 Use Python 3.11 for stream-functional jobs
We need to use Python 3.11 for all of our jobs now, so run these
jobs on Debian bookworm.

Change-Id: I302e98a24646f2323da64ca506a558529da498f9
2023-07-29 08:54:33 -07:00
James E. Blair 86e86ffbd8 Add a release note about Python 3.11
This is now the only version of Python tested with Zuul.  Note that
change and also update the pypi classifiers.

That requires that we run all the nox jobs with Python 3.11, so they
are updated as well.

Change-Id: I9bb7514d5eab32fed814a5a054cc6be642e91ab4
2023-07-27 17:09:54 -07:00
James E. Blair 60a8dfd451 Add Ansible 8
This is the currently supported version of Ansible.  Since 7 is out
of support, let's skip it.

Change-Id: I1d13c23189dce7fd9db291ee03a452089b92a421
2023-07-19 15:46:48 -07:00
James E. Blair e342d25cb9 Use buildx to build images
This is required in order to have speculative image builds with
images on quay.io.

Change-Id: I993ee98e160c27f98a363779ccd0630895935712
2023-05-22 06:55:24 -07:00
Tristan Cacqueray 0e3756bd5c Replace docker by podman for quick-start
Change-Id: Ic6ea0e350e7a6aae656823b4294e49ee871b3329
Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/883593
2023-05-22 06:54:41 -07:00
Clark Boylan c1b0a00c60 Only check bwrap execution under the executor
The reason for this is that containers for zuul services need to run
privileged in order to successfully run bwrap. We currently only expect
users to run the executor as privilged and the new bwrap execution
checks have broken other services as a result. (Other services load the
bwrap system bceause it is a normal zuul driver and all drivers are
loaded by all services).

This works around this by add a check_bwrap flag to connection setup and
only setting it to true on the executor. A better longer term followup
fixup would be to only instantiate the bwrap driver on the executor in
the first place. This can probably be accomplished by overriding the
ZuulApp configure_connections method in the executor and dropping bwrap
creation in ZuulApp.

Temporarily stop running the quick-start job since it's apparently not
using speculative images.

Change-Id: Ibadac0450e2879ef1ccc4b308ebd65de6e5a75ab
2023-05-17 13:45:23 -07:00
Clark Boylan 57cdfc978f Add container creation details
This adds extra bits of info that the ensure-quay-repo role expects to
be present.

Change-Id: I581c667515b5ee2eec4ee79e9f2b485247711195
2023-04-28 15:04:46 -07:00
James E. Blair 8ec8863eb8 Publish container images to quay.io
Change-Id: I142c99a5469fd5af5e558619cd9a4acc400221ff
2023-04-21 09:29:15 -07:00
James E. Blair 3ae02a193c Switch to nox-docs
Depends-On: https://review.opendev.org/868228
Change-Id: I95dd6f751bd3d64a146ed32ec660e48dbe473d81
2022-12-20 09:21:39 -08:00
James E. Blair 6c8855ede4 Add noxfile and switch to nox
Tox v4 behaves significantly differently than v3, and some of the
more complex things we do with tox would need an overhaul to
continue to use it.  Meanwhile, nox is much simpler and more
flexible, so let's try using it.

This adds a noxfile which should be equivalent to our tox.ini file.
We still need to update the docs build (which involves changes to
base jobs) before we can completely remove tox.

Depends-On: https://review.opendev.org/868134
Change-Id: Ibebb0988d2702d310e46c437e58917db3f091382
2022-12-20 08:57:53 -08:00
James E. Blair 07ab843dd3 Pin tox to 3
There are many issues with toxv4 that make it difficult to use with
Zuul.  Pin to tox version 3 while we find a solution.

Change-Id: I608b2ad4ab9407d8a0b77d5def5188922875e00f
2022-12-15 13:08:02 -08:00
Clark Boylan 9b2b8f82ff Add Python 3.11 testing
This adds python 3.11 testing and drops python3.10 in order to keep
testing only the bounds of what Zuul supports. Note that currently the
python 3.11 available for jammy is based on an RC release. This should
be fine as we do functional testing with a released python 3.11 and that
is what people will consume via the docker images.

Change-Id: Ic5ecf2e23b250d3dbf592983b17ec715d6e9722e
2022-10-27 16:47:14 -07:00
James E. Blair f9eb499870 Remove Ansible 5
Change-Id: Icd8c33dfe1c8ffd21a717a1a94f1783c244a6b82
2022-10-11 17:03:57 -07:00
James E. Blair 2d6b5c19ba Remove support for Ansible 2
Versions 2.8 and 2.9 are no longer supported by the Ansible project.

Change-Id: I888ddcbecadd56ced83a27ae5a6e70377dc3bf8c
2022-09-14 17:14:10 -07:00
James E. Blair 7949efd255 Add Ansible 6
Change-Id: I0d450d9385b9aaab22d2d87fb47798bf56525f50
2022-09-02 10:12:55 -07:00
James E. Blair 30fdd2c45b Add zuul change queue
So that we can add zuul-client to it.

Change-Id: Ia99d135fc6016895fe5f85ea93efeccc88f85a80
2022-08-16 10:35:50 -07:00
Zuul 78b14ec3c1 Merge "Bump up to python3.10" 2022-07-05 10:05:37 +00:00
Clark Boylan d665053fe9 Bump up to python3.10
This adds python3.10 testing on Jammy and switches the docker images to
python3.10 from 3.8.

We run sudo for postgres with -Hi to avoid non fatal errors when
postres' client attempts to write command history to Zuul's homedir (it
is running as the postgres user which can't write to zuul's homedir). We
also need to update the libffi package version for jammy to 8 in
bindep.txt. Finally, python_version values need to be quoted as "3.10"
is different than 3.10 which is equivalent to 3.1 when serialized by
yaml as a float.

Force setuptools to use stdlib (shipped by the distro) distutils to
avoid problems with virtualenvs not actually being virtualenvs.

Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy
as the default python there is 3.10.

Change-Id: I97b90bb7a23c90f108f23dda9fdd0e89f9f4dbca
2022-07-01 15:59:48 -07:00
James E. Blair ac67280c00 Increase unit test timeout to 2 hours
We're currently occasionally bumping into our limit of 90m and
it's starting to be a problem as we incrementally add more tests.

Increase the timeout to 2 hours.

Change-Id: I34a272e6905c4058f9d92751e4a7c2a4fa126fcb
2022-05-25 12:08:29 -07:00
James E. Blair ebf5c96d57 Add support for Ansible 5
This adds support for Ansible 5.  As mentioned in the reno, only
the major version is specified; that corresponds to major.minor in
Ansible core, so is approximately equivalent to our current regime.

The command module is updated to be based on the current code in
ansible core 2.12.4 (corresponding to community 5.6.0).  The previous
version is un-symlinked and copied to the 2.8 and 2.8 directories
for easy deletion as they age out.

The new command module has corrected a code path we used to test
that the zuul_stream module handles python exceptions in modules,
so instead we now take advantage of the ability to load
playbook-adjacent modules to add a test fixture module that always
raises an exception.  The zuul stream functional test validation is
adjusted to match the new values.

Similarly, in test_command in the remote tests, we relied on that
behavior, but there is already a test for module exceptions in
test_module_exception, so that check is simply removed.

Among our Ansible version tests, we occasionally had tests which
exercised 2.8 but not 2.9 because it is the default and is otherwise
tested.  This change adds explicit tests for 2.9 even if they are
redundant in order to make future Ansible version updates easier and
more mechanical (we don't need to remember to add 2.9 later when
we change the default).

This is our first version of Ansible where the value of
job.ansible-version could be interpreted as an integer, so the
configloader is updated to handle that possibility transparently,
as it already does for floating point values.

Change-Id: I694b979077d7944b4b365dbd8c72aba3f9807329
2022-04-14 13:33:53 -07:00
James E. Blair 6360bf85a8 Update testing to Python 3.8 and 3.9
Python 3.6 is EOL and Ansible 5 does not support it.  Bump our
testing to 3.8 + 3.9 to match Nodepool.

Change-Id: Ica6094c05afb87cd5bbc8752c6362d89c13eca0a
2022-04-14 13:33:39 -07:00
James E. Blair 2a8b29aa94 Remove built-in ARA support
This has been pinned to a very old version of ARA for some time, and
newer versions of Ansible are no longer compatible with the old version
of ARA.  Since this isn't receiving maintenance keeping it up to date,
remove it.

Note that if there is desire for support for this or other callback
plugins, it would be quite reasonable and relatively straightforward
to add the ability to generically configure additional callback plugins.
This would have the advantage of not requiring tight internal integration
between Zuul and other callback plugins.  Such a change would likely
be welcome.

Change-Id: I733e48127f2b1cf7d2d52153844098163e48bae8
2022-04-13 16:44:34 -07:00
Clark Boylan bb52902303 Update nodejs runtime to version 16
We had been using version 14 which is the previous LTS. Now there are
problems running:

  npx browserslist@latest --update-db

running out of memory. Update to the current nodejs LTS version to
ensure we are running an up to date runtime that hopefully performs more
consistently with the browserslist command.

Change-Id: Ib20c1090ea0f30b7dac2780b6ed963dd6e4b6f77
2022-03-14 15:29:31 -07:00
Ian Wienand f592c31263 nodepool-zuul-functional: remove
This job has been broken for a long time, and the paths it tests are
better covered by the quickstart tests which bring up nodepool with
Zuul.

It's a bit of an odd job because nodepool sets itself up, but then the
test calls the tox "nodepool" environment in Zuul here.  So remove the
job inclusion, but also the tox/unit-tests being run by the job (it's
already failing and non-voting on nodepool, so this won't affect
things).

Change-Id: I8483b1d66a6a58d4bd2f2fce82b023d8f0446ae7
2022-01-28 08:18:02 +11:00
Zuul 24acd74776 Merge "Bump test nodes up to Focal" 2022-01-26 01:38:28 +00:00
Clark Boylan b3a5a77d60 Bump test nodes up to Focal
We were running on Bionic because Zuul's inclusion of a pinned Gear
conflicted with TLS policy on Focal. With Gear gone we can bump up to
Focal safely now.

Followup changes can bump testing platforms ahead to 3.9 or newer as
well.

Change-Id: I4cfef79ebc97753994edaf36a1deca0d3b37ad17
2022-01-25 16:08:56 -08:00
James E. Blair c7928742a0 Revert "Don't run non-docs jobs on docs changes"
This reverts commit 23dc619838.

The immediate need to rebuild docs quickly has passed, we can
return to our regular config now.

Change-Id: I2896b2517cacfbc553add3ca34066db96df551a9
2022-01-20 14:13:49 -08:00
James E. Blair 23dc619838 Don't run non-docs jobs on docs changes
This is a temporary change to avoid running a bunch of jobs for
docs-only changes so we can iterate on doc changes quickly.

Change-Id: I2e700fe1fcd631f7095d95fcfc9b57685ae3cd70
2022-01-20 07:39:47 -08:00
Zuul 3f86be1d16 Merge "Add nonvoting multischeduler job" 2021-12-01 02:49:04 +00:00
Zuul 9235323880 Merge "Remove success-url and failure-url" 2021-11-30 19:34:50 +00:00
Clark Boylan b15636d95d Add nonvoting multischeduler job
This adds a nonvoting python38 tox unittest job that runs with
ZUUL_SCHEDULER_COUNT set to 2. We do this despite knowing the job will
probably fail so that we can start getting this consistent feedback on
changes in pre merge testing.

Change-Id: I08fac586a1a7140433f225988e490a1054cc69dd
2021-11-30 08:58:28 -08:00
Clark Boylan 9b7b42b627 CI image requires consistency cleanup
We build the docker images on bullseye not buster so we should set the
job requires values to bullseye specific artificats.

Change-Id: Ic29d9053524777b39aea436b7d5b9cee0296bd85
2021-10-25 20:16:20 +00:00
James E. Blair 072cdf7885 Remove success-url and failure-url
These options are deprecated and ignored by Zuul.

Change-Id: Ie0fd1895f61ea3d83db9d5a7b5e1c68781388d53
2021-07-29 16:45:52 -07:00
James E. Blair e0a8c37d3d Increase unit test job timeout to 90 minutes
In one particular cloud region, we reliably get nodes slower than
others, and appear to just barely timeout the job.  Increase the
timeout to 90 minutes to give us more room.

Change-Id: I3dc2a60037a9bdf20694d2811a9570ebe4f2f600
2021-06-24 13:16:11 -07:00
Albin Vass 9c79d86988 Only promote zuul-build-dashboard when it's built
Change-Id: I7aadaf5337ea0dfd3db96df77dc33f9ed854be48
2021-03-23 17:51:32 +01:00
James E. Blair 74a9c9de9b Use ZooKeeper TLS in tests
This mirrors the configuration in Nodepool for using TLS-enabled
ZooKeeper in tests.  We use the ensure-zookeeper role in order
to get a newer ZooKeeper than is supplied in bionic.

Change-Id: I14413fccbc9a6a7a75b6233d667e2a1d2856d894
2021-03-08 06:49:57 -08:00
James E. Blair f1e7ffab93 Remove zuul-migrate
This was intended as a one-time helper program to help people upgrade
from Zuul v2 to v3.  It did not cover all use cases, and has not been
kept up to date or improved.  It's time to remove it before the v4
release.

Change-Id: I12cdcedb5baabd8fa0937a6ea21590259093ead1
2021-02-17 09:22:46 -08:00
Jan Kubovy d518e56208 Prepare Zookeeper for scale-out scheduler
This change is a common root for other
Zookeeper related changed regarding
scale-out-scheduler. Zookeeper becoming
a central component requires to increase
"maxClientCnxns".

Since the ZooKeeper class is expected to grow
significantly (ZooKeeper is becoming a central part
of Zuul) a split of the ZooKeeper class (zk.py) into
zk module is done here to avoid the current god-class.

Also the zookeeper log is copied to the "zuul_output_dir".

Change-Id: I714c06052b5e17269a6964892ad53b48cf65db19
Story: 2007192
2021-02-15 14:44:18 +01:00
Guillaume Chauvel 83b2d68a91 tutorial: Rework quick-start and prepare for other tutorials
-quick-start steps are modified and fit more to what a reader would do
-quick-start test code is mainly splitted into 2 files, one which is a
 setup part as a role, the second one starts with cloning the test
 repository, just like all followings tutorial will do
-some elementary steps when manipulating or checking gerrit are being
 added as roles

tutorial ssh config: test ssh configuration has been modified to allow
 using a known_hosts file for both someone executing localtest and
 opendev.org's zuul. A reader executing the tutorial would still have to
 accept the fingerprint. To do so, commit-msg hook is fetched manually,
 otherwise it would be downloaded by git-review throught scp. Alas,
 git-review doesn't allow to pass options to scp to provide a new
 known_hosts file.
 User's ssh key is used if ~/.ssh/id_rsa.pub is available, otherwise use
 a generated one.

- "to_json | from_json | json_query" in test is due to an issue between
 ansible and jmespath [1]

[1] https://github.com/ansible-collections/community.general/issues/320

Change-Id: Id5c669537ff5afc7468352139980ebade167d534
2021-01-11 17:02:38 +01:00
James E. Blair 14a86c4d89 Revert "Revert "Update images to use python 3.8""
This reverts commit fe58e85bb1.

We resolved the twine issue with https://review.opendev.org/#/c/755670/

Depends-On: https://review.opendev.org/#/c/755670/
Change-Id: Iab73cf93dd7556609f7bddb13ab06ac7ab879d6b
2020-10-01 17:23:15 -06:00
Tobias Henkel fe58e85bb1
Revert "Update images to use python 3.8"
This breaks ensure-twine in opendev which is currently incompatible
with py38.

This reverts commit 429d73e93d.

Change-Id: I983b871d6b42b6d190bc055b57ba0a5f1a4618c0
2020-10-01 15:57:20 +02:00