Ansible 6 is EOL and Ansible 9 is available. Remove 6 and add 9.
This is usually done in two changes, but this time it's in one
since we can just rotate the 6 around to make it a 9.
command.py has been updated for ansible 9.
Change-Id: I537667f66ba321d057b6637aa4885e48c8b96f04
In I9628e2770dda120b269612e28bb6217036942b8e we switched zuul.change from
a plain string tagged with !unsafe to base64 encoded and no !unsafe tag.
The idea was to make the inventory file parseable by external tools while
avoiding accidental interpolation of the commit message by Ansible.
That doesn't work in all cases -- it's not hard to construct a scenario
where after base64 decoding the message any further processing by Ansible
causes it to undergo interpolation. Moreover, since then we have made
many changes to how we deal with variables; notably, the inventory.yaml
is no longer actually used by Zuul's Anisble -- it is now there only
for human and downstream processing. We call it the "debug inventory".
The actual inventory is much more complex and in some cases has lots of
!unsafe tags in it.
Given all that, it now seems like the most straightforward way to deal
with this is to tag the message variable as !unsafe when passing it to
Zuul's Ansible, but render it as plain text in the inventory.yaml.
To address backwards compatability, this is done in a new variable called
zuul.change_message. Since that's a more descriptive variable anyway,
we will just keep that one in the future and drop the current base64-
encoded zuul.message variable
Change-Id: Iea86de15e722bc271c1bf0540db2c9efb032500c
Inventory testing was opening yaml files to parse them and not
explicitly closing them when done. Fix this through the use of with
open() context managers.
Change-Id: I41a8ee607fcf13e86dd800cefb00d7e120265ed4
The Ansible version is sometimes used for selecting the correct linter
or for implementing feature switches to make roles/playbooks backward
compatible.
With the split of Ansible into an "ansible" and "ansible-core" package,
the `ansible_version` now contains the version of the core package.
There seems to be no other variable that contains the version of the
"Ansible community" package that Zuul is using.
In order to support this use-case for Ansible 5+ we will add the Ansible
version to the job's Zuul vars.
Change-Id: I3f3a3237b8649770a9b7ff488e501a97b646a4c4
Zuul ignores nodes with network_cli connections when running the
setup playbook in order to support the use-case where a network
appliance begins a job in an unreachable state.
Starting with Zuul v4.6.0, for security reasons, host variables
require a functioning Ansible connection in order to be set.
This includes the Nodepool variables such as public and private
IP addresses. Causing a network_cli host to become online after
the start of a job requires this information, and so this
functionality has not worked since 4.6.0.
To correct this, we now always add in the nodepool host vars as
originally set regardless of whether the variable freeze playbook
runs for a particular host. This means that even if we have no
other variables set for that host, we at least know the IP address
of the host and can interact with it directly in order to bring it
online.
Additionally, the freeze playbook did not correctly check the
return value, due to a typo. This accidentally allowed Zuul to
continue to function in this situation (but without the nodepool
variables accessible). With this change we continue to ignore
the return value, but intentionally so.
A test for this use-case is added, along with a releaso note.
Change-Id: Icd8a2c035e6c04a7c198281adbd07fef422a6c63
Story: 2009226
This adds support for Ansible 5. As mentioned in the reno, only
the major version is specified; that corresponds to major.minor in
Ansible core, so is approximately equivalent to our current regime.
The command module is updated to be based on the current code in
ansible core 2.12.4 (corresponding to community 5.6.0). The previous
version is un-symlinked and copied to the 2.8 and 2.8 directories
for easy deletion as they age out.
The new command module has corrected a code path we used to test
that the zuul_stream module handles python exceptions in modules,
so instead we now take advantage of the ability to load
playbook-adjacent modules to add a test fixture module that always
raises an exception. The zuul stream functional test validation is
adjusted to match the new values.
Similarly, in test_command in the remote tests, we relied on that
behavior, but there is already a test for module exceptions in
test_module_exception, so that check is simply removed.
Among our Ansible version tests, we occasionally had tests which
exercised 2.8 but not 2.9 because it is the default and is otherwise
tested. This change adds explicit tests for 2.9 even if they are
redundant in order to make future Ansible version updates easier and
more mechanical (we don't need to remember to add 2.9 later when
we change the default).
This is our first version of Ansible where the value of
job.ansible-version could be interpreted as an integer, so the
configloader is updated to handle that possibility transparently,
as it already does for floating point values.
Change-Id: I694b979077d7944b4b365dbd8c72aba3f9807329
These tests use internal Zuul functions from outside Zuul and
unsurprisingly, that breaks our assumptions about holding locks
and ZK contexts.
Rather than trying to get a pipeline lock and ZK context into this
test, just make it a more accurate simulation by abandoning the
changes instead of directly canceling execution.
Change-Id: I8f6bb6cac1ca78cd63ca7ac67384056dea522474
This removes the local builds list (executor.builds) from the executor
client.
To make this work, we have to adapt a few tests which are still using
this list. To provide a similar functionality for those tests, the
ZuulTestCase class now provides a method getCurrentBuilds() that looks
up the builds from the active pipelines/queues.
Change-Id: If74bc7164d1156cdb7ba06f12868c60bf1617ea8
This is the second part of I5de26afdf6774944b35472e2054b93d12fe21793.
It uses the executor api.
Three tests are disabled until the next change.
Change-Id: Ie08fa9dfb4bb3adb9a02e0a2e8b11309e1ec27cd
We pass zuul vars as extra vars, which means they don't need to be
in the inventory file. But we also write a "debug" inventory file
which isn't actually used by Zuul, but is useful for examining the
build. Go ahead and write zuul vars into that to maintain
compatability with the old behavior.
Change-Id: I06e1303beb507a45575a20264dd2071d58d8b39d
There are some special variables that should be only set by nodepool
and not on job level [1]. Overriding those could make mitm attacks
possible. Fix this by blocking those variables in the job definition
and data return.
[1] https://docs.ansible.com/ansible/latest/reference_appendices/special_variables.html#connection-variables
Story: 2008672
Task: 41964
Change-Id: Ie85fe110c092df7ef816af20356a55426cbebcb2
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
Freze Zuul job variables when starting a build so that jinja
templates can not be used to expose secrets. The values will be
frozen by running a playbook with set_fact, and that playbook
will run without access to secrets. After the playbook
completes, the frozen variables are read from and then removed
from the fact cache. They are then supplied as normal inventory
variables for any trusted playbooks or playbooks with secrets.
The regular un-frozen variables are used for all other untrusted
playbooks.
Extra-vars are now only used to establish precedence among all
Zuul job variables. They are no longer passed to Ansible with
the "-e" command line option, as that level of precedence could
also be used to obtain secrets.
Much of this work is accomplished by "squashing" all of the Zuul
job, host, group, and extra variables into a flat structure for
each host in the inventory. This means that much of the variable
precedence is now handled by Zuul, which then gives Ansible
variables as host vars. The actual inventory files will be much
more verbose now, since each host will have a copy of every "all"
value. But this allows the freezing process to be much simpler.
When writing the inventory for the setup playbook, we now use the
!unsafe YAML tag which is understood by Ansible to indicate that
it should not perform jinja templating on variables. This may
help to avoid any mischief with templated variables since they
have not yet been frozen.
Also, be more strict about what characters are allowed in ansible
variable names. We already checked job variables, but we didn't
verify that secret names/aliases met the ansible variable
requirements. A check is added for that (and a unit test that
relied on the erroneous behavior is updated).
Story: 2008664
Story: 2008682
Change-Id: I04d8b822fda6628e87a4a57dc368f20d84ae5ea9
Ansible needs to know which shell type the node uses to operate
correctly, especially for ssh connections for windows nodes because
otherwise ansible defaults to trying bash. Nodepool now allows this
setting in most driver configurations and this change makes Zuul
utilize that setting in the inventory file.
Change-Id: I55389ae8fa30be70c3939737f8c67282aad0ae47
Ansible 2.7 is in security fix only maintenance mode since quite some
time and will be end of life soon. It further blocks upgrade of zuul
to Python 2.8 due to incompatibilities. Thus drop support.
Change-Id: I13802db3314450ad149fdadacd1e2e70dd8468ef
Depends-On: https://review.opendev.org/727345
Replace `self.executor_client` with `scheduler.executor`.
This change only touches tests.
Change-Id: I22b01f2eff881e18633e5bab1ec390f3b5367a4d
Story: 2007192
With the event id in the inventory we can use it in the emit-job-header
role. This can aid in debugging, e.g. when a job is still running.
Change-Id: Id2d38bb3d121fa9cf959b9765ac8025d42001c02
The config/inventory fixture defines seven jobs, each needs merges and a
configuration to be prepared but the tests only assert against a single
job.
To reduce the overhead, hold the jobs at Gearman level to prevent them
from triggering merge operations and config loading.
Release solely the job for which we want to test the inventory.
When tearing down the job, cancel all remaining builds directly in
Gearman. That has shown to be faster than crafting an abandoned change.
The canceled jobs will bubble up to the pipeline manager and please our
suite assertFinalState() which requires that no items are left in any
pipeline managers queue.
By saving all the overhead of merge / ansible run etc, the tests locally
run more than twice faster than before.
Change-Id: Id7ffe741dcf82b1cb60099abc331ddc95cac8b3c
Github returns different urls for pull requests. One for api use and
one for browser use. The change url we're interested in is the one for
browser usage. However zuul takes the wrong one from the pull request
data structure but most of the time overwrites the change url
generated from event metadata. This leads to the issue that the change
url is sometimes not correct.
This can be fixed by taking the html_url from the pull request data
structure. Further due to the fact that the url is always set on a pr
and stays static throughout the whole lifecycle of the pr, be safe and
just don't overwrite the url from event metadata.
Change-Id: I2030b9dddd5bc618231b73d73ae64e2552231769
Zuul always sets ansible_python_interpreter as a host var. However
a user may want to set that as a regular var (to apply to the all
group) or a group var. If that happens, disable Zuul's own setting
of the value. Note that users can still override the all-var or
group-var with a host-var of their own.
Change-Id: Id130ec1718efa25b260b39ea0587ec5794e8e2cf
As this is the lastest stable version of ansible.
Change-Id: I02a0e9703a61b3976cec559a1069b51616c3d447
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Record the number of attempts zuul has made to run this job in the job
inventory. This will help expose reliability information in job logs and
in job log indexers. We want to try and expose job reliability as much
as possible and this is one way to do that.
Change-Id: I2f7c31ce510f59bc569c5db89ce6626d9e3ef436
The nodepool "python-path" config variable makes it's way through from
the node arguments and ends up as the "ansible_python_interpreter"
variable for the inventory when running the job.
Notably, Python 3 only distributions require this to be set to
/usr/bin/python3 to avoid what can often be confusing red-herring
errors (e.g. things like dnf packages incorrectly appearing to be
missing on Fedora, for example [1]).
Upstream is aware of this often confusing behaviour and has made an
"ansible_python_interpreter" value of "auto" to, essentially, "do the
right thing" [2] and choose the right python for the target
environment. This is available in Ansible >=2.8 and will become
default in 2.12.
This allows, and defaults to, an interpreter value of "auto" when
running with Ansible >=2.8. On the supported prior Ansible releases,
"auto" will be translated into "/usr/bin/python2" to maintain
backwards compatability. Of course a node explicity setting
"python-path" already will override this.
Nodepool is updated to set this by default with
I02a1a618c8806b150049e91b644ec3c0cb826ba4.
I think this is much more user friendly as it puts the work of
figuring out what platform has what interpreter into Ansible. It
alleviates the need for admins to know anything at all about
"python-path" for node configurations unless they are actually doing
something out of the ordinary like using a virtualenv. At the moment,
if you put a modern Python-3 only distro into nodepool, Zuul always
does the wrong thing by selecting /usr/bin/python2; you are left to
debug the failures and need to know to go and manually update the
python-path to Python 3.
Documentation is updated. Detailed discussion is moved into the
executor section; the README is simplified a bit to avoid confusion.
A release note is added.
A test-case is added. Note that it is also self-testing in that jobs
using Ansible 2.8 use the updated value
(c.f. I7cdcfc760975871f7fa9949da1015d7cec92ee67)
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1696404
[2] https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html
Change-Id: I2b3bc6d4f873b7d653cfaccd1598464583c561e7
This expands the discussion of executor-only jobs with some additional
notes.
Additionally a unit test is added to explicitly test executor-only
(i.e. blank nodeset) jobs.
Change-Id: I8fd2f932290e49da5a3605737e8940425cd092f4
This just mocks out we can modify the node.python_path setting in
nodepool. To confirm we are actually changing the value in
zuul-executor.
Change-Id: I064d9e4d1f8fcb79bc1dadc00a9561c2c9b5ac00
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Zuul recently added zuul.message which needs to be protected against
interpretation by jinja in ansible. This was initially done by marking
it with the !unsafe tag. However this has the disadvantage that the
inventory is no longer parsable by standard yaml parsers without
teaching them the !unsafe tag.
There is a similar simple possibility that doesn't rely on this tag by
base64 encoding the commit message. Ansible has filters for decoding
this so it is still quite easy to deal with base64 encoded vars in
ansible via '{{ zuul.message | b64decode }}'.
Change-Id: I9628e2770dda120b269612e28bb6217036942b8e
Having the change message available via the Zuul vars simplifies cases
where a job e.g. needs to update a GitHub/Jira/... ticket.
Those ticket numbers are usually referenced in the commit/PR message.
This avoids having to deal with secrets etc. to get this information
'out-of-band'.
Change-Id: Ib88db7f724dadfb8a4f86e76692f3e1c2c63a258
Both the operation and the read timeout can now be configured in the
zuul-executor main configuration file. If the network is flaky,
increasing these numbers from their defaults might help to lower the
rate of aborted Windows builds.
Change-Id: I4c25ca6027fc4150ec1c9c49ed286e7b4f20d4dd
When writing ansible inventory files as yaml, we should be using add
them under the children key per the official documentation:
https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#hosts-and-groups
This is the first step to allowing groups of groups for an inventory.
Change-Id: Id1e14d0364dd06af0371ca027031e46fb5f73e92
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
By default, Zuul uses runAnsibleSetup on all inventory nodes prior
to running job playbooks.
This translates to doing an 'ansible -m setup' against all nodes, but
this won't work on nodes where Python is not available, like network
nodes.
This change adds a specific setup_inventory.yaml file, which will not contain
nodes where setup module cannot work.
Change-Id: Ieb02a19036854b8d9089bcd4cc9dd0b46e3ce2fc
For supporting windows nodes we need the connection type to be
configurable. This adds the ansible_connection host variable if
nodepool defines it.
Change-Id: I6d2f81c7586ae0d533add95ea96a9ea8ce8c3ab5
Nodepool knows the username that you should ssh with at image build time
and includes this information in the node data. Zuul should use this
username for the executor target.
Change-Id: I1e677061b9fd495b192d25a5825362c81e40d0c6
Depends-On: Ife0daa79f319aea04ed32513f99c73c460156941
Today it is possible to create the following ansible inventory file:
[foo]
foo01 ansible_host=192.168.1.1
[bar]
bar01 ansible_host=192.168.1.1
Which allows a user to create multiple host aliases for a single
connection. This could be done with ansible groups, however there is
some functional differences on how ansible runs in that configuration.
We could also request 2 nodes from nodepool, however in this case, it
would be a waste of CI resources because every alias would need a new
node from nodepool.
Now, a user is able to alias multiple host names to a single node from
nodepool by doing the following:
nodeset:
nodes:
- name:
- foo
- bar
label: ubuntu-xenial
This would result in a single node request from nodepool, but create
an inventory file with 2 alaises sharing and single ansible_host
variable.
Change-Id: I674d6baac26852ee1503feb1ed16c279bf773688
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Instead of using a vars.yaml file and a -e command, just put the
variables into the all group in the inventory file. One less file to
manage, single file to look at for debugging.
Change-Id: I5b1f149ecca649b1434488392cc8232de20cd4fc
All of the zuul config files are yaml. Ansible supports yaml formatted
inventory files. If we use yaml, then displaying inventories to users
in raw log files should be easier to read for non-Ansible users.
Variables are set under the hostname, rather than as key=value strings
on the same line. We can also stop writing the vars file and just add
the variables to the top level vars of the same inventory file, but
let's do that next.
Construct the dict as an unattached function not a method to allow for
easy interative unittesting as context isn't needed to validate inputs
and outputs.
Change-Id: Ife7a909ea0e54015bddbf5426343dd5c5911953c
James E. Blair