The previous change added a bunch of tox v4 compatiblity, now we switch
to as our default. This should only happen after people have had
sufficient time to add tox v4 compatibility or manually pin back to <4
in their projects.
Change-Id: I16742fb933dba2162a73d532e1a9613af69021aa
Tox 4 released today and is a complete rewrite with many backward
incompatible changes. We need to update a number of things to support
that in the zuul-jobs tox role and elsewhere. A possibly incomplete
list of what was changed in this commit to make this work:
* Don't run tox --showconfig with {{ tox_extra_args }} as -vv is
in tox_extra_args by default and results in interleaved debug
output in the ini output making it invalid ini content.
* Update the tox siblings tox config parser to look for renamed
environment dir locations.
* Stop using whitelist_externals and use allowlist_exteranls
because whitelist_externals is removed and external commands that
are not explicitly allowed produce errors.
* Make the tox version configurable in ensure-tox as some users
may not be able to easily upgrade to tox v4.
* Escape literal # chars in tox.ini as they are treated as comments
when in the command strings now.
https://github.com/tox-dev/tox/issues/2617
Change-Id: I38e13b4f13bb1b2d6fb7e5c70b708e9bb016a455
Tox v4 just released and has a number of breaking changes. One of which
(yet to be properly identified) has completely broken Zuul's tox
siblings processing and produces errors like:
Traceback (most recent call last):
File "<stdin>", line 107, in <module>
File "<stdin>", line 99, in _ansiballz_main
File "<stdin>", line 47, in invoke_module
File "/usr/lib/python3.10/runpy.py", line 224, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/tmp/ansible_tox_install_sibling_packages_payload_0af4x4lc/ansible_tox_install_sibling_packages_payload.zip/ansible/modules/tox_install_sibling_packages.py", line 397, in <module>
File "/tmp/ansible_tox_install_sibling_packages_payload_0af4x4lc/ansible_tox_install_sibling_packages_payload.zip/ansible/modules/tox_install_sibling_packages.py", line 328, in main
File "/usr/lib/python3.10/configparser.py", line 724, in read_string
self.read_file(sfile, source)
File "/usr/lib/python3.10/configparser.py", line 719, in read_file
self._read(f, source)
File "/usr/lib/python3.10/configparser.py", line 1097, in _read
raise DuplicateOptionError(sectname, optname,
configparser.DuplicateOptionError: While reading from '<string>' [line 45]: option 'root' in section 'testenv:docs' already exists
Let's pin tox for now and a change that removes the pin can also fix tox
siblings (the change to remove the pin should be self testing
thankfully).
Also we make the zuul-tox-docs job temporarily non voting as it runs
ensure-tox non speculatively out of the opendev/base-jobs repo. We have
plenty of tox jobs runing against this changeto show it should work
fine.
Change-Id: Idcebd397f47cfef718721d2576cb43dbdb70801d
We can't do this all the time, because of rootless environments.
But sometimes people have root and want to be able to use something
from scripts from normal path.
Change-Id: I3f57a6108f8f53ebfdd12f04ecb3d8c68c5b4a60
On a static node where tox isn't present in the PATH, tox would be always
reinstalled even if already present in venv because it's not checked.
This patch adds tox_venv check, the first matching line is used to get
executable path
Change-Id: Ia27af308cd3b5e2ce5efe36508fe82468e88cbc7
While making test with a node:
- with tox installed using "pip install --user" as a non-root user
- PATH modified using ~/.ssh/environment
- declared in nodepool to run as this non-root user
tox checks produce an error:
"Check if tox is installed" evalues correctly (shell, bash)
"Output tox version" cannot find tox (command, no sh, no etc ...)
This change sets fact using the output of the first install check
Change-Id: I5886858bcd57afc8ea5fd121ad44bd82afb1a671
This reverts commit bac2bf6c45.
The problem was really that ensure-pip was giving us a
"ensure_pip_virtualenv_command" that didn't work on some systems.
Iaa3ecd05b64af6dd9b2ee17a39bcbe6cde8686ba fixes this underlying issue,
so we can revert to the original change that uses that to install the
tox environment.
Change-Id: I8ce9dceb721474d3220f6e72409481dc89875ee0
This reverts commit 6d78fc4f90.
In some environments, this error is being seen:
2020-05-07 18:36:02.139422 | debian-buster | "msg": "stdout: The virtual environment was not created successfully because ensurepip is not\navailable. On Debian/Ubuntu systems, you need to install the python3-venv\npackage using the following command.\n\n apt-get install python3-venv\n\nYou may need to use sudo with that command. After installing the python3-venv\npackage, recreate your virtual environment.\n\nFailing command: ['/home/zuul/.local/tox/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']\n\n"
Change-Id: I6d38bf16cc38020c53815dfa6ab94f8cab4de0a2
This currently installs with pip --user which cases problems if you
try to run this version of tox as another user. This is done in
system-config, for example, where we run tox with "become: yes" to run
testinfra.
By installing tox into a venv, we can call it as another user and it
just works because it's all encapsulated in the venv. We use the
virtualenv commands exported by ensure-pip to create this.
I think the original motivation for installing tox like this was to
ensure it is done without sudo permissions. This also doesn't require
permissions, but ensures the resulting tox_executable is able to be
executed in more contexts.
Needed-By: https://review.opendev.org/712819
Change-Id: Iebee8cb72cce7944c537fdb91b6c98ed51878661
There are over 490 .yaml files but only a few .yml, let's rename to be
consistent.
Add a test to block .yml files.
Change-Id: I2f1354de82f231154d926b51d9812b1e9c1a6202
Several roles are setting facts that are expected to be consumed by
roles following them; generally things like the path to installed
tools, etc.
Add a section to policy about this behaviour in general, noting it
should use the cachable flags to persist the values.
Add specific documentation notes in an "Output Variable" section for
roles currently implementing this behaviour.
Change-Id: I05657fec198176c7d7345e84293f4902c81fa30c
Use the ensure-pip role to ensure that `pip` is available. If the
existing flag `tox_prefer_python2` is set, we pass along the flag to
ensure-pip to install Python 2 pip as well.
Story: #2007386
Task: #39308
Change-Id: I5053e315df8d2ea9892911800c88afcbe31cc05b
This role currently does
which tox || pip install --user tox
and then sets "tox_exectuable" iff it has installed tox into the
user's pip environment.
This isn't idempotent -- if you run ensure-tox again, it will again
run "which tox" and miss that it already installed tox in the private
pip environment. Instead, make it look for
"which {{ tox_executable }}" ... this way it will find the install it
just did if it runs again.
There seems to be no reason to only set "tox_exectuable" in the case
this role had to install tox. Make this *always* set tox_exectuable
so that you can run ensure-tox and then be sure tox_exectuable is
something sane.
The current testing assumes that tox is installed in the base image.
We are working to remove this assumption; remove this part of the
test.
Rework the testing to first clear out any existing tox, which is
present on most images but will have no effect on the -plain images.
Then test when we install fresh it goes into the user pip install
area, and then test if we explicitly set tox_exectuable it overrides.
Change-Id: I36cfb62f2a2e2b9c2bbe92b46ed0b8098a873732
The role currently defaults to installing tox with python2 in order
to maintain compatibility with existing jobs. This change makes the
default behaviour use python3 as python2 is EOL.
Change-Id: Ic41b8fee7ae8a8f5df6c1107c6ed812c14b8c8be
This patch enables the ability to set a value which prefers the
installation of tox within python2 over python3.
Change-Id: I29c9585aa9048c0e2855ec1eaf1f48041cfe46e2
If we install our own local version of tox, we should set a fact
with the tox_executable so other tasks can leverage it.
Change-Id: If6895bbb898261e88c0e3083d21210209f79995f
The
command -v pip pip3 | head -n1
introduced with Ie50928c9b782ea84db916bb1441567e1206ff466 has a very
subtle race; if "pip" and "pip3" exists and there are two lines
output, the "head -n1" will exit and depending on scheduling the
"command" might write to a broken pipe (this manifests as exit code
141).
Move this to a more explicit if statement.
Co-Authored-By: Jens Harbott <j.harbott@x-ion.de>
Change-Id: I80823a7bc6351925d6f0b20bdebca3eafef0b27d
Fixed issues failing to install tox on python3 only systems which have
only pip3 executable and not the pip one.
Needed-By: https://review.rdoproject.org/r/#/c/24584/
Change-Id: Ie50928c9b782ea84db916bb1441567e1206ff466
Clark Boylan