Several packages in calling "python setup.py --name ..." may return
warning message, e.g.:
...
_DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!
********************************************************************************
Requirements should be satisfied by a PEP 517 installer.
If you are using pip, you can try `pip install --use-pep517`.
********************************************************************************
!!
dist.fetch_build_eggs(dist.setup_requires)
WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
<PACKAGE_NAME>
and then this huge log places into the package_name variable. But
script expects that package_name will contain only package name. Because of this situation could not find siblings packages and
install them.
Change-Id: I5bf9a19233c48d1260b5ab17d749bfc58a8ef2fa
Tox 4 released today and is a complete rewrite with many backward
incompatible changes. We need to update a number of things to support
that in the zuul-jobs tox role and elsewhere. A possibly incomplete
list of what was changed in this commit to make this work:
* Don't run tox --showconfig with {{ tox_extra_args }} as -vv is
in tox_extra_args by default and results in interleaved debug
output in the ini output making it invalid ini content.
* Update the tox siblings tox config parser to look for renamed
environment dir locations.
* Stop using whitelist_externals and use allowlist_exteranls
because whitelist_externals is removed and external commands that
are not explicitly allowed produce errors.
* Make the tox version configurable in ensure-tox as some users
may not be able to easily upgrade to tox v4.
* Escape literal # chars in tox.ini as they are treated as comments
when in the command strings now.
https://github.com/tox-dev/tox/issues/2617
Change-Id: I38e13b4f13bb1b2d6fb7e5c70b708e9bb016a455
This commit in Ansible:
9142be2f6c
now allows Python modules to specify their interpreter with the shebang.
We expect our roles to use the discovered python interpreter on remote
nodes, and on the executor, we need them to use the virtualenv. Removing
the specific shebang accomplishes this under Ansible 6, and has no effect
under older versions of Ansible.
Without this, for example, the log upload roles would not have access to
their cloud libraries.
Also update our ansible/cli check in our module files. Many of our modules
can be run from the command line for ease of testing, but the check that we
perform to determine if the module is being invoked from the command line
or Ansible fails on Ansible 5. Update it to a check that should work in
all 4 versions of Ansible that Zuul uses.
Change-Id: I4e6e85156459cca032e6c3e1d8a9284be919ccca
The below error is coming on mol-centos7 jobs while
installing sibling for python2 packeges on centos7
Error: TypeError: initial_value must be unicode or None, not str
This patch handled the TypeError while installing sibling
for python packeges for centos7.
Closes-Bug: #1946641
Change-Id: Ie8058cca92d099e50af19b95b4c417c5a665da0d
With the original verbose output support in change
Iafeb88eaf9a596603ad4d2134a4574345d5189ab we looked for lines from
tox --showconfig output starting with an opening '[' but verbosity
also causes output from pip install activity to be included if a
tox.requires entry causes it to install packages before continuing
and these lines are prefixed by a process ID number wrapped in
brackets with the command string after that, which prematurely
triggered our search for the start of the INI content. Add a
stipulation that the first INI output line also end in ']' in order
to skip over those additional prepended lines.
Change-Id: If29e5a9abe3b92a145d87f5efc1b93350ea3908a
Unfortunately, when tox combines --showconfig with verbosity options
like -vv, some non-config output gets streamed to stdout before the
configuration is emitted. Filter this preamble in
tox_install_sibling_packages by discarding any initial lines of
output before the first section heading.
Also extend get_envlist() to deal with the fact that additional
verbosity adds a [tox] section in the --showconfig output, which it
was previously relying on to determine whether the config had been
filtered to a subset of env sections. Instead also check the
tox.args string to determine whether a -e option was passed on the
command line.
Change-Id: Iafeb88eaf9a596603ad4d2134a4574345d5189ab
This allows projects, that use tox but may not have a setup.cfg file
still use tox siblings. We do this to allow non-python project, to use
tox as an entry point for testing, and still have depends-on
requirements work in zuul.
Change-Id: I9b37117b27ff6b7e436d456b6cbae39ccb9b968c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
There's a couple of places that call subprocess.check_output that
might raise and exception (I'm seeing one finding the "--name"); catch
the CalledProcessError and include it's output to give more info.
Change-Id: I53cfb9c1b37bd4dbf9e5bf04fbab83d3cdad795d
siblings: python2.7 ConfigParser has no __getitem__
Constrain soupsieve to <2 for python2.7
Add python2-dev to bindep for subprocess32
Change-Id: If9d6a0ae1a62a94dcec11f6bf637ffee7f0f4fc9
Also we can parse the output from --showconfig to se which
testenvs we should act upon when installing siblings.
Change-Id: I438035c92d62102c80db9bed81e9f290e4c941b2
Since tox_envlist has a default value it cannot be undefined
so the fail task will never run. Instead handle the case when
tox_envlist is an empty string by getting the default configured
envlist from tox. Also handle the casewhen tox_envlist is 'ALL'.
This also updates tox_install_sibling_packages to correctly
handle multiple testenvs and uses configuration supplied by
'tox --showconfig -e <envlist>' instead of guessing where the
envdir and logdir are located.
We also cannot run tox inside python because it gets complicated
to know which tox_executable we should call during the python test
cases so run these commands in ansible and pass the output to
tox_install_sibling_packages.
Since role params have higher precedence than set_fact we set an
internal _tox_envlist fact that is a comma separated list of testenvs
that should be run.
Change-Id: I9e5a1b041f653cbcff7b8ed62e4a95a0a040fdd7
This fixes a byte vs str problem on python3 in the tox role's sibling
library.
As part of this fix add unit testing of the function that was failing
under python3 and functional testing of the role that covers siblings.
Change-Id: I2df4ddce91dfd53f5c4177a19288e97120bc3e3b
Under Python3, subprocess.check_output returns binary data,
convert to utf-8 so that it runs under both Python2 and 3.
Using decode works with both Py2 and Py3.
Note that in other places, we already convert the output of check_output
with decode, this change follows that pattern.
Change-Id: I727a2368233473f57978ba24947f6d861b02f7da
Some projects have a setup.cfg included, but don't have metadata with
the package name in it. In this case, skip on to finding it via
setup.py
Change-Id: I9339ac86832994b9f7f706c99e07441fc120b25a
This patch is a follow up for patch:
I3573c067f63a09f22377612f53c80c80252965e4
Comment explains the string matching lines for the pip freeze output.
Change-Id: Ifc5b2ae002859c1ca7405c8e1f64380b311e983d
In pip 19.0 release [1] the
Editable Git installs without a remote now freeze as editable. (#4759) [2]
bug is fixed. Now the output of pip freeze for such package looks like:
# Editable Git install with no remote (neutron==10.0.8.dev66)
-e /home/zuul/src/git.openstack.org/openstack/neutron
instead of:
## !! Could not determine repository location
neutron==10.0.8.dev66
Since for listing the package names the module uses the lines that contains
'==' the new behaviour leaves some garbage in front of the package name. So
in case the string contains '(' then remove the first part of the string.
[1] https://pip.pypa.io/en/stable/news/#id15
[2] https://github.com/pypa/pip/issues/4759
Change-Id: I3573c067f63a09f22377612f53c80c80252965e4
Some openstack packages show up in the requirements list under a
hyphenated name while their package metadata name contains only
underscores. (For example, the glance_store library is listed in
requirements as 'glance-store'.) This is causing problems for the
tox_install_sibling_packages task run by zuul. Pip is able to handle
this situation [0]; add code so the ansible task can handle it too.
[0] http://paste.openstack.org/show/722298/
Change-Id: I89ad9926647a011174815761e79372f2d7d43609
Needed-by: https://review.openstack.org/#/c/569225/
Closes-bug: #1774030
When python setup.py --name is run in ansible/ansible currently, a
warning is printed:
UserWarning: Normalizing '2.7.0dev0' to '2.7.0.dev0' normalized_version
With stderr=stdout, this goes into the package name causing sibling
detection to not work.
The stderr is not interesting to us from this command, so ignore it.
Change-Id: I38ea21960932dbddcac1b828e95a0cf963c3b94c
As of pip10 pip is no longer promising external stable apis (they never
really did anyways just failed to annotate the source as such). As a
result our use of direct pip internals breaks under pip10.
We replace this with a fork and exec of pip instead as the command line
is a stable api. We were already doing this anyways so shouldn't cause
anything to run slower and should be more reliable as an api.
Note that we also remove the import of pip as this does an unexpected
thing and uses pip outside of the virtualenv rather than pip inside the
virtualenv. Since we are only forking pip inside the virtualenv now we
don't need to check for external pip.
Change-Id: Ib2628003995530881d297782f044dfc8a1c72d08
The current logic uninstalls all detected siblings first, then it
executes a single install with all of the detected sibling git repos.
If one sibling (A) depends on another sibling (B), then the commands:
pip uninstall A
pip uninstall B
pip install ../A ../B
Will result in pip finding B in the requirements list for ../A and
re-installing the released version, then considering the requirement
satisfied when it gets to ../B and not installing it.
To solve that, do the uninstall and reinstall in a single-project loop
for each sibling. This results in:
pip uninstall A
pip install ../A
pip will find B in ../A's requirements, but B will already be installed
because it hasn't been uninstalled yet, so nothing will happen. Then:
pip uninstall B
pip install ../B
Which will result in both ../A and ../B being installed.
As if that wasn't enough, if A and B happen to appear in the opposite
order, doing:
pip uninstall B
pip install ../B
pip uninstall A
pip install ../A
Will wind up with pip uninstalling ../B and installing B if the version
requested for B is greater than the version reported by ../B (which is
the case for openstack projects operating from master after a release
but before the first release of the next cycle)
In order to combat that, do a second installation pass with --no-deps.
This way the first pass will be sure to get any transitive dependency
changes, but the second pass will ensure all the siblings are installed
instead of their non-git versions.
pip uninstall B
pip install ../B
pip uninstall A
pip install ../A
pip install --no-deps ../B
pip install --no-deps ../A
Change-Id: I060e188313391de7847adf851566468c4b032342
ansible/ansible doesn't use pbr, so we can't inspect its setup.cfg to
drive tox-nstall-siblings.
In case we find a setup.py but no name in setup.cfg, fallback to running
setup.py --name.
Change-Id: I6d234247e3d45befd264e159c8b0fe77f01c7fdc
When sibling installation fails, we only get MODULE_FAILURE and no
feedback. Wrap the guts in an exception handler so we can figure out
what's wrong.
Change-Id: I42b61ad58415bacbd0d7b220fb47534745ddaa26
There is a bunch of duplicate logic between tox and tox-siblings, the
tox-siblings logic is already protected with a flag - and having them be
split already led to not doing the right thing with constraints files in
siblings when we did it right in tox itself.
Combine them into one role.
This means the siblings code will run as part of run instead of as part
of pre- but that's how things have worked for years anyway until
siblings was introduced - and is also the behavior is the siblings flag
is false -so it's more consistent overall.
Leave a no-op tox-siblings role so that we can gracefully remove the use of
tox-siblings from castellan, python-openstacksdk, shade and tacker.
Change-Id: Id61ae52d48b28cfc2221cb556a1c1f7c6dfd60dd
Roman Kuznecov