Commit Graph

44 Commits

Author SHA1 Message Date
Clark Boylan a021a9bba2 Pin requests-oauthlib<1.4.0 under python2.7
While requests-oauthlib 1.4.0 reports compatibility with python2.7
evidence points to the contrary. Specifically:

      import requests_oauthlib as oauth
    File "/home/zuul/src/opendev.org/zuul/zuul-jobs/.tox/py27/lib/python2.7/site-packages/requests_oauthlib/__init__.py", line 5, in <module>
      from .oauth1_session import OAuth1Session
    File "/home/zuul/src/opendev.org/zuul/zuul-jobs/.tox/py27/lib/python2.7/site-packages/requests_oauthlib/oauth1_session.py", line 1, in <module>
      from urllib.parse import urlparse
  ImportError: No module named parse

The azure-storage-blob library depends on this oauthlib which is needed
for log upload role testing. We address that by pinning the lib under
python2.7.

Change-Id: Ia27f2df166ccb33dcc93334d884411670ac859f7
2024-03-12 10:19:51 -07:00
Ian Wienand fe5f51c32f test-requirements: bump to Ansible 2.8
We are outside the 4 week period that we dropped Ansible 2.7
(I13802db3314450ad149fdadacd1e2e70dd8468ef) so update the testing
version to 2.8 here.

The comment has become stale, as ansible-lint is now installed via
linters-requirements.txt; remove the bit about how it will pull in the
latest version of Ansible.  Also clarify slightly the version usage.

Later versions of Ansible only have paramiko support as a plugin.  We
have just one place we use it explicitly to generate some keys for
testing; so it is really a dependency of zuul-jobs.  Add it
explicitly.

Change-Id: Iad8622594a773af78487dec88e4ce5cc48806ded
2022-08-12 10:08:10 +10:00
Clark Boylan e6b747b32b Fix two testing problems
First up we need python2.7 specific msrest version as azure-storage-blob
depends on it and it doesn't properly specify python versions for its
releases.

Next we cannot use zuul/zuul to test ensure-pip can install packages
because zuul/zuul requires python3.8 or newer now. Some platforms we run
ensure-pip on still only have python3.6. We address this by using
zuul-client instead.

Change-Id: I3d91fd3f85e3d7845ab2d7615a1ebfb614064573
2022-06-16 10:30:00 -07:00
Zuul 9761561e5a Merge "Add upload-logs-ibm role" 2022-04-11 22:30:23 +00:00
James E. Blair a8b4bc6ff3 Add upload-logs-ibm role
This role uploads logs to IBM Cloud object storage.

Change-Id: Ibe1131f863a64051b427fcb03b126b1577c4843a
2022-04-11 14:20:49 -07:00
Jens Harbott cd3c622abb Fix test jobs
- voluptuous dropped support for py27. see also [0]
- drop centos-8 jobs for which we no longer have images

[0] https://review.opendev.org/c/opendev/glean/+/836194

Change-Id: I6626a5a59625640d1c0b4e9026743441059177e0
2022-04-02 13:47:50 +02:00
Jeremy Stanley f8f1d1c271 Pin protobuf<3.18 for Python<3.6
Our Google Cloud log upload role relies on google-cloud-storage,
which in turn needs protobuf. Unfortunately, protobuf dropped Python
2.7 and 3.5 support in its 3.18.0 release, so we use an environment
marker to pin it in our test requirements.

Change-Id: I89caf4e36850fc4a912b76d75f368144ddb5e15f
2021-09-16 19:17:41 +00:00
Zuul 98384d489f Merge "Use openstacksdk 0.45.0 for python2.7" 2021-06-01 01:01:06 +00:00
Albin Vass da8a64b638 Use openstacksdk 0.45.0 for python2.7
Change-Id: Ib5b2bbd148fcee73f0069288cd368759d7c10cfb
2021-04-06 16:53:30 +02:00
James E. Blair 120a11ef20 Add upload-logs-azure role
Add support for uploading logs to Azure blob storage.

Change-Id: I0347977324b880123c6ed83ded3c39eb210612e2
2021-03-30 22:27:25 -07:00
Jeremy Stanley ebf477511f Pin keystoneauth1 and cachetools on older Python
Recent versions of keystoneauth1 have dropped support for Python 2.7
and 3.5, but the versions of openstacksdk we're pinning on those
Python releases don't pin keystoneauth1, so we have to do it
ourselves. Same goes for cachetools 4.0.0 dropping Python 2.7
support.

Change-Id: Id14c88b5e8dbf294e5f6d9e8d809493e81abd41c
2020-11-24 01:28:51 +00:00
Tobias Henkel 9ff9fa5827
Pin openstacksdk to <0.48.0
The openstacksdk dropped py35 support in 0.48.0. Since we still have
py35 support we need to pin it temporarily until we've deprecated and
removed it.

Change-Id: Ia50ed54f43605fc1e12c95a2218b73abcadc66ed
2020-10-06 17:12:11 +02:00
Albin Vass 838b0c8877 Add upload-logs-s3
Change-Id: I6ce64734ed5f20a212e6cb953d09ea2769238bea
2020-07-19 21:22:36 +02:00
Albin Vass 4d8698e30f Drop support for ansible 2.6
Zuul is dropping support for ansible2.6, wait a couple of weeks
and bump ansible version in zuul-jobs to 2.7.

Depends-On: https://review.opendev.org/#/c/727157/
Change-Id: Iccfb963ee0e1c4ba26e1f678426fb5a680af27fb
2020-06-18 12:15:08 +02:00
Felix Edel 2f8439148a
Return upload_results in test-upload-logs-swift role
This will only change the test role and can be used for validation.

We are facing some issues where the log upload to swift fails, but the
role is always succeeding. To get some more information about the
upload failures, we let the upload() method return those to the Ansible
module and provide them in the module's JSON result.

Change-Id: Ic54b877cd5ea58031e21e514eb3a7c50ea735795
2020-06-15 08:19:11 +02:00
Sorin Sbarnea 5e9f91764e Bump ansible-lint to 4.3.0
- Bumps linter and make use of its auto-detection
- Temporary skips linting test-playbooks/ to match previous behavior
- Documents skips in a way that makes it easy to maintain the rules
- Keeps linter config in standard location, so it can be loaded
  regardless how is called.

Change-Id: Ic379c91fa9385473f6ec2af91e61953dc10c1f54
2020-05-28 16:48:14 +01:00
Albin Vass c5171330b2 Bumb ansible to version 2.6
Support for ansible 2.5 was dropped 5 months ago:
https://review.opendev.org/#/c/650431/

Let's bumb the supported version in zuul-jobs to 2.6.

Change-Id: If0df7ddadcadbd744d50f7d6e5f87622dddf3e8e
2020-05-07 11:14:45 +02:00
Sorin Sbarnea 33461bbecc Enable yamllint
Adds yamllint to the linters with a minimal configuration, some
rules are disabled to allow us to fix them in follow-ups, if
we agree on them.

Fixes invalid YAML file containing characters inside block.

Fixes few minor linting issues.

Change-Id: I936fe2c997597972d884c5fc62655d28e8aaf8c5
2020-05-04 17:47:11 +01:00
James E. Blair 2ae8616306 Remove bashate from test-requirements
It doesn't look like anything uses this.

Change-Id: Id1652f2b8902bc6aaa4037e2fde9718d60dcf45b
2020-03-27 14:59:21 +01:00
Andreas Jaeger c50aee5cf9 Cap stestr version
Cap stestr version for python2, version 3.0 does not support python2
anymore but misses proper marks.

Change-Id: I544f4c195643b73cae6b6a201bfb5e0b2cfc59ff
2020-03-27 14:07:48 +01:00
Monty Taylor c800638473 Use a zuul_* and add an .ansible-lint file
ansible-lint now supports settings in a file, including exclude_paths.
This lets us simplfy the ansible-lint command. Also, stop installing
zuul and just use a fake zuul_return and zuul_console
to fake out ansible-lint.

Change-Id: I1482a9ab915cec2d45695b60cdbeb93d58cb392c
2020-03-13 20:26:06 +01:00
Monty Taylor a4d68dd83b Remove enum34 from test-requirements
We put this here as the first step in trying to solve updates
to the upstream enum package - but those are really actually only
issues with trying to install zuul into python2.

Change-Id: Ied3b82c5d0fc104d6a9bbfde943be639565adcd1
2020-02-24 11:29:51 -06:00
Monty Taylor d36ba50765 Fix unittests for python2 and fedora
We need to pin enum34 on python2 because upstream broke it.

Also, we should not install zuul in python2, because it's not targetted
at python2. It's only here so that linters can deal with zuul_return
and other things from its library, so it should be fine to not install
it for python2 unittests.

GitPython was in here as a pin to the versions that support python3
for zuul. Since we're constraining zuul to only python3, we don't need
to mention GitPython at all.

Update the siblings unit test to use requests as the package we expect
to be already installed since we're no longer installing zuul for
python2.

Fedora 29 got archived upstream:

  http://mirror.sjc1.vexxhost.openstack.org/fedora/releases/29/README

So we need to switch to fedora 30.

Change-Id: I69e0b03f624ba45889916f89c0912df667aaf096
2020-02-24 10:50:13 -06:00
James E. Blair b8ce30f3cb Add upload-logs-gcs role
This uploads logs to Google Cloud Storage.  It is based on
upload-logs-swift.

Change-Id: I4d83c3f4eb7f186271302cbb6ebae6b1f280785b
2020-02-03 16:02:15 -08:00
Jeremy Stanley 2711d10186 Be consistent about spaces before and after vars
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.

Adjust the new violations accordingly and remove the rule
206 exclusion.

Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
2019-08-07 20:17:41 +01:00
James E. Blair a74ff55816 Add generate-zuul-manifest role
Also, pin gitpython since it's gone py3-only.

Change-Id: I0d626945908ec4df785aea793f6243c6fdfbdb14
2019-07-23 09:43:55 -07:00
Jeremy Stanley 1ed78d714f Advance ansible-lint cap to test with 4
Running latest ansible-lint raises violations for rules 204, 206 and
601 so skip them for now.

Change-Id: Ibc62833cfc4b2fae728bc6c944f263af3fc2b89d
2019-06-26 18:09:51 +00:00
Monty Taylor 47ceef043a Shift doc requirements to doc/requirements.txt
We have doc requirements in the normal test-requirements list.
This is causing problems for some sphinx related things which
want python3 conflicting with our py27 test job.

Change-Id: I1c4ddba56377097f1ebf39051a496577d38f0494
2019-05-09 13:58:56 +00:00
James E. Blair e620f0b74e Document how to build a buildset registry
Change-Id: I57e6ace5f6731045b8043995b18835a499d2250c
2019-03-15 08:37:37 -07:00
Monty Taylor 5be1cd99f5
Pin ansible-lint to < 4
4 explodes all over the place. Let's not deal with that right now.

Change-Id: I2c1a8501c696a9c853e6ca0ccb15b369f71a70cd
2019-01-04 22:03:40 +00:00
Tobias Henkel b35f47190a Revert "Extract pep8 messages for inline comments"
There seems to be a problem with openstack-zuul-jobs [1].

This reverts commit 421f533b61.

[1] http://logs.openstack.org/86/610386/3/gate/openstack-zuul-jobs-linters/a8c857d/job-output.txt.gz#_2018-10-18_07_14_42_431592

Change-Id: Iac5236221e33df4613cc2f7a2edcffc323902320
2018-10-18 09:11:38 +00:00
Monty Taylor 421f533b61 Extract pep8 messages for inline comments
Move the zuul requirement from test-requirements into the linters
section of tox.ini because we need it as a src install in the tox
env, so that we can reference the ansible library location.

Change-Id: I089c69b539107bdbc25791f5730502a4f46e7cf6
2018-10-17 13:45:00 -07:00
James E. Blair d661ad4042 Add zuul to the linters tox env
So that the syntax check which runs on playbooks can find zuul_return.

Add re2 library to bindep, it's needed for zuul.

Change-Id: I93213ee19652c0ec9f4bc699204e9445a1e891c8
2018-10-02 08:15:52 +02:00
Ian Wienand 9ee2aecd21 Bump ansible test version to 2.5
Iab951c11b23a24757cf5334b36bc8f7d12e19db0 was merged in June to update
Zuul to Ansible 2.5.  This is outside the 4 week update period for new
features, so we can update and use Ansible 2.5 features in zuul-jobs.

Add a little bit more of a note in the requirements about what's going
on while here.

Change-Id: Ic018b7259749148660e1bdb6c9a779551390fd09
2018-08-17 15:57:17 +10:00
Tobias Henkel 6330a47aad
Require at least openstacksdk 0.17.1
The swift upload at least needs openstacksdk 0.17.1 which fixes a bug
that prevented upload with the deflate filter. So state this in the
requirements file.

Change-Id: I64aa9ed8246006e7b77e94536e9a62dc1ff16b2a
2018-08-02 08:38:46 +02:00
James E. Blair 2bf16a421f Add upload-logs-swift role
This adds a role to upload log files to swift.  It generates index
files, sets expiration times, and creates partitioned containers as
needed.

This is largely based on the zuul_swift_upload.py script written for
Zuul v2 by Joshua Hesketh.

Change-Id: If520c849fe6a20833804609583ea758e8dd2a6f6
Co-Authored-By: Joshua Hesketh <josh@nitrotech.org>
2018-07-27 13:40:03 -07:00
James E. Blair 1548d82ffd Add a python unit test framework
Add a unit testing framework for python roles.  Thanks to Matthew
Treinish for the suggestion of how to perform discovery (and much of the
code which is copied from Tempest).

Change-Id: Iec95dd1026a41614def57c65c3faa0516a682a5a
2018-07-25 09:11:25 -07:00
Andreas Jaeger 7c4734f398 Run ansible-lint on repo
This change follows  I16186c929e7d0e6e34b35271559e555255a52b00 to run
openstack-zuul-jobs-linters on the repository.

Also, silence remaining lint role.

Depends-On: I46c53229f878fc707bab627654d820b891d9f625
Change-Id: I76abf21ea51c5f7553639475e408e06768fb1016
2017-10-27 07:49:02 +02:00
Dirk Mueller 0dd2f84d0b Remove pep8/pyflakes
flake8 pulls the right versions that it needs in already,
so listing them explicitly is redundant.

Change-Id: I3441696f055a51c18828d2b345c269c69c1b0e8c
2017-10-25 14:26:34 +02:00
David Moreau-Simard c8afe1b563
Pin ansible<2.4 in test-requirements for ansible-lint
Otherwise ansible-lint picks up the latest version, 2.4, which makes
the linters job horribly fail for the time being.

Change-Id: I31f9aea554b3853a4b9bc0af9a24c4f5c5388c22
2017-09-19 22:12:12 -04:00
Monty Taylor 82af975879
Update tox/test-requirements
In working on a different patch it became clear that flake8 wasn't
actually running such that it found errors. Updated the
test-requirements to match what's in zuul and all started working (and
failing) properly.

Change-Id: Icfdb1fedbd92ff49484b116a0879686581274a25
2017-09-02 14:17:32 -05:00
James E. Blair 4e6c7ef6a5 Switch to using zuul-sphinx
Remove the internal zuul domain in favor of installing from
zuul-sphinx.

Change-Id: Ia3bc2e6ad38625afedb3abd925801e1bcb40eb9e
2017-08-01 09:44:18 -07:00
Paul Belanger f40a7fbe1e
Install zuul to access zuul_console
When we lint our playbooks, we need to also have zuul install our
custom action plugins.

Also default to python3 so we can properly install zuul into tox.

Change-Id: Ib7e39e43005aa73c9d482af6becef3408d097e15
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-19 13:06:30 -04:00
James E. Blair e5ebc400cb Add initial license, docs, and other config
Change-Id: Ie016aa9a3494bea1e37b7eee4a970d5772317d47
2017-06-08 14:01:21 -07:00