Commit Graph

52 Commits

Author SHA1 Message Date
James E. Blair 509880073e Remove command.warn usage
This is no longer present in Ansible 9.

Removing these upsets ansible-lint, so those errors are ignored.

The base roles job has bitrotted on centos-7 and bionic due to
a bad voluptuous release used in an stestr test.  That is fixed in
this change as well.

Change-Id: I67886d5ad82ab590979f82bd102d6f974b9d4421
2024-02-14 15:17:52 -08:00
Clark Boylan f910bbac74 Pin virtualenv in tox environments
Virtualenv 20.22.0 dropped support for python<=3.6 (including 2.7). We
still want to run tests under 2.7. Work around this by capping off
virtualenv in tox's requires list.

Change-Id: I4868cd5a8a958f04ac782e0963c52a118b2f5ebf
2023-04-21 20:43:45 +00:00
Clark Boylan 41153f0653 Update zuul-jobs to handle tox3 and tox4
Tox 4 released today and is a complete rewrite with many backward
incompatible changes. We need to update a number of things to support
that in the zuul-jobs tox role and elsewhere. A possibly incomplete
list of what was changed in this commit to make this work:

  * Don't run tox --showconfig with {{ tox_extra_args }} as -vv is
    in tox_extra_args by default and results in interleaved debug
    output in the ini output making it invalid ini content.
  * Update the tox siblings tox config parser to look for renamed
    environment dir locations.
  * Stop using whitelist_externals and use allowlist_exteranls
    because whitelist_externals is removed and external commands that
    are not explicitly allowed produce errors.
  * Make the tox version configurable in ensure-tox as some users
    may not be able to easily upgrade to tox v4.
  * Escape literal # chars in tox.ini as they are treated as comments
    when in the command strings now.
    https://github.com/tox-dev/tox/issues/2617

Change-Id: I38e13b4f13bb1b2d6fb7e5c70b708e9bb016a455
2022-12-07 15:14:16 -08:00
Ian Wienand 66345d0ab7
linters: lint that library files don't start with #!
See notes in I4e6e85156459cca032e6c3e1d8a9284be919ccca, this ensures
the library files don't start with #! so they will run under the
Ansible interpeter on the remote node.

Change-Id: I8754d0962e8453d2030c589be23560e454e0052e
2022-09-16 07:11:58 +10:00
Ian Wienand 255ed06075 ansible-lint: disable progressive mode
Progressive mode

 ... makes the linter return a success even if some failures are
 found, as long the total number of violations did not increase since
 the previous commit.

I have found what I think is inconsistent matching of the errors
between runs and I'm not sure it isn't hiding problems.  We are
linter-clean and gate on the linter passing, so we don't need to do
this two-pass system.

Change-Id: I47be01a095d80dfb4d15f90da7bce49c3d42a2dd
2022-07-27 17:16:19 +10:00
Sorin Sbarnea 0eaa5cf59a Upgrade ansible-lint to 5.0
- bumps ansible-lint to 5.0
- updates our custom rules to make them compatible with 5.0
- replace custom module mocking with native ansible-lint ones
- remove custom call of ansible-playbook --syntax-check as now this
  is done by ansible-lint
- assured molecule vars are hosted under a vars/ folder in order to
  avoid confusing linter detection.
- replaced custom rule for loop var names in role as now this this an
  optional core feature of the linter (see config)
- replaced custom rule no-same-owner with opt-in one (see config)

Change-Id: I233fae8c9036d295968a97ee80e07fde8846c633
2021-03-17 15:44:43 +00:00
Jan Zerebecki 8434951b7c
Better error output for update-test-platforms.py
When update-test-platforms.py autogenerates parts that are checked into
git and the generated and commited content is different the CI error is
hard to understand. Make it explicit what caused this. Output the diff,
so one can see the difference.

Change-Id: Ib9751c6ad6dc55514177934e0db327c5d08b74eb
2020-12-11 16:49:44 +01:00
Sorin Sbarnea 3febd1423a Enable progressive mode with ansible-lint
Makes use of newer feature which produces failures only when current
commit is adding new violations, exiting ones being considered
as already known.

To prove it works, it also removes the E208 from warn, making it
an error. Still, the final linting result is a success because
these violations were already present before current commit.

Change-Id: Ia858f2a3e71d9634e9d90e890d82714105e8f8c9
2020-11-07 03:20:19 +01:00
Sorin Sbarnea 5e9f91764e Bump ansible-lint to 4.3.0
- Bumps linter and make use of its auto-detection
- Temporary skips linting test-playbooks/ to match previous behavior
- Documents skips in a way that makes it easy to maintain the rules
- Keeps linter config in standard location, so it can be loaded
  regardless how is called.

Change-Id: Ic379c91fa9385473f6ec2af91e61953dc10c1f54
2020-05-28 16:48:14 +01:00
Albin Vass a38becca99 Fix py27 tox
siblings: python2.7 ConfigParser has no __getitem__
Constrain soupsieve to <2 for python2.7
Add python2-dev to bindep for subprocess32

Change-Id: If9d6a0ae1a62a94dcec11f6bf637ffee7f0f4fc9
2020-05-27 17:12:32 +02:00
Zuul 60a1f1a729 Merge "Update flake8 ignore rules to match Zuul" 2020-05-19 21:09:11 +00:00
James E. Blair 9643fb1ace Update flake8 ignore rules to match Zuul
These are the standard flake8 ignore rules for Zuul; synchronize
them here for consistency.

Change-Id: Ibdd105b3ea09f394a548e2bec13d0b53dea93363
2020-05-18 12:34:34 -07:00
Monty Taylor 6ab78932a4 Set ignore_basepython_conflict to true
Otherwise we miss that we're not using 3.8 in the 38 env.

Fixes the cgi.escape issue that we didn't catch earlier.

Change-Id: I604bd18b70950580f60714598624d493f4a5060f
2020-05-09 20:43:52 +00:00
Sorin Sbarnea 33461bbecc Enable yamllint
Adds yamllint to the linters with a minimal configuration, some
rules are disabled to allow us to fix them in follow-ups, if
we agree on them.

Fixes invalid YAML file containing characters inside block.

Fixes few minor linting issues.

Change-Id: I936fe2c997597972d884c5fc62655d28e8aaf8c5
2020-05-04 17:47:11 +01:00
Sorin Sbarnea 0c3ab1ce8d Improve linters execution
- Avoid calling ansible-lint for each argument as this slows down
  its execution time considerably (105s -> 41s)
- Prints list of offending files when git reports dirty, avoiding
  confusions errors.

Change-Id: I4b6c6a7935febc4934d0fc6a55f7f9d3e6e87942
2020-04-23 14:00:18 +01:00
Zuul b3358c7559 Merge "Make linting use of find portable" 2020-04-21 20:00:15 +00:00
Sorin Sbarnea 523272a3a6 Make linting use of find portable
Current implementation failed to lint on BSD-like systems due
to the use of GNU specific find feature.

Change-Id: I0dbf635eb5b2e9b37dd0062ca165a5b51d090543
2020-04-21 13:25:27 +01:00
Andreas Jaeger 2026f1825b Use main.yaml, not .yml
There are over 490 .yaml files but only a few .yml, let's rename to be
consistent.

Add a test to block .yml files.

Change-Id: I2f1354de82f231154d926b51d9812b1e9c1a6202
2020-04-20 17:44:49 +02:00
Monty Taylor c800638473 Use a zuul_* and add an .ansible-lint file
ansible-lint now supports settings in a file, including exclude_paths.
This lets us simplfy the ansible-lint command. Also, stop installing
zuul and just use a fake zuul_return and zuul_console
to fake out ansible-lint.

Change-Id: I1482a9ab915cec2d45695b60cdbeb93d58cb392c
2020-03-13 20:26:06 +01:00
Ian Wienand 39bebd1623 Add tox env for update-test-platforms
It seems some versions of ruamel do a better job at preserving
comments than others.  Create a tox env to regenerate the lists with a
version we know works.

Change-Id: I13503890fd3c18487281e3394b003e9f36cd24f2
2020-02-07 09:58:10 +11:00
Sorin Sbarnea 0c8560de77 Making opensuse-tumbleweed jobs nv
Avoids ocasional failures caused by broken mirror repository.

We keep it, as it proves useful in discovering other issues from time
to time.

See https://zuul.opendev.org/t/zuul/builds?job_name=zuul-jobs-test-multinode-roles-opensuse-tumbleweed

Also assures linting catches errors caused by changes that missed to
regenerate the project stanza.

Change-Id: Id988a51175e8daa4f3c9725822dadcd7cd7aeb0e
2020-01-24 09:41:34 +01:00
Kevin Carter 7f156f22c7
Make pre-molecule tox playbook platform agnostic
This change updates the tox molecule playbook to make it platform
agnostic by moving the OS specific package lists into var files
that will be dynamically loaded based on the operating system
the playbook is being executed against. This is being done so
that we can use molecule on OS's like CentOS, Ubuntu, SUSE,
Debian, etc. While the only platforms supported at this time
are Red Hat based, this change will allow additional var files
to be added enabling additional platforms without needing any
chanages to our task structure.

> The linters job has been updated to exclude vars files from the
  ansible lint playbook commands. This is done because the playbook
  commands assume all found files are playbooks.

Change-Id: I88f3551838e3676374d0a795631c8769ba40fbf4
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2020-01-02 09:49:28 -06:00
James E. Blair ec8a58ddb7 use-buildset-registry: Vendor pytoml and remarshal
In order to edit the V2 registries.conf file used by podman, we
need to be able to manipulate toml from ansible.  There is no
standard library or Ansible support for that now, and we don't want
to install any python packages on the remote node.  Therefore,
vendor the remarshal and pytoml code into this role.

This is done in a standalone commit for easier review and auditing.

The originating projects are:

  https://github.com/dbohdan/remarshal
  https://github.com/avakar/pytoml

And both are MIT licensed.  Appropriate headers are added where
necessary.

Note that pytoml has been concatenated into one file in order to
adhere to Ansible's requirements for python modules.

Change-Id: I679ea5eb5cb29591be09d2f1b712400c49158abd
2019-12-03 14:10:23 -08:00
Jeremy Stanley 2711d10186 Be consistent about spaces before and after vars
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.

Adjust the new violations accordingly and remove the rule
206 exclusion.

Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
2019-08-07 20:17:41 +01:00
Jeremy Stanley b6d2616275 Don't compare to literal True/False
With the arrival of ansible-lint 4, comparisons to literal boolean
values are now forbidden. Adjust the new violations accordingly and
remove the rule 601 exclusion.

Change-Id: I18ba2d7d41fabaff35d10d520037188c7d9d1249
2019-08-01 19:01:37 +02:00
Jeremy Stanley 1ed78d714f Advance ansible-lint cap to test with 4
Running latest ansible-lint raises violations for rules 204, 206 and
601 so skip them for now.

Change-Id: Ibc62833cfc4b2fae728bc6c944f263af3fc2b89d
2019-06-26 18:09:51 +00:00
Monty Taylor 47ceef043a Shift doc requirements to doc/requirements.txt
We have doc requirements in the normal test-requirements list.
This is causing problems for some sphinx related things which
want python3 conflicting with our py27 test job.

Change-Id: I1c4ddba56377097f1ebf39051a496577d38f0494
2019-05-09 13:58:56 +00:00
James E. Blair 4b3adfeea6 Organize documentation by subject area
This splits all of the current job and role documentation into
files by subject area so that jobs and roles are easier for users to
find.

This will require that any future new jobs or roles add a line to the
appropriate area of the documentation, since that can no longer be
done automatically.  A linter check is added to ensure that every
job and role continue to be documented.

After this refactor, we can begin to enhance the documentation pages
so that they include narrative documentation and subsections.

Change-Id: Ia6f0e89b57e3cb0d7d1745206384c946506d7ea0
2019-03-25 14:21:04 -07:00
James E. Blair be0ae67dff Update zuul ansible location
With multi-ansible support, the location of the zuul plugins/modules
has moved.

Change-Id: I8e2cf1f2276bca5a912fed52062628b3b9d0d6d9
2019-03-18 09:13:11 -07:00
Paul Belanger c0a1ebb310 Set ANSIBLE_ACTION_PLUGINS for linters
The zuul_return plugin was recently converted to an action plugin, make
sure we also export that path for ansible-lint.

Change-Id: I9e497ef4258753d4d305f8caab6d47e469a38ccd
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-01-18 17:48:29 -05:00
Andreas Jaeger 93ddc0247d Fix flake8 3.6.0 errors
Flake8 3.6.0 now warns about both line break after and *before* binary
operator, you have to choose whether you use W503 or W504. Disable the
newer W504.

Also, ignore warning about invalid escape sequence in regex.

Change-Id: Ibfbd5df21e01d5a7bd44a216ff63bc805dd5c186
2018-10-25 08:09:28 +02:00
Monty Taylor 5cd8fbeb58
Update ANSIBLE_LIBRARY to use envsitepackagesdir
We install zuul via test-requirements, so the zuul files should be in a
directory under the site-packges of the virtualenv that tox installed it
in to. Update the path to point correctly to that location.

Remove the ansible-lint skip tags which should now work because the
library path should be pointing to a location that actually holds the
content.

Change-Id: If2d4b39267c4b9a3102a951143b568f8447af8d9
2018-10-18 09:23:25 -05:00
Tobias Henkel b35f47190a Revert "Extract pep8 messages for inline comments"
There seems to be a problem with openstack-zuul-jobs [1].

This reverts commit 421f533b61.

[1] http://logs.openstack.org/86/610386/3/gate/openstack-zuul-jobs-linters/a8c857d/job-output.txt.gz#_2018-10-18_07_14_42_431592

Change-Id: Iac5236221e33df4613cc2f7a2edcffc323902320
2018-10-18 09:11:38 +00:00
Monty Taylor 421f533b61 Extract pep8 messages for inline comments
Move the zuul requirement from test-requirements into the linters
section of tox.ini because we need it as a src install in the tox
env, so that we can reference the ansible library location.

Change-Id: I089c69b539107bdbc25791f5730502a4f46e7cf6
2018-10-17 13:45:00 -07:00
James E. Blair d661ad4042 Add zuul to the linters tox env
So that the syntax check which runs on playbooks can find zuul_return.

Add re2 library to bindep, it's needed for zuul.

Change-Id: I93213ee19652c0ec9f4bc699204e9445a1e891c8
2018-10-02 08:15:52 +02:00
James E. Blair 46389b5187 add-build-sshkey: Remove only the master key
This implements a module to directly interact with the ssh-agent
so that the master key may be removed from the ssh-agent without
removing any per-project keys.

Change-Id: Ife91ad8afa9b41b0e779a832e298aca8d61ae98b
2018-09-05 09:26:35 -07:00
Andreas Jaeger 834cd2dd21 Update docs build job
Use sphinx-build in tox.ini.

Change-Id: I5c8af6341ac8e84bfe0aa0fb55be3cd78df797ca
2018-08-21 21:12:22 +02:00
Ian Wienand eee316ce7b Enable Python 2.7 unit tests
Add a testenv:py27 environment that overrides basepython to 2.7

Unfortunately implicit namespace packages are a Python3 thing [1] so
we have to scatter a few __init__.py's around for the test loader
under python2 to be able to find the unit test directories.

Update documenation to mention this

Needed-By: https://review.openstack.org/592768

[1] https://www.python.org/dev/peps/pep-0420/

Change-Id: I9a653666e8a083fb7f3fbb92589fe0467a41e6e6
2018-08-17 10:23:20 +10:00
James E. Blair 1548d82ffd Add a python unit test framework
Add a unit testing framework for python roles.  Thanks to Matthew
Treinish for the suggestion of how to perform discovery (and much of the
code which is copied from Tempest).

Change-Id: Iec95dd1026a41614def57c65c3faa0516a682a5a
2018-07-25 09:11:25 -07:00
Colleen Murphy d1956bd9f7 Fix ansible-lint job
The 'linters' tox environment was not running on roles, it was only
running on playbooks. This change adds a command to the linters
environment to ensure all roles are linted.

Since these weren't being linted, there were some problems with them.
The first was a warning about usage of the shell module. Both of these
usages seemed appropriate, so this patch adds a skip_ansible_lint tag to
each task that was failing. The second is a warning "no action detected
in task" for zuul modules. This is due to the fact that the linter
cannot find the custom module. One option is to set the ANSIBLE_LIBRARY
path to point to the zuul ansible library directory[1], but the linter
virtualenv does not actually have zuul installed. Instead, we just
disable the linter for the failing tasks.

This also cleans up a comment in the tox file that was referring to a
nonexistent zuul job.

[1] https://github.com/willthames/ansible-review/issues/16

Change-Id: Ie49da9a09733b623bb25c5a4c8aa07eacacf4b33
2018-03-28 23:10:22 +02:00
Andreas Jaeger 66ec61d67c Simplify ansible invocations in tox.ini
We have no legacy files in here, so no need to skip them.
Also, sync invocations of ansible lint and ansible syntax checks.

Change-Id: I003cf6a46d1dc376d83118e5e5c405aa54d33f22
2017-10-28 16:56:30 +02:00
Andreas Jaeger 7c4734f398 Run ansible-lint on repo
This change follows  I16186c929e7d0e6e34b35271559e555255a52b00 to run
openstack-zuul-jobs-linters on the repository.

Also, silence remaining lint role.

Depends-On: I46c53229f878fc707bab627654d820b891d9f625
Change-Id: I76abf21ea51c5f7553639475e408e06768fb1016
2017-10-27 07:49:02 +02:00
Jeremy Stanley 4aa181a77c Stop running ansible-lint on playbooks
The following form raises "ANSIBLE0013 Use shell only when shell
functionality is required:

  shell: '! something'

Chances are its heuristic fails to look for use of the negation (!)
shell operator.

Rather than exclude ANSIBLE0013, just stop running it completely for
now since we've had to do the same on openstack-zuul-jobs in the
I14896450b16f8e65128804b44a643da63580812d change.

Change-Id: Ie9da24a4ab20d9dd0bf7ebb2e3d957b9d8563900
Depends-On: I60daa67e7154d2bf621305e0e0aa6f4db49033c1
2017-09-28 20:14:53 +00:00
David Moreau Simard 83bdedd4b2 Revert "Fix lint issue"
This also adds an exception to ansible-lint, ANSIBLE0006, because
we want to use "apt-get update" for updating the apt cache [1]

[1]: https://review.openstack.org/#/c/492716/

This reverts commit 5fe5b6ca1a.

Change-Id: Icd79198964e86c7a2c73102f3e4d845d161b924d
2017-09-07 10:29:14 -04:00
Monty Taylor 82af975879
Update tox/test-requirements
In working on a different patch it became clear that flake8 wasn't
actually running such that it found errors. Updated the
test-requirements to match what's in zuul and all started working (and
failing) properly.

Change-Id: Icfdb1fedbd92ff49484b116a0879686581274a25
2017-09-02 14:17:32 -05:00
Jenkins c327263467 Merge "Rework tox -e linters" 2017-07-20 05:14:24 +00:00
Paul Belanger f40a7fbe1e
Install zuul to access zuul_console
When we lint our playbooks, we need to also have zuul install our
custom action plugins.

Also default to python3 so we can properly install zuul into tox.

Change-Id: Ib7e39e43005aa73c9d482af6becef3408d097e15
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-19 13:06:30 -04:00
Paul Belanger 74d237a3f7
Rework tox -e linters
Move our syntax-check to be first, logic being if we have bad syntax
and ansible-lint will most likely fail.

Clean our redirects to only output failures. It would be nice is
ansible-playbook / ansible-lint could use the same redirects, but
sadly they cannot.

Change-Id: Ia3c5966d60d28b81e2f2c6a446116bb37df6ab23
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-18 10:48:54 -04:00
Paul Belanger 8826f3afd9
Rename pep8 to linters for tox
Using linters makes more sense, for playbooks / roles. Make the
switch.

Change-Id: I9694f95c3c041f9816c83d60680bfde50e1fb858
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-17 16:54:29 -04:00
Paul Belanger 8e812f1eb4
Include ansible-playbook syntax-check for tox pep8
Syntax check our playbooks / roles when we run pep8. This way we can
quickly tests if our playbooks are valid.

Change-Id: Id9b44216820a2b0702e1cfa46cd7603c89ed40d3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-17 16:53:55 -04:00