Commit Graph

146 Commits

Author SHA1 Message Date
James E. Blair a077776d2f Pin kazoo to 2.9.0
Kazoo recently make a release which appears to be having an effect
on our unit tests.  The number of times the unit test job encounters
connection errors which lead to timeouts has increased since the
release.

Until we understand further, keep to the old release.

Change-Id: I470f342321f8c4e514716f4eb4d5693d1586d515
2024-02-15 16:18:15 -08:00
Jeremy Stanley 4f90164ba7 Refresh Keycloak OpenID Howto for latest Keycloak
Going back through the Howto, many of the Keycloak admin UI screens
have changed layout and prompts, making the guide less
straightforward to follow. Update it to match Keycloak 23 (latest at
time of writing).

It also appears that pyJWT 2.5.0 fixed the bug which caused it to
break on encountering unsupported keys, so the noted step for
disabling rsa-enc-generated should no longer be necessary (this has
also become somewhat harder to do in more recent Keycloak versions).

Newer Keycloak seems to get confused by trailing slashes on origins,
which causes problems for its CORS headers, so fix it and add a
parenthetical note as a reminder.

Change-Id: I585439d4718fafd7075a3ed31ebcd10139dc355a
2024-02-14 17:05:41 +00:00
James E. Blair 81d3c86115 Pin paho-mqtt to <2
Change-Id: I98308d1fdaef08a1dfdb152bab8382beb10eb87f
2024-02-10 08:08:25 -08:00
Zuul bd11c4ff79 Merge "Add gcloud pubsub support to Gerrit driver" 2023-10-04 03:29:42 +00:00
James E. Blair 456205e7be Switch from fb-re2 to google-re2
At the time we began to adopt the re2 library, fb-re2 was the only
Python wrapper.  Google-re2 is available now, is developed and
released by the maintainers of re2 as part of the re2 project,
and distributes manylinux wheels which simplifies the Zuul build
process.  We have already adopted google-re2 for our Ansible
environments, so this simplifies packaging.

Change-Id: Id1b47d3b980faaf48a3d2b7959925d8c73bd3db1
2023-08-28 15:00:46 -07:00
James E. Blair 36276806b8 Add gcloud pubsub support to Gerrit driver
This adds support for the Google Cloud Pub/Sub service to the
Gerrit driver.  It is very similar to Kafka.

Change-Id: Ib6e4dc01058b74e1042bfd1deb9fa4f3f43f7a36
2023-08-02 14:50:28 -07:00
James E. Blair e6e978615f Add AWS Kinesis support
Gerrit has an event plugin for AWS Kinesis (which looks sort of
like Kafka, but without server side checkpoints.  or ordering.).

Add support to the Gerrit driver for it for sites which would
rather use that than ssh.

Change-Id: I942845ac16bf220664499f14ff7c4086ff65de2a
2023-07-25 11:04:19 -07:00
James E. Blair 4dc0962a49 Add Kafka support to Gerrit
Gerrit supports a number of pub-sub plugins which can act as
alternatives to stream-events.  These can often be easier for
users to configure than ssh access and have the advantage of
providing queueing and delivery guarantees for messages.

This change not only adds support for Kafka, but is meant as
a template for adding support for other Gerrit pub-sub plugins
as well.

Change-Id: Ib03d8cb9ef709b625d2717a09125930548c86a22
2023-07-15 14:41:23 -07:00
Clark Boylan 529368ffee Switch to sqlalchemy 2.0
Our test suite no longer complains with RemovedIn20 warnings for
sqlalchemy 2.0 incompatibilities. Check if this works now.

Change-Id: I0ffab3788493dcddf39a1262813293abb6611c19
2023-02-01 09:18:59 -08:00
James E. Blair c0985cff39 Pin sqlalchemy to <2.0 and revert "Update git"
2.0 has breaking api changes:
https://www.sqlalchemy.org/blog/2023/01/26/sqlalchemy-2.0.0-released/

Revert "Update git"

This reverts commit 944b9852c9.

Upstream Debian has updated git to 2.30.2-1+deb11u1 which patches git
for the issues we manually patched to cover. We don't need the manual
patch anymore and can switch to the distro hence this revert.

Reviewers should double check that the image build process installs the
expected 2.30.2-1+deb11u1 version.

Change-Id: I02c4746a0f6651dfcd04ea88ccfd7d62e481d490
Co-Authored-By: Clark Boylan <cboylan@sapwetik.org>
2023-01-30 11:45:27 -08:00
Clark Boylan 21ccbe8ba2 Unvendor kazoo locks recipes
These recipes were vendored so that we could carry this fix locally:
  https://github.com/python-zk/kazoo/pull/650

It appears that this fix has been merged and included in kazoo>=2.9.0 so
we include that as the minimum version and drop the vendored file.

This also fixes the isSet() deprecation warning as upstream kazoo has
switched to is_set().

Change-Id: Ide48e9f949e083b658775b74db3856b118fc5d69
2023-01-11 11:16:34 -08:00
James E. Blair 343904e1a4 Use unsafe_skip_rsa_key_validation with cryptography
This is a partial revert of c4476d1b6a
which added the use of a private flag to skip unecessary (for us)
cryptography checks.  The cryptography package has now normalized
that flag into a parameter we can pass, so use the new param and
update the dependency to require the version that supports it.

Change-Id: I1dfa203525e85020ccf942422ad3cc7040b851dd
2023-01-11 10:37:24 -08:00
James E. Blair 3780ed548c Unpin JWT and use integer IAT values
PyJWT 2.6.0 began performing validation of iat (issued at) claims
in 9cb9401cc5

I believe the intent of RFC7519 is to support any numeric values
(including floating point) for iat, nbf, and exp, however, the
PyJWT library has made the assumption that the values should be
integers, and therefore when we supply an iat with decimal seconds,
PyJWT will round down when validating the value. In our unit tests,
this can cause validation errors.

In order to avoid any issues, we will round down the times that
we supply when generating JWT tokens and supply them as integers
in accordance with the robustness principle.

Change-Id: Ia8341b4d5de827e2df8878f11f2d1f52a1243cd4
2022-11-15 13:52:53 -08:00
James E. Blair 6a5a2d0f6f Fix test_ensure_cloned and JWT tests
This is two changes in one since they both fix breakage due to
external dep changes:

Git no longer allows file urls in submodules by default, but one
of our unit tests relied on that behavior (in order to verify that
we can clone repos with submodules).  Run the submodule add command
in the test with a flag which allows file urls.

Pin PyJWT <2.6.0

2.6.0 causes the following error:

2022-10-20 15:58:04,800 cherrypy.error.140001002710128   ERROR    [req: 86efa7775e88473a9e9d5e54f0c83050] [20/Oct/2022:15:58:04] HTTP
Traceback (most recent call last):
  File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 702, in _do_respond
    response.finalize()
  File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 901, in finalize
    content = self.collapse_body()
  File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 859, in collapse_body
    new_body = b''.join(self.body)
  File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_json.py", line 24, in encode
    for chunk in _encode(value):
  File "/usr/lib/python3.10/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.10/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ImmatureSignatureError is not JSON serializable

Change-Id: I2e345f24ea0a62ce8d9dbe1c066438f194e7075c
2022-10-20 16:01:08 -07:00
James E. Blair ce40b29677
Add support for configuring and testing tracing
This adds support for configuring tracing in Zuul along with
basic documentation of the configuration.

It also adds test infrastructure that runs a gRPC-based collector
so that we can test tracing end-to-end, and exercises a simple
test span.

Change-Id: I4744dc2416460a2981f2c90eb3e48ac93ec94964
2022-09-19 08:42:28 +02:00
Andy Ladjadj 25a96ed82f Pin elasticsearch to <8.0.0
The latest release 8.0.0 of elasticsearch breaks Zuul when using elasticsearch connection

Trace:
...
File "/usr/local/lib/python3.9/dist-packages/zuul/driver/elasticsearch/con>
self.es = Elasticsearch(
TypeError: __init__() got an unexpected keyword argument 'use_ssl'

Change-Id: Idd4cf9d67f60374fa1e8b9a816845829de31e9f1
2022-03-16 16:02:32 +00:00
Benjamin Schanzel 73e3f025fb
Unpin github3.py<3.0.0 requirement
Since [1] was solved by the 3.1.0 release of github3.py, we can relax
the dependency constraint accordingly.

[1] https://github.com/sigmavirus24/github3.py/issues/1057

Change-Id: Ia31cb4eff50964fe077f9d1d538ca51b4e64c51c
2022-02-14 17:55:38 +01:00
James E. Blair 215c96f500 Remove gearman server
The gearman server is no longer required.  Remove it from tests and
the scheduler.

Change-Id: I34eda003889305dadec471930ab277e31d78d9fe
2022-01-25 06:44:17 -08:00
James E. Blair 0fa86b525c Pin tzlocal to avoid warnings
Apscheduler requires tzlocal/pytz, and they have introduced a warning
which we can avoid by pinning to an earlier version.  This is not likely
to be fixed in pytz 3.x, but will be in 4.x.

See https://github.com/agronholm/apscheduler/discussions/570

Change-Id: I9c0555ef107d411b8e2fac9dabc7547459e5ffa7
2022-01-12 09:56:43 -08:00
Tobias Henkel 1a08b7542f
Pin github3.py to <3.0.0
The latest release 3.0.0 of Github3.py breaks Zuul when using Github
Enterprise <3.1 [1].

[1] https://github.com/sigmavirus24/github3.py/issues/1057

Change-Id: I1c199cd43985458f88137b218916b156972494ad
2021-11-03 16:43:12 +01:00
Tobias Henkel fc8a7df5d0
Pin CacheControl to <0.12.7
A recent update of CacheControl broke parts of the github api in zuul
which resulted in empty bodies in the response [1] when requests were
cached.

[1] Trace:
GET https://(...)/api/v3/(...)/files?per_page=100 result: 200, size: 0, duration: 67
ERROR zuul.GithubEventProcessor: Exception when processing event:
Traceback (most recent call last):
  File "/opt/zuul/lib/python3.8/site-packages/github3/models.py", line 168, in _json
    ret = response.json()
  File "/opt/zuul/lib/python3.8/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 346, in run
    self._process_event()
  File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 404, in _process_event
    change = self.connection._getChange(
  File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 1334, in _getChange
    pull = self.getPull(change.project.name, change.number,
  File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 1706, in getPull
    pr['files'] = [f.filename for f in probj.files()]
  File "/opt/zuul/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 1706, in <listcomp>
    pr['files'] = [f.filename for f in probj.files()]
  File "/opt/zuul/lib/python3.8/site-packages/github3/structs.py", line 90, in __iter__
    json = self._get_json(response)
  File "/opt/zuul/lib/python3.8/site-packages/github3/structs.py", line 137, in _get_json
    return self._json(response, 200)
  File "/opt/zuul/lib/python3.8/site-packages/github3/models.py", line 170, in _json
    raise exceptions.UnexpectedResponse(response)
github3.exceptions.UnexpectedResponse: 200 [No message]

Change-Id: I5921e98d4252d4c96d116fe514b8637fad86a7fb
2021-10-30 11:33:54 +02:00
James E. Blair 9f2e3e23e9 Pin gear < 0.16.0
Gear is ready to make a release, but we're about to remove it.
Pin to less than the upcoming release to retain stability while
we finish up removal.

Change-Id: I4822c212f0d1fc018e92d01d162868e274dbd6ba
2021-07-29 12:47:05 -07:00
Tristan Cacqueray 0dbd8c0784 prometheus: add options to start the server and process collector
This change adds a new prometheus_port option to start a metric server
to be scrapped by a prometheus service. By default, the server exposes
process informations.

Change-Id: Ie329df6adc69768dfdb158d00283161f8b70f07a
2021-04-26 14:47:36 +00:00
Paul Belanger c1e39ff457 Include database requirements by default
An SQL database is an additional requirement so we should also include
the client libraries for it.

Change-Id: Id3a8b55928d9ce9e8764f18a64a42bb2f14f5ecb
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2021-02-23 16:38:47 -05:00
Zuul dc2d6c847d Merge "Bump pyjwt to 2.0.0" 2021-01-23 01:35:22 +00:00
Matthieu Huin daa51d766f Bump pyjwt to 2.0.0
The release of pyjwt 2.0.0 changed the behavior of some functions, which
caused errors. Fix the errors, use pyjwt 2.0.0's better handling of JWKS,
and pin requirement to 2.X to avoid future potential API breaking changes.

Change-Id: Ibef736e0f635dfaf4477cc2a90a22665da9f1959
2021-01-14 12:35:18 +00:00
Zuul b514d50c57 Merge "A Zuul reporter for Elasticsearch" 2021-01-13 18:51:47 +00:00
Fabien Boucher 89b6803085 A Zuul reporter for Elasticsearch
It has the capability to index build and buildset results.

With the help of tools like Kibana, advanced analytics
dashboard could be built on top of the Zuul Elasticsearch
index.

Optionally job's variables and zuul_return data can be
exported along with build results under the job_vars and
job_returned_vars fields.

Change-Id: I5315483c55c10de63a3cd995ef681d0b64b98513
2021-01-13 16:44:22 +01:00
Sorin Sbarnea c82282aa7e Remove importlib-resources pinning
Removes outdated workaround on importlib-resources.

This fixes issue with py39+, where stdlib already provides it.

Change-Id: I82264c293ff9072e970e81417658fc934cccea2d
2021-01-08 16:33:47 +00:00
Matthieu Huin 9092c1f43a Pin pyjwt to 1.7.1
Release 2.0.0 on December 22, 2020 brought breaking changes. Pin to the
last working version until we can fix the breaks properly.

Change-Id: I2e6cc36d373f8c774eda5f5caec9868f74023729
2020-12-23 09:39:06 -08:00
Ian Wienand 17d48efc50 Revert "requirements: temporarily pin urblib3 for zuul-client"
This reverts commit a58b29c6bf.  The
dependency is uncapped in zuul-client now.

Change-Id: I9d31326c444cf225442c0b46e040312be9a2ff73
2020-11-13 11:36:54 +11:00
Ian Wienand a58b29c6bf requirements: temporarily pin urblib3 for zuul-client
The reasoning is explained inline.  We can remove this when
zuul-client is fixed.

Change-Id: I431d9202cabcc9bf7b60fd643c55b557f6505185
2020-11-13 11:15:52 +11:00
Zuul a1d4aa0e55 Merge "ConfigItem*Exception : add failure context" 2020-09-23 13:34:12 +00:00
Tobias Henkel b9c2d25dce
Don't match branch protection rule patterns locally
Branch protection rules in github are fn patterns which are currently
matched locally in zuul. This is error prone and can lead in edge
cases to wrong matches resulting in wrong enqueue decisions into gate
pipelines. When requesting branch protection rules in github we also
can request the matching refs along with the rules. This is much safer
since we can plain text match them against the change branch.

Change-Id: Ic995d4b2e16a5d741f0209fa9236959d8f4d10b9
2020-09-14 13:21:34 +02:00
Ian Wienand c8d93c955c ConfigItem*Exception : add failure context
These various exceptions are raised as certain parts of the
configuration are parsed.  Although they give you a lot of information
about the issue, they don't give you any context for what part of your
configuration caused the error to be raised.

They are all working on a particular item of the configuration that I
think it would be very useful to show in the error output.

This was inspired by me putting playbooks in the .zuul.d directory,
where they were read as config files.  In hindsight it was obvious,
but it took me a while to figure out what was going on.  With this it
would have reported something like:

  Zuul encountered a syntax error while parsing its configuration in
  the repo org/project1 on branch master.  The error was:

  Configuration item has more than one key.  Each zuul.yaml
  configuration file must be a list of dictionaries with a
  single key, for example:

  - job:
      name: foo
  ... blah ...

  The incorrect values are:

  hosts: all
  tasks:
  - debug:
      msg: blah
    name: foo

which would have clued me in immediately.

This truncates the message if it is too long; which might be the case
if dumped a playbook or foreign config file.  However I think
practically most errors are typo level on jobs/templates/etc. that
will show nicely.

The YAML version is updated to support non-sorted dumping of the keys.
This is particularly helpful with Python 3.7+ and default ordered
dicts.

Change-Id: I4851a5a796fa452a023e0e6193fc724ae0967a44
2020-08-06 10:50:32 +10:00
James E. Blair 6fae92ec71 Require kazoo 2.8.0
This has a fix required for using TLS.

Change-Id: I6d29ca3bf31b1f50cebc245d3db2095ab65921c4
2020-07-15 09:33:48 -07:00
Fabien Boucher e8935676a8 Update cherrypy and cheroot requirement pinning
Cheroot 8.3.1 have been released with the https://github.com/cherrypy/cheroot/issues/263
fix.

Change-Id: I83aad30a6145ed2c7689d6c25c624c453c07ed2b
2020-07-13 16:51:23 +00:00
Tristan Cacqueray 170542089a requirements: add explicit reference to dateutil
Zuul is using a dateutil provided transitively. This change
makes the requirements explicit.

Change-Id: I672d409a1bb241e5d199c77acdd51a8260d99ee2
2020-05-26 11:36:35 +00:00
Zuul a655aee0f1 Merge "Ensure we use recent enough virtualenv" 2020-05-13 19:50:09 +00:00
Sorin Sbarnea 268bab1362 Fix cherrypy dependency conflict
Current set of dependencies brings latest cherrypy which is
incompatible with cheroot we need. This pins cherrypy to last
version that is compatible. The current verision of pip will
still install cherrypy, but the upcoming depsolver is unable to
resolve the conflict.

We also add a fail-safe `pip check` that runs the new depsolver
and should prevent a similar regressions from appearing.

Change-Id: I2b0fa43b906d0919854d30da0ee825002444b201
2020-04-29 22:07:44 +00:00
Jan Kundrát 7fa388cae3 Ensure we use recent enough virtualenv
I installed Zuul 18.0 on a CentOS 7 with Python 3.6 from RH SCL repos
which comes with virtualenv==15.1.0. This is effectively too old to
provision the virtual environments for Ansible, likely because that
version of virtualenv hardcodes PIP 9. During the installation,
zuul-executor dies with this error:

 Installing collected packages: MarkupSafe, jinja2, PyYAML, pycparser, cffi, six, pynacl, cryptography, bcrypt, paramiko, ansible, XStatic-Bootstrap-SCSS, pyfakefs, XStatic-Patternfly-Bootstrap-Treeview, wcwidth, pyparsing, pyperclip, cmd2, pbr, stevedore, PrettyTable, cliff, XStatic, XStatic-Patternfly, wrapt, debtcollector, msgpack, pytz, netaddr, iso8601, netifaces, Babel, oslo.i18n, oslo.utils, oslo.serialization, itsdangerous, Werkzeug, click, Flask, SQLAlchemy, Flask-SQLAlchemy, XStatic-jQuery, Mako, python-editor, python-dateutil, alembic, Flask-Migrate, pygments, junit-xml, Flask-Script, Frozen-Flask, extras, linecache2, traceback2, argparse, unittest2, fixtures, python-mimeparse, testtools, python-subunit, XStatic-DataTables, ara, os-service-types, urllib3, chardet, idna, certifi, requests, keystoneauth1, jsonpointer, jsonpatch, appdirs, munch, decorator, jmespath, requestsexceptions, dogpile.cache, openstacksdk, dictdiffer, websocket-client, pyasn1, rsa, pyasn1-modules, cachetools, google-auth, oauthlib, requests-oauthlib, kubernetes, python-string-utils, ruamel.yaml.clib, ruamel.yaml, openshift, google-resumable-media, protobuf, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-storage
 Running setup.py install for PyYAML: started
 Running setup.py install for PyYAML: finished with status 'done'
 Running setup.py install for ansible: started
 Running setup.py install for ansible: finished with status 'done'
 Running setup.py install for XStatic-Patternfly-Bootstrap-Treeview: started
 Running setup.py install for XStatic-Patternfly-Bootstrap-Treeview: finished with status 'error'
 Complete output from command /var/lib/zuul/ansible-bin/2.6/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-24z_rrjk/XStatic-Patternfly-Bootstrap-Treeview/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6ie_p40b-record/install-record.txt --single-version-externally-managed --compile --install-headers /var/lib/zuul/ansible-bin/2.6/include/site/python3.6/XStatic-Patternfly-Bootstrap-Treeview:
 Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/setuptools/__init__.py", line 10, in <module>
 from setuptools.extern.six.moves import filter, filterfalse, map
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/setuptools/extern/__init__.py", line 1, in <module>
 from pkg_resources.extern import VendorImporter
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3017, in <module>
 @_call_aside
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside
 f(*args, **kwargs)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3045, in _initialize_master_working_set
 dist.activate(replace=False)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2577, in activate
 declare_namespace(pkg)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2151, in declare_namespace
 _handle_ns(packageName, path_item)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2091, in _handle_ns
 _rebuild_mod_path(path, packageName, module)
 File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2120, in _rebuild_mod_path
 orig_path.sort(key=position_in_sys_path)
 AttributeError: '_NamespacePath' object has no attribute 'sort'

This points to https://github.com/pypa/pip/issues/4216, so the fix is
apparently to make sure that there's new enough PIP + setuptools in the
created virtual env, and these versions are determined by host's
virtualenv module (thanks to webknjaz on IRC for explaining this to me).

After a restart zuul-executor runs normally, but as a side effect not
all required Python modules are installed for Ansible's usage. For me,
this meant that (e.g.) `upload-logs-swift` won't work ebcause there's no
"openstack" module.

I don't know how to make zuul-executor error persistent, but let's at
least make the life easier for the next poor sod installing this SW on
a system where "virtualenv" is "too old" -- which is also the latest
available Ubuntu if I'm checking their revision numbers correctly.
Require virtualenv >= 20 while preserving the blacklist from commit
7c6503aba. Given that Zuul's own CI uses PIP to pull the latest
virtualenv for the test suite, make sure that the users get what was
tested.

Change-Id: I2b53e532b63a9ae20e817d9599c46cccf613e425
2020-04-08 14:45:09 +02:00
Jeremy Stanley 7e53f87d06 Declare support for Python3.8
Replace our tox-py37 job with tox-py38, extend the list of trove
classifiers for Python versions in package metadata, and replace the
"py35" in the tox.ini envlist with just "py3" so that folks running
`tox` unqualified on their systems will use whatever python3
interpreter they have on hand (odds are it's in our supported range
these days). Also uncap python-daemon so we use a version compatible
with Python >=3.8.

Change-Id: I4b382bb89847d3ae2b20292d923c3c2211dfb254
2020-03-11 20:22:21 +00:00
Zuul 80bdb338c1 Merge "Optimize canMerge using graphql" 2020-03-06 18:08:03 +00:00
Zuul 7f8081bd02 Merge "Uncap GitPython" 2020-03-06 18:03:35 +00:00
James E. Blair f6d6371c39 Fix py35 by pinning importlib-resources
The global tox installation on our test nodes is affected by an
upstream issue[1].  Additionally, the virtualenv created by tox
under those conditions is also affected.  To work around this,
pin the version of importlib-resources in the global tox install,
as well as in Zuul's own requirements.

[1] https://gitlab.com/python-devs/importlib_resources/issues/83

Change-Id: I31ed50185a71d867a2ad512ef9b526c5b607ed5c
2020-03-02 10:52:38 -08:00
Tobias Henkel 4c972f00bd
Optimize canMerge using graphql
The canMerge check is executed whenever zuul tests if a change can
enter a gate pipeline. This is part of the critical path in the event
handling of the scheduler and therefore must be as fast as
possible. Currently this takes five requests for doing its work and
also transfers large amounts of data that is unneeded:

* get pull request
* get branch protection settings
* get commits
* get status of latest commit
* get check runs of latest commit

Especially when Github is busy this can slow down zuul's event
processing considerably. This can be optimized using graphql to only
query the data we need with a single request. This reduces requests
and load on Github and speeds up event processing in the scheduler.

Since this is the first usage of graphql this also sets up needed
testing infrastructure using graphene to mock the github api with real
test data.

Change-Id: I77be4f16cf7eb5c8035ce0312f792f4e8d4c3e10
2020-02-28 09:43:56 +01:00
Tobias Henkel e749e68183
Uncap GitPython
GitPython was capped to <3.0.0 due to increase test times. According
to the upstream issue [1] this should, be fixed upstream. See if we
can uncap it now.

[1] https://github.com/gitpython-developers/GitPython/issues/906

Change-Id: I62a2731c4045f72060a67b8ad3eb89f6b07a0e97
2020-02-19 10:30:21 +01:00
James E. Blair 1e4501612d Avoid gear version 0.15.0
There may have been a behavior change with polling that causes a
surplus of log lines.  Avoid that version until we understand more.

Change-Id: I246acda9352267eb6b9be6d1b06fb87b133709ec
2020-02-17 13:32:02 -08:00
Clark Boylan 7c6503aba0 Simplify virtualenv install and execution
virtualenv==20.0.0 introduced new symlinking of "seed" packages that
broke within a bwrap context. We fixed this by forcing those seed
packages to be installed by pip avoiding any symlinking. Since then
upstream virtualenv has realize this was bad default behavior and
virtualenv==20.0.2 does not symlink by default.

Take advantage of this and exclude 20.0.0 and 20.0.1 as they need
special handling. This allows other versions (new and old) to function.

Change-Id: I3b993e72b52e2234e046a657280b008887c535a3
2020-02-12 09:46:37 -08:00
Tobias Henkel b6166bf81e Uncap virtualenv
New virtualenv has changed behavior and cli arguments. In particular the
seed packages are by default installed in a common lib location and not
updated. These packages are then symlinked to in the virtualenv. This
causes problems for zuul when we run ansible in bwrap and that common
lib path isn't bind mounted in the container.

We can fix this by telling virtualenv to install the seed packages uses
pip and not its internal bundled app data. This is preferable for Zuul
because then we don't have to manage permissions and locations of a
separate seed package library location.

Note that we force virtualenv to be >=20.0.0 as old virtualenv does not
have this new seeder option.

Change-Id: Ib9b2e0b51e6acd744d1c23b4a27067f0202a679d
2020-02-10 14:05:57 -08:00