The existing openapi spec document (used to generate the swagger
ui page in the web app as well as the rst documentation) is
both incomplete and wrong due to bitrot.
This change adds a script which automatically generates much of
the api documentation from the code. The output is still incomplete,
but it does include at least the same endpoints currently documented,
and of those, all of the inputs and outputs.
Due to its automatic generation, all of the endpoints and their
inputs are now documented. Only some outputs are missing (as well
as explanatory text, which was pretty thin before).
It does the following:
* Inspects the cherrypy router object to determine the endpoints to
include, and identifies their HTTP methods and the python functions
that implement them.
* It inspects the function python docstring to get summary documentation
for the endpoint.
* It inspects the function arguments and compares them to the
router path to determine if each is a path or query parameter,
as well as whether each is required.
* It merges type and descriptive information from the python docstring
about each parameter.
* For output, a schema system similar to voluptuous is used to describe
the output names and types, as well as optional descriptive information.
One of two function decorators are used to describe the output.
It removes the documentation for the status page output format. This API
is specially optimized for the Zuul status page, is very complex, and we
should therefore not encourage end-users to develop against it. The
endpoint itself is documented as such, but the response value is
undocumented.
Future work:
More descriptive text and output formats can be documented.
Change-Id: Ib1a2aad728c4a7900841a8e3b617c146f2224953
The pg_dump command is picky about client/server versions, so to
make it easier for developers to run tests locally using the pgsql
container, provida an option to use pg_dump from the container
instead of whatever version is installed locally, which may not
match.
Change-Id: I9add7af40b8f8d92a88d983eacbc59a01f0e7b9f
This reverts commit 1c188ad440.
Reason for revert: This change increases test failures; investigation is ongoing.
Change-Id: I3a101656ee0904e1cf6c856dba845385c333260f
I'm not sure if this is a good idea or not yet. Testing with MySQL
actually exposed the alembic transaction thing that is fixed in a
grandparent change. That said much of the world seems to be moving to
MariaDB so it might be a good idea for us to test that more explicitly?
We update bindep.txt which will install mariadb during CI job runs. We
also update our local test setup tooling to roughly match what will be
used in CI. For example the mariadb:10.6 docker image is chosen because
this is roughly equivalent to what Jammy packages.
Another good reason for this change is that Debian ARM64 does not
package MySQL and only packages MariaDB. This should give us better
compatibility for testing on architectures other than x86-64.
Change-Id: I9d00557ca5823da34278c3e9ecda2daaf13440b7
MySQL 8 no longer supports implicitly creating a user using the GRANT
statement. Use a separate CREATE USER statement instead.
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the
right syntax to use near 'identified by 'openstack_citest' WITH GRANT
OPTION' at line 1
Change-Id: I4cab4c1855d1ba97cbfc9dd0835b3d302d73aa62
Recently I debugged an issue that reproduced on mysql 8.0 on Jammy but
not my local system. It took quite some time for me to realize that
there was a difference in database versions which ended up being
an important detail. Update our docker-compose to better match what we
get in CI.
Change-Id: I7de268acb81680f3e6b7d3b1aa057e7babd3fa62
Tox v4 behaves significantly differently than v3, and some of the
more complex things we do with tox would need an overhaul to
continue to use it. Meanwhile, nox is much simpler and more
flexible, so let's try using it.
This adds a noxfile which should be equivalent to our tox.ini file.
We still need to update the docs build (which involves changes to
base jobs) before we can completely remove tox.
Depends-On: https://review.opendev.org/868134
Change-Id: Ibebb0988d2702d310e46c437e58917db3f091382
The Zuul unittest jobs were timing out which prompted me to take a look
at what might be taking extra time. One thing I noticed is that we're
running the yarn build (which runs react-scripts) multiple times when we
really only need to run it once. The reason for that is our check to see
if zuul web has already built is looking for a file that the builds no
longer produce. Update that check to look for a current file and we'll
save a bit of time in our jobs and when running things locally.
Change-Id: Iae3604fbaf072d53895db850cfc989a832b12b27
When not using a ROOTCMD when running the test-setup-docker.sh the
script will fail with the following error message:
++ id -u
+ USER_ID=1000 docker-compose up -d
./test-setup-docker.sh: line 50: USER_ID=1000: command not found
Due to Bash's simple command expansion[0] variable assignments will be
interpreted as the command when no ROOTCMD is given. To work around this
we use default ROOTCMD to the `env` command.
[0]: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Simple-Command-Expansion
Change-Id: Id696069a4eec2b2c2513ac449098b0ef73dc3906
Update the TESTING.rst file to reflect that just installing and
starting zookeeperd is no longer sufficient now that we require TLS
and auth for the connection, as well as running database servers.
Suggest the container-based setup script instead. Also improve that
script to allow it to be invoked as a normal user, with root command
escalation tool choice (e.g. "sudo") supplied through a ROOTCMD
environment variable, so that things created inside the git worktree
like the CA don't end up root-owned.
Related, the tox-docker plugin previously suggested in the document
is no longer a viable option for the same reasons, so clean up the
plumbing for it in tox.ini as well.
Change-Id: Iac32799425a5bd4b1bdbf56f34a2310241ac4499
This adds python3.10 testing on Jammy and switches the docker images to
python3.10 from 3.8.
We run sudo for postgres with -Hi to avoid non fatal errors when
postres' client attempts to write command history to Zuul's homedir (it
is running as the postgres user which can't write to zuul's homedir). We
also need to update the libffi package version for jammy to 8 in
bindep.txt. Finally, python_version values need to be quoted as "3.10"
is different than 3.10 which is equivalent to 3.1 when serialized by
yaml as a float.
Force setuptools to use stdlib (shipped by the distro) distutils to
avoid problems with virtualenvs not actually being virtualenvs.
Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy
as the default python there is 3.10.
Change-Id: I97b90bb7a23c90f108f23dda9fdd0e89f9f4dbca
A couple of locations continue to reference actiongeneral which has been
removed. Update these locations to use action as the current location
for these plugins.
Change-Id: I71c03d2c0a84592be66fa0d84bc684684a392a27
The zookeeper containers entrypoint will try to switch to the user
'zookeeper' if the container is started with uid '0'. This can cause issues
when the uid for the zookeeper user inside the container and the
users uid outside the container differ since it will lead to zookeeper
getting access denied when trying to read the certificates.
This also adds logging configuration to make it easier to debug
zookeeper in the future.
Change-Id: I51db53fe093a294e804148f682053123f54adfe1
We had been using version 14 which is the previous LTS. Now there are
problems running:
npx browserslist@latest --update-db
running out of memory. Update to the current nodejs LTS version to
ensure we are running an up to date runtime that hopefully performs more
consistently with the browserslist command.
Change-Id: Ib20c1090ea0f30b7dac2780b6ed963dd6e4b6f77
Since the pipeline state is stored in ZooKeeper, there could be cases
where the change_queues key is missing in the status json. This makes
API requests fail:
2022-02-22 17:56:18,390 ERROR cherrypy.error.139989033522128: [22/Feb/2022:17:56:18] HTTP
Traceback (most recent call last):
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 638, in respond
self._do_respond(path_info)
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond
response.body = self.handler()
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/lib/encoding.py", line 223, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/lib/jsontools.py", line 59, in json_handler
value = cherrypy.serving.request._json_inner_handler(*args, **kwargs)
File "/opt/zuul/lib/python3.8/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
return self.callable(*self.args, **self.kwargs)
File "/opt/zuul/lib/python3.8/site-packages/zuul/web/__init__.py", line 1050, in status_change
return result_filter.filterPayload(payload)
File "/opt/zuul/lib/python3.8/site-packages/zuul/web/__init__.py", line 193, in filterPayload
for change_queue in pipeline['change_queues']:
KeyError: 'change_queues'
Fix this by using a .get() call rather than directly accessing the
dictionary key by name.
A similar issue was already fixed in [1].
[1]: https://review.opendev.org/c/zuul/zuul/+/829018
Change-Id: I947f58f02c3da7dad35d1fc186c7026800f7cbdd
This is like the real zk-shell except:
1) It supports SSL
2) It automatically decompresses znodes if necessary
3) It handles sharded data
4) It doesn't have many commands
Change-Id: I2124f216f08f68ca63b763cd6f6adce452c07f79
These may be useful for zuul developers to understand issues with the
ZK data storage.
zk-dump will dump an approximation of the contents of ZK to the
filesystem for manual examination.
zk-analyze will perform some analysis on the tree to identify objects
which may be execessively large.
Change-Id: I1a90cce42da719eee0a5e50242034390722d518e
This can be used immediately to manually build old docs with
updated themes and URLs. We can iterate on this to make it a
periodic job so it happens automatically.
Change-Id: I021e236ffcfd436e1169e9e804efee013cbe10aa
In case for whatever reason there is already another
MySQL server running on the standard port developers
may want to run the MySQL test-server on a different
port. This can be done by specifying an environment
variable "ZUUL_MYSQL_PORT":
export ZUUL_MYSQL_PORT=3307
tools/test-setup-docker.sh
tox -e py3
Change-Id: Id32bac1a87619803d973b066762d525bd9a4f913
Change Ib4037da2f7a0f803aca24ce880dbc262375db6a4 introduced 2 extra
parameters for the zuul.model.SourceContext constructor. These
parameters don't really matter for decrypt_secret.py. We just pass the
None value.
Change-Id: Ic4f09558865349c8c479defe1b314b0c4c92dd0f
The gearman RPC methods are being removed, so this needs a change
to work with one of the web-based methods. The most simpla end
forward-looking method is zuul-client, so update it to use that.
Change-Id: If3f6ca4bae2b2beddb3bb71b36fdcba112722186
When executing the unit tests with multiple schedulers in parallel a lot
of them are failing becasue there are too many open SQL connections.
However, executing those tests one after another doesn't end up in the
connection limit being exhausted. So this doesn't look like a shutdown
problem in Zuul.
Having a look at the MySQL server when it's under load during the tests
revealed that the default connection limit of 151 on the server side is
exhausted very quickly when a lot of tests are running in parallel (each
test running with two schedulers).
Therefore, this change increases the default MySQL connection limit to
300. We should keep in mind that this connection limit has to go in hand
with the number of schedulers used in the unit tests and the number of
tests executed in parallel. Maybe on the long term it might make sense
to cap the latter parameter somehow in the tox.ini file.
The same applies to the ZooKeeper connection limit which is configured
in tools/zoo.cfg.
Change-Id: Iff76e99ec82edc8e8bc110a22a096bb689d8dd1f
Debian bullseye includes skopeo, so we can drop the kubic repository
(which as I write this is having synchronization errors) in favor of
just using the version from the underlying OS.
Change-Id: Ifde147c8c690dd6a421b0133dbabcff7dd9b9649
This adds a modified kazoo.DataWatch class which does not set
watches on paths that don't exist. This is mostly so that when
a DataWatched path is deleted, we don't leave a watch in place.
Without that, the executor api will leak watches on the ZK cluster.
We vendor the entire watches.py file from kazoo (ASL2 licensed) to
avoid any issues with API changes. Separately we will see if they
are interested in this class upstream.
This updates the tests to use the wchp 4lw to test that we don't
leak watches.
Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/799334
Change-Id: Ie4491eef03b58d858d95c78ba9454839d169cff1
We're seeing occasional npm install failures in jobs; retry the
"yarn install" command if it fails.
Also, pass the YARN_REGISTRY env var in the tox remote job.
Change-Id: Ic80dccf0869ce805368f678eafcfdf48f8bf9bea
This mirrors the configuration in Nodepool for using TLS-enabled
ZooKeeper in tests. We use the ensure-zookeeper role in order
to get a newer ZooKeeper than is supplied in bionic.
Change-Id: I14413fccbc9a6a7a75b6233d667e2a1d2856d894
This change is a common root for other
Zookeeper related changed regarding
scale-out-scheduler. Zookeeper becoming
a central component requires to increase
"maxClientCnxns".
Since the ZooKeeper class is expected to grow
significantly (ZooKeeper is becoming a central part
of Zuul) a split of the ZooKeeper class (zk.py) into
zk module is done here to avoid the current god-class.
Also the zookeeper log is copied to the "zuul_output_dir".
Change-Id: I714c06052b5e17269a6964892ad53b48cf65db19
Story: 2007192
Now that zuul-client's encrypt subcommand covers the same
functionalities as encrypt_secret.py, add a deprecation
message when running the script. Document the zuul-client
encrypt command in the doc section about secrets.
Change-Id: Id5437ffbb688cb80b2744db3beeaa28c97080d90
Depends-On: https://review.opendev.org/765313
Opendev runs a caching mirror for npm packages. Use it when running
yarn install during tox tests. Due to the way yarn.lock files work at
the moment we need to patch the lockfile on the fly to inject the
mirror urls [1]. This can be handlel inside the pip.sh where we patch
the lockfile, run yarn and reset it afterwards.
[1] https://github.com/yarnpkg/yarn/issues/2566
Change-Id: Id1f2daac8327f3110e6beeb2d6391baccfd3ce34
This change adds a script to check if a zuul deployment is
using untrusted command on the executor host.
Operator needs to take action on the affected job to avoid
failure after upgrading to zuul v3.19.1.
Change-Id: I5d18df3fd120e8f4c189ce0b59aa89553e9df816
Zuul was designed to block local code execution in untrusted
environments to not only rely on bwrap to contain a job. This got
broken since the creation of a command plugin that injects the
zuul_job_id which is required for log streaming. However this plugin
doesn't do a check if the task is a localhost task. Further it is
required in trusted and untrusted environments due to log
streaming. Thus we need to fork this plugin and restrict the variant
that is used in untrusted environments.
We do this by moving actiongeneral/command.py back to action/*. We
further introduce a new catecory actiontrusted which gets the
unrestricted version of this plugin.
Change-Id: If81cc46bcae466f4c071badf09a8a88469ae6779
Story: 2007935
Task: 40391
The js content tarball creation was broken. Instead of leaving the
symlink to a non-existing directory which gets created during python
setup we should just remove that symlink as well and create the
symlink and the static dir during the python setup. This way nothing
will be in the way of the javascript content generation.
This reverts commit eb7b18b38e.
Change-Id: I5f8bfa62cd2d4d9823b86dbcda14885230847a82
Starting from mysql version 8 it's not possible to create a user
implicitely when using GRANT.
This patch makes the behavior compatible with that.
Change-Id: I1dc25194de384bf14607aff4e0f230e394f00582
The zk-ca.sh tool for generating certificates fails with
and error that ./tools/openssl.cnf cannot be opened if a
relative path is supplied on ubuntu-bionic. So expand the
path to an absolute path instead.
http://paste.openstack.org/show/794091/
Change-Id: Ib30e9754d4520b956e83cae014c1be181ae44631
James E. Blair