Change I216b76d6aaf7ebd01fa8cca843f03fd7a3eea16d unified the
service stop sequence but omitted changes to zuul-web. Update
zuul-web to match and make its sequence more robust.
Also remove unecessary sys.exit calls from the merger.
Change-Id: Ifdebc17878aa44d57996e4bdd46e49e6144b406b
This is a new reconfiguration command which behaves like full-reconfigure
but only for a single tenant. This can be useful after connection issues
with code hosting systems, or potentially with Zuul cache bugs.
Because this is the first command-socket command with an argument, some
command-socket infrastructure changes are necessary. Additionally, this
includes some minor changes to make the services more consistent around
socket commands.
Change-Id: Ib695ab8e7ae54790a0a0e4ac04fdad96d60ee0c9
This is a framework for making upgrades to the ZooKeeper data model
in a manner that can support a rolling Zuul system upgrade.
Change-Id: Iff09c95878420e19234908c2a937e9444832a6ec
To facilitate automation of rolling restarts, configure the prometheus
server to answer readiness and liveness probes. We are 'live' if the
process is running, and we are 'ready' if our component state is
either running or paused (not initializing or stopped).
The prometheus_client library doesn't support this directly, so we need
to handle this ourselves. We could create yet another HTTP server that
each component would need to start, or we could take advantage of the
fact that the prometheus_client is a standard WSGI service and just
wrap it in our own WSGI service that adds the extra endpoints needed.
Since that is far simpler and less resounce intensive, that is what
this change does.
The prometheus_client will actually return the metrics on any path
given to it. In order to reduce the chances of an operator configuring
a liveness probe with a typo (eg '/healthy/ready') and getting the
metrics page served with a 200 response, we restrict the metrics to
only the '/metrics' URI which is what we specified in our documentation,
and also '/' which is very likely accidentally used by users.
Change-Id: I154ca4896b69fd52eda655209480a75c8d7dbac3
With this change zuul-web will generate the status JSON on its own by
directly using the data from Zookeeper. This includes the event queue
lengths as well as the pipeline summary.
Change-Id: Ib80d9c019a15dd9de9d694cb62fd34030016c311
This uses the configloader in zuul-web to load the system config and
tenant layouts directly from ZooKeeper.
Doing so will allow us to provide the necessary information for most API
endpoints directly in zuul-web without the need to ask the scheduler via
RPC for it.
Change-Id: I4fe19c4e41f3357a07b2fda939c5ffb4e7055e37
This change adds a new prometheus_port option to start a metric server
to be scrapped by a prometheus service. By default, the server exposes
process informations.
Change-Id: Ie329df6adc69768dfdb158d00283161f8b70f07a
Currently, the ZooKeeper connection is initialized directly in the cmd
classes like zuul.cmd.scheduler or zuul.cmd.merger and then passed to
the server instance.
Although this makes it easy to reuse a single ZooKeeper connection for
multiple components in the tests it's not very realistic.
A better approach would be to initialize the connection directly in the
server classes so that each component has its own connection to
ZooKeeper.
Those classes already get all necessary parameters, so we could get rid
of the additional "zk_client" parameter.
Furthermore it would allow us to use a dedicated ZooKeeper connection
for each component in the tests which is more realistic than sharing a
single connection between all components.
Change-Id: I12260d43be0897321cf47ef0c722ccd74599d43d
To make the tests more like running the apps, use fromConfig in the
tests rather than directly creating a client from a string connection
description. Note that since fromConfig requires TLS and the tests
don't use it, a temporary argument has been added to that method to
allow non-tls connections; it's only used within the test suite.
In future changes, we can configure the tests to use TLS connections,
remove the argument, and then move client instantiation into the
server classes themselves (which will remove additional differences
between test and production startup code).
Change-Id: Ic45c0e60c464ed8eeb44cb32bab039403f73ec69
On the way towards a fully scale out scheduler we need to move the
times database from the local filesystem into the SQL
database. Therefore we need to make at least one SQL connection
mandatory.
SQL reporters are required (an implied sql reporter is added to
every pipeline, explicit sql reporters ignored)
Change-Id: I30723f9b320b9f2937cc1d7ff3267519161bc380
Depends-On: https://review.opendev.org/621479
Story: 2007192
Task: 38329
The commands are managed as entry-points so remove
ununecessary shebangs. Also lib/re2util.py does not
require a shebang as well.
zuul_return.py does not have a main and is not supposed
to be run directly.
Ununecessary shebangs for non executable script causes
rpmlint issues.
Change-Id: I6015daaa0fe35b6935fcbffca1907c01c9a26134
One of the last things the primary thread in zuul-web does after
configuring and starting the service is to call signal.pause(). This
causes it to wait for any signal to arrive and once a signal has arrived
and been handled we stopped zuul-web and killed the process.
This is a problem because SIGUSR2 is expected to toggle preformance and
debug tooling in the process. But since we exit after signal.pause()
returns we can't do that properly.
Fix this by removing the signal.pause() loop entirely and instead wait
for the web command thread to die. This command thread is the last thing
stopped by ZuulWeb.stop() so we can join() on it to wait for a complete
stop of the ZuulWeb service.
Change-Id: I66fbb86c25479823faeded8ede9c6204846c35c1
This adds a script to generate TLS certs for zookeeper.
It also adds new config file options for specifying certs for a
TLS connection, adds a howto document to advise admins on how
to configure ZK for TLS.
It also removes the 'required' flag for the SASL auth parameters,
since they are not actually required.
Include the default openssl.cnf file since some distros modify it
to specify paths that are incompatbile with the zk-ca.sh script.
Change-Id: Icd976cc32dfd9f75f8cfb1c9ad11e08af31723d6
Add an "authorize_user" RPC call allowing to test a set of claims
against the rules of a given tenant. Make zuul-web use this call
to authorize access to tenant-scoped privileged actions.
Change-Id: I50575f25b6db06f56b231bb47f8ad675febb9d82
Allow an operator to define authorization rules.
Allow an operator to add authorization rules to a tenant.
Add a rule parser and a rule registry.
The authZ engine is not plugged in yet.
Change-Id: I3a86c6c7d62ad2bce68a98dbd2fff18549b94fb9
A user with the right JSON Web Token (JWT) can trigger a autohold,
reenqueue or dequeue a buildset from the web API.
The Token is expected to include a key called "zuul.admin" that
contains a list of the tenants the user is allowed to perform
these actions on.
The Token must be passed as a bearer token in an Authorization header.
The Token is validated thanks to authenticator declarations in Zuul's
configuration file.
Change-Id: Ief9088812f44368f14234ddfa25ba872526b8735
Zuul uses the command processor for most components to send commands
to a running service. We should add this to zuul-web as well starting
with the stop command. In a later change we'll add commands for
starting/stopping a repl server.
Change-Id: I1d02ec30341be0890afb332dcf9f32f10a52ead5
This change adds a Pagure driver for Zuul.
Pagure is a Github like forge https://pagure.io/pagure/.
This driver is usable starting with Pagure 5.3.
For history: first Pagure PR gated by Zuul is
https://pagure.io/test-zuul/pull-request/22
Change-Id: I1711653355ae26a3fff3bb6de3c6fca7113cdd01
This default is unlikely to be correct and has caused confusion
for us in the past. Remove it (which matches the documentation).
Change-Id: I3453b0e918fb1c6783514c470f40f4e973fd683a
If there are errors setting up the connections (eg, a recently seen
possible example related to sql migrations), we should log them.
Change-Id: I39a51514c1c2277106e11682dedde410372b50e5
flake 3.6.0 introduces a couple of new tests, handle them in the zuul
base:
* Disable "W504 line break after binary operator", this is a new warning
with different coding style.
* Fix "F841 local variable 'e' is assigned to but never used"
* Fix "W605 invalid escape sequence" - use raw strings for regexes.
* Fix "F901 'raise NotImplemented' should be 'raise
NotImplementedError'"
* Ignore "E252 missing whitespace around parameter equals" since it
reports on parameters like:
def makeNewJobs(self, old_job, parent: Job=None):
Change "flake8: noqa" to "noqa" since "flake8: noqa" is a file level
noqa and gets ignored with flake 3.6.0 if it's not at beginning of line
- this results in many warnings for files ./zuul/driver/bubblewrap/__init__.py and
./zuul/cmd/migrate.py. Fix any issues there.
Change-Id: Ia79bbc8ac0cd8e4819f61bda0091f4398464c5dc
This change adds a zk client to the ZuulWeb service to implement a
labels route for returning the available labels.
Change-Id: Iecf2948e4895829da3f090734c93fcd0c8a497b5
A recent change to scope sql connections to tenants was incorrect
because it assumed zuul-web would have access to a tenant's layout,
but that is only in the zuul-scheduler's memory. Instead, this change
causes zuul-web to perform an RPC call to the scheduler to ask which
connection to use before performing sql queries.
This slipped through testing because the zuul web test fixture used
the same connection registry object as the scheduler, and therefore
*was* able to access data which normally would not be available to it.
This updates the zuul web test fixture to use a separate connection
registry.
Also, because zuul-web doesn't need any other kinds of connections at
the moment, add a facility to allow it to only load sql connections,
so that it doesn't do anything with the gerrit or github drivers.
Change-Id: Iac6d8c6960f4a31eb2f78511664b704758e41687
* Aiohttp (and related libraries) have a python support policy
which is causing us problems.
* Cherrypy supports threads which integrates well with the rest
of Zuul.
Change-Id: Ib611df06035890d3e87fc5ad92fdfc7ac441edce
The only rest API endpoint that uses sql queries is
/api/tenant/{tenant}/builds. There's no connection in there, which
means it doesn't make sense for that to be attached to a sql connection
(which is currently the case). Moreover, it doesn't make sense for
*every* tenant's endpoint to be attached to the *same* connection.
In other words, the current situation only allows for a single sql
connection system-wide, even if someone is using different connections
per tenant.
Moving the handler for the endpoint into the sql driver means that it
can dispatch the query to the appropriate connection for a given tenant
(since a tenant is always implied by the REST endpoint).
Moreover, the *rest* of the system actually allows multiple connections
within a single tenant, and we should support that here, but I don't
immediately have a solution of how to handle pagination across queries
that span multiple connections. This is an improvement in that it is
now tenant-scoped, but it's not ideal.
This also removes the (undocumented!) sql_connection_name config file
option.
It also uses the tenant name from the path to constructe the query so
that it always includes the correct tenant (this eliminates the
inadvertant ability for one tenant to query another tenant's builds).
The internal API here isn't great, but it will get cleaned up in the
next patch which converts to cherrypy.
Change-Id: Ie1f19f0b392d4c010ef43dc6220ff1c8667f5a4a
yarn drives package and dependency management. webpack handles
bundling, minification and transpiling down to browser-acceptable
javascript but allows for more modern javascript like import statements.
There are some really neat things in the webpack dev server. CSS
changes, for instance, get applied immediately without a refresh. Other
things, like the jquery plugin do need a refresh, but it's handled just
on a file changing.
As a followup, we can also consider turning the majority of the status page
into a webpack library that other people can depend on as a mechanism
for direct use. Things like that haven't been touched because allowing
folks to poke at the existing known status page without too many changes
using the tools seems like a good way for people to learn/understand the
stack.
Move things so that the built content gets put
into zuul/web/static so that the built-in static serving from zuul-web
will/can serve the files.
Update MANIFEST.in so that if npm run build:dist is run before the
python setup.py sdist, the built html/javascript content will be
included in the source tarball.
Add a pbr hook so that if yarn is installed, javascript content will be
built before the tarball.
Add a zuul job with a success url that contains a source_url
pointing to the live v3 data.
This adds a framework for verifying that we can serve the web app
urls and their dependencies for all of the various ways we want to
support folks hosting zuul-web.
It includes a very simple reverse proxy server for approximating
what we do in openstack to "white label" the Zuul service -- that
is, hide the multitenancy aspect and present the single tenant
at the site root.
We can run similar tests without the proxy to ensure the default,
multi-tenant view works as well.
Add babel transpiling enabling use of ES6 features
ECMAScript6 has a bunch of nice things, like block scoped variables,
const, template strings and classes. Babel is a javascript transpiler
which webpack can use to allow us to write using modern javascript but
the resulting code to still work on older browsers.
Use the babel-plugin-angularjs-annotate so that angular's dependency
injection doesn't get borked by babel's transpiling things (which causes
variables to otherwise be renamed in a way that causes angular to not
find them)
While we're at it, replace our use of var with let (let is the new
block-scoped version of var) and toss in some use of const and template
strings for good measure.
Add StandardJS eslint config for linting
JavaScript Standard Style is a code style similar to pep8/flake8. It's
being added here not because of the pep8 part, but because the pyflakes
equivalent can catch real errors. This uses the babel-eslint parser
since we're using Babel to transpile already.
This auto-formats the existing code with:
npm run format
Rather than using StandardJS directly through the 'standard' package,
use the standardjs eslint plugin so that we can ignore the camelCase
rule (and any other rule that might emerge in the future)
Many of under_score/camelCase were fixed in a previous version of the patch.
Since the prevailing zuul style is camelCase methods anyway, those fixes
were left. That warning has now been disabled.
Other things, such as == vs. === and ensuring template
strings are in backticks are fixed.
Ignore indentation errors for now - we'll fix them at the end of this
stack and then remove the exclusion.
Add a 'format' npm run target that will run the eslint command with
--fix for ease of fixing reported issues.
Add a 'lint' npm run target and a 'lint' environment that runs with
linting turned to errors. The next patch makes the lint environment more
broadly useful.
When we run lint, also run the BundleAnalyzerPlugin and set the
success-url to the report.
Add an angular controller for status and stream page
Wrap the status and stream page construction with an angular controller
so that all the javascripts can be bundled in a single file.
Building the files locally is wonderful and all, but what we really want
is to make a tarball that has the built code so that it can be deployed.
Put it in the root source dir so that it can be used with the zuul
fetch-javascript-tarball role.
Also, replace the custom npm job with the new build-javascript-content
job which naturally grabs the content we want.
Make a 'main.js' file that imports the other three so that we just have
a single bundle. Then, add a 'vendor' entry in the common webpack file
and use the CommonsChunkPlugin to extract dependencies into their own
bundle. A second CommonsChunkPlugin entry pulls out a little bit of
metadata that would otherwise cause the main and vendor chunks to change
even with no source change. Then add chunkhash into the filename. This
way the files themselves can be aggressively cached.
This all follows recommendations from https://webpack.js.org/guides/caching/https://webpack.js.org/guides/code-splitting/ and
https://webpack.js.org/guides/output-management/
Change-Id: I2e1230783fe57f1bc3b7818460463df1e659936b
Co-Authored-By: Tristan Cacqueray <tdecacqu@redhat.com>
Co-Authored-By: James E. Blair <jeblair@redhat.com>
There are a few pieces of information that are useful to know in the web
layer.
websocket_url is a config setting that, if set, is needed by the
console streaming. We currently pass this in appended to the streaming
url as a url parameter (which since it's a URL is a bit extra odd)
The endpoint is normally relative to the webapp,
but may need to be overridden in cases like publishing the html and
javascript to a disconnected location such as the draft output into the
log server in openstack or publishing built html/javascript to swift.
Add WebInfo and TenantWebInfo objects and corresponding /info and
/{tenant}/info routes. As an alternative, we could collapse WebInfo
and TenantWebInfo to just WebInfo and leave the tenant field set to None
for the /info route.
Some of the API functions are optionally provided by
plugins. The github plugin provides webhook URLs and the SQLReporter
plugin is needed for the builds endpoints. Add a Capabilities object
that can report on the existance of such things and pass it to plugin
route registration so that capabilities can be registered.
Add support for configuring stats_url
The old zuul status page had sparklines and other graphs on it, which
are not present in the current one because the graphite server wasn't
parameterized.
Add a config setting allowing a URL to a graphite server to be set and
expose that in the /info endpoint. Since statsd itself can emit to multiple
different backends, add a setting for the type of server, defaulting to
graphite.
Change-Id: I606a3b2cdf03cb73aa3ffd69d9d64c171b23b97a
Rather than special-casing github and sql in zuul-web, add a
registration method to Connection to allow a Connection to register
routes.
Provide two types of WebHandler classes - a 'raw' class and a 'driver'
class. The driver class prepends '/connection/{connection}' to the
provided path.
Shift the github webhook and SQL web methods to use the plugin interface.
Change-Id: I065937b20447248e7894e1dfeec269faa6fd30d2
We want to have zuul-web to handle all http serving stuff so also the
github webhook handling needs to be moved to zuul-web.
Note that this changes the url of the github webhooks to
/driver/github/<connection_name>/payload.
Change-Id: I6482de6c5b9655ac0b9bf353b37a59cd5406f1b7
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
Co-Authored-by: Tobias Henkel <tobias.henkel@bmw.de>
This reverts commit 00d7ea51fd. It
intended to refactor common code paths for signal handling. However in
our dockerized deployment this seems to completely break signal
handling. Thus it needs to be reverted.
Change-Id: Id5731557ff9a363c7a3d9438a8efcd476e38380c
Almost all zuul apps use the method term_handler for SIGINT and
SIGTERM. Defining this centrally in ZuulDaemonApp makes this much
simpler and without repitition.
Change-Id: I68f8d1bf52b0e16340818d2bcc44cd9fc5868ca7
We want the stack dump handler to be present in all zuul apps so this
can be registered in a central place.
Change-Id: I0c4a97d6ee983aa4d57928682dfb6eeffd050197
This change adds a SqlHandler to query the sql reporter database from
zuul-web through the /{tenant}/builds.json controller.
This change also adds a /{tenant}/builds.html basic web interface.
Change-Id: I423a37365316cc96ed07ad0895c7198d9cff8be5
Adopt some of the structure from nodepool to make daemon process
handling more consistent. Handle some argument parsing centrally.
Change the default pid file structure to match nodepool:
/var/run/zuul/<processname>
Attempt to use the pidfile before daemonizing so that errors are
immediately reported.
Drop the config validation test since it is almost useless at this
point.
Change-Id: I4a9d9473ce028e0b0cd32a8c48598c1682e1c329
zuul now provides socket-based console streaming, which is super cool.
In order to have jenkins parity with web streaming, we need to provide a
websocket (javascript in browsers can't really connect to random ports
on servers)
After surveying the existing python websocket options, basically all of
them are based around twisted, eventlet, gevent or asyncio. It's not
just a thing we can easily deal with from our current webob/paste
structure, because it is a change to the fundamental HTTP handling.
While we could write our own websocket server implementation that was
threaded like the rest of zuul, that's a pretty giant amount of work.
Instead, we can run an async-based server that's just for the
websockets, so that we're not all of a sudden putting async code into
the rest of zuul and winding up frankensteined. Since this is new code,
using asyncio and python3 seems like an excellent starting place.
aiohttp supports running a websocket server in a thread. It also
supports doing other HTTP/REST calls, so by going aiohttp we can set
ourselves up for a single answer for the HTTP tier.
In order to keep us from being an open socket relay, we'll expect two
parameters as the first message on the websocket - what's the zuul build
uuid, and what log file do we want to stream. (the second thing,
multiple log files, isn't supported yet by the rest of zuul, but one can
imagine a future where we'd like to support that too, so it's in the
protocol) The websocket server will then ask zuul over gearman for the
IP and port associated with the build and logfile and will start
streaming it to the socket.
Ultimately we'll want the status page to make links of the form:
/console.html?uuid=<uuid>&logfile=console.log
and we'll want to have apache map the websocket server to something like
/console.
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
Change-Id: Idd0d3f9259e81fa9a60d7540664ce8d5ad2c298f
James E. Blair