This supplies a per-project default value for Zuul's default-branch
based on what the default branch is set to in GitHub. This means
that if users omit the default-branch setting on a Zuul project
stanza, Zuul will automatically use the correct value.
If the value in GitHub is changed, an event is emitted which allows
us to automatically reconfigure the tenant.
This could be expanded to other drivers that support an indication
of which branch is default.
Change-Id: I660376ecb3f382785d3bf96459384cfafef200c9
This updates the branch cache (and associated connection mixin)
to include information about supported project merge modes. With
this, if a project on github has the "squash" merge mode disabled
and a Zuul user attempts to configure Zuul to use the "squash"
mode, then Zuul will report a configuration syntax error.
This change adds implementation support only to the github driver.
Other drivers may add support in the future.
For all other drivers, the branch cache mixin simply returns a value
indicating that all merge modes are supported, so there will be no
behavior change.
This is also the upgrade strategy: the branch cache uses a
defaultdict that reports all merge modes supported for any project
when it first loads the cache from ZK after an upgrade.
Change-Id: I3ed9a98dfc1ed63ac11025eb792c61c9a6414384
When exclude-unprotected-branche is in effect and a project doesn't have
any protected branches (e.g. a wrong branch protection rule in Github or
none at all) the branch cache will contain an empty list.
Since the empty list was so far used to indicate a fetch error, those
projects showed up with a config error in zuul-web ("Will not fetch
project branches as read-only is set").
For the branch cache we need to distinguish three different cases:
1. branche cache miss (no fetch yet)
2. branch cache hit (including empty list of branches)
3. fetch error
Instead of storing an empty list in case of a fetch error we will store
a sentinel value of `None` in those cases. However, with this change we
can't use `None` as an indicator for a cache miss anymore, so we are now
raising a `LookupError` instead.
Change-Id: I5b51805f7d0a5d916a46fe89db7b32f14063d7aa
When an operator issues a full-reconfigure or tenant-reconfigure
command to the scheduler, the branch cache is cleared. It will
be fully populated again during the subsequent full-reconfiguration,
or eventually populated more slowly over time after a
tenant-reconfiguration. But during the time the branch cache is
clear, zuul-web will be unable to update layouts since it is incapable
of populating the branch cache itself.
This produces a short time period of errors during a full-reconfig
or a potentially long time period of errors after a tenant-reconfig.
To correct this, we now detect whether a layout update in zuul-web
is incomplete due to a branch cache error, and retry the update
later. In the scheduler, we only clear the projects from the branch
cache that are affected by the tenants we are reloading (still all the
projects for a full-reconfigure). This limits the time during which
zuul-web will be unable to update the layout to only the time that
the scheduler spends actually performing a reconfiguration.
(Note that in general, the error here is not because zuul-web is
loading the layout for the tenant that zuul-scheduler is reconfiguring,
but rather it is loading a layout for a tenant which has projects
in common with the tenant that is being reconfigured.)
Change-Id: I6794da4d2316f7df6ab302c74b3efb5df4ce461a
The full reconfiguration event is primarily used to correct invalid
states which may result from missed events, such as a branch creation.
However, it did not invalidate or update the actual data in the branch
cache. Correct this by completely clearing the branch cache of every
connection when starting a full reconfiguration. This will cause the
connections to query their upstream sources the next time that we ask
for a list of branches.
Change-Id: I1529fe1dd6092de612a94c14ec0527ab76f62e47
The drivers are expected to populate the change cache before
passing trigger events to the scheduler so that all the difficult
work is done outside the main loop. Further, the cache cleanup
is designed to accomodate this so that events in-flight don't have
their change cache entries removed early.
However, at several points since moving the change cache into ZK,
programming errors have caused us to encounter enqueued changes
without entries in the cache. This usually causes Zuul to abort
pipeline processing and is unrecoverable.
We should continue to address all incidences of those since they
represent Zuul not working as designed. However, it would be nice
if Zuul was able to recover from this.
To that end, this change allows missing changes to be added to the
change cache.
That is primarily accomplished by adjusting the Source.getChange
method to accept a ChangeKey instead of an Event. Events are only
available when the triggering event happens, whereas a ChangeKey
is available when loading the pipeline state.
A ChangeKey represents the minimal distinguishing characteristics
of a change, and so can be used in all cases. Some drivers obtain
extra information from events, so we still pass it into the getChange
method if available, but it's entirely optional -- we should still
get a workable Change object whether or not it's supplied.
Ref (and derived: Branch, Tag) objects currently only store their
newrev attribute in the ChangeKey, however we need to be able to
create Ref objects with an oldrev as well. Since the old and new
revs of a Ref are not inherent to the ref but rather the generating
event, we can't get that from the source system. So we need to
extend the ChangeKey object to include that. Adding an extra
attribute is troublesome since the ChangeKey is not a ZKObject and
therefore doesn't have access to the model api version. However,
it's not too much of a stretch to say that the "revision" field
(which like all ChangeKey fileds is driver-dependent) should include
the old and new revs. Therefore, in these cases the field is
upgraded in a backwards compatible way to include old and newrev
in the standard "old..new" git encoding format. We also need to
support "None" since that is a valid value in Zuul.
So that we can continue to identify cache errors, any time we encounter
a change key that is not in the cache and we also don't have an
event object, we log an error.
Almost all of this commit is the refactor to accept change keys
instead of events in getChange. The functional change to populate
the cache if it's missing basically consists of just removing
getChangeByKey and replacing it with getChange. A test which deletes
the cache midway through is added.
Change-Id: I4252bea6430cd434dbfaacd583db584cc796dfaa
This adds reporting of compressed and uncompressed data sizes for
the serialized pipeline states and the change caches.
These can be used to observe trends over time in data usage and
potentially find areas for optimization.
Change-Id: I6504e5c2b539295b58644a4ac3d7501da48fe7c1
This is a framework for making upgrades to the ZooKeeper data model
in a manner that can support a rolling Zuul system upgrade.
Change-Id: Iff09c95878420e19234908c2a937e9444832a6ec
We didn't include the branch cache ltimes when loading the initial
config in zuul-web, so if zuul-web starts before the branch cache
is updated, it won't be told to refresh it when loading the layout.
This matches what we do in zuul-scheduler, and adds a skipped test.
The test does unusual things with the startup sequence and does not
appear to be reliable when run upstream, so it is disabled for now.
This also adds a debug line and adjust another to remove an apostraphe
to make grepping easier.
Change-Id: Ib184f4cadb21124fd3dedeea3797b01bb9c431cc
This is an early preparation step for removing the RPC calls between
zuul-web and the scheduler.
In order to do so we must initialize the ConfigLoader in zuul-web which
requires all connections to be available. Therefore, this change ensures
that we can load all connections in zuul-web without providing a
scheduler instance.
To avoid unnecessary traffic from a zuul-web instance the onLoad()
method initializes the change cache only if a scheduler instance is
available on the connection.
Change-Id: I3c1d2995e81e17763ae3454076ab2f5ce87ab1fc
This keeps the branch cache continuously populated. When part of
the cache is invalidated by a branch creation/deletion event, instead
of clearing the cache, this updates the data.
That can be slightly more efficient in that it may reduce the number
of ZK transactions, and it may also help us when adding layout loading
to zuul-web, in that those connections will not need to make any remote
calls to the code review system.
Change-Id: I0ad3ea0ecab50ae5261f9510e34ffb27ca83ac1e
In case a project(-branch) cache is cleared we add the cache ltime to
the trigger event. This ltime can later be used to determine if the
project branch cache needs to be refreshed.
Change-Id: I25d2ffb144e3a80856b10c920b42c37ef5ac9235
Remove the duplicate methods in the connections that use a
Zookeeper-backed change cache by moving them to a common mixin-class.
Change-Id: I527522e79444dfb69b41c70e7698a281166b2128
We need to periodically remove change data nodes that are not referenced
by any cache entry. This needs to happen more often than the periodic
cache maintenance (every 5min vs. every hour).
For this we need to introduce a new method `cleanupCache()` in the
connection interface.
In each run of the change cache's cleanup method we will identify
candidates of dangling data nodes to be cleaned up on the next run. The
reason for this delayed cleanup is that we don't want to remove data
nodes where a driver is in the process of creating/updating a new cache
entry.
Change-Id: I9abf7ac6fa117fe4a093ae7a0db1df0da5ae3ff3
With the persistent change cache in Zookeeper we need to periodically
remove changes that are outdated and no longer required by any active
item in the pipelines.
The cache maintenance will be performed together with the other general
cleanup every hour.
Change-Id: I62e75ab8c5b43f830e01b0e4c08b25ecdc5eed08
Clearing of connection caches doesn't seem to be needed in practice and
is removed as it can't be easily coordinated between multiple
schedulers.
Change-Id: I663fa1f1752d0dd599af06a7275a14235faaff2f
In order to expose the length of the connection event queues we need to
introduce `getEventQueue()` as a new interface in the connection base
class.
The queue length will be available in the status JSON output for display
in zuul-web as well as a statsd gauge metric:
zuul.scheduler.eventqueues.connection.<connection-name>
This gives an operator more insight into possible bottlenecks during
event pre-processing.
Change-Id: Ie2e09e0c7cd166dce5b884c3e3d12ead42b05397
So far the connection registry was added after the Scheduler was
instantiated.
We can make the ConnectionRegistry mandatory to simplify the
Scheduler instantiation.
Change-Id: Iff7b1a597c97f2cd13bea75f9f23585b0e7f76b3
The pull request based events already have the information if the
target branch is protected. Hence we don't need to re-query this
information again.
Change-Id: I547db8090e13f1103918ce9c5f33002e9a42fcef
This change adds a /connections route to return the list of connections
and their configuration. This will be consumable as a way to merge
change locally.
Change-Id: Ia392faa2aea0740756ccd97f10cccb5aec51545a
The current attempt to caches branches is ineffective -- we
query the list of branches during every tenant reconfiguration.
The list of branches for a project is really global information;
we might cache it on the Abide, however, drivers may need to filter
that list based on tenant configuration (eg, github protected
branches). To accomodate that, just allow/expect the drivers to
perform their own caching of branches, and to generally keep
the list up to date (or at least invalidate their caches) by
observing branch create/delete events.
A full reconfiguration instructs the connections to clear their
caches so that we perform a full query. That way, an operator
can correct from a situation where the cache is invalid.
Change-Id: I3bd0cda5875dd21368e384e3704a61ebb5dcedfa
This info parameter to getWebController is unused (and redundant as the
zuul_web parameter has the info attribute attached to it). Clean it out
to avoid confusion.
Change-Id: I6ce12b535f56f8fa367a9a99d3e46bc26c2a50d8
* Aiohttp (and related libraries) have a python support policy
which is causing us problems.
* Cherrypy supports threads which integrates well with the rest
of Zuul.
Change-Id: Ib611df06035890d3e87fc5ad92fdfc7ac441edce
There are a few pieces of information that are useful to know in the web
layer.
websocket_url is a config setting that, if set, is needed by the
console streaming. We currently pass this in appended to the streaming
url as a url parameter (which since it's a URL is a bit extra odd)
The endpoint is normally relative to the webapp,
but may need to be overridden in cases like publishing the html and
javascript to a disconnected location such as the draft output into the
log server in openstack or publishing built html/javascript to swift.
Add WebInfo and TenantWebInfo objects and corresponding /info and
/{tenant}/info routes. As an alternative, we could collapse WebInfo
and TenantWebInfo to just WebInfo and leave the tenant field set to None
for the /info route.
Some of the API functions are optionally provided by
plugins. The github plugin provides webhook URLs and the SQLReporter
plugin is needed for the builds endpoints. Add a Capabilities object
that can report on the existance of such things and pass it to plugin
route registration so that capabilities can be registered.
Add support for configuring stats_url
The old zuul status page had sparklines and other graphs on it, which
are not present in the current one because the graphite server wasn't
parameterized.
Add a config setting allowing a URL to a graphite server to be set and
expose that in the /info endpoint. Since statsd itself can emit to multiple
different backends, add a setting for the type of server, defaulting to
graphite.
Change-Id: I606a3b2cdf03cb73aa3ffd69d9d64c171b23b97a
Rather than special-casing github and sql in zuul-web, add a
registration method to Connection to allow a Connection to register
routes.
Provide two types of WebHandler classes - a 'raw' class and a 'driver'
class. The driver class prepends '/connection/{connection}' to the
provided path.
Shift the github webhook and SQL web methods to use the plugin interface.
Change-Id: I065937b20447248e7894e1dfeec269faa6fd30d2
We want to have zuul-web to handle all http serving stuff so also the
github webhook handling needs to be moved to zuul-web.
Note that this changes the url of the github webhooks to
/driver/github/<connection_name>/payload.
Change-Id: I6482de6c5b9655ac0b9bf353b37a59cd5406f1b7
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
Co-Authored-by: Tobias Henkel <tobias.henkel@bmw.de>
The change cache is used to holf references to active Change objects
so that if something about them changes, the connection driver can
immediately update the data in memory and all enqueued items, etc,
will reflect the new data (because each logical change has a single
Change object that they reference).
However, once these objects are no longer referenced by any items
in pipelines or pending events, we don't need to keep them in the
cache. We used to manually prune the cache, but that was difficult
and we stopped doing it some time ago. Currently, this is a slow
but intentional memory leak.
Instead, let the python GC handle it for us by using weak references
to the Change objects in the cache. Once the only references to
these objects are the cache itself, they will be discarded.
Change-Id: Id659888a41dd60b7e72e891a47f1f742c30d0cc4
The automatic statsd configuration based on env variables has
proven cumbersome and counter-intuitive. Move its configuration
into zuul.conf in preparation for other components emitting stats.
Change-Id: I3f6b5010d31c05e295f3d70925cac8460d334283
* The "Adding .. event" log messages are redundant -- whatever is adding the
event logs it just as well, and it makes filtering for Scheduler messages
difficult. Remove them.
* The gerrit/github connections log both the driver and connection name.
the driver can be inferred from the connection, and very frequently
this just ends up saying "Scheduling gerrit event from gerrit" which
is weird. Remove the driver name.
* Make the pipeline logs say which pipeline they are for. This is way more
useful than the manager name (which is implied by the pipeline name anyway).
Change-Id: I8fcaef87ddcd8428776ee76a4519e4764f2d9c5b
It exists only for py2/py3 compat. We do not need it any more.
This will explicitly break Zuul v3 for python2, which is different than
simply ceasing to test it and no longer declaring we support it. Since
we're not testing it any longer, it's bound to degrade overtime without
us noticing, so hopefully a clean and explicit break will prevent people
from running under python2 and it working for a minute, then breaking
later.
Change-Id: Ia16bb399a2869ab37a183f3f2197275bb3acafee
Just happened to notice that the scheduler did a statsd call about
events with the gerrit name. That's not accurate anymore.
Pull that code into the connection so that we can use the driver name
and the connection name in the statsd calls and in the log message
emitted.
Change-Id: Ia5ed9458c0b28573392c576db7789b61f4d87f3b
Allow connections to register their own handlers for HTTP URIs inside
the zuul's webapp HTTP server. That way, connections can listen for
events comming through HTTP.
Story: 2000774
Change-Id: Ic5887d00ff302f67469df5154e9df10b99f1cfcd
Apparently, there's been a change in recent upstream Gerrit which
changed the JSON serialization of change numbers in the output of
`gerrit stream-events`. Previously, the change number was being
transmitted as a string. That has changed and now it's being sent as a
number. That results in a failure later on:
2017-02-14 15:35:55,294 ERROR zuul.IndependentPipelineManager: Exception while launching job XXX for change <Change 0x7f3efc185c90 64,1>:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/zuul/scheduler.py", line 1520, in _launchJobs
dependent_items)
File "/usr/lib/python2.7/site-packages/zuul/launcher/gearman.py", line 318, in launch
destination_path = os.path.join(item.change.getBasePath(),
File "/usr/lib/python2.7/site-packages/zuul/model.py", line 913, in getBasePath
self.number[-2:], self.number, self.patchset)
TypeError: 'int' object has no attribute '__getitem__'
I would love to have this extended this with a proper testcase. Please
feel free to push an updated version over this patchset, or just let me
know which place is the most appropriate and I can do it myself, too.
Change-Id: I68f5fa7fa4fac0b4d2eb232fe37df11e57157ecf
This will allow us to enter results from all jobs for
use with the openstack-health dashboard.
Depends-On: I08dbbb64b3daba915a94e455f75eef61ab392852
Change-Id: I28056d84a3f6abcd8d9038a91a6c9a3902142f90
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This change, while substantial, is mostly organizational.
Currently, connections, sources, triggers, and reporters are
discrete concepts, and yet are related by virtue of the fact that
the ConnectionRegistry is used to instantiate each of them. The
method used to instantiate them is called "_getDriver", in
recognition that behind each "trigger", etc., which appears in
the config file, there is a class in the zuul.trigger hierarchy
implementing the driver for that trigger. Connections also
specify a "driver" in the config file.
In this change, we redefine a "driver" as a single class that
organizes related connections, sources, triggers and reporters.
The connection, source, trigger, and reporter interfaces still
exist. A driver class is responsible for indicating which of
those interfaces it supports and instantiating them when asked to
do so.
Zuul instantiates a single instance of each driver class it knows
about (currently hardcoded, but in the future, we will be able to
easily ask entrypoints for these). That instance will be
retained for the life of the Zuul server process.
When Zuul is (re-)configured, it asks the driver instances to
create new connection, source, trigger, reporter instances as
necessary. For instance, a user may specify a connection that
uses the "gerrit" driver, and the ConnectionRegistry would call
getConnection() on the Gerrit driver instance.
This is done for two reasons: first, it allows us to organize all
of the code related to interfacing with an external system
together. All of the existing connection, source, trigger, and
reporter classes are moved as follows:
zuul.connection.FOO -> zuul.driver.FOO.FOOconnection
zuul.source.FOO -> zuul.driver.FOO.FOOsource
zuul.trigger.FOO -> zuul.driver.FOO.FOOtrigger
zuul.reporter.FOO -> zuul.driver.FOO.FOOreporter
For instance, all of the code related to interfacing with Gerrit
is now is zuul.driver.gerrit.
Second, the addition of a single, long-lived object associated
with each of these systems allows us to better support some types
of interfaces. For instance, the Zuul trigger maintains a list
of events it is required to emit -- this list relates to a tenant
as a whole rather than individual pipelines or triggers. The
timer trigger maintains a single scheduler instance for all
tenants, but must be able to add or remove cron jobs based on an
individual tenant being reconfigured. The global driver instance
for each of these can be used to accomplish this.
As a result of using the driver interface to create new
connection, source, trigger and reporter instances, the
connection setup in ConnectionRegistry is much simpler, and can
easily be extended with entrypoints in the future.
The existing tests of connections, sources, triggers, and
reporters which only tested that they could be instantiated and
have names have been removed, as there are functional tests which
cover them.
Change-Id: Ib2f7297d81f7a003de48f799dc1b09e82d4894bc
Expose the ability to refresh a change in zuul/connection/gerrit.py
too, which is needed for our unit testing.
Change-Id: Iefd09d9b8deef563299e0f209d95e25b61aa4c1e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This lets us ask a source for all of the branches for a project.
This uses the git protocol for now, but this can get much nicer
in the future if we switch to using Gerrit's REST API. It should
also be easy to do with github.
The included comment indicates why it's being added -- implementation
to follow in subsequent changes.
Change-Id: I0dfcd61f343a235dcf935aea434b9772d6e746d9
Some gerrit clients connect through stateful firewalls,
which are occasionally "flushed", which leads to hanging of the connection.
This could be avoided by tcp keepalives.
To turn keepalives on, set "keepalive" in gerrit connection attributes to non-zero
(0 means no keepalives sent)
The default is turn it on with timeout of 60 seconds.
Change-Id: Ic3d11e0ac2108f02832e2114ba8bc1adcc87cb4f
James E. Blair