This change completes the circular dependency refactor.
The principal change is that queue items may now include
more than one change simultaneously in the case of circular
dependencies.
In dependent pipelines, the two-phase reporting process is
simplified because it happens during processing of a single
item.
In independent pipelines, non-live items are still used for
linear depnedencies, but multi-change items are used for
circular dependencies.
Previously changes were enqueued recursively and then
bundles were made out of the resulting items. Since we now
need to enqueue entire cycles in one queue item, the
dependency graph generation is performed at the start of
enqueing the first change in a cycle.
Some tests exercise situations where Zuul is processing
events for old patchsets of changes. The new change query
sequence mentioned in the previous paragraph necessitates
more accurate information about out-of-date patchsets than
the previous sequence, therefore the Gerrit driver has been
updated to query and return more data about non-current
patchsets.
This change is not backwards compatible with the existing
ZK schema, and will require Zuul systems delete all pipeline
states during the upgrade. A later change will implement
a helper command for this.
All backwards compatability handling for the last several
model_api versions which were added to prepare for this
upgrade have been removed. In general, all model data
structures involving frozen jobs are now indexed by the
frozen job's uuid and no longer include the job name since
a job name no longer uniquely identifies a job in a buildset
(either the uuid or the (job name, change) tuple must be
used to identify it).
Job deduplication is simplified and now only needs to
consider jobs within the same buildset.
The fake github driver had a bug (fakegithub.py line 694) where
it did not correctly increment the check run counter, so our
tests that verified that we closed out obsolete check runs
when re-enqueing were not valid. This has been corrected, and
in doing so, has necessitated some changes around quiet dequeing
when we re-enqueue a change.
The reporting in several drivers has been updated to support
reporting information about multiple changes in a queue item.
Change-Id: I0b9e4d3f9936b1e66a08142fc36866269dc287f1
Depends-On: https://review.opendev.org/907627
The drivers are expected to populate the change cache before
passing trigger events to the scheduler so that all the difficult
work is done outside the main loop. Further, the cache cleanup
is designed to accomodate this so that events in-flight don't have
their change cache entries removed early.
However, at several points since moving the change cache into ZK,
programming errors have caused us to encounter enqueued changes
without entries in the cache. This usually causes Zuul to abort
pipeline processing and is unrecoverable.
We should continue to address all incidences of those since they
represent Zuul not working as designed. However, it would be nice
if Zuul was able to recover from this.
To that end, this change allows missing changes to be added to the
change cache.
That is primarily accomplished by adjusting the Source.getChange
method to accept a ChangeKey instead of an Event. Events are only
available when the triggering event happens, whereas a ChangeKey
is available when loading the pipeline state.
A ChangeKey represents the minimal distinguishing characteristics
of a change, and so can be used in all cases. Some drivers obtain
extra information from events, so we still pass it into the getChange
method if available, but it's entirely optional -- we should still
get a workable Change object whether or not it's supplied.
Ref (and derived: Branch, Tag) objects currently only store their
newrev attribute in the ChangeKey, however we need to be able to
create Ref objects with an oldrev as well. Since the old and new
revs of a Ref are not inherent to the ref but rather the generating
event, we can't get that from the source system. So we need to
extend the ChangeKey object to include that. Adding an extra
attribute is troublesome since the ChangeKey is not a ZKObject and
therefore doesn't have access to the model api version. However,
it's not too much of a stretch to say that the "revision" field
(which like all ChangeKey fileds is driver-dependent) should include
the old and new revs. Therefore, in these cases the field is
upgraded in a backwards compatible way to include old and newrev
in the standard "old..new" git encoding format. We also need to
support "None" since that is a valid value in Zuul.
So that we can continue to identify cache errors, any time we encounter
a change key that is not in the cache and we also don't have an
event object, we log an error.
Almost all of this commit is the refactor to accept change keys
instead of events in getChange. The functional change to populate
the cache if it's missing basically consists of just removing
getChangeByKey and replacing it with getChange. A test which deletes
the cache midway through is added.
Change-Id: I4252bea6430cd434dbfaacd583db584cc796dfaa
This is a framework for making upgrades to the ZooKeeper data model
in a manner that can support a rolling Zuul system upgrade.
Change-Id: Iff09c95878420e19234908c2a937e9444832a6ec
Currently, the GitConnection doesn't provide a 'source' attribute. So
far this wasn't a problem as the source attribute was only accessed
within the same connection/driver.
With the ChangeCache we are now accessing the source attribute outside
of the specific connection/driver combination which fails in case of a
GitConnection with the following exception:
Traceback (most recent call last):
File "/workspace/zuul/zk/vendor/watchers.py", line 177, in _log_func_exception
result = self._func(data, stat, event)
File "/workspace/zuul/zk/change_cache.py", line 201, in _cacheItemWatcher
self._get(key, data_uuid, zstat)
File "/workspace/zuul/zk/change_cache.py", line 306, in _get
change = self._changeFromData(data)
File "/workspace/zuul/zk/change_cache.py", line 410, in _changeFromData
project = self.connection.source.getProject(change_data["project"])
AttributeError: 'GitConnection' object has no attribute 'source'
This seems to be only a problem in a multi scheduler setup when the
ChangeCache tries to load a change from ZooKeeper.
To fix this we simply add the source attribute to the GitConnection.
This uses the same approach like other connections by calling
driver.getSource().
Change-Id: I9503bf117f7fab31038e11eef9db65205d55e2e5
This is an early preparation step for removing the RPC calls between
zuul-web and the scheduler.
In order to do so we must initialize the ConfigLoader in zuul-web which
requires all connections to be available. Therefore, this change ensures
that we can load all connections in zuul-web without providing a
scheduler instance.
To avoid unnecessary traffic from a zuul-web instance the onLoad()
method initializes the change cache only if a scheduler instance is
available on the connection.
Change-Id: I3c1d2995e81e17763ae3454076ab2f5ce87ab1fc
This adds a ChangeKey class which is essentially a structured universal
identifier for a change-like object (Ref, Branch, Change, PR, whatever).
We can use this in ZK objects to reference changes, and by doing so, we
can in many cases avoid actually referencing the change objects
themselves.
This also updates the actual keys in ZK to be sha256sums of the structured
key (for brevity and simplicity of encoding).
Change-Id: I6cd62973d48ad3515f6aa8a8172b9e9c19fcda55
Move common methods to AbstractChangeCache so that concrete
implementations only need to defined the mapping from change type as
string to the change class.
Change-Id: I78c1bdfad1c0986aa6aef387424ea35b6c2aa71d
Remove the duplicate methods in the connections that use a
Zookeeper-backed change cache by moving them to a common mixin-class.
Change-Id: I527522e79444dfb69b41c70e7698a281166b2128
We need to periodically remove change data nodes that are not referenced
by any cache entry. This needs to happen more often than the periodic
cache maintenance (every 5min vs. every hour).
For this we need to introduce a new method `cleanupCache()` in the
connection interface.
In each run of the change cache's cleanup method we will identify
candidates of dangling data nodes to be cleaned up on the next run. The
reason for this delayed cleanup is that we don't want to remove data
nodes where a driver is in the process of creating/updating a new cache
entry.
Change-Id: I9abf7ac6fa117fe4a093ae7a0db1df0da5ae3ff3
With the persistent change cache in Zookeeper we need to periodically
remove changes that are outdated and no longer required by any active
item in the pipelines.
The cache maintenance will be performed together with the other general
cleanup every hour.
Change-Id: I62e75ab8c5b43f830e01b0e4c08b25ecdc5eed08
This is the second part of I767c0b4c5473b2948487c3ae5bbc612c25a2a24a.
It uses the MergerAPI.
Note: since we no longer have a central gearman server where we can
record all of the merge jobs, some tests now consult the merger api
to get the list of merge jobs which were submitted by that scheduler.
This should generally be equivalent, but something to keep in mind
as we add multiple schedulers.
Change-Id: I1c694bcdc967283f1b1a4821df7700d93240690a
Currently if two triggers of the same connection type need to trigger on
different events it's not possible to do so since the events are never
filtered on which connection they came from.
For example with the following setup where gerrit-org-1 only wants to
trigger on changes to 'master' and gerrit-org-2 only wants to trigger on
changes to 'develop' they will instead both trigger on 'master' and
'develop'since the events are never filtered on which connection they
came from.
- pipeline:
name: check
trigger:
gerrit-org-1:
- event: patchset-created
branch: 'master'
gerrit-org-2:
- event: patchset-created
branch: 'develop'
Change-Id: Ia0476d71dee59c8b80db7630ac7a524bce87e6f9
Trigger events will now be dispatched via Zookeeper. The event queues
are namespaced by tenant since the event processing will later require a
tenant lock in a multi scheduler deployment.
Gitlab events hold their labels as a non-serializable set attribute; this
change adjusts them to be held in a list (but set operations are still
used for de-duplication).
Change-Id: Ie54fc16488ab8cbc15f97d003f36c12b8a648ed4
Each scheduler in tests needs its own connection registries.
This change only affects tests.
Change-Id: I2ad188cf5a72c46f7486c23ab653ff574123308b
Story: 2007192
So that other drivers which may need to poll for ref updates can
use the same code as the git driver, extract that to a helper
class.
Change-Id: I89a9f5e461bfa5daecd74943d8a5521363cfd559
Sometimes, e.g. during reconfiguration, it can take quite some time
between the trigger event and when a change is enqueued.
This change allows tracking the time it takes from receiving the event
until it is processed by the scheduler.
Change-Id: I347acf56bc8d7671d96f6be444c71902563684be
When looking for git source references, we only need the heads.
This reduce the load of using https://opendev.org/zuul/zuul-jobs
which now contains all the gerrit refs.
Change-Id: I72532592312062dee17f095f3d56866d16e429dc
This change adds a /connections route to return the list of connections
and their configuration. This will be consumable as a way to merge
change locally.
Change-Id: Ia392faa2aea0740756ccd97f10cccb5aec51545a
Put the logging under the zuul hierarchy, and also add the ref
to the event representation. Both of these changes are in aid
of better visibility of git events.
Change-Id: Iac00be7536ad236e366cd793366ac6819357c17c
The current code uses a format string with an unconditional injection
of a /. This can result in a double-slash which can then cause git to
be unable to clone things. That in turn can lead to an indefinite hang
of a job trying to use the repo from the git driver.
Replace the format string with os.path.join which will correctly join
the project to the baseurl whether it's a url or a bare path, and
whether or not it has a trailing slash.
Change-Id: Id0e529b916ff75d6b6df98d978d2349f6037499c
These tests relied on sleeps which can cause races when running
the full test suite in parallel. Instead, wait for the events
we know will happen to happen.
Also remove the dependency on yarl now that aiohttp has made a
release which works with yarl 1.0 (however, it does not work with
<1.0 which is why this needs to be combined with this change to
fix tests).
Change-Id: Ib1c626cdd3f083dd1d23a3c6547bd7163b66567e
This patch improves the existing git driver by adding
a refs watcher thread. This refs watcher looks at
refs added, deleted, updated and trigger a ref-updated
event.
When a refs is updated and that the related commits
from oldrev to newrev include a change on .zuul.yaml/zuul.yaml
or zuul.d/*.yaml then tenants including that ref is reconfigured.
Furthermore the patch includes a triggering model. Events are
sent to the scheduler so jobs can be attached to a pipeline for
running jobs.
Change-Id: I529660cb20d011f36814abe64f837945dd3f1f33
When using a branch and pull model on a shared repository there are
usually one or more protected branches which are gated and a dynamic
number of temporary personal/feature branches which are the source for
the pull requests. These temporary branches while ungated can
potentially include broken zuul config and therefore break the global
tenant wide configuration.
In order to deal with this model add support for excluding unprotected
branches. This can be configured on tenant level and overridden per
project.
Change-Id: I8a45fd41539a3c964a84142f04c1644585c0fdcf
It exists only for py2/py3 compat. We do not need it any more.
This will explicitly break Zuul v3 for python2, which is different than
simply ceasing to test it and no longer declaring we support it. Since
we're not testing it any longer, it's bound to degrade overtime without
us noticing, so hopefully a clean and explicit break will prevent people
from running under python2 and it working for a minute, then breaking
later.
Change-Id: Ia16bb399a2869ab37a183f3f2197275bb3acafee
So that we can store the canonical hostname. Also use this to
find and store the connection name to keep the initializer signature
small.
Story: 2000953
Change-Id: Ie10f86ff3412016b411bcc511b4d9ad3af163d61
Not sure if schema requirements changed but it is no longer valid to
use:
Schema({}, extra=True)
The docs say that should be avoided and instead you should use:
Schema(dict)
To indicate a dict that can have any key: value pairs. Additionally
extra=True appears to be replaced with extra=ALLOW_EXTRA so update that
in cases where we have some subset of desired keys.
Change-Id: I3dc966dae534416f2b601ec8d299e1d0e3651e7b
This is the start of the implementation of:
http://lists.openstack.org/pipermail/openstack-infra/2017-March/005208.html
It lets us associate a canonical hostname with every connection
that we will later use to uniquely identify source code repos.
Story: 2000953
Change-Id: I7f2e64944d46f304e63a54078e682fd5e1682f27
This adds a very basic git driver, essentially so that Zuul can
use it to load its configuration from a non-subject system. This
facilitates third-party testing (where the configuration is under
the control of the zuul operator in a git repo, but the subject
system is otherwise unrelated).
Change-Id: I26829dbcbeeb23fec3a7978767071d9d28ea5597
James E. Blair