This refactors the the config error handling based on nested
context managers that add increasing amounts of information
about error locations. In other words, when we start processing
a Job object, we will:
with ParseContext.errorContext(info about job):
do some stuff
with ParseContext.errorContext(info about job attr):
do some stuff regarding the job attribute
with ParseContext.errorContext(next job attr):
do stuff with a different attr
We store a stack of error contexts on the parse context, and at any
point we can access the accumulator for the most recent one with
ParseContext.accumulator in order to add a warning or error. If we
have an attribute line number, we'll use it, otherwise we'll just
use the object-level information.
We also collapse the exception hnadlers into a single context
manager which catches exceptions and adds them to the accumulator.
This lets us decide when to catch an exception and skip to the next
phase of processing separately from where we narrow our focus to
a new object or attribute. These two actions often happen together,
but not always.
This should serve to simplify the configloader code and make it
easier to have consistent error handling within.
Change-Id: I180f9b271acd4b62039003fa8b38900f9863bad8
This adds the ability for the configuration loader to accumulate
warning-level configuration errors. Additionally, the
driver-specific triggers can do the same.
An initial pair of warnings is added for two recently deprecated
attributes on the gerrit trigger.
Change-Id: I61f856b3aea3e0ad3147c52c7aa724f40734d1f5
Currently if two triggers of the same connection type need to trigger on
different events it's not possible to do so since the events are never
filtered on which connection they came from.
For example with the following setup where gerrit-org-1 only wants to
trigger on changes to 'master' and gerrit-org-2 only wants to trigger on
changes to 'develop' they will instead both trigger on 'master' and
'develop'since the events are never filtered on which connection they
came from.
- pipeline:
name: check
trigger:
gerrit-org-1:
- event: patchset-created
branch: 'master'
gerrit-org-2:
- event: patchset-created
branch: 'develop'
Change-Id: Ia0476d71dee59c8b80db7630ac7a524bce87e6f9
The zuul driver also emits logs that should be annotated. Further it's
an event source and thus should generate uuid for the events it
creates.
Change-Id: I8c55a6baa792293b4fc9863557fefd59950f0333
It exists only for py2/py3 compat. We do not need it any more.
This will explicitly break Zuul v3 for python2, which is different than
simply ceasing to test it and no longer declaring we support it. Since
we're not testing it any longer, it's bound to degrade overtime without
us noticing, so hopefully a clean and explicit break will prevent people
from running under python2 and it working for a minute, then breaking
later.
Change-Id: Ia16bb399a2869ab37a183f3f2197275bb3acafee
This change, while substantial, is mostly organizational.
Currently, connections, sources, triggers, and reporters are
discrete concepts, and yet are related by virtue of the fact that
the ConnectionRegistry is used to instantiate each of them. The
method used to instantiate them is called "_getDriver", in
recognition that behind each "trigger", etc., which appears in
the config file, there is a class in the zuul.trigger hierarchy
implementing the driver for that trigger. Connections also
specify a "driver" in the config file.
In this change, we redefine a "driver" as a single class that
organizes related connections, sources, triggers and reporters.
The connection, source, trigger, and reporter interfaces still
exist. A driver class is responsible for indicating which of
those interfaces it supports and instantiating them when asked to
do so.
Zuul instantiates a single instance of each driver class it knows
about (currently hardcoded, but in the future, we will be able to
easily ask entrypoints for these). That instance will be
retained for the life of the Zuul server process.
When Zuul is (re-)configured, it asks the driver instances to
create new connection, source, trigger, reporter instances as
necessary. For instance, a user may specify a connection that
uses the "gerrit" driver, and the ConnectionRegistry would call
getConnection() on the Gerrit driver instance.
This is done for two reasons: first, it allows us to organize all
of the code related to interfacing with an external system
together. All of the existing connection, source, trigger, and
reporter classes are moved as follows:
zuul.connection.FOO -> zuul.driver.FOO.FOOconnection
zuul.source.FOO -> zuul.driver.FOO.FOOsource
zuul.trigger.FOO -> zuul.driver.FOO.FOOtrigger
zuul.reporter.FOO -> zuul.driver.FOO.FOOreporter
For instance, all of the code related to interfacing with Gerrit
is now is zuul.driver.gerrit.
Second, the addition of a single, long-lived object associated
with each of these systems allows us to better support some types
of interfaces. For instance, the Zuul trigger maintains a list
of events it is required to emit -- this list relates to a tenant
as a whole rather than individual pipelines or triggers. The
timer trigger maintains a single scheduler instance for all
tenants, but must be able to add or remove cron jobs based on an
individual tenant being reconfigured. The global driver instance
for each of these can be used to accomplish this.
As a result of using the driver interface to create new
connection, source, trigger and reporter instances, the
connection setup in ConnectionRegistry is much simpler, and can
easily be extended with entrypoints in the future.
The existing tests of connections, sources, triggers, and
reporters which only tested that they could be instantiated and
have names have been removed, as there are functional tests which
cover them.
Change-Id: Ib2f7297d81f7a003de48f799dc1b09e82d4894bc
This addresses a recent increase in test_idle failures. The
existing apscheduler was not being shut down during reconfigurations
which caused us to end up with two apschedulers running during the
test. This could cause a trigger event to fire while the test
was expecting the system to be idle.
The triggers were not previously shut down during reconfigurations.
Instead, the timer trigger relied on having its postConfig method
called on each reconfiguration, where it would begin by cleaning
up any existing jobs. However, since the connections changes,
triggers are stopped, discarded, and recreated during reconfiguration.
Because the stop method of the timer trigger was not actually being
called, a new trigger was created each time we reconfigured, and old
ones were never cleaned up.
This likely had a production impact as well, however, it was not
likely to be visible unless a configuration change altered the
scheduled times for periodic queues (in that case, we would see jobs
run at both the old and new times).
Change-Id: Ia7c61984a9c47a9b1554a4ccb99309674dffec11
The connection registry should not have to know about the scheduler,
rather, the inverse is true.
(NB, connections themselves still know about the scheduler, but
that's okay, that happens after the connection registry is created.)
Drivers should be able to access the global configuration when
being created, so store that when the connection registry configures
itself.
Change-Id: Iea4b8fe3888b5eefd3df9ce385225b885f2caa0b
This patch upgrades zuul to support APScheduler 3.0. For the most
part, 3.0 was a rewrite but our changes seem to be limited.
Change-Id: I0c66b5998122c3f59ed06e3e7b3ab3199f94f478
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This is a large refactor and as small as I could feasibly make it
while keeping the tests working. I'll do the documentation and
touch ups in the next commit to make digesting easier.
Change-Id: Iac5083996a183d1d8a9b6cb8f70836f7c39ee910
Move EventFilter configuration into the triggers to allow dynamic
inclusion of triggers rather than specifying each one in the sechduler.
Change-Id: I4ed345058ff1ffdd662fafa854e36782cc7f047b
and test the all triggers adhere to the set contract.
Also standardise the trigger class names to NameTrigger.
This will make it easier to do more triggers in the future and also
add the possibility of loading triggers dynamically.
Co-Authored-By: Gregory Haynes <greg@greghaynes.net>
Change-Id: I283bffad97edb91a6a2258163d4162a15b904757
This is to further differentiate between sources and triggers.
Eventually allowing for multiple triggers per pipeline.
Still to come is separating connections from everything.
Change-Id: I1d680dbed5f650165643842af450f16b32ec5ed9
3rd party CI layout usually has only a few projects defined,
so it's possible that some changes depend on projects
which are unknown to Zuul scheduler.
These items had None as a "item.change.project", which
is not handled in many places, for ex. in reconfiguration.
These cases could be handled by defining these projects in layout
as "foreign" projects: no jobs, no other non-standard attributes.
Changes to those projects are also dropped, unless
they came as dependencies.
Change-Id: I7912197fb86c1a7becb7f43ca36078101f632715
If a review label uses NoOp, then it will indicate 'MAY' as the
status in the submit record for that label (indicating that there
are no criteria for that label to be satisfied). Treat that in
the same way as OK so that a merge can occur.
Change-Id: I24be23d16aa133a3323b79bd775d42678ebc1dfe
The gerrit event processing thread performs its own queries and
updates of change objects outside of the main thread, possibly
while those objects are being used in the main thread. Generally
this is okay because typically only new data are being added to
changes, and if any of the new data are relevant, the main thread
will receive an event about it and update accordingly.
However, in the process of updating changes in the event processing
thread, some attributes of the change may be temporarily set to
incorrect values, such as empty lists. Switch to atomic updates
of those values so that if the main thread consults the change object,
it has reasonably valid values.
This may have caused sporadic failures of test_dependent_changes_dequeue,
but is also capable of happening in production.
Change-Id: I134d50fe2da8ef344ee8d4fb68a1eed0692b04e2
When querying a change immediately after a patchset upload, Gerrit
may return incorrect data about dependent changes. In order to
avoid this, ensure at least 5 seconds have elapsed before delivering
any Gerrit events to Zuul.
Change-Id: I95c459f52562556ca1722878e32388d51033e86c
In order to find reverse dependencies specified by cross-repo
"Depends-On" headers, search for the occurance of a given change-id
in commit messages, and then filter the result set for "Depends-On"
headers.
Add those cross-repo reverse dependencies to the already supplied
Gerrit git reverse dependencies. This should cause behaviors
such as automatic enqueuing of reverse dependencies on merges to
apply to CRD as well as git-dependencies.
When collecting reverse CRD, update cached Zuul change records.
This corrects a bug where if change A depends on B in a different
repo, and change B is updated with a new patchset, Zuul's cached
record for change A will still point to the old patchset of change
B. This change corrects that by causing B to trigger a refresh of
A when the reverse CRD is noted.
Change-Id: I1bcf1f0fa0ea1c20b01ea478a83f179b612a0ae9
Parse commit messages for "Depends-On: <changeid>" and treat
matching changes as changes that the given change depends on.
This will treat any changes in any branch of any project as
such. If the projects share a dependent change queue, the
changes will be enqueued in order. If they do not share a
change queue in a dependent pipeline, then the latter one will
be unable to be enqueued until the change it depends on merges.
If the dependencies result in a cycle, Zuul will log the error
but otherwise the problematic changes will be ignored.
Dependent changes in independent pipelines are not yet addressed.
Change-Id: I90c173f86d11e6c44d1f408646589b7c75b1cd52
Does not actually handle more than one change correctly yet.
Just turns the attribute into a list and makes current processing
iterative.
Change-Id: I1acd62125b315fdcf04594cb52a71b16093884a7
The pep8 jobs aren't running in gate at the moment.
Fix up the pep8 issues and remove the tox filter that stopped it
from running.
Also ignore E129 (visually indented line with same indent as next
logical line) as we don't follow it.
Change-Id: I394708ba96797bbc6fcd951e6436a104be0a3746
gerrit defaults to a limit of 500 changes returned. Nova regularly has
more than 500 open reviews. This means that some operations aren't
seeing all the open reviews, and causing some oddness (like merge
check).
Implement implicit paging.
In looking at this code I also believe I found a double -1 slicing
error which would mean that we would miss some reviews being listed.
Change-Id: I00d47c05e4a740b61c690510593fd29eaef49679
gerrit.query does return the owner information, we can save that
in the Change object and serialize it to status.json
Change-Id: Ib505ef8703bc60058d2775f522fcc3d02910b05f
Invalid layout could be used and cause zuul to crash. If a ref
was specified for an event that did not supply a ref,
(e.g. patchset-created) then zuul would crash trying to match
the regex supplied in layout.yaml to event.ref, which would be
None. This results in all jobs failing, so this patch
makes sure that only the ref-updated gerrit event can have
a ref pattern specified.
Change-Id: I297e38f3749166ce54a946f6cbcf708744f20fa1
A previous change removed the trigger cache cleanup. When Zuul
is live-reconfigured, it creates new Project objects and
attaches them to all changes currently in pipelines. Because
the cache contained changes outside of pipelines, those changes
were not being updated with new Project objects.
This restores the maintainCache method and uses it during the
reconfigure event to clear the cache of all changes not currently
in pipelines. This corrects the error introduced by the previous
change as well as gives Zuul an opportunity to release unused
memory (as now the cache will at least be emptied on each
reconfiguration).
The project-change-merged test is updated to explicitly test this
situation.
Change-Id: I67736fca08f2e14ab733bd9f143820da19839ef9
The zuul trigger ends up querying a large number of changes on
each merge, most of which are not in any pipelines. To avoid
needlessly querying gerrit for data that do not change, keep
all changes that Zuul ever sees in the cache.
We could consider removing changes that are both not in pipelines
and closed (ie, keep all open changes), but we still end up querying
a number of merged changes each time due to Zuul following
dependencies of open changes.
Change-Id: Ie78df9aa43ec5ac35bdea79dcdafdfdd41d51d5b
Against a live gerrit, we observed that sometimes gerrit may not
return data for a change. In the case of the zuul trigger's
project-change-merged event, it's best to just ignore errors
for those changes and proceed.
Also, the gerrit trigger was attempting to parse the timing
results from the query. The test did not catch this because it
did not supply them in the mocked method. Correct this as well.
Also, double check that the query used was the one expected in
the test.
Change-Id: I792127d29f67f53a419eb94e9e0afb83b6e1bcb2
Function updateChange short-cuts dependency checking for merged changes, but
some attributes that need to be updated were after the short-cut return.
This resulted in zuul saying status is MERGED, but then not matching
Changeish filter for status MERGED.
Closes-Bug: 1356662
Change-Id: I954b2716a5af75a959d4129ba88d7dae0750d2a5
This adds the ability for a pipelite to have multiple triggers.
This also adds a "Zuul" trigger which is used to generate trigger
events based on internal actions Zuul has taken.
It supports two event types:
* parent-change-enqueued: This can be used so that other pipelines
can enqueue children of parents that are enqueued in a different
pipeline. Specifically, this lets OpenStack enqueue changes in
check when their parents are enqueued in gate (which may be
necessary because of our clean check rules).
This could be used to replace the internal logic that enqueues
children in dependent pipelines (moving that into explicit
configuration instead).
One can also imagine a future 'change-enqueued' event so that a
pipeline could react directly to a change in another.
* project-change-merged: This can be used to trigger changes on all
open changes for a project when a change is merged to that project.
Specifically, this lets us perform light-weight merge checks on all
open changes whenever a change is merged.
Change-Id: I2a67699dbed92a6b9c143a77795cb126f1f4dd57
A TriggerEvent may originate from a trigger that does not represent
the canonical location of the project source. For instance, the
timer trigger strangely depends on the gerrit trigger to actually
handle Git operations behind the scenes. Instead, make an explicit
association between pipelines and their source triggers so that
their event trigger does not need to have that implicit association.
This is a step toward having pipelines support multiple triggers
(they already support multiple reporters).
Change-Id: Ie80ffde411fe40fddfc4496b7adb0004f660c48c
Add a generalized pipeline requirements option that allows the
user to specify that a change must meet certain pre-requisites
before being enqueued into the pipeline. This is intended to
replace the "requires-approval" event filter on triggers (which
due to the automatic enqueuing of dependencies may not always
behave as the user intends). It also adds the ability to
specify that a change must be either open or closed or have one
of a number of specified statuses in order to be enqueued.
Change-Id: I7a55aa33fa8e1dcb405796261085e31138d37653
Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
* zuul/trigger/gerrit.py(Gerrit.isMerged): Make sure to return False
if we fell through all the conditionals, rather than defaulting to
None. This should have no current impact since all comparisons to
the returned value elsewhere are boolean and not "is None" or
similar.
Change-Id: I05360833f1e5b35fbfb1894dadcb46e82ff78a11
* zuul/trigger/gerrit.py(Gerrit._isMerged): Only consider MERGED
status to indicate a change is successfully merged. SUBMITTED can
exist on changes which Gerrit was unable to merge due to nonexistent
parent/dependent commits, and possibly for other reasons too.
Change-Id: I673be5010031f8bc26ab31db9afcc7c28854db85
For very large projects, when Gerrit is under load, it can take
longer than 60 seconds for the submit action to be processed and the
associated commit to replicate to its local mirror, causing Zuul to
think the change did not merge and resetting any remaining changes
in that dependent queue. Increase this to 5 minutes, since the
impact from these "phantom gate resets" is more significant than any
potential increase in merge wait time.
Change-Id: If57d5ed2720707c1ec3a3477dd229f042789fa1f
Connect it to Zuul via Gearman. Any number of mergers may be
deployed.
Directly find the pipeline for a build when processing a result,
so that the procedure is roughly the same for build and merge
results.
The timer trigger currently requires the gerrit trigger also be
configured. Make that explicit inside of the timer trigger so
that the scheduler API interaction with triggers is cleaner.
Change-Id: I69498813764753c97c426e42d17596c2ef1d87cf
This feature allows Zuul to consider existing (or new) approval
votes associated with a change when determining whether an event
matches. For example, it can be used to require that a Verified
vote of a certain age be present before a change is enqueued in
a pipeline.
Change-Id: I81344713d71b345b08576334568b9c49c810c7e9
As it stands, if multiple pipelines
have timers with the same timespec, they will all respond to
the events generated for all of them.
Instead of matching on the timespec, associate a timer trigger
with a pipeline directly.
Change-Id: I6a799cc3b59bd7527ace9ee1048bf633dcaa4cd9
Add a command line client called 'zuul' that supports one command
to start with: 'enqueue'. It allows an operator (one with access
to the gearman server) to enqueue an arbitrary change in a specified
pipeline. It uses gearman to communicate with the Zuul server, which
now has an added RPC listener component to answer such requests via
gearman.
Add tests for the client RPC interface.
Raise an exception if a Gerrit query does not produce a change. Unlike
events from Gerrit, user (or admin) submitted events over the RPC bus
are more likely to reference invalid changes. To validate those, the
Gerrit trigger will raise an exception (and remove from its cache) changes
which prove to be invalid.
Change-Id: Ife07683a736c15f4db44a0f9881f3f71b78716b2
Allows multiple reports per a patchset to be sent to pluggable
destinations. These are configurable per pipeline and, if not
specified, defaults to the legacy behaviour of reporting back only
to gerrit.
Having multiple reporting methods means only certain success/failure
/start parameters will apply to certain reporters. Reporters are
listed as keys under each of those actions.
This means that each key under success/failure/start is a reporter and the
dictionaries under those are sent to the reporter to deal with.
Change-Id: I80d7539772e1485d5880132f22e55751b25ec198
Several assignments in updateChange would actually just keep
appending data causing immensely large data structures (which
are later traversed putting Zuul into a significant busy loop).
Make sure that data are replaced instead of augmented.
Change-Id: I8c6528adbbe24d30f8d5bb8b55bb731fefd9941a
Add the ability for Zuul to accept inputs from multiple trigger
sources simultaneously.
Pipelines are associated with exactly one trigger, which must now
be named in the configuration file.
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
Change-Id: Ief2b31a7b8d85d30817f2747c1e2635f71ea24b9
Change-Id: I69563ee47dd6f3777a52b67999ff1a03247f1e1e
Reviewed-on: https://review.openstack.org/35324
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Store the results of the configuration (pipelines, jobs, and all)
in a new Layout object. Return such an object from the parseConfig
method in the scheduler. This is a first step to reloading the
configuration on the fly -- it supports holding multiple
configurations in memory at once.
Change-Id: Ide56cddecbdbecdc4ed77b917d0b9bb24b1753d5
Reviewed-on: https://review.openstack.org/35323
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
James E. Blair