This was deprecated quite some time ago and we should remove it as
part of the next major release.
Also remove a very old Zuul v1 layout.yaml from the test fixtures.
Change-Id: I40030840b71e95f813f028ff31bc3e9b3eac4d6a
After upgrading Gerrit to 2.13 our gate stopped working. The reason
for this is that after a successful gate run zuul does something like
'gerrit review --label verified=2 --submit'. The verified label in
Gerrit by default is configured as 'Verified'. The newer version of
gerrit behaves different now. It accepts the +2 vote on verified but
doesn't submit the patch anymore if the casing is not correct. This
forces us to specify the label in the same casing as gerrit
expects. In that case the tolower() in canMerge prevents the patch
from entering the gate.
In order to avoid confusion and be consistent, avoid any case
conversions and use the labels exactly as defined in Gerrit.
Note that this patch requires changes to the pipelines such that the
labels are spelled exactly as defined in Gerrit.
Change-Id: I9713a075e07b268e4f2620c0862c128158283c7c
With a little tweak, test_jobs_launched and test_job_branch sufficiently
test that jobs are launched on the correct node.
Change-Id: I6f65b4574e13f56409f7d1e4e88fc1cd15ba1fda
There is a slight different in the zuulv3 syntax, we appear to lose
the ability to define template variables. The zuul configuration has
been updated accordingly.
Change-Id: I060eab9df841ce7ebcc6ae367e2122c5e7c01f55
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Create a new configuration project specific to one-job-project. Once
created using the new zuulv3 syntax, the test just worked.
Change-Id: Iba61970ddb2aafbebd0c326dce4b85ae72e35f02
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This allows us to add arbitrary string 'tags' to jobs. These may
then be inspected inside of parameter functions for any purpose.
In OpenStack, we will likely pass these through untouched to the
the build so that they can be picked up by the logstash worker
and we can record extra metadata about jobs.
Change-Id: Ibc00c6d30cdfe4678864adb13421a4d9f71f5128
This begins a lot of related changes refactoring config loading,
the data model, etc., which will continue in subsequent changes.
Change-Id: I2ca52a079a837555c1f668e29d5a2fe0a80c1af5
Unittest scenario:
1) Project with only one job in check pipeline (3rd party CI scenario)
2) That job has "files" restriction, which the default test change doesn't match.
3) Item is not reported, and is deleted from the queue.
Change-Id: I2ba6558111f67061476f8857b4f10019d2750a68
Since reporters must be specified now (after we added the ability
to have multiple reporters), the right way to make a silent pipeline
is to have no reporters, and the right way to have an experimental
pipeline (non-voting but reporting) is to have a reporter with no
parameters.
Change-Id: I5c979f0d2a6aafe5de4d60d8ce245d9bdcd73dae
This reverts commit 3aa3727225.
There are several more places in Zuul where it is assumed that job
names are unique. Fixing them all would be complicated and perhaps
not worth the benefit. Go back to requiring that they be unique for
a project.
Change-Id: I5b8aa33d6aed9cb086f44594ce59947c0190cc11
Allow an optional queue-name parameter to be set for a job.
As projects with that job are combined with others into shared
change queues, give the queue that name. This allows us to, say,
set the queue name of the tempest gate job to 'integrated' and
end up with the shared change queue of all the OpenStack integrated
projects named 'integrated'.
With that, we can do things like emit stats for the 'integrated'
queue.
Change-Id: Iafd218d7cd519312ccbf97de7c070e8d3b82038c
Provide the short name of a project (anything after the last '/') to
project templates as the variable 'name'. If 'openstack/nova' invoked
a template, the variable 'name' would automatically be set to 'nova'
within the template.
Ideally this means that most template invocations in OpenStack's layout
will not need any variables defined.
Change-Id: I596744917c30c92041b8ea5b1f518d50fb64e59b
Accept multiple template invocations per project, and also allow
adding individual jobs to a project that uses templates.
Change-Id: I6c668dd434c12bec96b9a27afd9fd2eca7a11d0a
Update the scheduler algorithm to NNFI -- Nearest Non-Failing Item.
A stateless description of the algorithm is that jobs for every
item should always be run based on the repository state(s) set by
the nearest non-failing item ahead of it in its change queue.
This means that should an item fail (for any reason -- failure to
merge, a merge conflict, or job failure) changes after it will
have their builds canceled and restarted with the assumption that
the failed change will not merge but the nearest non-failing
change ahead will merge.
This should mean that dependent queues will always be running
jobs and no longer need to wait for a failing change to merge or
not merge before restarting jobs.
This removes the dequeue-on-conflict behavior because there is
now no cost to keeping an item that can not merge in the queue.
The documentation and associated test for this are removed.
This also removes the concept of severed heads because a failing
change at the head will not prevent other changes from proceeding
with their tests. If the jobs for the change at the head run
longer than following changes, it could still impact them while
it completes, but the reduction in code complexity is worth this
minor de-optimization.
The debugging representation of QueueItem is changed to make it
more useful.
Change-Id: I0d2d416fb0dd88647490ec06ed69deae71d39374
Allows multiple reports per a patchset to be sent to pluggable
destinations. These are configurable per pipeline and, if not
specified, defaults to the legacy behaviour of reporting back only
to gerrit.
Having multiple reporting methods means only certain success/failure
/start parameters will apply to certain reporters. Reporters are
listed as keys under each of those actions.
This means that each key under success/failure/start is a reporter and the
dictionaries under those are sent to the reporter to deal with.
Change-Id: I80d7539772e1485d5880132f22e55751b25ec198
Add the ability for Zuul to accept inputs from multiple trigger
sources simultaneously.
Pipelines are associated with exactly one trigger, which must now
be named in the configuration file.
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
Change-Id: Ief2b31a7b8d85d30817f2747c1e2635f71ea24b9
Allow, eg, jobs in a gate pipeline to take precedence over
a check pipeline.
Change-Id: Idf91527704cc75b00a336291f91b908286f8e630
Reviewed-on: https://review.openstack.org/36552
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Change-Id: I147491d2c08fa1040f41f0c299113e3eb8968252
Reviewed-on: https://review.openstack.org/29158
Reviewed-by: Antoine Musso <hashar@free.fr>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Remove the Jenkins launcher and add a new Gearman launcher (designed
to be compatible with Jenkins) in its place.
See the documentation for how to set up the Gearman Plugin for
Jenkins.
Change-Id: Ie7224396271d7375f4ea42eebb57f883bc291738
On setup where Zuul ends up triggering hundreds of projects, you end up
having projects using roughly the same pipeline/jobs. Whenever one want
to add a job in all the similiar project, he has to edit each project
one by one.
To save some precious time, this patch introduces the concept of project
templates. It lets you define a set of pipeline and attached jobs
though the job names can be passed parameters defined on a per project
basis. Thus, updating similiar projects is all about editing a single
template.
A basic example is provided in the documentation.
The voluptuous schema has been updated. It does check whether all
parameters are properly passed to a template but does NOT check whether
the resulting job name exist.
The parameter expansion in templates is borrowed from Jenkins Job
Builder (deep_format function). It has been tweaked to also expand
dictionary keys.
Layout test plan:
$ nosetests -m layout --nocapture
Test layout file validation ...
<...>
bad_template1.yaml
required key not provided @
data['projects'][0]['template']['project']
bad_template2.yaml
extra keys not allowed @
data['projects'][0]['template']['extraparam']
good_template1.yaml
ok
<...>
$
A basic test hasbeen added to verify whether a project-template properly
triggers its tests:
$ nosetests --nocapture \
tests/test_scheduler.py:testScheduler.test_job_from_templates_launched
Test whether a job generated via a template can be launched ... ok
----------------------------------------------------------------------
Ran 1 test in 0.863s
OK
$
Change-Id: Ib82e4719331c204de87fbb4b20c198842b7e32f4
Reviewed-on: https://review.openstack.org/21881
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add an additional job parameter, 'file', that will cause that
job to only run if the change touches files that match the
specification.
Change-Id: I8c8fd3d029e02e338fd1dd266443b9ac56c0e5ac
Reviewed-on: https://review.openstack.org/23710
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When a new patchset is created for a change that is in a pipeline,
cancel running builds and dequeue that change (and possibly
dependent changes that can no longer merge).
Make this optional (and document the option).
Fixes bug 1022643.
Change-Id: I8f591956cf86645443e4b6075b8cdfc95a939e4f
Reviewed-on: https://review.openstack.org/20948
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
By setting parameters on a pipeline, you can change the default
"Build successful/failed" introductory messages left on Gerrit.
Change-Id: Ie05c8c87a47994faced4ee67fda4ab4e64ea0d20
Reviewed-on: https://review.openstack.org/19677
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Move most of the change queue logic into the model. This should
be a little cleaner, and it should be easier to follow what
happens when a build is complete.
Run launch jobs over the whole queue.
Collapse addChange into the base manager.
Makes reading the logic around adding a change simpler. Several
hooks are provided for the dependent manager subclass to extend
the method around dependent changes.
Move onbuildcompleted and reporting methods to base class.
Simplifies the code path when a build is completed. Move to a
more generalized model where updates are applied to changes as
received, then search for changes ready to be reported, then
search for jobs that need to be launched.
Also,
* test check queue functionality
* assert all queues are empty at ends of tests
* move formatting methods to pipeline/queue classes
* add queue-only mode notification to status screen
Change-Id: I336ac289af6ebd23567ac54b359c9f38af7f2ac7
Reviewed-on: https://review.openstack.org/12277
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Fix a bug where change objects were being compared incorrectly
when determining whether or not a change was in the queue
after a failed change had been dequeued. Added a test-case
that simulates the real-world conditions that exposed the bug.
Change-Id: I94a7915353335d80ab42b6c10c19595cb27788ae
Reviewed-on: https://review.openstack.org/12078
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
Change-Id: I5a9ccc857c28d458e3a8fbac03a9d29cc49b1da0
Reviewed-on: https://review.openstack.org/11904
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
This situation exposed a problem where tests for the second change
would not be run if the change in front of it had one job marked
hold and succeeded.
Change-Id: I8f3c3bdb9d3631efc8b207a61d5a9086808cafb4
Reviewed-on: https://review.openstack.org/11818
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add git repo management and merging. When collecting changes to
be tested together, merge or cherry-pick those changes into the
zuul-managed repos, and create a unique ref for that configuration.
Pass the ref to Jenkins instead of the string description of the
changes, so that Jenkins only needs to checkout that one ref.
This moves the complexity of merging and managing multiple commits
out of Jenkins and into Zuul.
The GERRIT_CHANGES variable is deprecated (along with the rest of
the GERRIT_* variables) and will be removed in a future patch
(which will contain a documentation update).
Change-Id: I126c9030223c07a30f7092e2273ebd7605d9f3df
Reviewed-on: https://review.openstack.org/11349
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
If there are no jobs for a project in a pipeline, an exception
would be raised. This change ensures that case is checked.
The test fixtures are updated to simulate this condition as well.
Change-Id: Id8e5bdef6288e5574d8553c11f5ac8e8c6c61704
Reviewed-on: https://review.openstack.org/10973
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
This is a refactoring of the data model with the following goals:
* Call top-level queues pipelines -- because too many other things
are already called queues. Pipelines convey the idea that there
are a number of tasks to be performed (jobs), and that those
tasks can be applied to different changes in parallel.
* Eliminate references to queue_name from within a Change.
Instead, methods that need to understand the pipeline that were
accessed previously via the change are now located in the
Pipeline class, taking a change as an argument. Essentially,
many methods involving changes (and builds, jobs, etc) must now
be called in the context of a pipeline.
* Add a changeish object to encompass the things that change and
ref events have in common.
Change-Id: Iaf8ed0991f3c5b2bf7ded2c340a60725f7f98eaf
Reviewed-on: https://review.openstack.org/10757
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add two tests:
* Changes behind failed changes get retested
* Shared change queues
Also, add the ability to simulate that a job for a change failed.
Change-Id: Ia266a9a6a8920ef49b24025f8109c4414c563e8c
Reviewed-on: https://review.openstack.org/10578
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add support for testing, and some initial functional tests.
Some threads in Zuul now have proper shutdown methods, since
that's needed for testing. The threads that are faked in the
testing infrastructure still lack shutdown methods.
Fix some bugs uncovered by tests:
Reverse the order of commits that are given to Jenkins (the
commits that preceded the one being tested were listed in
revers order, this corrects that).
Return 0 in reportChange if the change has already been
reported. The old return value of True was misinterpreted
as an error.
Change-Id: I4979d0efd8581061a2b856b892d4ecdb75284a1b
Reviewed-on: https://review.openstack.org/10572
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
James E. Blair