The re2 library does not support negative lookahead expressions.
Expressions such as "(?!stable/)", "(?!master)", and "(?!refs/)" are
very useful branch specifiers with likely many instances in the wild.
We need to provide a migration path for these.
This updates the configuration options which currently accepts Python
regular expressions to additionally accept a nested dictionary which
allows specifying that the regex should be negated. In the future,
other options (global, multiline, etc) could be added.
A very few options are currently already compiled with re2. These are
left alone for now, but once the transition to re2 is complete, they
can be upgraded to use this syntax as well.
Change-Id: I509c9821993e1886cef1708ddee6d62d1a160bb0
This supplies a per-project default value for Zuul's default-branch
based on what the default branch is set to in GitHub. This means
that if users omit the default-branch setting on a Zuul project
stanza, Zuul will automatically use the correct value.
If the value in GitHub is changed, an event is emitted which allows
us to automatically reconfigure the tenant.
This could be expanded to other drivers that support an indication
of which branch is default.
Change-Id: I660376ecb3f382785d3bf96459384cfafef200c9
This updates the Gerrit driver to match the pattern in the GitHub
driver where instead of specifying individual trigger
requirements such as "require-approvals", instead a complete ref
filter (a la "requirements") can be embedded in the trigger
filter.
The "require-approvals" and "reject-approvals" attributes are
deprecated in favor of the new approach.
Additionally, all require filters in Gerrit are now available as
reject filters.
And finally, the Gerrit filters are updated to return
FalseWithReason so that log messages are more useful, and the
Github filters are updated to improve the language, avoid
apostraphes for ease of grepping, and match the new Gerrit
filters.
Change-Id: Ia9c749f1c8e318fe01e84e52831a9d0d2c10b203
This mimics a useful feature of the Gerrit driver and allows users
to configure pipelines that trigger on events but only if certain
conditions of the PR are met.
Unlike the Gerrit driver, this embeds the entire require/reject
filter within the trigger filter (the trigger filter has-a require
or reject filter). This makes the code simpler and is easier for
users to configure. If we like this approach, we should migrate the
gerrit driver as well, and perhaps the other drivers.
The "require-status" attribute already existed, but was undocumented.
This documents it, adds backwards-compat handling for it, and
deprecates it.
Some documentation typos are also corrected.
Change-Id: I4b6dd8c970691b1e74ffd5a96c2be4b8075f1a87
Github uses libgit2 to compute merges without requiring a worktree [0].
In some cases this can lead to Github detecting a merge conflict while
for Zuul the PR merges fine.
To prevent such changes from entering dependent pipelines and e.g. cause
a gate reset, we'll also check if Github detected any merge conflicts.
[0] https://github.blog/2022-10-03-highlights-from-git-2-38/
Change-Id: I22275f24c903a8548bb0ef6c32a2e15ba9eadac8
This adds the "draft" PR status as a pipeline requirement to the
GitHub driver. It is already used implicitly in dependent pipelines,
but this will allow it to be added explicitly to other pipelines
(for example, check).
This also fixes some minor copy/pasta errors in debug messages related
to github pipeline requirements.
Change-Id: I05f8f61aee251af24c1479274904b429baedb29d
Github now emits a webhook event when a branch protection rule changes.
Add support for that in the Github driver. We will update the branch
protection status in the branch cache immediately, and emit a trigger
event which will cause tenant reconfigurations if the protection status
of a branch has changed in any tenants.
Note: a branch protection rule applies to any number of branches, so
we may generate multiple Zuul trigger events from a single Github
webhook event in this case.
Change-Id: I0a7af786f9c69cf67eaaf4c75f437f8cf64a054a
Not all attributes of a GitHub PullRequest were properly
(de-)serialized which makes some test fail when running with
multiple schedulers.
Change-Id: Ic2c97992f8b2b4eaee89ca81bc202433d6715109
We only need to call the refreshDeps method if the Depends-On
list has changed. That can only happen with a new patchset
(gerrit) or the PR body has changed (github et al). Add a method
to determine if the PR body has changed so we can reduce the
times where we need to call this method.
Change-Id: Iaa50a274c29347397bc4e10e2c3cefc25e442879
Currently if two triggers of the same connection type need to trigger on
different events it's not possible to do so since the events are never
filtered on which connection they came from.
For example with the following setup where gerrit-org-1 only wants to
trigger on changes to 'master' and gerrit-org-2 only wants to trigger on
changes to 'develop' they will instead both trigger on 'master' and
'develop'since the events are never filtered on which connection they
came from.
- pipeline:
name: check
trigger:
gerrit-org-1:
- event: patchset-created
branch: 'master'
gerrit-org-2:
- event: patchset-created
branch: 'develop'
Change-Id: Ia0476d71dee59c8b80db7630ac7a524bce87e6f9
Since trigger events will be stored in Zookeeper we need a way to
(de-)serialze them from/to dictionaries.
Change-Id: I1698e22b61947761ddeb10264b84ec157609495b
- Fix wrong declaration and use of "check_run" attribute in Github
driver (was declared as "check_runs")
- Add missing fields to driver specific trigger event classes
Change-Id: I207fceaa699339ffeed747c718b5d706bb07431b
The pull request based events already have the information if the
target branch is protected. Hence we don't need to re-query this
information again.
Change-Id: I547db8090e13f1103918ce9c5f33002e9a42fcef
The 'status' field is a driver specific field and should therefore be
defined and initialized in the driver specific subclass of a change and
not in the 'Change' base class.
Change-Id: I0a20041ae8b9f5e4c359fea86c21d1c5c5cbfaca
Instead of synchronously checking via the GitHub API if a pull request
is mergeable, we can already get the necessary information during Github
event processing so the canMerge() check operates on cached data similar
to how this is done for the Gerrit driver already.
With this we can also remove the additional API requests for the commit
status and check runs since this info can be retrieved via the same GraphQL
request.
Change-Id: I6b4848dcb5d27071deeb6a59642c0e3c03a799da
So far, github use the subject of the first change in the commit
message body. This is most of the time redundant with the PR title.
Ths idea is to use instead the change body.
Change-Id: I9a2bc8aa3ff60a28b2ed62a4ee5fa9f7a2c9dcda
Utilizing the checks API to report the build state to Github provides
some additional functionality that is not supported by the status API.
Those are:
- Defining custom actions to e.g. cancel a running build
- Report line-based file annotations
This change implements the basic checks API workflow. Once this is in
place, the additional features could simply be added on top.
Change-Id: I7e790783ee35971085863b5487ff094fa0b23d65
Story: 2007268
Task: 38672
This change improves filter matches method to return a reason when
the filter fails. This helps debug why an event doesn't enter a
pipeline.
Change-Id: Ic31a059813a54b7ec9fae66312923cdd7d1fd7de
The GithubTriggerEvents currently don't log any detail of the event
itself. This makes it hard to analize issues using the log. We should
add more details about the action, project and pull request if
available.
Change-Id: Id07aa2c0bcfe743a0bfa34d635e77e9ecac4dc48
A tenant reconfiguration might be caused by a branch creation,
deletion, or change to files. Cause TriggerEvents to repr themselves
with flags indicating which of those are true. Have the
GithubTriggerEvent also repr itself with the associated delivery id
for easier tracing. Finally, have the scheduler log these events
when performing a tenant reconfiguration so we can see all the info.
Change-Id: I2cfc7615dfb3533aeb9d42b009cc67ce88a5ee48
Currently when creating a post pipeline with the github driver we can
only use the push event. This has the drawback that it ignores
protected branches and thus also works speculatively on any branch if
you want to have a generic post pipeline. This imposes problems with
secrets.
So instead using the push event we want to trigger on pull_request
closed events. However this currently doesn't work as we cannot
distinguish between merged and abandoned pull requests. Adding a
merged requirement nicely solves that problem.
Change-Id: I46670c6aa036976c430a6034a6b1da0e23fa9f92
The github status requirements matching and trigger filter are
currently plain text matching based. This currently limits sharing of
pipeline definitions between tenants as zuul reports the status as
'<tenant>/<pipeline>'. This currently makes it necessary to define
trigger filter for each tenant [1] and completely blocks pipeline
requirements.
A solution to this is regex matching which makes it possible to define
the filter once [2].
Further this enables an interesting further use case to trigger on any
successfull status [3]. This makes it easier to cooperate with other
CI systems or github apps which also set a status.
Directly use re2 as this will be used in the future for regex
matching.
[1] Trigger filter snippet
trigger:
github:
- event: pull_request
action: status
status:
- zuul:tenant1/check:success
- zuul:tenant2/check:success
- zuul:tenant3/check:success
- zuul:tenant4/check:success
[2] Regex trigger filter snippet
trigger:
github:
- event: pull_request
action: status
status:
- zuul:.+/check:success
[3] Generic success filter snippet
trigger:
github:
- event: pull_request
action: status
status:
- .*:success
Change-Id: Id1b9d7334db78d0f13db33d47a80ffdb65f921df
The documentation claims that reject is a mirror of require, but only
reviews were included in the actual implementation. With these changes
we can now say the documentation is correct.
Change-Id: I3b321a38e0a2b36ec333ae99a97038cabbbc42d5
On github, if somebody has the 'admin' permission, they implicitly have
write access to the repo. So rather than a straight up string match,
we need to implicitly assume anybody who has 'admin' also has 'write'.
Change-Id: Idf159c6746757fb3c9ce55f1d0cae432d3f8e478
In case the layout includes some regex approval filters (like
username) the dynamic layout creation fails during deepcopy of the
pipelines [1]. This is caused by Github/GerritApprovalFilter overwriting
stuff in the original data structure. Fix this by using the already
existing deepcopied data structures.
To test this an unused pipeline with approval filters for Github and
Gerrit is added to the dynamic layout test cases. This triggers the
deepcopy error without the fix in every test_dynamic_* test case.
[1] Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/zuul/manager/__init__.py", line 453, in _loadDynamicLayout
include_config_projects=False)
File "/usr/lib/python3.6/site-packages/zuul/configloader.py", line 1478, in createDynamicLayout
config = tenant.config_projects_config.copy()
File "/usr/lib/python3.6/site-packages/zuul/model.py", line 2150, in copy
r.pipelines = copy.deepcopy(self.pipelines)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
...
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
y = copier(memo)
TypeError: cannot deepcopy this pattern object
Change-Id: I4f7f74787aa91e938d0e2f07bd15b4e21d49cb88
Projects that do not use github reviews may wish to instead use github
labels as a way to trigger pipelines. As such, these projects may also
wish to require labels exist on a pull request when processing other
events.
Change-Id: I8f73c438c58db38790ea7e5bf435fbda01324e77
Story: 2000774
Task: 4632
Parts of this were already added, the schema and even a test layout that
used the option. However it was never fully plumbed into the github
model. This change completes that work and adjusts a test to exercise
the reject filter.
Story: 2000774
Task: 4633
Change-Id: I632b71a0ea3d8c6ba0e92f22595d38684705dbe8
Push and ref-updated events do not have an associated change or
pull-request. Thus, requirements that are specific to changes or
pull-requests cannot possibly be met.
To make this work, Ref filters should only be applied to changes that
come from a matching connection. This was not previously enforced when
filters became connection specific.
Change-Id: I5d67bef264db0ad5ba3a7180ce74e3670ba822ce
Story: 2000774
Task: 4624
Require that the commit from the event is the latest commit in the pull
request.
Also fix a problem with faked github status grabs. Now we're sending an
event where the sha of the event isn't the head sha, and that was
tripping up our fakes.
Change-Id: I269c97d096e42f0a2d4a0f1b0e57eb238e0b7baf
This allows using a connection specific requirement on the status of the
head of a PR. A list of statuses is accepted, just like the pipeline
requirement.
Update GithubRefFilter class to be more clear that it deals with
required statuses, whereas the event filter works with the status being
provided.
Change-Id: Ib91f6527bf1d8ff5fbc6434c8adaca1cd5e1ba6d
This updates the github source to only use the most recent review
from each user reviewing a PR. This avoids having obsolete negative
reviews trump a newer positive review.
I've added a new test for good measure to ensure it works for the
github case, where we are doing some conversion of github formatted
timestamps to internal timestamps.
Change-Id: I5607901def856c9363ec786a4116bfec19c9c97c
Co-Authored-By: Jesse Keating <omgjlk@us.ibm.com>
Signed-off-by: Adam Gandelman <adamg@ubuntu.com>
Github reviews are a new pipeline requirement that is driver specific.
Reviews can be approved, changes_requested, or comment. They can come
from people with read, write, or admin access. Access is hierarchical,
admin level includes write and read, and write access includes read.
Review requirements model loosely the gerrit approvals, allowing
filtering on username, email, newer-than, older-than, type, and
permission.
Brings in an unreleased Github3.py code. Further extends that code to
determine if a user has push rights to a repository.
Documentation is not included with this change, as the docs need
restructuring for driver specific require / reject.
Change-Id: I3ab2139c2b11b7dc8aa896a03047615bcf42adba
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
This adds support for triggering on github status updates.
Config schema for the github trigger has been updated to accept a list
of statuses, in the "github_user:context:status" format.
Change-Id: I15aef35716ddbcd1e66f84a73d27ca2689c936e4
Co-Authored-By: Jesse Keating <omgjlk@us.ibm.com>
Signed-off-by: Adam Gandelman <adamg@ubuntu.com>
When creating a github pull request change object, the statuses of the
head commit are fetched, and the latest status of each user and context
set is appended into a list of statuses. The string is
'user:context:state', where state can be success, error, or failure.
Context is freeform, and the user is the login of the entity that set
the status.
Tests have been added to validate that github based requirements on
pipelines are honored.
Change-Id: I45abbd6cbddd36b8491bdf9bb8d545216537ad2f
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
As we expand the Github driver, we're seeing a need to specify driver-specific
pipeline requirements. To accomplish this, bump the require/reject pipeline
keywords down a level underneath connection names. This lets users specify
per-source pipeline requirements.
This adds new API methods for sources to create the new pipeline filters
(by returning instances or subclasses of RefFilter, which used to be called
ChangeishFilter).
This change also creates and/or moves driver-specific subclasses of EventFilter
and TriggerEvent in(to) their respective drivers.
Change-Id: Ia56c254e3aa591a688103db5b04b3dddae7b2da4
James E. Blair