Since we require deleting the state (except config cache) for the
circular dependency upgrade, we can now remove any zookeeper
backwards compatability handling not in the config cache. This
change does so. It leaves upgrade handling for two attributes
in the branch cache (even though they are old) because it is
plausible that even systems that have been upgrading regularly
may have non-upgraded data in the branch cache for infrequently
used branches, and the cost for retaining the upgrade handling
is small.
Change-Id: I881578159b06c8c7b38a9fa0980aa0626dddad31
The re2 library does not support negative lookahead expressions.
Expressions such as "(?!stable/)", "(?!master)", and "(?!refs/)" are
very useful branch specifiers with likely many instances in the wild.
We need to provide a migration path for these.
This updates the configuration options which currently accepts Python
regular expressions to additionally accept a nested dictionary which
allows specifying that the regex should be negated. In the future,
other options (global, multiline, etc) could be added.
A very few options are currently already compiled with re2. These are
left alone for now, but once the transition to re2 is complete, they
can be upgraded to use this syntax as well.
Change-Id: I509c9821993e1886cef1708ddee6d62d1a160bb0
Change I0421c5a446d0b75194096b1d8f0e0866dae3b8f0 began including
BranchMatcher objects instead of strings in the SourceContext class
if an implied-branches pragma was supplied. If a syntax error
occurs in a repo with such a pragma, the source context needs to
be serialized to zk, and that would fail. To correct this, add
serialization methods to the source context and branch matchers.
Also correct an unrelated error where the syntax "- job:" would
fail to report the error to the user because the value None was
added to the list of config objects, and we always assume that
is a dict. To correct this, raise the syntax error earlier in
that case.
Change-Id: Id7c1dff5e9865d21c3215bdc32c8fa2cb6603aaa
If a project had a branch with a "+" in the name (or some other
special regex chars), then implied branch matchers would not match
the branch and jobs would not run. Instead of treating implied
branch matchers as regexes, treat them as requiring exact matches.
To facilitate creating the correct type of branch matcher, create
them in the configloader instead of in the model classes.
Change-Id: I0421c5a446d0b75194096b1d8f0e0866dae3b8f0
To try to approach a more intuitive behavior for jobs which apply
to tags but are defined in-repo (or even for centrally defined
jobs which should behave differently on tags from different branches),
look up which branches contain the commit referenced by a tag and
use that list in branch matchers.
If a tag item is enqueued, we look up the branches which contain
the commit referenced by the tag. If any of those branches match a
branch matcher, the matcher is considered to have matched.
This means that if a release job is defined on multiple branches,
the branch variant from each branch the tagged commit is on will be
used.
A typical case is for a tagged commit to appear in exactly one branch.
In that case, the most intuitive behavior (the version of the job
defined on that branch) occurs.
A less typical but perfectly reasonable case is that there are two
identical branches (ie, stable has just branched from master but not
diverged). In this case, if an identical commit is merged to both
branches, then both variants of a release job will run. However, it's
likely that these variants are identical anyway, so the result is
apparently the same as the previous case. However if the variants
are defined centrally, then they may differ while the branch contents
are the same, causing unexpected behavior when both variants are
applied.
If two branches have diverged, it will not be possible for the same
commit to be added to both branches, so in that case, only one of
the variants will apply. However, tags can be created retroactively,
so that even if a branch has diverged, if a commit in the history of
both branches is tagged, then both variants will apply, possibly
producing unexpected behavior.
Considering that the current behavior is to apply all variants of
jobs on tags all the time, the partial reduction of scope in the most
typical circumstances is probably a useful change.
Change-Id: I5734ed8aeab90c1754e27dc792d39690f16ac70c
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
This adds exceptions that were already applied
to the irrelevant files matcher.
It implements some version of option D discussed in [1].
Added comments to both matchers implementations.
The FileMatcher is now a stub holding relevant regex only.
[1] https://review.opendev.org/660856
Change-Id: Icf5df145e4cd351ffd04b1e417e9f7ab8c5ccd12
Story: 2005040
Task: 29531
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
When having the two branches stable and stable-foo with distinct
project pipelines that pull in different project-templates currently
both project-templates match. The project-templates in this case get
implicit branch matchers. However the implied branch matcher currently
only does a partial match and so a change to stable-foo matches the
project-templates of both stable and stable-foo.
Fix this by changing the implied branch matchers to do full matching.
Change-Id: I569db50f05c1b53cb1fb26e84ea0af377e3df73c
Adding an implied branch matcher to jobs on in-repo project defs
works great when an item *has* a branch. But some items, such as
tags, don't. With recent changes, it is now impossible for a
project to add a job in-repo that runs in a tag pipeline.
To correct this, we need to drop some of the optimizations which
assumed we could match the implied branch against existing branch
matchers, and instead, when adding a job in-repo, simply add a new
kind of branch matcher, an ImpliedBranchMatcher, that is evaluated
in a boolean 'and' with any existing branch matchers.
The ImpliedBranchMatcher only fails if the item has a branch, and
the branch doesn't match. If the item doesn't have a branch, it
always succeeds.
This means that when a project adds a job to a tag pipeline in-repo,
it will most likely only have the ImpliedBranchMatcher, which will
simply succeed.
It also means that the multiple project configurations present in
the project's multiple branches can all add jobs to tag pipelines,
and so to remove such a job, changes may need to be made to all
branches of a project. However, there's not much that can be done
about that at the moment.
Change-Id: Id51ddfce7ef0a6d5e3273da784e407ac72a669db
Based on Jim's feedback, change the branch matching logic to always
have priority over ref matching. And v3 will always have refs, so no
need to check if that attribute exists. Also adds a test that checks the
current breakage of branch matching logic.
Change-Id: Iba148b73a77b3300ad84db1c05c083d2c82cd950
Allow to configure jobs to run only when certain files are changed.
Github does not list the /COMMIT_MSG in the changed files as gerrit
does. Therefore the matcher now returns False only if the single file is
the /COMMIT_MSG one.
Change-Id: I4fa8a328f2ba430c25377e50e1eff7c45829eba6
This makes everything a child of Ref. As a result of this rearrangement,
NullChanges are awkward to use, and thus, have been replaced by enqueued
Refs where the 'ref' attribute is still unknown. As a result,
status.json will show timer-created jobs with an ID, where they used to
show a null, which is why that change to test_timer is necessary.
Change-Id: Ief0d3dde089b5529b9df7a804f6fea72b8b7dc48
Story: 2000781
Task: 3300
An earlier change dealt with inheritance for pre and post playbooks;
they are nested so that parent job pre and post playbooks run first
and last respectively.
As for the actual playbook, since it's implied by the job name, it's
not clear whether it should be overidden or not. We could drop that
and say that if you specify a 'run' attribute, it means you want to
set the playbook for a job, but if you omit it, you want to use the
parent's playbook.
However, we could keep the implied playbook behavior by making the
'run' attribute a list and adding a playbook context to the list each
time a job is inherited. Then the launcher can walk the list in order
and the first playbook it finds, it runs.
This is what is implemented here.
However, we need to restrict playbooks or other execution-related
job attributes from being overidden by out-of-repo variants (such
as the implicit variant which is created by every entry in a
project-pipeline). To do this, we make more of a distinction
between inheritance and variance, implementing each with its own
method on Job. This way we can better control when certain
attributes are allowed to be set. The 'final' job attribute is
added to indicate that a job should not accept any further
modifications to execution-related attributes.
The attribute storage in Job is altered so that each Job object
explicitly stores whether an attribute was set on it. This makes
it easier to start with a job and apply only the specified
attributes of each variant in turn. Default values are still
handled.
Essentially, each "job" appearance in the configuration will
create a new Job entry with exactly those attributes (with the
exception that a job where "parent" is set will first copy
attributes which are explicitly set on its parent).
When a job is frozen after an item is enqueued, the first
matching job is copied, and each subsequent matching job is
applied as a varient. When that is completed, if the job has
un-inheritable auth information, it is set as final, and then the
project-pipeline variant is applied.
New tests are added to exercise the new methods on Job.
Change-Id: Iaf6d661a7bd0085e55bc301f83fe158fd0a70166
If there is some skip-if condition containing all-files-match-any, then
Zuul skips jobs for changes without modified files (merge commits),
because it always matches '/COMMIT_MSG'.
So test files for regexes only if CR has more than one modified file,
because '/COMMIT_MSG' is always included even for empty merge commits.
Change-Id: Iad78d9eb8212beea3238728321c1ba74efa991e2
There was previously no way to skip a job if the contents of a given
patchset did not require it to be run. This meant that tempest jobs
would run in response to patchsets that would not affect their
execution, like those that only modified documentation or unit tests.
This change introduces a 'skip-if' configuration option that allows
jobs to be configured to not run when specified criteria are met.
Change-Id: I9b261f03ec00426160d9396f3a20312e89b624d4
James E. Blair