Reorganizing docs as recommended in:
https://www.divio.com/blog/documentation/
This is simply a reorganization of the existing documents and changes
no content EXCEPT to correct the location of sphinx doc references.
Expect followup changes to change document names (to reflect the new
structure) and to move content from existing guides (e.g., to move the
pipeline/project/job structure definitions out of the "Project Configuration"
reference guide into their own reference documents for easier locatability).
All documents are now located in either the "overview", "tutorials",
"discussions", or "references" subdirectories to reflect the new structure
presented to the user. Code examples and images are moved to "examples" and
"images" root-level directories.
Developer specific documents are located in the "references/developer"
directory.
Change-Id: I538ffd7409941c53bf42fe64b7acbc146023c1e3
Combining the supercedent pipeline manager with file filters can lead
to unexpected results so add a note about this.
Change-Id: I6d00b63b4fa926dd2d71240d7837ca49aa739d27
This facilitates integration with the gerrit checks API (and may
prove useful for other similar APIs). It will allow us to report
that a change has no jobs in a particular pipeline. A Zuul
pipeline will correspond to a Gerrit check, which means we can
update the status for that check from "SCHEDULED" to "NOT_RELEVANT"
if we determine that no jobs should run for the change. This
closes out the status of the check in Gerrit when a project is
configured to participate in a check/pipeline but no jobs are
actually configured.
Test coverage for this will be added in change
Ida0cdef682ca2ce117617eacfb67f371426a3131.
Change-Id: Ide2a332b294d7efe23601d80eeb92b5af1d4c21b
This facilitates integration with the gerrit checks API (and may
prove useful for other similar APIs). It will allow us to report
that a change has been enqueued in a particular pipeline. A Zuul
pipeline will correspond to a Gerrit check, which means we can
update the status for that check from "NOT_STARTED" to "SCHEDULED"
when it enters the pipeline. This is important for our check
polling loop, and it will cause that check to stop appearing in
the list of pending checks.
Test coverage for this is added in change
Ida0cdef682ca2ce117617eacfb67f371426a3131.
Change-Id: I9ec329b446fa51e0911d4d9ff67eea7ddd55ab5d
This expands the discussion of executor-only jobs with some additional
notes.
Additionally a unit test is added to explicitly test executor-only
(i.e. blank nodeset) jobs.
Change-Id: I8fd2f932290e49da5a3605737e8940425cd092f4
This causes file matchers to automatically match when the
configuration of the job itself changes. This can be used instead
of matching "^.zuul.yaml$" which may cause too many jobs to run
in larger repos.
Change-Id: Ieddaead91b597282c5674ba99b0c0f387843c722
If you define pragma.implied-branches on a branch, it needs to include
that branch name or changes will not be applied. Let's document this so
that other don't wonder why their zuul layout changes on a branch don't
actually trigger anything.
Change-Id: Ia9bbc54a199cd759580973f756956a9cf07d6b3e
To handle the case where an untrusted project defines a job with
a secret which another project would like to run, allow a config
project to attach that job to a project-pipeline and have it run
regardless of the allowed-projects setting.
Normally, untrusted jobs with secrets have an implicit and
non-overridable allowed-projects setting of only that project, to
avoid a situation where another project with a trusted post-review
pipeline gains access to the secret by using a Depends-On to a
change which lifts the allowed-projects restriction. This change
allows a config project to bypass this, in effect saying that the
projects involved trust each other sufficiently (or else, do not
have access to a post-review pipeline which could be used to
obtain secrets).
Change-Id: I52ab193d0e39a37de64c8b3cb6953538e4073b43
Many projects running on Github want to use the squash merge strategy
of github to keep their history clean. Now that the github reporter
forwards the merge-mode that can be configured on a project it is easy
to also support squash merge.
Change-Id: I1f4131002dd380f2860557a9aebcc0bcf4dc2af8
Instead of creating a merge commit, Github also allows for two other
options, squash and rebase. This change makes the github reporter
aware of the merge-mode which already can be set. For now only merge
and merge-resolve are supported which both map to the Github merge
mode 'merge'. The other two merge modes that are supported by Github
are not yet supported by the merger and will be implemented in later
changes.
Change-Id: I3d51261f248072d2e3ee7e3212377f81b9a41010
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
In some cases like resource constrained environments it is beneficial
to report on changes in a fail fast manner to immediately report if
one job fails. This can be useful especially if a project has many
expensive long-running jobs. This introduces a fail-fast flag in the
project pipeline that let's the project choose the trade off between
full information and quick feedback.
Change-Id: Ie4a5ac8e025362dbaacd3ae82f2e8369f7447a62
This section mentions 3 out of the 4 places zuul configs can be;
add the fourth so that people aren't misled.
Change-Id: Ib5dc22dd2f1c5b3a4f17acb03e7a76a29d736f1b
Currently the default ansible version is selected by the version of
zuul itself. However we want to make this configurable per deployment
(zuul.conf), tenant and job.
Change-Id: Iccbb124ac7f7a8260c730fbc109ccfc1dec09f8b
A "soft" dependency can be used to indicate that a job must run
after another completes, but only if it runs at all. For example,
a deployment job which depends on a build job with different file
matcher criteria.
Change-Id: I4d7fc2b40942569323da273c4529fdb365a3b11a
Like pre-run and post-run, allow a user to run a list of playbooks for
a job. One example would be your job workflow would be to run multiple
playbooks over using a site.yaml file with include_playbook commands.
A second use case, more related to job design. With multiple playbooks
support for job.run, the first playbook would be use deploy your server
and the second playbook to validate the server was provisioned properly.
Today, this can be done using a single run and post-run playbooks,
however if post-run fails, zuul will return POST_FAILURE, not FAILURE.
Not a large issue, but could be confusing to users when POST_FAILURE is
returned.
While it is possible a user could create a single site.yaml playbook,
and use multiple include_playbook statements to get this functionality,
there are downsides to this approach (mostly with the leaking of
variables). Today, operators simply run ansible-playbook multiple times
with the specific playbooks they only want.
Story: 2002543
Task: 22101
Change-Id: I6268d9944e745cc07407ea7dd040fbfeb79dad4d
Related-To: https://review.openstack.org/519596
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Adds support for expressing artifact dependencies between jobs
which may run in different projects.
Change-Id: If8cce8750d296d607841800e4bbf688a24c40e08
It is possible to circumvent the use of `allowed-projects` in
untrusted projects by creating a change which `Depends-On` a
change which alters a project definition. This behavior may be
unexpected, so documentation has been updated with warnings to
avoid relying on it in sensitive cases.
It may have been possible to expose a secret, or use resources
protected by a secret, if a job using a secret was defined in an
untrusted project on a system with an independent pre-merge
post-review pipeline -- that is, a pipeline with `post-review` set
to true, `manager` set to `independent`, and which operated on
changes before they merged.
To prevent disclosure or use in this situation, `allowed-projects`
is now automatically set to the current project when a secret is
used in a job defined in an untrusted project, and it can not be
overridden.
The test_trusted_secret_inheritance_gate test is removed because
it only tested that jobs with secrets in an untrusted repo were
able to run in a trusted repo. That is no longer possible.
Change-Id: I77f6a011bca08a2433137dc29597b7cc2757adb1
Story: 2004837
Task: 29037
This allows jobs to pass secrets to parent jobs. That will allow
for jobs which are designed to be used with secrets with the actual
secrets supplied by child jobs.
Change-Id: I544aec77c86f4cb1c11bf6e14a7d93efea0421d6
When calculating relative_priority for independent pipelines,
use shared change queues just as is done for dependent pipelines.
To implement this, we now calculate shared change queues for all
pipelines, not just dependent ones, though we don't use those
queues for any purpose other than this.
Change-Id: I59b1090ca1f4fcc72276445e6ff4c5cf4f2f5030
Currently when jobs use semaphores they first get and lock the build
nodes and then aquire the semaphore. If there are many jobs waiting
for the semaphore this can block a substantial part of the available
resources. In order to make this safe default to acquire the semaphore
before requesting the nodes.
However in some cases when jobs with a semaphore shall run as fast as
possible in a consecutive manner then it might be preferrable to
accept some waste of resources. In order to support this use case the
job using a semaphore can override this behavior and still acquire the
semaphore after getting the nodes.
Change-Id: Id6f582ec29219d280d05319d1b822c7934437b7a
This adds a "vars:" entry to projects and project-templates to make
available global values for each job. This can be useful to avoid
repeating the same variable definitions for the same job in different
pipelines, or to pass a project-specific value to jobs in a templating
situation.
Change-Id: I9fb468743a21067773979d113e714b5c3e908d02
Currently, variables using the extra-vars flags always win precedence
over any other variable in ansible. There is also a 2nd use case where
playbooks variables for serial, hosts, etc can only be set using
extra-vars CLI flag.
While this could be achieved by using secrets today, it doesn't feel
like the correct way to use them. Additionally, secrets are
dictionary values in ansible, making them hard to use the filters like
default().
Change-Id: I6d8018661f8d13b7324a012cdbf9614e983e5114
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This allows us to encode complex data structures, such as entire
openstack clouds.yaml cloud configs, as secrets.
Change-Id: I5fbafa41b9ab180842a4dd7bd82e2603f38e6644
The project pipelines themselves have branch matchers now, so we should
not add implied branch matchers to project-pipeline job variants. This
error would cause centrally defined tag/release jobs not to run when added
to in-repo project-pipeline definitions in multi-branch repos because
these project-pipeline variants would end up with branch matchers.
There remains a similar case where if a job is defined in a multi-branch
repo with no explicit branch matchers and added to a tag/release pipeline,
it will not run because the job definition itself will not match the tag.
Currently the only solution to this is to add an explicit branch matcher
to one or more of the top-level job definitions. A more intuitive solution
is difficult because in the case of multiple variants, it's not clear which
should apply.
Removing the implied branch matchers from project-pipeline jobs also removes
them from project-template jobs. We previously added branch matchers to
project configs, but did not do the same for project-templates. This change
requires that we do so. Now all projects and project-templates are given
implied branch matchers if appropriate, and these are used to determine if
their jobs are added. This is a further behavior change in that a project
which invokes a template defined in another project which is branched will
(absent the disabling of implicit branch matchers) no longer use that template
on branches other than the one where it is defined.
Change-Id: I55cec1897b0d64fa61d43ef5dbeb8a3c37bf7862
David Shrewsbury