Reorganizing docs as recommended in:
https://www.divio.com/blog/documentation/
This is simply a reorganization of the existing documents and changes
no content EXCEPT to correct the location of sphinx doc references.
Expect followup changes to change document names (to reflect the new
structure) and to move content from existing guides (e.g., to move the
pipeline/project/job structure definitions out of the "Project Configuration"
reference guide into their own reference documents for easier locatability).
All documents are now located in either the "overview", "tutorials",
"discussions", or "references" subdirectories to reflect the new structure
presented to the user. Code examples and images are moved to "examples" and
"images" root-level directories.
Developer specific documents are located in the "references/developer"
directory.
Change-Id: I538ffd7409941c53bf42fe64b7acbc146023c1e3
Add some developer docs to the zuul_action plugin source file, and
add a section to the documentation about using the 'file' parameter
with zuul_return.
Change-Id: I5d592d7ae0aa2467926cf2aab939d6ba68465a90
Link to zuul variables, where available, when referenced instead of just
highlighting.
Also correct statement about NODE_FAILURE status that can result from
more than just quota errors.
Change-Id: I13cb682e26e2d0ab72791d7998bc9d03b3615797
Record the number of attempts zuul has made to run this job in the job
inventory. This will help expose reliability information in job logs and
in job log indexers. We want to try and expose job reliability as much
as possible and this is one way to do that.
Change-Id: I2f7c31ce510f59bc569c5db89ce6626d9e3ef436
Zuul recently added zuul.message which needs to be protected against
interpretation by jinja in ansible. This was initially done by marking
it with the !unsafe tag. However this has the disadvantage that the
inventory is no longer parsable by standard yaml parsers without
teaching them the !unsafe tag.
There is a similar simple possibility that doesn't rely on this tag by
base64 encoding the commit message. Ansible has filters for decoding
this so it is still quite easy to deal with base64 encoded vars in
ansible via '{{ zuul.message | b64decode }}'.
Change-Id: I9628e2770dda120b269612e28bb6217036942b8e
In order to make it simple for multiple independent playbooks to return
artifacts, append the values of zuul.artifacts in zuul_return rather than
the usual method of overwriting lists.
Change-Id: I09e6076b4bb354023c5414d149d9bfa59fb3ea4b
This is a partial revert of f12453f6cb.
The use case that change was designed to address is poorly served by
that change. The intent was to make it easier to return multiple
artifacts in multiple playbooks independently by relying on the dictionary
merge behavior of zuul_return. However, in the entirely likely case
of artifacts with generated names, it becomes difficult because Ansible
does not run jinja on dictionary keys.
Therefore, revert to the previous list behavior. A subsequent change
will add a feature to zuul_return to address the underlying issue of
returning multiple artifacts from different playbooks.
Change-Id: I0581aa68fcef320ab27c11ddd6338a15eef38ceb
A recent attempt to use the artifact return feature of zuul_return
exposed some rough edges. These two changes should make it much
easier to use.
First, return artifacts as a dictionary instead of a list. This
requires that they have unique names (which is no bad thing -- what
would two artifacts named "docs" mean anyway?). But mainly it allows
the dict merging behavior of zuul_return to be used, so that one
playbook may use zuul_return with some artifacts, and another playbook
may do the same, without either needing to load in the values of
the other first (assuming, of course, that they use different artifact
names).
Second, add a metadata field. In the database and API, this is JSON
serialized, but in zuul_return and zuul.artifacts, it is exploded into
separate fields. This lets jobs do things like associate versions or
tags with artifacts without having to abuse the url field.
Change-Id: I228687c1bd1c74ebc33b088ffd43f30c7309990d
Adds support for expressing artifact dependencies between jobs
which may run in different projects.
Change-Id: If8cce8750d296d607841800e4bbf688a24c40e08
Having the change message available via the Zuul vars simplifies cases
where a job e.g. needs to update a GitHub/Jira/... ticket.
Those ticket numbers are usually referenced in the commit/PR message.
This avoids having to deal with secrets etc. to get this information
'out-of-band'.
Change-Id: Ib88db7f724dadfb8a4f86e76692f3e1c2c63a258
The plan for the idea of a "promote" pipeline is to fetch
previously uploaded artifacts from the build log server
and move them to the final publication location. However,
jobs which store data (such as documentation builds,
tarballs, or container images) on the log server should not
need to know the configuration of the log server in order
to return the artifact URL to zuul. To support this, if
the job returns a relative URL for an artifact, assume it
is relative to the log URL for the build and combine the
two when storing the artifact info.
Change-Id: I4bce2401c9e59fd469e3b3da2973514c07faecf2
This change adds executor support for generic build resources.
Using the connection_port, a kube config file is generated to enable ansible
tasks to use the resource. Context and pod name is also exported in a
zuul.resources variable to enable job direct access to the build resource.
Currently this change supports 'kubectl' resource for containers that behave
like a machine, and 'project'/'namespace' resource for native containers.
Change-Id: Icdb9e800177dc770c58f65b02456a6b904be0666
Add an artifact table to the SQL reporter, and allow builds to store
artifact URLs using zuul_return. The web API will return the urls.
Change-Id: I8adfc25cc93327ca73c98bbe170e8f39a0864f7f
After a build finishes, if it returned file comments, the executor
will use the repo in the workspace (if it exists) to map the
supplied line numbers to the original lines in the change (in case
an intervening change has altered the files).
A new facility for reporting warning messages is added, and if the
executor is unable to perform the mapping, or the file comment syntax
is incorrect, a warning is reported.
Change-Id: Iad48168d41df034f575b66976744dbe94ec289bc
There are some use cases where it would be very useful to have some
resources that stay during the livetime of the buildset. E.g.:
* Spinning up a docker registry for child jobs that automatically gets
cleaned up after a test run.
* Efficiently caching intermediate build results that are needed in
child jobs without needing and overloading a central system for
this.
In order to provide this functionality a job now can pause itself
after the run phase and let its children run. After all children are
finished it will be resumed. This can be easily triggered in a job by
using the zuul_return module together with the ability to forward any
data via ansible variables to the children.
- name: Pause job
zuul_return:
data:
zuul:
pause: true
registry_ip: "{{ hostvars[groups.all[0]].ansible_host }}"
Change-Id: I1a078bf22b9d3f9a33bd1517e73234749932e1bf
This documents the process for leaving line comments via a reporter
and implements it in the Gerrit reporter.
See https://review.openstack.org/579033 for a method of mapping to
the correct line numbers in an environment with speculative merging.
Change-Id: Ice396eeb51cea9c1c998c0fb6cc09572f945b31a
This adds a "vars:" entry to projects and project-templates to make
available global values for each job. This can be useful to avoid
repeating the same variable definitions for the same job in different
pipelines, or to pass a project-specific value to jobs in a templating
situation.
Change-Id: I9fb468743a21067773979d113e714b5c3e908d02
The return values section in the docs is a single section that covers
multiple topics about returning variables for child jobs and different
special use cases. Restructure this that each special use case gets
its own sub section. This makes it easiert to spot that these special
cases exist.
Change-Id: Iddefd849952e0e8a0646f0f25eaf7edcf5fb912e
Currently, variables using the extra-vars flags always win precedence
over any other variable in ansible. There is also a 2nd use case where
playbooks variables for serial, hosts, etc can only be set using
extra-vars CLI flag.
While this could be achieved by using secrets today, it doesn't feel
like the correct way to use them. Additionally, secrets are
dictionary values in ansible, making them hard to use the filters like
default().
Change-Id: I6d8018661f8d13b7324a012cdbf9614e983e5114
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The "variables" section discusses the 4 sources of variables
(site-wide, secrets, job and parent job) but then the zuul.* variable
reference has ended up in the middle of them.
Extract the the zuul variable reference and move it up a level, after
the discussion of the other variables. Enumerate the list of
variables and add a few links while we're here.
Change-Id: I127c5c3730e950e11695cb6e1b660537b4bc6107
Expose the ability for a user to skip child jobs using zuul_return and
zuul.child_jobs.
Change-Id: I21ea9b2d3b1711f0d55bbe3d626ac7dde5db2919
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
A job may want to know information about the first level of child jobs
that will run after it, to provide this information we now include
zuul.child_jobs (as list) in the inventory file.
Change-Id: I17386d4aa1409b7ac996518981501c2d22899434
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The docs tell us to use the zuul_success variable together with the
'boolean' filter. In fact the filter is named 'bool'. Fix that and add
an example.
Change-Id: Ibb4dc6b10f6f46ec6746bcf3230e6847cb9d0c93
Switch to use https://zuul-ci.org/docs/ for documentation links.
Change-Id: Ie52f025994e982ff2298696f79e8f0718946e7b3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Before merging an item we set the remote ref to point to the current
branch HEAD.
* Instead of deleting the 'origin' remote in the executor we just set
it to a bogus value.
This feature enables jobs to perform quality checks on a per
commit basis (e.g. commit message checks).
For pull requests it was not possible before to determine which
commits are part of the current change.
Change-Id: I5de51917fbdfaa6ffee21dff7b63c520706c59eb
This adds a post-timeout setting which applies as a timeout to each
post-run playbook. This is separate and independent of the normal job
timeout which now only applies to the pre-run and run playbooks in a
cumulative fashion.
The reason for this change is when a pre-run or run playbook hits the
timeout and the job fails you still want to do your best to copy all of
the log data that you can find so that you can debug the timeout.
Similarly to timeout, if post-timeout is not set then post-run playbooks
will have no timeout and can run indefinitely.
Change-Id: I830a6a14d2623f50fbc3f05396cc909d79de04bb
This variable has been transitioned to zuul.projects in the dependent
changes, so is no longer required.
Depends-On: Ib71241a862cf29f6e542dc1d4a2c5cd554d5d7d8
Depends-On: I1408a15d4da274f3fc911f03d5572f4818bc4ebb
Change-Id: I283e66293110aa3bc99acb1cbc6eb3e0cc11f750
In the User's Guide section dealing with Git Repositories, link to
the zuul-base-jobs documentation to solve half of the TODO comment
there.
Change-Id: Ic333278f1f00184f033ab01ab4568e53b2e40fea
The zuul_return task needs to be executed on the executor node,
otherwise it fails with KeyError: 'ZUUL_JOBDIR'
Change-Id: Ib02648f40b1602c85b8c70ee0965f0fde2d227fe
There was a second location where the old "list" type was used in the
docs that was missed in the change to dict of dicts. Fix this minor
documentation detail to say dictionary instead of list.
Change-Id: I4c9f90f201040e3044e818b839fc715e906f049d
It's been suggested this variable would be more convenient to use
if it were simply omitted rather than set to null if the job does
not set an override_checkout value.
Also, remove a stray TODO which is done.
Change-Id: Iededec3007857f4aebabcd027c40e9e0fc101ced
David Shrewsbury