In case of aws spot instances, the node can drop the connection anytime.
Ansible loops do not fail when the first item fails, instead they execute all items before reporting the failure.
This leads to a big overhead, if we sync repositories, when the node has dropped the connection beforehand.
Therefore i used include_tasks, which fails for the first unsuccessful item.
Change-Id: Id6079a2cda10a825384d52d47750d9c05d323e00
The prepare_workspace_sync_required_projects_only variable allows
users to define which projects to sync to the node. This can prevent
syncing of unnecessary repositories. For some builds e.g. the
depends-on repositories dont need to be synced. The projects are
filtered based on the 'required' flag present in each zuul.project
entry and the required projects list also does not contain projects
which are present due to Depends-On or gate queue sequencing.
Having unnecessary repos in the workspace can for example also break
the analysis phase of bazel.
Change-Id: I3cc36cbfc60c81956caf5137da63973aeade4e21
Co-Authored-By: James E. Blair <jim@acmegating.com>
Co-Authored-By: Bernhard Berg <bernhard.berg@bearingpoint.com>
This change is preparation for https://review.opendev.org/c/zuul/zuul-jobs/+/887917
In the beginning, there was only prepare-workspace[0] which rsynced repos.
Then we added mirror-workspace-git[1] to make it more efficient by using git operation, but it required some openstack-specific code in project-config to work.
Then we added prepare-workspace-git[2] which completed the git-based sync solution by locating everything requried in zuul-jobs. It used mirror-workspace-git by reference and added this TODO:
# TODO(tobiash): we might want to deprecate the role mirror-workspace-git-repos
# and move it here.
This change completes that TODO by moving the mirror-workspace-git-repos code into prepare-workspace-git and places the repo in a sensible and maintainable state with two simple and good options:
* prepare-workspace (rsync)
* prepare-workspace-git (git)
In the unlikely event anyone is still using mirror-workspace-git-repos standalone (OpenStack/OpenDev is not, and that solution was haphazard as described above) they would be well served by a notification that there is a better alternative which is what most of the community actually uses now.
[0] cfffd4431b
[1] 348598e96a
[2] 7cee7156bc
Change-Id: Ib80e0447d49363182fd0d4c4d4e269841bc3aa95
Use zuul_workspace_root as the destination folder for preparing and
mirroring the workspace, which points to ansible_user_dir by default.
(As is currently the case).
The override of this variable can be useful when using a static node
with multiple slots to use a separate workspace for each slot.
Change-Id: If50abc7ccc2dfb6f4ea9f9d5c6a5091ff6967aef
Our prepare-workspace-git role was trying to express a few related steps
as ansible tasks. The problem is that each one of these steps had to
iterate through the entire project list incurring the ansible startup
time for each task of each loop iteration. We can speed things up a bit
if we use a single loop and a slightly more complicated shell task that
checks if things like directories exist before cloning/initing git
repos.
Note this is still slower than if we removed the Ansible task loop
entirely and move this logic into an ansible module that can run a loop
internally. However, making a change like that is significantly more
involved as we'd likely end up exec'ing git with python and need to
manage logging, errors, and so on. The shell module already handles that
for us in a nice way.
Change-Id: Ic87eb182cc4ca4bd0acdd1aa46c2d72dc1165e90
This is breaking jobs in zuul.ansible.com, as there is an issue with
order with DNS. In our base jobs, we first configure ensure git repos
are installed on the node, then configure the node for infra things
like dns, package mirrors, etc. In our case, dns hasn't been fully
configured, which makes the package install fail.
I would propose we revert this change and re-release diskimage-builder. Or
have git considered an image based dependency.
A workaround for now has been to revert DIB, to rebuild images adding
back git. However, this was exposed due to us missing one image.
This reverts commit 5d09503c88.
Change-Id: I533cbec83f7525175b9a94d70a61ba7424154aa6
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.
Adjust the new violations accordingly and remove the rule
206 exclusion.
Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
Currently the incremental push of the git repos to the nodes is split
into a part in project-config and a part in zuul-jobs and only works
within the OpenStack context. The prepare-workspace-git role is an
attempt to make this generically available in zuul-jobs.
It first checks for each needed project if it's cached. If yes it's
cloned to the destination, if not it's just initialized as an empty
git repo. After that we can generically use git operations to sync the
correct state of the repos to the destination.
Change-Id: I87938a02d51e561b25ca2cb20f53d62f3cd3ae73
Lukas Kranz
yatinkarel