We occasionally see the this task fail for the first element in the
zuul.projects list with a MODULE FAILURE and a return code of -13
(SIGPIPE) [1]. So far we couldn't identify the root cause, so try to
mitigate this issue by retrying on failure. This solution is similar to
the one used for the "Synchronize repos" task[2].
There is a bug report in Ansible that fits
Since it's only the first element in the loop that is failing while
subsequent elements are successful, we currently have two assumptions:
1. As the task before is using a `delegate_to: localhost' [3],
there might be a problem with Ansible when switching the connection
from localhost to the remote host (node).
2. Since the task before is using the same SSH connection [4] that is
used by Ansible to push the git repository, there might be some
"leftovers" on the connection that make the next task fail.
3. There is also a bug report in Ansible [5] which might be causing
that error.
[1]:
{
"ansible_loop_var": "zj_project",
"changed": false,
"failed": true,
"module_stderr": "",
"module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": -13,
"zj_project": {...}
}
[2]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L32)
[3]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L25)
[4]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L16)
[5]: https://github.com/ansible/ansible/issues/81777
Change-Id: I0c4cb87bb076b9b40c9c446dbe5db437daff5897
This change is preparation for https://review.opendev.org/c/zuul/zuul-jobs/+/887917
In the beginning, there was only prepare-workspace[0] which rsynced repos.
Then we added mirror-workspace-git[1] to make it more efficient by using git operation, but it required some openstack-specific code in project-config to work.
Then we added prepare-workspace-git[2] which completed the git-based sync solution by locating everything requried in zuul-jobs. It used mirror-workspace-git by reference and added this TODO:
# TODO(tobiash): we might want to deprecate the role mirror-workspace-git-repos
# and move it here.
This change completes that TODO by moving the mirror-workspace-git-repos code into prepare-workspace-git and places the repo in a sensible and maintainable state with two simple and good options:
* prepare-workspace (rsync)
* prepare-workspace-git (git)
In the unlikely event anyone is still using mirror-workspace-git-repos standalone (OpenStack/OpenDev is not, and that solution was haphazard as described above) they would be well served by a notification that there is a better alternative which is what most of the community actually uses now.
[0] cfffd4431b
[1] 348598e96a
[2] 7cee7156bc
Change-Id: Ib80e0447d49363182fd0d4c4d4e269841bc3aa95
Use zuul_workspace_root as the destination folder for preparing and
mirroring the workspace, which points to ansible_user_dir by default.
(As is currently the case).
The override of this variable can be useful when using a static node
with multiple slots to use a separate workspace for each slot.
Change-Id: If50abc7ccc2dfb6f4ea9f9d5c6a5091ff6967aef
It appears this setting inadvertently had an extra word "ignore"
included in the key name, and later versions of the git module in
Ansible (5) throw an error in that case. Remove it so that this
role will work in all versions.
This has been previously tested in a base-test cycle via
test-mirror-workspace-git-repos.
Change-Id: I597aa8d331e53955e826f09aa51a579e12c94f85
If the checkout failed the mirror-workspace-git-repos must fail the
job. Otherwise this can lead to a broken gate because a different
state than the state that should be tested can be tested silently.
Change-Id: I98e948776ec4b749bc1475ee3a17e37007591118
The mirroring of git workspaces to k8s pod nodes is done by piping a git
push command through `kubectl exec`. Without specifying a k8s context
for kubectl, the `current-context` from the kubeconfig is used. This is
problematic in multi-node scenarios where different nodes reside in
different namespaces and thus, are reachable via separate contexts.
Ansible cannot handle the context switch automatically as it would via
the inventory and its kubectl connection plugin because we are running
the git push from the executor/localhost.
This change makes sure the correct context of the corresponding node is
passed to `kubectl exec` explicitly, thus pushing the git repos to the
correct k8s pod.
Change-Id: Ic0b11b5860bb0062b3a1c068f4f963fb76018715
The `mirror-workspace-git-repos` uses Git to mirror repos from the
executor to build nodes. This comes in handy when the node already
holds a copy of the repo(s), as Git can incrementally update them.
Currently, `mirror-workspace-git-repos` assumes the build node is
directly reachable via ssh. This is (usually) not the case when
running against a Kubernetes Pod build node, thus we can not
efficiently update repos on such nodes using Git.
This change allows pushing Git repos to Kubernetes Pods by bridging
the Git communication over `kubectl` and thereby allowing for the
same behaviour as for standard, non-container, ssh-reachable build
nodes.
(cf. https://git-scm.com/docs/git-remote-ext)
Change-Id: I71919d3bc93e48d5bb24234265fca9c84ce23b87
In case of reused workspaces on static nodes remote
non-bare repos needs to be cleaned up to make sure
that any untracked files from any earlier jobs are removed.
Change-Id: I00dda298f499b241a224c19657ebbb8a9015eb4d
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.
Adjust the new violations accordingly and remove the rule
206 exclusion.
Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
Especially when using Github branch and push workflows repos can
easily have hundreds of branches. The role mirror-workspace-git-repos
which pushes them onto the node prints all of them in every
build. This can get quite messy. Thus we need to optionally be able to
silence the git push operation. This is done on purpose by the --quiet
parameter instead of using no_log because this way errors are still
logged normally and normal operation produces no log output for this
task.
Change-Id: Id77cc00972df9139e4544dd79954d91f4ab6b60b
This expilicitly notes the HEAD of the checked out tree in the logs.
Currently you'll see something like
... HEAD is now at 9dfb46a18 [wip] install ansible as editable during devel jobs
... Already on 'master'
... ok: Item: Runtime: 0:00:00.079626
for each project as this loop happens. However there's no explicit
mention of the project name and you have to reverse engineer what
menioned change relates to what project.
Where this would be useful is when you have an external project
dependency and suddenly your testing stops working due to a change in
it. You can quickly compare the last good run with the failing HEAD
mentioned here to help bisect the problem.
Change-Id: I84d9a8a460e99d3bc558b3a7584d787b22f2ed33
In the case of using a custom SSH port to connect to the ansible host,
the task of pushing the repository failed, as it assumed the SSH port to
be the default 22.
Now the command is aware of the custom SSH port, as it is specified in
the host facts.
Note that we have to escape the hostname if it is an ipv6 address
(because : is a special character in ipv6 addresses). Thankfully ansible
provides a filter to do that for us:
https://docs.ansible.com/ansible/2.5/user_guide/playbooks_filters_ipaddr.html#wrapping-ipv6-addresses-in-brackets
Change-Id: I76969850fc3a57078370fe38efdebac146720797
This reverts commit 34cfe44ca5.
The :port suffix doesn't work if the hostname is an ipv6 ip address as
the hostname needs to be wrapped in []s in that case. Revert this so
that we can fix this in a followup.
Change-Id: I2dfee56d40a4cdd096946a2f67c37117fff74cce
In the case of using a custom SSH port to connect to the ansible host,
the task of pushing the repository failed, as it assumed the SSH port to
be the default 22.
Now the command is aware of the custom SSH port, as it is specified in
the host facts.
Change-Id: I3418016ad5a57e3024a8a553340a7e69b7d1c2fe
We need to set the remote HEAD of the repos after we mirror them. We
have been doing that by looking at git status on the local repos to find
what their head has been set to. However, we have 'checkout' in
zuul.projects now, so we can really just use that.
Change-Id: I251da11b7767793278c3f08dcf64aa008e759669
Depends-On: https://review.openstack.org/540948
Sometimes the task to determine the local HEAD of projects we've
checked out is failing in a way that we can't parse. Add a step before
the one that fails to show the full git status output to help with
debugging.
Change-Id: I095489249593442e1e9a3b346aed1ecf4135825f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Zuul has transitioned zuul.projects to a dict that mirrors
zuul._projects. Convert users back to zuul.projects so we can remove
the underscore version in zuul.
Change-Id: I1408a15d4da274f3fc911f03d5572f4818bc4ebb
Depends-On: I3c011f72933e98ccbf8badf0e9197c8659766c51
This is a revert of commit f28cc33bc3
which was a revert of the original attempt at moving to zuul._projects
(df98773573).
The issue was that the order of projects in "Determine local HEAD" was
not the same as "Update remote repository state" and we ended up
checking out the wrong thing.
To avoid this, build the projects and their current heads on the
exectutor into a separate dict, and as we iterate the projects look up
the correct values in that.
Change-Id: I0d9a66435714efe42fb653aafc6cb71b08ca6b8a
This reverts commit df98773573.
The order in which we iterate over items in the "determine local HEAD"
task is not always the order in which we iterate over mirror-workspace-
git-repos.
Change-Id: I181d6a2cfa2c3104730f20362bbdc10fa582339f
Zuul exports the projects list as a dictionary in _projects in the
depends-on. The idea is that we switch these jobs to that, and do
some testing. When ready, we switch "projects" to be the dictionary
view, and revert "_project" calls here to "project".
Change-Id: Id9a7c137ca5bed25d81087201091157c8401576a
Depends-On: I4476b9d4915d107e29b91229287865bff0ada305
Fix a few problems noticed when using this in project-config:
* Ignore apt-get warning in configure-mirrors
* All tasks should be named in upload-logs, emit-job-header
* use git_config in mirror-workspace-git-repos
* Ignore git warnings in mirror-workspace-git-repos
Change-Id: Ibcdc8d5a42fb081b5faa0cc89460e4cf88d369ba
There are no playbooks that use the roles in this repo in a way that
would lint them, however running linters in project-config found
numerous issues in these roles. Correct the roles.
Change-Id: I017532c8aed8fd90474052423c7ba2510cf84631
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
When we're pushing to a non-bare repo, git has a sad if we push to the
branch that's checked out. But that's ok - we're going to update the
checked out state anyway as a follow up. Tell git it's ok.
Then, after we've pushed, we need to reset after having pushed branch HEAD
to a non bare repo. Clean up after ourselves by removing the config
setting we added to allow the push in the first place.
Change-Id: Id84e9af9d33519e515b8e6d330b375a8e17c14f5
Felix Edel