Since squash merges will not create a merge commit Zuul is creating one
separately. However, this step can fail when there are no changes to
commit. This might for example happen in change-triggered post
pipelines.
To fix this issue we will restore the behavior used prior to
I3379b19d77badbe2a2ec8347ddacc50a2551e505 and allow creating empty
merge commits.
Change-Id: Ic18cb98f12260338799e963b8dc915c8be1d421b
Zuul packs refs directly rather than rely on git to do so. The reason
for this is it greatly speeds up repo resetting. Typically there are two
pieces of information for each packged ref (sha and refname). Git
annotated and signed tags are special because they have the sha of the
tag object proper, the tag refname, and finally the sha of the object
the tag refers to.
Update Zuul's ref packing to handle this extra piece of information for
git tags.
Co-Authored-By: James E. Blair <jim@acmegating.com>
Change-Id: I828ab924a918e3ded2cd64deadf8ad0b4726eb1e
This adds information about the changes associated with a
circular dependency queue item. Currently the bundle_id can be
used to identify which of the items in zuul.items is related to
the current dependency cycle. That variable is deprecated, so
zuul.buildset_refs can be used to replace that functionality.
Since it repeats some of the information at the top level (eg
zuul.change, zuul.project, etc), the code is refactored so they
can share the dictionary construction. That is also used by
zuul.items. This results in a few extra fields in zuul.items
now, such as the change message, but that is relatively
inconsequential, so is not called out in the release notes.
The src_dir is similarly included in all of these places. In
writing this change it was discovered that
zuul.items.project.src_dir always used the golang scheme, but
zuul.project.src_dir used the correct per-job workspace scheme.
This has been corrected so that they both use the per-job scheme
now.
A significant reorganization of the job variable documentation is
included. Previously we had a section with additional variables
for each item type, but since many of these are duplicated at the
top level, in the item list, and now in the refs list, that
structure became difficult to work with. Instead, the
documentation for each of these sections now exhaustively lists
all of the possible variables. This makes for some repitition,
but it also means that a user can see at a glance what variables
are available, and we can deep-link to each one. To address the
variation between different item types, the variables that mutate
based on item type now contain a definition list indicating what
types they are valid for and their respective meanings.
Change-Id: Iab8f99d4c4f40c44d630120c458539060cc725b5
In addition to the safeguard in
Iebf49a9efe193788199197bf7846e336d96edf19 we will only return the final
config for a project-branch as part of the merge result.
Change-Id: I1eb3b75d8762aff4e1ebd057661869df985a79e2
The recent update to make setting refs more efficient could encounter
an edge case if a branch or tag was removed from the upstream repo
after the repo state was retrieved by zuul. If removing the ref caused
the underlying objects to be removed not not be sent in a fetch, then
our blindly setting the repo state with a ref pointed to an unresolvable
object could leave the repository in a corrupted state.
To recover from any potential corruption that may have somehow happened,
this change adds an additional case where we will remove the underlying
repo and re-clone.
To prevent any such corruption from happening in the first place, we add
a check that each hexsha is resolvable before we set it when restoring
the repo state. This does add a small amount of overhead, but should be
much less than manipulating the loos refs one at a time. A copy of nova
with 10,000 refs adds 100ms for this checking.
Change-Id: Ifd298905e634f83a147644d35ff3ea1c143b3d1f
When the executor clones a repo from the cache to the workspace,
it performs a lot of unecessary work:
* It checks out HEAD and prepares a workspace which we will
immediately change.
* It copies all of the branch refs, half of which we will immediately
delete, and in some configurations (exclude_unprotected_branches)
we will immediately delete most of the rest. Deleting refs with
gitpython is much more expensive than creating them.
This change updates the initial clone to do none of those, instead
relying on the repo state restoration to take care of that for us.
Change-Id: Ie8846c48ccd6255953f46640f5559bb41491d425
When using the rebase merge-mode a failed "merge" will leave the repo in
a state that Zuul so far could not recover from. The rebase will create
a `.git/rebase-merge` directory which is not removed when the rebase
fails.
To fix this we will abort the rebase when it fails and also remove any
existing `.git/rebase-merge` and `.git/rebase-apply` directories when
resetting the repository.
DEBUG zuul.Merger: [e: ...] Unable to merge {'branch': 'master', 'buildset_uuid': 'f7be4215f37049b4ba0236892a5d8197', 'connection': 'github', 'merge_mode': 5, 'newrev': None, 'number': 71, 'oldrev': None, 'patchset': 'e81d0b248565db290b30d9a638095947b699c76d', 'project': 'org/project', 'ref': 'refs/pull/71/head'}
Traceback (most recent call last):
File "/opt/zuul/lib/python3.10/site-packages/zuul/merger/merger.py", line 1099, in _mergeChange
commit = repo.rebaseMerge(
File "/opt/zuul/lib/python3.10/site-packages/zuul/merger/merger.py", line 626, in rebaseMerge
repo.git.rebase(*args)
File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 542, in <lambda>
return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 1005, in _call_process
return self.execute(call, **exec_kwargs)
File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 822, in execute
raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git rebase 39fead1852ef01a716a1c6470cee9e4197ff5587
stderr: 'fatal: It seems that there is already a rebase-merge directory, and
I wonder if you are in the middle of another rebase. If that is the
case, please try
git rebase (--continue | --abort | --skip)
If that is not the case, please
rm -fr ".git/rebase-merge"
and run me again. I am stopping in case you still have something
valuable there.
Change-Id: I8518cc5e4b3f7bbfc2c2283a2b946dee504991dd
I8e1b5b26f03cb75727d2b2e3c9310214a3eac447 introduced a regression that
prevented us from re-cloning a repo that no longer exists on the file
system (e.g. deleted by an operator) but where we still have the cached
`Repo` object.
The problem was that we only updated the remote URL of the repo object
after we wrote it to the Git config. Unfortunately, if the repo no
longer existed on the file system we would attempt to re-clone it with a
possibly outdated remote URL.
`test_set_remote_url` is a regression test for the issue described
above.
`test_set_remote_url_invalid` verifies that the original issue is fixes,
where we updated the remote URL attribute of the repo object, but fail
to update the Git config.
Change-Id: I311842ccc7af38664c28177450ea9e80e1371638
When the git command crashes or is aborted due to a timeout we might end
up with a leaked index.lock file in the affected repository.
This has the effect that all subsequent git operations that try to
create the lock will fail. Since Zuul maintains a separate lock for
serializing operations on a repositotry, we can be sure that the lock
file was leaked in a previous operation and can be removed safely.
Unable to checkout 8a87ff7cc0d0c73ac14217b653f9773a7cfce3a7
Traceback (most recent call last):
File "/opt/zuul/lib/python3.10/site-packages/zuul/merger/merger.py", line 1045, in _mergeChange
repo.checkout(ref, zuul_event_id=zuul_event_id)
File "/opt/zuul/lib/python3.10/site-packages/zuul/merger/merger.py", line 561, in checkout
repo.head.reset(working_tree=True)
File "/opt/zuul/lib/python3.10/site-packages/git/refs/head.py", line 82, in reset
self.repo.git.reset(mode, commit, '--', paths, **kwargs)
File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 542, in <lambda>
return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 1005, in _call_process
return self.execute(call, **exec_kwargs)
File "/opt/zuul/lib/python3.10/site-packages/git/cmd.py", line 822, in execute
raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git reset --hard HEAD --
stderr: 'fatal: Unable to create '/var/lib/zuul/merger-git/github/foo/foo%2Fbar/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.'
Change-Id: I97334383df476809c39e0d03b1af50cb59ee0cc7
This is two changes in one since they both fix breakage due to
external dep changes:
Git no longer allows file urls in submodules by default, but one
of our unit tests relied on that behavior (in order to verify that
we can clone repos with submodules). Run the submodule add command
in the test with a flag which allows file urls.
Pin PyJWT <2.6.0
2.6.0 causes the following error:
2022-10-20 15:58:04,800 cherrypy.error.140001002710128 ERROR [req: 86efa7775e88473a9e9d5e54f0c83050] [20/Oct/2022:15:58:04] HTTP
Traceback (most recent call last):
File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 638, in respond
self._do_respond(path_info)
File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 702, in _do_respond
response.finalize()
File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 901, in finalize
content = self.collapse_body()
File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 859, in collapse_body
new_body = b''.join(self.body)
File "/home/corvus/git/zuul/zuul/.tox/py310/lib/python3.10/site-packages/cherrypy/_json.py", line 24, in encode
for chunk in _encode(value):
File "/usr/lib/python3.10/json/encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/lib/python3.10/json/encoder.py", line 438, in _iterencode
o = _default(o)
File "/usr/lib/python3.10/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ImmatureSignatureError is not JSON serializable
Change-Id: I2e345f24ea0a62ce8d9dbe1c066438f194e7075c
The fix including 2 parts:
1. For Gtihub, we use the base_sha instead of target branch to
be passed as "tosha" parameter to get precise changed files
2. In method getFilesChanges(), use diff() result to filter out
those files that changed and reverted between commits.
The reason we do not direcly use diff() is that for those
drivers other than github, the "base_sha" is not available yet,
using diff() may include unexpected files when target branch
has diverged from the feature branch.
This solution works for 99.9% of the caseses, it may still get
incorrect list of changed files in following corner case:
1. In non-github connection, whose base_sha is not implented, and
2. Files changed and reverted between commits in the change, and
3. The same file has also diverged in target branch.
The above corner case can be fixed by making base_sha available in
other drivers.
Change-Id: Ifae7018a8078c16f2caf759ae675648d8b33c538
It can happen that the remote ref (corresponding to the branch in
cache) is not available when local workspace is cloned.
Fix this issue by creating the remote ref when it does not exist.
Change-Id: I68244e0b5aa3c8b6e15693ffc2897d4f416e0d5c
The reverted change can lead to the listing of files that are not
changed in the referenced commit(s). This can e.g. happen if the base
branch (e.g. master) has diverged from the feature branch.
This is now also tested to avoid regressions in the future. The issue
related to files that are added/removed in the same range of commits
(e.g. a PR) needs to be addressed in a separate change.
This reverts commit e63d7b0cdb.
Change-Id: I07bc4a09bf162fdbc4c2daeecb19e12d81241801
The original implementation takes into account the changed fils from
all commits of a PR.
It causes a bug when files get changed and reverted in those commits.
e.g. A file is added in first commit then removed in second commit,
this file should should not be considered as a changed file in the PR.
Change-Id: I7db8b9d3f3267073c5e1a71f52e75939ffa91773
The scheduler depends on merge completed events in order to advance
the lifecycle of a queue item. Without them, items can be stuck in
the queue indefinitely.
In the case of certain merge errors, we may not have submitted a
result to the event queue. This change corrects that.
Change-Id: I9527c79868ede31f1fa68faf93ff113ac786462b
At the end of this test, we check the merger api cache to see if the
merge requset is cleaned up. We may need to give the cache time to
update, so use iterate_timeout.
We should also be consistent about which merger api object we use;
to that end, the test is updated to only use the api object owned by
the merger client when checking the cache contents (since that's the
one that we use to run the cleanup method).
Finally, it appears this was subject to the same issue that was fixed
in 4f96125007 where we were not handling
deleted znodes correctly.
Change-Id: Ic39b01b2d2eb138cb79c7f5b412c6390c07112bf
The Merger and executor APIs have a lot in common, but they behave
slightly differently. A merger needs to sometimes return results.
An executor needs to have separate queues for zones and be able to
pause or cancel jobs.
This refactors them both into a common class which can handle job
state changes (like pause/cancel) and return results if requested.
The MergerApi can subclass this fairly trivially.
The ExecutorApi adds an intermediate layer which uses a
DefaultKeyDict to maintain a distinct queue for every zone and then
transparently dispatches method calls to the queue object for
that zone.
The ZK paths for both are significantly altered in this change.
Change-Id: I3adedcc4ea293e43070ba6ef0fe29e7889a0b502
This is the second part of I767c0b4c5473b2948487c3ae5bbc612c25a2a24a.
It uses the MergerAPI.
Note: since we no longer have a central gearman server where we can
record all of the merge jobs, some tests now consult the merger api
to get the list of merge jobs which were submitted by that scheduler.
This should generally be equivalent, but something to keep in mind
as we add multiple schedulers.
Change-Id: I1c694bcdc967283f1b1a4821df7700d93240690a
This adds the concept of a 'scheme' to the merger. Up to this point,
the merger has used the 'golang' scheme in all cases. However it is
possible with Gerrit to create a set of git repositories which collide
with each other using that scheme:
root/example.com/component
root/example.com/component/subcomponent
The users which brought this to our attention intend to use their repos
in a flat layout, like:
root/component
root/subcomponent
To resolve this we need to do two things: avoid collisions in all cases
in the internal git repo caches of the mergers and executors, and give
users options to resolve collisions in workspace checkouts.
In this change, mergers are updated to support three schemes:
* golang (the current behavior)
* flat (new behavior described above)
* unique
The unique scheme is not intended to be user-visible. It produces a
truly unique and non-conflicting name by using urllib.quote_plus. It
sacrifices legibility in order to obtain uniqueness.
The mergers and executors are updated to use the unique scheme in their
internal repo caches.
A new job attribute, 'workspace-scheme' is added to allow the user to
select between 'golang' and 'flat' when Zuul prepares the repos for
checkout.
There is one more kind of repo that Zuul prepares: the playbook repo.
Each project that supplies a playbook to a job gets a copy of its repo
checked out into a dedicated directory (with no sibling repos). In that
case there is no risk of collision, and so we retain the current behavior
of using the golang scheme for these checkouts. This allows the playbook
paths to continue to be self-explanatory. For example:
trusted/project_0/example.com/org/project/playbooks/run.yaml
Documentation and a release note are added as well.
Change-Id: I3fa1fd3c04626bfb7159aefce0f4dcb10bbaf5d9
Several changes in an attempt to clarify exactly when updates and
resets should and do happen:
* Remove the repo_state argument from Merger.getRepo()
It was unclear under what circumstances the low-level repo object
honored repo_state (not much). Remove it entirely and rely on
high-level Merger methods to deal with repo_state.
* Have merger.setRepoState() operate on one project instead of a
list of items
Part of the reason we were passing repo_state to low-level
methods was to reset the state for required projects in the
executor. Essentially there were three cases: projects of change
items, projects of non-change items, and projects of neither but
in required-projects. The low-level repo_state usage only
handled the last, the first is easy, and the second we handled by
creating a list of non-change items and passing it to
setRepoState on the merger.
A simpler method of handling all of that is to reduce it to two
cases: projects of change items (which need to be merged) and the
rest (which need to be restored). If we do that, we can maintain
a set of projects we've seen while merging in the first case,
then iterate over all the remaining projects and call
setRepoState on each in the second.
* Remove the update call from Repo.reset()
This lets us call Repo.reset() frequently (i.e., at the start of
any operation that writes to the merger's git repo working dir)
without performing a git fetch. We need to make sure we call
Repo.update() where necessary.
* Remove the reset call from Merger.updateRepo()
This will now only call repo.update(), and even that will only
happen if the repo_state says we should. So we can safely call
this before any significant operations and know that it will
update the repo if necessary.
* Add an update() call to getRepoState()
Because we removed the update() call from Repo.reset(), we need
to add one here next to the existing call to reset().
* Add a reset call to getFiles()
It relied on the reset in updateRepo.
* Set execution_context to False on the executor's main merger
The execution_context parameter determines whether we manipulate the
origin remotes to point at the previous commit. This should be set
for mergers that operate on the build work dir, but it should not
be set for the main merger within the executor (so the main merger
behaves just like a standalone merger). It previous was erroneously
set for the executor's main merger and this change corrects that.
* Add Merger.updateRepo() calls in the merger server merge method
The merger needs to update and reset each repo before merging changes.
Currently _mergeItem resets the repo the first time it encounters it.
But we still need to update the repo. We don't want to update within
the merger method because the executor performs batch updates in
parallel before starting a merge and we don't want to re-do that work.
So instead we add it to the merger server invocation, so it's only
used in the merger:merge gearman function code path.
Change-Id: I740e958357dc7bf0a6506474c5991da12ab6264e
If we pass a repo state to checkoutBranch, we should check out the
repo/branch as specified by the repo state.
FYI, there are two theoretical ways we could call checkoutBranch:
1) Without a repo state (in which case we would expect to checkout
the current value of the branch in the repo; presumable this would
be done after an update() call also without a repo state, which means
the current branch would match upstream).
2) With a repo state, in which case we would expect to check out
the branch as specified by the repo state.
Currently Zuul only uses option 2.
Change-Id: Icad68a337b3ff5fc80af32ee0a4845cc83daa14b
The isUpdateNeeded call was not operating on the actual dictionary
format that is passed in. The tests did not catch this because they
pass in the format that is expected. Update both the tests and the
calling code in the merger to fix.
This breaks the just-added fast-forward test, which shows us that
the current behavior really is broken.
Change-Id: I34b7dbe1d4f7032d217bca30ca9a8d3c986c1915
I modified this test slightly, but it's generally what Clark wrote
in https://review.opendev.org/784142
That test fails against master, but this does not:
I don't think the Zuul ref check at the end is necessary, and I
added a checkoutBranch call, since what we really want to do
is make sure that we get the correct thing checked out.
With those changes, along with a corrected form of the repo state
dict, this passes on master and can be used as a regression test
for the global repo state work.
Note, however, that this only passes due to the mismatch of the
repo state dict format. That will be corrected in a followup
change.
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
Change-Id: I7f01910f189984f6bd80c09a2af2ec3536d102fe
A badly configured .gitmodules committed to a repo can currently break
zuul by not allowing it to git fetch [1]. There is already logic handling
merge conflicts in .gitmodules by resetting the repo but this is not
enough if the current commit is faulty. Instead fix this by trying to
reset to a commit before .gitmodules was introduced.
[1] Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/zuul/executor/server.py", line 2935, in _innerUpdateLoop
self.merger.updateRepo(
File "/usr/local/lib/python3.8/site-packages/zuul/merger/merger.py", line 805, in updateRepo
repo.reset(zuul_event_id=zuul_event_id, build=build,
File "/usr/local/lib/python3.8/site-packages/zuul/merger/merger.py", line 397, in reset
self.update(zuul_event_id=zuul_event_id, build=build)
File "/usr/local/lib/python3.8/site-packages/zuul/merger/merger.py", line 601, in update
self._git_fetch(repo, 'origin', zuul_event_id, tags=True, prune=True)
File "/usr/local/lib/python3.8/site-packages/zuul/merger/merger.py", line 265, in _git_fetch
repo.git.fetch(remote, ref_to_fetch,
File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 542, in <lambda>
return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 1006, in _call_process
return self.execute(call, **exec_kwargs)
File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 823, in execute
raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git fetch -f --prune --tags origin
stderr: 'fatal: bad config line 9 in file /var/lib/zuul/executor-git/.../.gitmodules'
Change-Id: I33f8f5905167ebb95ab58f9ef192359573495927
Gerrit's refs are left-padded with zeroes if the change's number is
below 10, for example 9,1 -> refs/changes/09/9/1.
Fix an error in computing the change's ref when the change's number is
below 10.
Modify the test framework to emulate Gerrit ref naming convention more
faithfully in tests.
Change-Id: I54a3c3dcaa9a08cff97bfd701e28b6f240fdb77d
In case a branch name contained the '@' character this was interpreted
by Gitpython according to the format supported by git-rev-parse when
resolving the ref name.
Since we already have the commit for the ref we can use it directly,
that way avoiding any issues caused by parsing the ref name.
Change-Id: I49665c62389245f937317e70f093d33a4bf759d3
GitPython is trying to be nice and thinks that we want the head '<name>' when trying
to create the head 'refs/heads/<name>'.
Avoid this by always prepending 'refs/heads/'.
Change-Id: I90768fd678b02a7fbfc0675456dc4105b51d7f06
We've noticed that if zuul executors (and presumably mergers) don't shut
down gracefully that they may leak git index.lock files in the .git dirs
of the merger repos. Since these repos should be dedicated to zuul's use
without outside interference we can reasonably safely remove any present
index.lock files when starting zuul mergers (and executors).
This implementation does an os.walk under the merger repos root looking
for .git dirs and once it has found them checks for any index.lock
files. This happens before starting the gearman worker which should
avoid any races with these resources.
Change-Id: Ie043453bcdf4500a3718da6f705c882431acafdf
Since executors can also handle merge jobs and those merges happen in
the executor's repo cache we need to protect temporary merger refs from
being garbage collected.
Because the executor's update jobs might reset the local branch heads in
between merges, we create the refs for the speculative branch state in
'refs/zuul' instead. Those refs are cleaned up when the related branch
no longer exists.
Branch names for the Zuul refs are hashed (SHA1) in order to avoid
issues with empty directories when the branch name contains slashes.
E.g. the speculative state of the master branch will be referenced by
'refs/zuul/4f26aeafdb2367620a393c973eddbe8f8b846ebd'
Change-Id: Idd2b0bd2dfeba22f3961f851f8a463bc5c9d37ff
This tests that a renamed branch can be reset by our merger code.
This can (and probably should be?) expanded to cover functional testing
of the code review event -> merger merge -> job run pipeline.
Change-Id: I86aba36b560896d1c1971c40031390424881fbd7
When merging changes on different branches it might happen that objects
are garbage collected in-between merges. To avoid this, we will always
update the local head to match the (temporary) speculative state for a
branch.
Change-Id: I1866d44e3b40efcea8865c2367b0a06edd21ed84
The reason for occasionally having invalid FETCH_HEADS has been
tracked to git's behavior of ignoring FETCH_HEAD when checking for
unrechable objects [1]. In order to solve this issue change the fetch
command to map the fetched ref to refs/zuul/fetch. This ref is unused
but will ensure that git won't prune the referenced object.
This reverts commit 9b300bc8df.
[1] https://public-inbox.org/git/20160708025948.GA3226@x/
Change-Id: Iea8d376b840a6486a9fa51b6654641d3b14f90f3
Zuul was not cleaning/pruning stale refs in the correct order which can
be problematic e.g. with branche names containing slashes.
The issue can be reproduced by creating a branch "foobar" on the remote
that is then also cloned to a local repository. Now, when the remote
branch "foobar" is deleted and another branch "foobar/sub" is created, a
fetch without the prune option will lead to the following exception:
2020-01-08 09:57:00,934 zuul.Repo ERROR Retry 1: Fetch /tmp/tmp4g49pxfl/zuul-test/workspace origin None
Traceback (most recent call last):
File "/tmp/zuul/zuul/merger/merger.py", line 225, in _git_fetch
**kwargs)
File "/tmp/zuul/.tox/py36/lib/python3.6/site-packages/git/cmd.py", line 545, in <lambda>
return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
File "/tmp/zuul/.tox/py36/lib/python3.6/site-packages/git/cmd.py", line 1014, in _call_process
return self.execute(call, **exec_kwargs)
File "/tmp/zuul/.tox/py36/lib/python3.6/site-packages/git/cmd.py", line 825, in execute
raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(1)
cmdline: git fetch --tags origin
stderr: 'error: cannot lock ref 'refs/remotes/origin/foobar/sub': 'refs/remotes/origin/foobar' exists; cannot create 'refs/remotes/origin/foobar/sub'
From /tmp/tmp4g49pxfl/zuul-test/upstream/org/project1
! [new branch] foobar/sub -> origin/foobar/sub (unable to update local ref)'
To fix this problem the Repo.update() method will now pass prune=True
when fetching from the remote end.
However, this was not enough to correctly reset the repo. Zuul was still
trying to update local heads to match the remote before cleaning up
stale refs.
The fix for this is to reset the working directory to the default remote
branch that 'origin/HEAD' is referencing. This is necessary since the
locally checked out branch might be stale and must be deleted.
Afterwards all stale local heads that no longer exist on the remote end
can be cleaned up. The last step is to update all local heads to match
the remote.
Further older git releases leaked empty directories when deleting
referenced. E.g. creating and deleting refs/heads/foo/bar leaked the
directory refs/heads/foo which blocks creation of the branch foo in
the future. This has been fixed in git 2.13.0 [1]. In order to account
for that cleanup empty directories when dealing with git versions
older than 2.13.
[1] Fix for directory leak in git:
446397774a
Change-Id: Ic379848f7f14643a33c43248deb1850ce89cb34c
TestMergerRepo.test_repo_repr attempts to clone from a remote named
"remote". Since that is most probably is never reacheable, it would
times out after 60 seconds, only to then assert against the object
representation.
The test has little value by itself, the Repo.__repr__ method is
explicit. In the future, git blame can be used to identify why
local_path is added to the repr (id is not enough to disambiguate).
Change-Id: I96b4245f1c1ce441240d24231b3d3c07017f3ef0
If a new branch is created off of an existing commit then the rev
(commit) for the new branch is likely already in the repos on the
executors. Unfortunately we were checking if a repo needs to be updated
based only on the presence of the commit and not whether or not the ref
(branch) exists.
This means in some cases zuul would fall back to the default branch for
the repo instead of testing changes proposed to new branches because it
thought that the local executor git repo did not need to be updated. Not
updating the local git repo results in the branch not being created
which fails branch validity checks and we fall back to the default.
You can see that happening in this log snippet:
2019-08-26 16:35:40,747 DEBUG zuul.ExecutorServer: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] Got executor:execute job: b405f247cf3a441b924bce4c35b2f635
2019-08-26 16:35:41,255 INFO zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Beginning job openstack-tox-py36 for ref refs/changes/58/677258/5 (change https://review.opendev.org/677258)
2019-08-26 16:35:41,255 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Job root: /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635
2019-08-26 16:35:41,256 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Updating project {'connection': 'gerrit', 'canonical_name': 'opendev.org/x/ranger-agent', 'override_branch': None, 'default_branch': 'master', 'name': 'x/ranger-agent', 'override_checkout': None}
2019-08-26 16:35:41,258 INFO zuul.ExecutorServer: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Updating repo gerrit/x/ranger-agent
2019-08-26 16:35:41,406 INFO zuul.Merger: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Skipping updating local repository gerrit/x/ranger-agent
2019-08-26 16:35:41,419 DEBUG zuul.ExecutorServer: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Finished updating repo gerrit/x/ranger-agent
2019-08-26 16:35:41,998 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Git updates complete
2019-08-26 16:35:42,960 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Cloning gerrit/x/ranger-agent
2019-08-26 16:35:43,032 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Cloning from /var/lib/zuul/executor-git/opendev.org/x/ranger-agent to /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,202 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Merging for change 677258,5
2019-08-26 16:35:43,202 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Processing ref refs/changes/58/677258/5 for project gerrit/x/ranger-agent / python3 uuid 2ebedc56e15d47118f60e2e248b19d62
2019-08-26 16:35:43,203 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] No base commit found for ('gerrit', 'x/ranger-agent', 'python3')
2019-08-26 16:35:43,203 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Resetting repository /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,207 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Updating repository /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,570 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Reset to kilo
2019-08-26 16:35:43,573 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Restore repo state for project gerrit/x/ranger-agent
2019-08-26 16:35:43,594 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/python3 at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,617 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/master at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,619 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/ocata at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,620 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/newton at 9e01ee002f36ecae2a21ded4e9e14de00c78de6c in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,621 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/kilo at 6ef65d8d15b730129ff86c83d6103d97f65193e9 in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,624 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Delete reference refs/remotes/origin/HEAD
2019-08-26 16:35:43,697 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Updating remote reference origin/python3 to e796eb53956f1e83ddc53a8dce9cf3957b3b59ef
2019-08-26 16:35:43,702 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Checking out e796eb53956f1e83ddc53a8dce9cf3957b3b59ef
2019-08-26 16:35:43,974 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Merging refs/changes/58/677258/5 with args ['-s', 'resolve', 'FETCH_HEAD']
2019-08-26 16:35:44,099 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/master at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,258 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/ocata at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,290 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/newton at 9e01ee002f36ecae2a21ded4e9e14de00c78de6c in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,323 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/python3 at e5ef4f738e6d916103c178aedb65e6aa7d2e9dfd in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,348 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/kilo at 6ef65d8d15b730129ff86c83d6103d97f65193e9 in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,511 INFO zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Checking out opendev.org/x/ranger-agent project default branch master
2019-08-26 16:35:44,515 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Checking out master
Change-Id: I1858a2ce54312b16524db915d093ccef2b0f892e
Currently we update each cached repo that is involved in a job during
the start phase of the job. After updating we clone each of these
repos into the work dir. However when running a job we already got a
repo state containing all references we need. So we can check if the
cached repo already has everything we need. If the repo state doesn't
contain revisions we don't have yet we can skip updating the repo and
thus safe a fetch.
This can improve job startup times significally especially for
expensive jobs with many required projects.
Change-Id: I9364e438d581b068fa19c9dfc24adab60479c385
If we have an event we should submit its id also to the merger so
we're able to trace merge operations via an event id.
Change-Id: I12b3ab0dcb3ec1d146803006e0ef644e485a7afe
When a VM hosting an executor or merger crashes it can leave created
but unwritten object files behind. Those render the cached git repo
useless and the only way to recover currently is to manually delete
the repo. Otherwise all jobs using that repo will end up with the
MERGER_FAILURE result [1]. We can easily catch this error and delete
the cached repo in this case to force a clean clone when retrying.
[1] Trace:
2019-04-26 11:01:29,699 ERROR zuul.Merger: Unable to update org/project
Traceback (most recent call last):
File "/opt/zuul/lib/python3.7/site-packages/zuul/merger/merger.py", line 598, in updateRepo
repo.reset()
File "/opt/zuul/lib/python3.7/site-packages/zuul/merger/merger.py", line 255, in reset
self.update()
File "/opt/zuul/lib/python3.7/site-packages/zuul/merger/merger.py", line 432, in update
self._git_fetch(repo, 'origin', tags=True)
File "/opt/zuul/lib/python3.7/site-packages/zuul/merger/merger.py", line 216, in _git_fetch
**kwargs)
File "/opt/zuul/lib/python3.7/site-packages/git/cmd.py", line 548, in <lambda>
return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
File "/opt/zuul/lib/python3.7/site-packages/git/cmd.py", line 1014, in _call_process
return self.execute(call, **exec_kwargs)
File "/opt/zuul/lib/python3.7/site-packages/git/cmd.py", line 825, in execute
raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git fetch --tags origin
stderr: 'error: object file .git/objects/b8/b6c200917c5c0c9e7d329d9278645153f2740e is empty
error: object file .git/objects/b8/b6c200917c5c0c9e7d329d9278645153f2740e is empty
fatal: loose object b8b6c200917c5c0c9e7d329d9278645153f2740e (stored in .git/objects/b8/b6c200917c5c0c9e7d329d9278645153f2740e) is corrupt
fatal: The remote end hung up unexpectedly'
Change-Id: I499a89a154c03f3efcb364434d27917cd43739b0
In case of an execption we want to have the local path of the repo for
debugging. The object id alone doesn't provide enough information.
Change-Id: I6675f4f3a65ba975456687de91827694273862e1
When processing changes with dependencies in pipelines using the
independent pipeline manager we enqueue non-live items that represent
the dependent changes but don't run tests. Currently we perform a
merge operation also with every non-live item. Especially when
rebasing large stacks in Gerrit this can lead to a huge overhead of
merges. In this case we perform 1 + 2 + ... + n merges where n is the
size of the stack in Gerrit. E.g. a rebase of a stack of 20 changes we
perform 210 merge operations.
Those merges are needed to get the changed config of every item ahead
because we need the resulting config changes to produce a correct
layout. Fortunately the merger already returns a list of file changes
per item. We can leverage this to populate the changed files of all
non-live changes ahead when we receive the merge result of the live
change. This makes it possible to skip the merges of the non-live
changes ahead and have one merge operation per change.
In the rebase example above this optimization reduces the number of
performed merges from 210 to 20.
Change-Id: I82848367bd6f191ec5ae5822a1f438070cde14e1
Using git merge-bases will not work properly for some edge-cases.
Using git rev-list will return the set of commits that are reachable
from the PR head but without those already in the base branch.
Change-Id: I25c75a70892bbac52db783fb63513f6aa37cb5f6
Simon Westphahl