This was intended as a one-time helper program to help people upgrade
from Zuul v2 to v3. It did not cover all use cases, and has not been
kept up to date or improved. It's time to remove it before the v4
release.
Change-Id: I12cdcedb5baabd8fa0937a6ea21590259093ead1
The commands are managed as entry-points so remove
ununecessary shebangs. Also lib/re2util.py does not
require a shebang as well.
zuul_return.py does not have a main and is not supposed
to be run directly.
Ununecessary shebangs for non executable script causes
rpmlint issues.
Change-Id: I6015daaa0fe35b6935fcbffca1907c01c9a26134
flake 3.6.0 introduces a couple of new tests, handle them in the zuul
base:
* Disable "W504 line break after binary operator", this is a new warning
with different coding style.
* Fix "F841 local variable 'e' is assigned to but never used"
* Fix "W605 invalid escape sequence" - use raw strings for regexes.
* Fix "F901 'raise NotImplemented' should be 'raise
NotImplementedError'"
* Ignore "E252 missing whitespace around parameter equals" since it
reports on parameters like:
def makeNewJobs(self, old_job, parent: Job=None):
Change "flake8: noqa" to "noqa" since "flake8: noqa" is a file level
noqa and gets ignored with flake 3.6.0 if it's not at beginning of line
- this results in many warnings for files ./zuul/driver/bubblewrap/__init__.py and
./zuul/cmd/migrate.py. Fix any issues there.
Change-Id: Ia79bbc8ac0cd8e4819f61bda0091f4398464c5dc
zuul/cmd/migrate.py contained two variables that are assigned but never
used, flake 3.6.0 found these. They were hidden by the earlier "flake8:
noqa" in the file that earlier flake8 versions treated as file level
ignore wherever it appeared.
Remove the two variables found.
Change-Id: I7dca305da5bf605386590d96bec0150e06b8ff6d
Zuul now requires playbooks to have .yaml extensions in the job
stanza.
Change-Id: I71af1cc421b2a0c3560f133244663b7fe3c7c481
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
In the case of RDO, there is logic in JJB we don't actually want to
convert with zuul-migrate. We could modify JJB and remove this logic,
however it is also just as easy to add the ability to a user to choose
which macros to skip for migration.
Change-Id: I0b169fa66be2d7030c271caaf223a2cbc1ca5db2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
It is possible for a project not to have a '/' in its name, add
support for it.
Change-Id: Ic773da4ec552a55c8b647fb38e54e300368d8be1
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
It is possible for layout.yaml not to use project-templates sections,
in this case allow zuul-migrate to skip converting them.
Change-Id: I00e7fe252d951281d96fd10022289a912948a520
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Jobs that need this expect it as an environment variable.
Update the migration script in case it gets used again.
Change-Id: Ic4d372a795e17a33116d19e9e4603fb5a08fe152
In order to find project templates that need to be expanded from job
matchers that are not project specific, we need to make a pass through
all of the projects assuming all templates need to be expanded. We then
look at the result of the expansion to see if anything was actually
done. As part of this, we also collect same-expansions on a job basis to
track if a given job always has the same matcher expansion. If it does,
we can apply that to the job definition and not to the project-pipeline
defintion, which could lower the number of templates that need to be
expanded.
This may be the ugliest code I've ever written. I'm sorry.
Also fix a bash bug in the run-migration script that caused final to
always get run regardless of flag setting. Whoops.
Change-Id: I523909e5242e0db125b7560cbdcd9ac41ca6c72f
Turns out 99 isn't a good prefix.
Also, remove the move argument now that we're merging with the existing
projects.yaml.
Also, stop running zuul unittests on migration script changes. They are
not relevant.
Change-Id: I10ed8cae64c82ed5afd01bb03a74ffc4fd2d87ee
It's important to have legacy files for legacy jobs, project-templates
and a legacy dir for legacy playbooks. HOWEVER - the project pipeline
configs are the project pipeline configs. Rather than emitting
99legacy-projects -if there is already a zuul.d/projects.yaml, assume
that it contains a list of projects, read it in, and then merge the
generated project-pipeline definitions with the existing ones. This will
give us a unified project-pipeline definition list. (It loses the
comments in the original file, but there were only a couple that can be
added back by hand.
Change-Id: Ia2f440df47aac0a5b1e6baae3c081bf69d7add03
We prefer output in files like this to be sorted - so sort these.
Also, rename the files from 99converted- to 99legacy- and split jobs
into jobs and project-templates to match the other split.
Change-Id: I126fe240b64987684e433cbf8f9f72a729e416f1
Putting them directly into shell: breaks here docs. We can also use
the same dict for setting executable.
Change-Id: I050a17085f342e5765215bcde140649a9c740773
If a job hits the logic as a string, it gets overwritten, but since it's
just a string element of a list it's not a reference member so it does
not update the original list.
Instead, make a new list that we return and then overwrite the old list.
This gets the exclusions added properly.
Also, throw in a set conversion in the middle so that we don't
accumulate multiple copies of the same file exclusions. One copy of each
one is just fine.
Change-Id: Ia8a0d32f3221b252fa093662e6fcd860ba0cabf7
These run zuul-cloner to clone puppet-openstack-integration and then
run a script in puppet-openstack-integration to clone the base repos.
We can't detect that normally - but we can map them to a base job
similar to the dsvm jobs.
Change-Id: I0d994e5c2a02f826e9cafd7df3870e2912e4db12
In v2, PROJECTS was fed to zuul-cloner and it would then try to clone
things. That's not how v3 works. In v3, we need required-projects stated
in the job definition so that Zuul can properly clone things.
Extract PROJECTS and DEVSTACK_PROJECT_FROM_GIT from job definitions to
try to attach appropriate required-projects lists.
dsvm jobs still have a larger list this doesn't account for though.
This also removes support for specifying a required-projects list in the
mapping, since we weren't actually using it and the semantics of which
thing should win were confusing.
Change-Id: I01156b38a1e907677f69042a003eb580ba3d224c
Instead of publishing to docs-draft, we're just going to pull the
results into the log dir and set the success-url.
Change-Id: Ibb7cfa82b9e01233fb9062ed5d5dddde0b8f5fcc
A setting has been added to migration.yaml to allow listing jobs that
should not result in a shared queue. noop is an example.
Change-Id: I2df7202fea591190e7962e04e71f42cd75ccbe0b
Unmatched single quotes is what killed us before. Add an option which
does a test with ansible-playbook --syntax-check to validate whether a
playbook emitted with shell instead of script works.
Running that way takes around 90 minutes. So default to shell, but
provide --syntax-check as a way to check using shell and fallback to
script. This can then be used to fix pre-migration jobs if desired.
Also separate task entries with an extra '\n' for readability.
And only collect the generated playbooks - not all of them.
Change-Id: I147a7562e6578ab300ef56ae7b1c9d9d3f8f920f
The docs-draft jobs all need to publish to docs-draft. We have a base
job for that now, use it.
(Re-proposed from I52b9b72e29560e5667895f2d99c642ffa375d4d8)
Change-Id: Id7ea9eb94cc3e84a348fe8c411ff9f8b60bd0b1a
When we mark a job in the mapping, it's becaues we have a new job for
it. There is, therefore, no need to emit one.
Change-Id: Ic1eee1e86915b5d984c66f281aec295259849a9d
We need to put jobs and project_templates into openstack-zuul-jobs and
we need to put project pipelines into project-config.
Change-Id: I97c45547f2a550a3d6e188637b1d9958006d1178
James E. Blair