While YAML does have inbuilt support for anchors to greatly reduce
duplicated sections, anchors have no support for merging values. For
diskimages, this can result in a lot of duplicated values for each
image which you can not otherwise avoid.
This provides two new values for diskimages; a "parent" and
"abstract".
Specifying a parent means you inherit all the configuration values
from that image. Anything specified within the child image overwrites
the parent values as you would expect; caveats, as described in the
documentation, are that the elements field appends and the env-vars
field has update() semantics.
An "abstract" diskimage is not instantiated into a real image, it is
only used for configuration inheritance. This way you can make a
abstrat "base" image with common values and inherit that everywhere
without having to worry about bringing in values you don't want.
You can also chain parents together and the inheritance flows through.
Documentation is updated, and several tests are added to ensure the
correct parenting, merging and override behaviour of the new values.
Change-Id: I170016ef7d8443b9830912b9b0667370e6afcde7
If, during a long DIB image build, we lose the ZooKeeper session,
it's likely that the CleanupWorker thread could have run and removed
the ZK record for the build (its state would be BUILDING and unlocked,
indicating something went wrong). In that scenario, when the DIB
process finishes (possibly writing out DIB files), it will never get
cleaned up since the ZK record would now be gone. If we fail to update
the ZK record at the end of the build, just delete the leaked DIB files
immediately after the build.
Change-Id: I5cb58318efe51b5b0c3413b7a01f02a50215a8b6
By making this a static method, we can allow code other than the
CleanupWorker thread to call it for DIB file removal.
This change helped to identify that the builder unit test
test_cleanup_failed_image_build was not valid because the fake
dib creation process was exiting before creating any files.
Since the test was attempting to validate those files were
supposed to be deleted, it would always succeed. Moving the
exit to after file creation fixes that.
Change-Id: Iaa5f02abbaebd9866b6cc16c977e0d42d804eee1
A builder thread can wedge if the build process wedges. Add a timeout
to the subprocess. Since it was the call to readline() that would block,
we change the process to have DIB write directly to the log. This allows
us to set a timeout in the Popen.wait() call. And we kill the dib
subprocess, as well.
The timeout value can be controlled in the diskimage configuration and
defaults to 8 hours.
Change-Id: I188e8a74dc39b55a4b50ade5c1a96832fea76a7d
We have been using DIB_CHECKSUM for a few weeks now, and things
appear to be working well. Now switch to passing --checksum via
disk-iamge-create and properly clean up the checksum files when we
delete our images.
Change-Id: Ia849aa179750648be13455d1f38a6a01423bd8c0
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
fake-image-create currently just ignores options it doesn't know
about. I'd rather see us being strict about this, so if we introduce
typos, etc, we find it. Expand the getopt call for the extra args
passed in currently.
(if this gets too complicated, a quick re-write in python with
argparse would make things easier)
Change-Id: If1eb1774549463644fc1d667d17773c4e30cbb60
DIB can output an image_name.d directory containing image manifests.
These have a very similar naming convention to the images we build which
confuses builders.
Change-Id: If5c2fd5bb6c5da33ba489df250a103f6c154093a
With mutliple cloud providers supporting different image formats and
versions nodepool needs to express the requirements providers have on
images. Add a provider.image-type setting to providers that determine
which image formats diskimage-builder should output.
Change-Id: I808571e2c9cc10fb7dfcd6e30262769872cd75cc
We currently branch based on the names of providers, images, etc in
order to test with our fakes. We can get a lot more coverage by
monkeypatching our fakes over the clients they are faking for.
Change-Id: If80eb1f4d914f8b2fbd914311ecda2cf66e7a803
Test that when a dib image fails to build nodepool continues to build
nodes using images it does have available.
To test that dib actually fails check the logged subprocesses for the
tests for exit status of 127. This is the value that is returned when
the fake dib builds are explicitly told to fail.
Change-Id: Ie7c13af319141ec3cf8d43155358157e7d9274b8
If the "release" is numeric, such as for Fedora, it gets interpreted
as an integer and blows up when we try to insert it into the
environment (see I0d4e8519972d14fc100f46dae1b4549743b438f2). Although
you can type the yaml input, automatically convert it to a string to
follow principle of least-surprise.
Test-case is updated. The whole image is made to look "fedora-like"
to avoid confusion.
Change-Id: I0ea4dc54edad7ef9b5d4ad59922f762ed8bbcd89
In change Ic6c21f715441a16a1d832163dc71921f25d654df I made the mistake
of specifying env-var's as a list, rather than a dict, in the yaml.
In this change we check that env-vars is a dictionary and soft-fail if
it is not.
Additionally a script 'fake-image-create' is added to be using during
testing. This is spawned in-place of disk-image-create and can check
arbitrary things, returning non-zero if it finds them. We check for
some added environment variables as a first step.
The documentation is also clarified slightly with an expanded example.
Change-Id: I7fe1972fdcc7e8b0231e9d232485da9c5dfdb31b
Ian Wienand