Reorganizes the unit tests into a new subdirectory, and
adds a new functional test subdirectory (that will be further
divided into subdirs for drivers).
Change-Id: I027bba619deef9379d293fdd0457df6a1377aea8
Currently nodepool is defaulting to the qcow2 if no format was given
in the diskimage config. However nodepool also can auto-detect the
image format based on clouds.yaml. This is not taken into account
leading to qcow2 added when using raw-only clouds and auto-detection.
This can be fixed by delaying the fallback to the point where we
actually do the build.
Change-Id: Ic1d9b5abd837a3225cc13da30a80ebf1dda61b64
This updates the builder to store individual build logs in dedicated
files, one per build, named for the image and build id. Old logs are
automatically pruned. By default, they are stored in
/var/log/nodepool/builds, but this can be changed.
This removes the need to specially configure logging handler for the
image build logs.
Change-Id: Ia7415d2fbbb320f8eddc4e46c3a055414df5f997
This change adds a plugin interface so that driver can be loaded dynamically.
Instead of importing each driver in the launcher, provider_manager and config,
the Drivers class discovers and loads driver from the driver directory.
This change also adds a reset() method to the driver Config interface to
reset the os_client_config reference when reloading the OpenStack driver.
Change-Id: Ia347aa2501de0e05b2a7dd014c4daf1b0a4e0fb5
The pep8 rules used in nodepool are somewhat broken. In preparation to
use the pep8 ruleset from zuul we need to fix the findings upfront.
Change-Id: I9fb2a80db7671c590cdb8effbd1a1102aaa3aff8
This change is a follow-up to the drivers spec and it makes the fake provider
a real driver. The fakeprovider module is merged into the fake provider and
the get_one_cloud config loader is simplified.
Change-Id: I3f8ae12ea888e7c2a13f246ea5f85d4a809e8c8d
This change moves OpenStack related code to a driver. To avoid circular
import, this change also moves the StatsReporter to the stats module so that
the handlers doesn't have to import the launcher.
Change-Id: I319ce8780aa7e81b079c3f31d546b89eca6cf5f4
Story: 2001044
Task: 4614
This change adds a generic Provider meta class to the common
driver module to support multiple implementation. It also renames
some method to better match other drivers use-cases, e.g.:
* listServers into listNodes
* cleanupServer into cleanupNode
Change-Id: I6fab952db372312f12e57c6212f6ebde59a1a6b3
Story: 2001044
Task: 4612
We currently use the hostname of the builder host as the
identifier for the owner of built images. This value is
recorded in ZooKeeper and is used to make sure that only
the owner of an image build ever deletes the entry from
ZooKeeper.
Using hostname can be problematic if it ever changes. It will
cause orphaned builds that will not get deleted. This change
allows us to use a UUID as the identifier, deprecating use of
hostname altogether (although we continue storing that info).
The UUID will be stored in a file in the images directory so
that it may persist across nodepool-builder restarts.
In order to help with transitioning existing builders to using
UUID instead of hostname, the code will always compare the UUID
value AND the hostname so that existing ZK entries will be
matched until they age away.
Change-Id: Ifafbab9fb0f41564cc1af595586fa7353ce1d0d0
Since we are working towards python3 support, lets rename nodepool.py
to launcher.py to make relative imports nicer, otherwise we'd have to
use:
from . import foo
Change-Id: Ic38b6a8c2bf25d53625e159cb135b71d383b700c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We currently have a race condition between our cleanup worker and our
unit test. My hope is, if we agree to disable the CleanupWorker thread
for the test, we still consider this a valid test.
Change-Id: I04b87ef044de7f99cc9cbd0c08747e53d383693b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This test is currently flapping and needs to be reworked. Disable
again as not to side track the PTG effort.
Change-Id: I23ed4024fc3d2b906c2ae4702042ed65ce020a6b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Remove gearman and zmq settings from yaml file.
Change-Id: I8f0938ce132af89b1c7fcd2e9b1d217f23d4df0f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
But leave the current failing tests disabled. These will be cleaned up
in a follow up patch.
Change-Id: Iea7d56ceb14684c77cb991d362bff992ca061590
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Disables all tests that depend on a working nodepoold and removes
obsolete tests since the database and gearman are going away.
Change-Id: Ic973c8a657fbfa38523e73231bdb5fce53a81f3a
Users can now use nodepool-builder as a front end to
diskimage-builder. The main reason to do this, is we can leverage the
nodepool.yaml file to store the diskimage configuration. Plus
distributed builders.
To actually support this, we need to expose a formats field into the
diskimages section.
Change-Id: I1c0c196828c5c1bf2bf26e278598dbb680d0263e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
If disk-image-create fails, we also mark the image as failed.
However, when we run the cleanup worker, we expect images to exist.
In the case of a failed build, this will never happen.
Change-Id: I6f27a30e77ca11499f6ae2ea91922b379692f499
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Casting state_time with int(), which lost us time precision, is a
holdover from the old builder code. It is unnecessary and causes
strange problems in rare cases where we sort by state_time and that
precision is lost (see I2dc127b7cd42d3ee7afb1f0677810554a800e8a8
which is reverted by this change).
Change-Id: I6785232f760e29468b21be41224efe7f89543c4d
We have a very rare failure: waiting for build 001 to be deleted
sometimes fails in test_image_rotation. This is because if things
happen quickly enough, builds 001 and 002 will get the same upload
state_times, which can throw off the CleanupWorker's calculation
of "most recent uploads" (it would think 001 is more recent than 002,
due to sorting on state_time). Add a slight delay to make sure they
have different times.
Change-Id: I2dc127b7cd42d3ee7afb1f0677810554a800e8a8
We had a race condition waiting for image002 being uploaded to the
cloud. Now, ensure we are waiting for the right images in our
waitForImage() calls.
Change-Id: I814b97360a62b6c447baa7c5eddc783c5ed82c8c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Hard upload fails (those where we could not record the state in
ZooKeeper as FAILED, thus they remain as UPLOADING) are not cleaned
up from current builds, only when the build ages out. This adds
code to clean those up by removing the ZK records for them.
Change-Id: Ie1e06fc897f87ec1226ad48a355e096c659e5090
Story: 2000811
It is possible for external_name on our upload images to be empty.
Either we have written the data to zookeeper incorrect or some other
unknown.
As a result, check if external_name is set, then delete the image.
This will allow our cleanup worker to properly delete unneeded
diskimages from the HDD.
Change-Id: I2bad5f2c2585c54879497de3c8997dd6234be5a7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Test that a failed upload is recorded as FAILED in ZooKeeper.
While coding this test, discovered that failed uploads were
being left in the UPLOADING state, rather than FAILED. That
is also fixed here.
Change-Id: I26bb641c8673a5c62900b9a81f2dd11eaad78eba
Add test coverage to ensure images in nodepool are rotated properly.
Change-Id: I8fe545b87dcd7c94bf301820b34cd640ab6bbeeb
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add a unit test to validate our rebuild schedule is working correctly
for diskimage-builder.
Change-Id: Id36a64191c13938aede657d4cb70c20595cd7dd3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
When a build is no longer used, make sure that all of its uploads
are deleted, then the build itself.
Change-Id: I56fc6677c27fa653889b0720a5d9b06aac32612f
Now that we only support DIB for testing, start to simplify our
configuration files for testing. For example, replace fake-dib-image
with fake-image.
Change-Id: Iaf7a7ff714ab2d22f220e497def7c1cb6987c386
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The function is the same, but the name reflects that this class
is the central piece of the builder.
Change-Id: I7baa0d1b43140f63ad62b105d462f447c83dbb57
Having two start and stop methods was a holdover for when the
builder and scheduler were separate classes. Move the private
_start method into the public startBuilder method, and just call
it start for consistency. Same for stop.
Change-Id: I924c718a8435adc12831e34ad8803a41bf81346f
This is no longer necessary as the start method is synchronous.
We also move setting the flag to the beginning of the method to
avoid potential race conditions where the sub-threads might
exit before the _running flag is set.
Also, set the stub-method sleep calls to 0.1 seconds rather than
5 seconds so that tests finish faster.
Change-Id: I75baff21ce0e9180b12a7f126c9328692d5527c1
The NodePoolBuilder class was only serving as a stop/start manager for
the scheduler. We can move that functionality directly into the
BuilderScheduler class and eliminate use of NodePoolBuilder altogether.
We keep NodePoolBuilder class around for reference until all
functionality can be moved into the new scheduler class.
Change-Id: I89b126ed033a86a1d2b4092edee61981aee60c09
Originally, the thread started with the NodePoolBuilder.runForever()
method did all the work of the builder. This functionality is moving
to inside the BuildScheduler. As a result, the thread used for
NodePoolBuilder is no longer necessary. The intention is to (eventually)
totally decouple these classes so that NodePoolBuilder is used only to
start and stop the builder thread of execution. It's run() method is
renamed to start() to better reflect its purpose.
Change-Id: Ief00e4a06bd919e99dbff07f7238ac51035b06ef
This is the building block for the structure that nodepool builder will
take going forward. It introduces a new scheduling thread, and moves all
current build and upload worker threads underneath that. A new thread
for receiving ZooKeeper watch events is also introduced.
This code will run, but isn't currently functional because the worker
and watch threads just sleep. Later reviews will begin to add the
functionality.
A supporting developer's guide is added that shows the main components
and communication channels.
Since this is a partial implementation, several tests have to be
temporarily disabled until the ZooKeeper functionality is in place.
Change-Id: I210bb6ba1580403ddc3e5d87eb9bf60ffc7433d8
Allows defining a cluster of ZooKeeper servers within the nodepool
configuration file. Validation of the new config values is supported,
and the new values are documented.
Change-Id: I24401d08135a071dee1d00711fdd82c1d728b31a
It can be handy to test against an already running zookeeper, rather
than requiring one be installed in your test environment. Allow for
that by using the NODEPOOL_ZK_HOST environment variable.
This also gives each ZKTestCase based test a pre-existing client
connection that can be used for testing. To avoid znode path conflicts,
each connection is chroot'ed under /nodepool_test/{uniqueInt}. We
attempt to remove the {uniqueInt} znode and children as part of the
test cleanup (also removed at setUp time if cleanup fails). This will
leave a single znode for /nodepool_test, but if many tests fail for
some reason, it is easier to remove a single root node rather than
potentially many of them if we do not chroot.
Change-Id: I4fd61b16353c16bed2744e0d863ce188019430cd
David Shrewsbury