As we continue to debug performance, it can be useful to have access
to a REPL as in Zuul. Since there is no command socket in Nodepool
as there is in Zuul, the REPL must be engaged with a CLI argument
at startup.
Change-Id: Ieece1628494f79f39bd04216fc9ad7b725e541d8
Having python files with exec bit and shebang defined in
/usr/lib/python-*/site-package/ is not fine in a RPM package.
Instead of carrying a patch in nodepool RPM packaging better
to fix this directly upstream.
Change-Id: I5a01e21243f175d28c67376941149e357cdacd26
This change allows you to specify a dib-cmd parameter for disk images,
which overrides the default call to "disk-image-create". This allows
you to essentially decide the disk-image-create binary to be called
for each disk image configured.
It is inspired by a couple of things:
The "--fake" argument to nodepool-builder has always been a bit of a
wart; a case of testing-only functionality leaking across into the
production code. It would be clearer if the tests used exposed
methods to configure themselves to use the fake builder.
Because disk-image-create is called from the $PATH, it makes it more
difficult to use nodepool from a virtualenv. You can not just run
"nodepool-builder"; you have to ". activate" the virtualenv before
running the daemon so that the path is set to find the virtualenv
disk-image-create.
In addressing activation issues by automatically choosing the
in-virtualenv binary in Ie0e24fa67b948a294aa46f8164b077c8670b4025, it
was pointed out that others are already using wrappers in various ways
where preferring the co-installed virtualenv version would break.
With this, such users can ensure they call the "disk-image-create"
binary they want. We can then make a change to prefer the
co-installed version without fear of breaking.
In theory, there's no reason why a totally separate
"/custom/venv/bin/disk-image-create" would not be valid if you
required a customised dib for some reason for just one image. This is
not currently possible, even modulo PATH hacks, etc., all images will
use the same binary to build. It is for this flexibility I think this
is best at the diskimage level, rather than as, say a global setting
for the whole builder instance.
Thus add a dib-cmd option for diskimages. In the testing case, this
points to the fake-image-create script, and the --fake command-line
option and related bits are removed.
It should have no backwards compatibility effects; documentation and a
release note is added.
Change-Id: I6677e11823df72f8c69973c83039a987b67eb2af
This patch makes the nodepool process avoid starting up as a daemon in
the Docker images, as it's not meant to become a background process
within a container. In order to have consistent logging like in the
daemonized mode we need to add a new foreground option that runs in
foreground but without debug logging.
Change-Id: I77e9e6e4f94cf726336419a2b22916cc1e974e62
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
After daemonization, relative paths may be different, so resolve
all of the command line pathname arguments before then.
Change-Id: I2092c3f8e4595d8f874c769f41dacdf47554d681
The secure config file has largely been unused and ignored for v3.
This add support for reading ZooKeeper credentials from the secure
file. Note that actually specifying authentication credentials is
left for future work, but this adds the framework necessary for that.
ZooKeeper creds can be in both the normal config file and the secure
file. If specified in both, the data in the secure configuration wins.
Change-Id: I5d9c12c00f5e85ef258128337cdb99809f86b8ed
There is a lot of commonality between the nodepool apps that is
replicated between each of the files. This will mean adding more again
when new apps are added.
Separate a base NodepoolApp and NodepoolDaemonApp that handle the common
parts of setting up a nodepool app and use them in the existing apps.
First Proposed as: I098b0a1d749e9dc400d45355f9295ca07058768b
Then reverted: I7d8f1cd866b768326157f4503c1f729ebb703c0c
Change-Id: I81e80af8e2c18e56db8b6dca59ceb279dc9d591c
Logging is currently broken due to this change. Nothing is logged.
This reverts commit 03c41ccc35.
Change-Id: I7d8f1cd866b768326157f4503c1f729ebb703c0c
There is a lot of commonality between the nodepool apps that is
replicated between each of the files. This will mean adding more again
when new apps are added.
Separate a base NodepoolApp and NodepoolDaemonApp that handle the common
parts of setting up a nodepool app and use them in the existing apps.
Change-Id: I098b0a1d749e9dc400d45355f9295ca07058768b
This allows the builder to be run in an otherwise production-like
setting but with the actual image build stubbed out.
Change-Id: If41428605c13f263da78ebe382ac83b4c1858c42
The function is the same, but the name reflects that this class
is the central piece of the builder.
Change-Id: I7baa0d1b43140f63ad62b105d462f447c83dbb57
Having two start and stop methods was a holdover for when the
builder and scheduler were separate classes. Move the private
_start method into the public startBuilder method, and just call
it start for consistency. Same for stop.
Change-Id: I924c718a8435adc12831e34ad8803a41bf81346f
The NodePoolBuilder class was only serving as a stop/start manager for
the scheduler. We can move that functionality directly into the
BuilderScheduler class and eliminate use of NodePoolBuilder altogether.
We keep NodePoolBuilder class around for reference until all
functionality can be moved into the new scheduler class.
Change-Id: I89b126ed033a86a1d2b4092edee61981aee60c09
Originally, the thread started with the NodePoolBuilder.runForever()
method did all the work of the builder. This functionality is moving
to inside the BuildScheduler. As a result, the thread used for
NodePoolBuilder is no longer necessary. The intention is to (eventually)
totally decouple these classes so that NodePoolBuilder is used only to
start and stop the builder thread of execution. It's run() method is
renamed to start() to better reflect its purpose.
Change-Id: Ief00e4a06bd919e99dbff07f7238ac51035b06ef
This is the building block for the structure that nodepool builder will
take going forward. It introduces a new scheduling thread, and moves all
current build and upload worker threads underneath that. A new thread
for receiving ZooKeeper watch events is also introduced.
This code will run, but isn't currently functional because the worker
and watch threads just sleep. Later reviews will begin to add the
functionality.
A supporting developer's guide is added that shows the main components
and communication channels.
Since this is a partial implementation, several tests have to be
temporarily disabled until the ZooKeeper functionality is in place.
Change-Id: I210bb6ba1580403ddc3e5d87eb9bf60ffc7433d8
Clarify code in nodepool/cmd/builder.py so we don't have two classes
named NodePoolBuilder and multiple variables named 'nb'.
Change-Id: Icb77ca2edea49ed3b787e600cf3754160ee1eff8
* Builders were interfering with the gear shutdown procedure
by overriding the use of the 'running' variable on gear workers.
Instead, just rely on the built-in shutdown process in the gear
worker class.
* Have the builder shutdown provider managers as well.
* Correctly handle signals in the builder.
* Have the nodepool daemon shut down its gearman client.
* Use a condition object so that we can interrupt the main loop
sleep and exit faster.
Both the builder and the daemon now exit cleanly on CTRL-C when
run in the foreground.
Change-Id: Iefd5ef7df74e701725f4bafe4df51b8276088fe5
This allows a user to tweak nodepool-builder image workers over using
static settings.
Change-Id: I5268e0ca34ab91ffae7a203ca710eecf571585e5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Because we may want to colocate the nodepoold and nodepool builder
daemons they need to use different pidfiles. Make that the case by
default.
Change-Id: I563e59855f70c3e28cc1daff8bd0a9d063e179c0
This works well for zuul, so it should work for nodepool.
Change-Id: I536694494d58c7cb4bb9dd1b10d24ddbdaf127ae
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add entrypoint so builders can be run on their own using the
nodepool-builder command. Also update config processing so builders do
not require things like a zmq-publisher.
Change-Id: Ied1142990ca630f22044a472db77920daa4c2e5a
James E. Blair