This change is a follow-up to the drivers spec and it makes the fake provider
a real driver. The fakeprovider module is merged into the fake provider and
the get_one_cloud config loader is simplified.
Change-Id: I3f8ae12ea888e7c2a13f246ea5f85d4a809e8c8d
It's possible that it's easier for a nodepool user to just specify a
name or id of a flavor in their config instead of the combo of min-ram
and name-filter.
In order to not have two name related items, and also to not have the
pure flavor-name case use a term called "name-filter" - change
name-filter to flavor-name, and introduce the semantics that if
flavor-name is given by itself, it will look for an exact match on
flavor name or id, and if it's given with min-ram it will behave as
name-filter did already.
Change-Id: I8b98314958d03818ceca5abf4e3b537c8998f248
This was an unused setting which was left over from when we supported
snapshots.
Change-Id: I940eaa57f5dad8761752d767c0dfa80f2a25c787
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Before os-client-config and shade, we would include cloud credentials
in nodepool.yaml. But now comes the time where we can remove these
settings in favor of using a local clouds.yaml file.
Change-Id: Ie7af6dcd56dc48787f280816de939d07800e9d11
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
It no longer makes sense to have nodepool execute 'ready-scripts' on a
remote node. With zuulv3, we have ansible and are able to convert our
ready-scripts into ansible-playbooks.
Change-Id: I07b63a16a668bb9a37fb3f763ac29f307f6c3a65
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Remove files and fakes related to Jenkins. Since the 'targets'
config section was for mapping to Jenkins, this isn't needed either.
Change-Id: Ib5c615a95fcdce5234b3c63957171d77b8fbc65d
As we move closer to zuulv3, the concept of sub-nodes no longer
applies. As a result, we can start to remove the legacy code making
it easier to re-enable our unit tests in the future.
Change-Id: If964e082bb56b32c8fbc0f3539b83629976fe041
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Sets up a shim for the new nodepool.NodePool.run() method that does
not require any service except ZooKeeper and removes all references
to Gearman/ZMQ.
Change-Id: I452c24d631592f47eb3f4cbffb56f3252f36c298
Script-dir was only used for snapshot building. Since that is no
longer supported, remove it.
It was also passed through to DIB as an evironment variable.
However, the OpenStack infra element that used it defaults to
a sensible value. Users who still want to use that can add it
as an environment variable to appropriate diskimage entries.
Change-Id: Ie72bb9675fdad6364589a6e3823f4ef9a0228e86
We don't have plans to support building images from snapshots in the
future. If we add it back, it will require a new builder implementation
(with zookeeper interface, etc) anyway.
In addition to removing essentially dead code, this also begins the
process of simplifying the configuration file. It removes 'base-image'
and 'setup' from the provider image attributes since they are only used
for snapshot images.
Change-Id: I68e0d2c7017cfebe35341189b5aedb0556bcd78b
Instead, move this feature to be an attribute of the diskimages.
A "rebuild-age" is set for each diskimage (defaults to 24 hours),
and when a given diskimage becomes older than this age, the builder
will attempt to rebuild it (regardless of the time of day).
This makes more sense for a system that will retry failed builds
as well as being generally more idempotent, but also means that
when builds and uploads occur will be slightly less predictable.
Change-Id: I4725383158d5baf8fb67e86cb014d2db122e71a0
With OSC and shade patches, we lost the ability to run nodepoold
in the foreground with fakes. This restores that ability.
The shade integration unit tests are updated to use the string
'real' rather than 'fake' in config files, as they are trying to
avoid actually using the nodepool fakes, and the use of the string
'fake' is what triggers their use in many cases.
Change-Id: Ia5d3c3d5462bc03edafcc1567d1bab299ea5d40f
Depends on a new secure.conf file to grab mysql
connection and jenkins credentials. By this way,
nodepool.yaml is just a flat file without credentials,
that can be passed externally.
Co-Authored-By: Ramy Asselin <ramy.asselin@hpe.com>
Depends-On: Ie9381740e3644feaee1f1b201499e3a253677f39
Change-Id: Ifa4d500c52974b9fd3a0b7fd9c28cf8f52899ca9
Write the provider, region, and az names into /etc/nodepool/provider
in a form that can be sourced by a shell script. Potentially used
by a ready script.
Change-Id: I48933ddb89a4a89cb22e1cb005469833f91c1f66
This patch adds the ability to specify a net-label
instead of using a net-id (network UUID).
Rather than use network UUID's in our nodepool.yaml
config files it would be nice to use the more meaningful
network labels instead. This should make the config file
more readable across the various cloud providers and
matches how we use image names (instead of image
UUIDs) as well.
The current implementation relies on the os-tenant-networks
extension in Nova to provide the network label lookup.
Given that nodepool is currently focused around using
novaclient this made the most sense. We may at some point
in the future want to use the Neutron API directly for
this information or perhaps use a combination of both
approaches to accommodate a variety of provider API
deployment choices.
Tested locally on my TripleO overcloud using two
Neutron networks.
Change-Id: I9bdd35adf2d85659cf1b992ccd2fcf98fb124528
Write information about the node group to /etc/nodepool, along
with an ssh key generated specifically for the node group.
Add an optional script that is run on each node (and sub-node) for
a label right before a node is placed in the ready state. This
script can use the data in /etc/nodepool to setup access between
the nodes in the group.
Change-Id: Id0771c62095cccf383229780d1c4ddcf0ab42c1b
Labels replace images as the basic identity for nodes. Rather than
having nodes of a particular image, we now have nodes of a particular
label. A label describes how a node is created from an image, which
providers can supply such nodes, and how many should be kept ready.
This makes configuration simpler (by not specifying which images
are associated with which targets and simply assuming an even
distribution, the target section is _much_ smaller and _much_ less
repetitive). It also facilitates describing how a nodes of
potentially different configurations (e.g., number of subnodes) can
be created from the same image.
Change-Id: I35b80d6420676d405439cbeca49f4b0a6f8d3822
An image can specify that it should be created with a number of
subnodes. That number of nodes of the same image type will also
be created and associated with each primary node of that image
type.
Adjust the allocator to accomodate the expected loss of capacity
associated with subnodes.
If a node has subnodes, wait until they are all in the ready state
before declaring a node ready.
Change-Id: Ia4b315b1ed2999da96aab60c5c02ea2ce7667494
For debugging it is really inconvenient to have to wait for puppet to
*work* to log into a box and analyse what went wrong. This small patch
makes making one-time keypairs optional. Setting provider.keypair to
the name of a nova keypair will use that keypair in provisioning - but
you'll need to have the real key available via an agent for nodepool
to login itself.
Change-Id: I768ba656368ff00f9f7374ec0a46e57cd563ee61
Provider.networks can now be specified as a list of dicts containing
the same parameters that 'nova boot --nic ....' would take - e.g.
net-id: 'someuuid'.
Change-Id: Ie34c3636f8a2fce6aae0ceb249294eaba6a92d35
Co-Authored-By: Robert Collins <rbtcollins@hp.com>
Use state change times to determine job start and end times for a
run, and log runtime stats for the job, job+branch, job+branch+image,
job+branch+image+provider, as well as build counts for each of those.
Update the fake yaml file to align the ZMQ port with the fake servers.
Add statsd to the fake servers.
Add instructions on how to run a fake nodepool that emits statsd info.
Change-Id: I5296cafd191f363698a088fd50d5da24902fd104
This patch adds the necessary explanations on how to
use floating-ip, and add a parameter "pool" to fix the issue
which raises FloatingIpPoolNotFound when using nodepool.
Change-Id: I2344acd8b49f990eeb02c1b6016d1869f9d665f5
Closes-Bug: #1245829
Rackspace have recently added 'Performance' flavors. These flavors
distinguish themselves from other flavors only by a difference in
their name. The current config can only filter by RAM. Add a config
option for indicating a string that should be found in the flavor name.
Change-Id: I42448dde02f405182e887bf928b152cf97d2a0cb
Use information from Gearman to determine the immediate load
requirements of the system and spin up as many nodes as required
to meet the demand. Use the existing information about the
min and max servers to determine the ratio of servers to spin
up from each provider.
Replaces the several fake server scripts with one script that
implements statsd, zmq, and gearman to ease testing.
Change-Id: Ic0dedc7ef2760ff664912f771377e02967ad5633
The existing db session strategy was inherited from a bunch of
shell scripts that ran once in a single thread and exited.
The surprising thing is that even worked at all. This change
replaces that "strategy" with one where each thread clearly
begins a new session as a context manager and passes that around
to functions that need the DB. A thread-local session is used
for convenience and extra safety.
This also adds a fake provider that will produce fake images and
servers quickly without needing a real nova or jenkins. This was
used to develop the database change.
Also some minor logging changes and very brief developer docs.
Change-Id: I45e6564cb061f81d79c47a31e17f5d85cd1d9306
Tristan Cacqueray