This job has been broken for a long time, and the paths it tests are
better covered by the quickstart tests which bring up nodepool with
Zuul.
It's a bit of an odd job because nodepool sets itself up, but then the
test calls the tox "nodepool" environment in Zuul here. So remove the
job inclusion, but also the tox/unit-tests being run by the job (it's
already failing and non-voting on nodepool, so this won't affect
things).
Change-Id: I8483b1d66a6a58d4bd2f2fce82b023d8f0446ae7
This change follows up on a few TODOs left by the lock/unlock nodes on
executor change.
When locking the nodes on the executor we used the AnsibleJob as a
replacement for the old build parameter that was provided to the
nodepool client methods as they were originally called by the scheduler.
However, the AnsibleJob class should only be used internally by the
executor, so we now provide all parameters directly to the nodepool
methods.
This also annotates the logger in the updated nodepool client methods
and fixes an outdated method signature in
test_scheduler.TestSemaphore.test_semaphore_zk_error.
Remove two comments about storing timestamps on the build request in
ZooKeeper as this doesn't make much sense. It sounded like a good idea
in the beginning, but with the current solution, the scheduler doesn't
need to care about the build request anymore after it was submitted
(except for canceling/cleanup purposes) and the result data is
self-contained.
Change-Id: I2d1005f69904c6ace8f79523133f382af0024c52
The internal zuul.nodepool.Nodepool.requests dictionary is used so
the scheduler can keep track of its requests. Since we will have
multiple schedulers emitting requests, we can't use that any more.
Remove any remaining uses of it.
The NodeRequest uid was only used to index that dictionary (and
was used to persist a request across resubmission). Since it isn't
needed any more, it is removed.
Change-Id: I7c82485d95979c6c9a246c3dc3954bae3c65ac13
Moving the parent provider determination into the pipeline manager
allows us to remove the buildset and job objects from the NodeRequest
constructor. This way we can fully serialize the NodeRequest to
ZooKeeper and restore it without missing important information.
This has also an impact on the NodeRequest's priority property. As this
is only needed to determine the znode path when the NodeRequest is
submitted, we can provide it directly as parameter to the
submitNodeRequest call (and the related update callbacks).
To ensure that NodePool doesn't strip those additional information when
it fulfills the NodeRequest, we use the new "requestor_data" field which
is implemented in [1].
To make this work, we also have to look up the buildset by its UUID from
the active tenants and pipelines when the NodesProvisioned event is
handled in the scheduler. Something similar was already done for
handling the other result events as well.
[1]: https://review.opendev.org/c/zuul/nodepool/+/798746/
Depends-On: https://review.opendev.org/c/zuul/nodepool/+/798746/
Change-Id: Id794643dcf26b0565499d20adba99d3b0518fdf1
To lock/unlock the nodes directly in the executor server, we have to
make the Nodepool API work without a scheduler instance.
To keep the stats emitting intact, we provide a statsd client directly
to the Nodepool instance.
This leaves only one place where the scheduler is used in the Nodepool
class, which is the onNodesProvisioned() callback.
This callback won't be necessary anymore when the nodes are locked on
the executor and thus this function call and the scheduler parameter
itself can be removed.
Change-Id: I3f3e4bfff08e244f68a9be7c6a4efcc194a23332
The ZooKeeperClient now provides a fromConfig() method that parses all
necessary configuration values to instantiate a ZooKeeperClient.
Previously, this needed to be done in every component to initialize the
connection to ZooKeeper.
Change-Id: I5fa4ddab5f85c658291f1262ee0392a60086846e
This change is a common root for other
Zookeeper related changed regarding
scale-out-scheduler. Zookeeper becoming
a central component requires to increase
"maxClientCnxns".
Since the ZooKeeper class is expected to grow
significantly (ZooKeeper is becoming a central part
of Zuul) a split of the ZooKeeper class (zk.py) into
zk module is done here to avoid the current god-class.
Also the zookeeper log is copied to the "zuul_output_dir".
Change-Id: I714c06052b5e17269a6964892ad53b48cf65db19
Story: 2007192
Add a relative_priority field to node requests and continuously
adjust it for each queue item based on the contents of queues.
This allows for a more fair distribution of build resources between
different projects. The first item in a pipeline from a given
project (or, in the case of a dependent pipeline, group of projects)
has equal priority to all other first-items of other projcets in
the same pipeline. Second items have a lower priority, etc.
Depends-On: https://review.openstack.org/620954
Change-Id: Id3799aeb2cec6d96a662bfa394a538050f7ea947
Several fixes to this job:
Use pip3 in the setup script as nodepool is v3 only; this whole script
needs to be made native, but this works for now.
Make sure we pass a list as the Node() request name
Update accpetNodes call to pass id
Change-Id: I3d064e6c1df02f1dad2a08855927703a0e23a047
The nodepool tests set up zookeeper connections and a fake nodepool.
These test resources spawned threads that were leaking and causing
subsequent tests to run with zookeeper and nodepool threads running.
Clean that up.
Change-Id: Ib799d4a40eb51e7bbe71673a082f345235c41019
In order to get the method resolution correct for setUp in the tests we
need to call super on the current Class not the parent.
Change-Id: I1cf7f45c107a1fad2b70e3b18dc23b8bcc6c4b33
This code was only used in our nodepool integration tests, so remove
it and update our documentation.
Change-Id: I5698321992f58064683a772720e1349742d96d25
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This uses the python unittest framework to run a test similar to
the existing zuul-nodepool unit test, but expecting an actual
running nodepool.
Change-Id: I769e1421c146cc3545dce606487c8c72e3d3a4c5
Ian Wienand