This change is a common root for other
Zookeeper related changed regarding
scale-out-scheduler. Zookeeper becoming
a central component requires to increase
"maxClientCnxns".
Since the ZooKeeper class is expected to grow
significantly (ZooKeeper is becoming a central part
of Zuul) a split of the ZooKeeper class (zk.py) into
zk module is done here to avoid the current god-class.
Also the zookeeper log is copied to the "zuul_output_dir".
Change-Id: I714c06052b5e17269a6964892ad53b48cf65db19
Story: 2007192
This adds a script to generate TLS certs for zookeeper.
It also adds new config file options for specifying certs for a
TLS connection, adds a howto document to advise admins on how
to configure ZK for TLS.
It also removes the 'required' flag for the SASL auth parameters,
since they are not actually required.
Include the default openssl.cnf file since some distros modify it
to specify paths that are incompatbile with the zk-ca.sh script.
Change-Id: Icd976cc32dfd9f75f8cfb1c9ad11e08af31723d6
Return the hold request IDs returned from the zookeeper API as
a sorted list so that they will appear in sorted order in the
zuul CLI output.
Change-Id: I3a3d738ac2bebb8b446cb0710bf9f5452c232372
An earlier change alters the autohold held node IDs from a list
to a dict structure. If someone upgrades to this change, but has
existing autoholds with the old format, we will break them. This
change handles both cases.
Change-Id: Ib6381e459a9694cae61ba3229d48ddfe8f55f392
Instead of storing a flat list of nodes per hold request, this
change updates the request nodes attribute to become a list of
dictionary with the build uuid and the held node list.
Change-Id: I9e50e7ccadc58fb80d5e80d9f5aac70eb7501a36
Marking the nodes as USED will allow nodepool to delete them.
If we are unsuccessful in marking any of the held nodes as used,
we simply log the error and try again at some future point until
all nodes are eventually marked, allowing the hold request to be
deleted.
Change-Id: Idd41c58b5cce0aa9b6cd186fa5c33066012790b8
Storing autohold requests in ZooKeeper, rather than in-memory,
allows us to remember requests across restarts, and is a necessity
for future work to scale out the scheduler.
Future changes to build on this will allow us to store held node
information with the change for easy node identification, and to
delete any held nodes for a request using the zuul CLI.
A new 'zuul autohold-delete' command is added since hold requests
are no longer automatically deleted.
This makes the autohold API:
zuul autohold: Create a new hold request
zuul autohold-list: List current hold requests
zuul autohold-delete: Delete a hold request
Change-Id: I6130175d1dc7d6c8ce8667f9b14ae9377737d280
The recently introduced change to request child nodes of paused jobs
with a higher priority causes the priority to be actually lower in
case the pipeline precedence is set to 'high'.
Due to the lexical sorting of node requests in Nodepool, 99 will be
treated as the lowest prio in this case.
This is especially apparent when Nodepool is up against quota.
Change-Id: I094dee4f357c9974b6d9e95fcd70b02115d9de93
Add a relative_priority field to node requests and continuously
adjust it for each queue item based on the contents of queues.
This allows for a more fair distribution of build resources between
different projects. The first item in a pipeline from a given
project (or, in the case of a dependent pipeline, group of projects)
has equal priority to all other first-items of other projcets in
the same pipeline. Second items have a lower priority, etc.
Depends-On: https://review.openstack.org/620954
Change-Id: Id3799aeb2cec6d96a662bfa394a538050f7ea947
A node can get removed from underneath us when we iterate through
all of them. Handle that better.
Change-Id: I28eb479eba4d59ef15e53c99f2abb58fb669ad39
This change adds a zk client to the ZuulWeb service to implement a
labels route for returning the available labels.
Change-Id: Iecf2948e4895829da3f090734c93fcd0c8a497b5
This change mitigates connection issue when zuul is starting before
zookeeper is ready, for example when the control plane is restarted.
Change-Id: Ia0b077b5e0429cfd6b4020254dc1b472ef2ecfbf
Our autohold can linger longer than we requested if we get an
exception during node iteration. Let's handle that particular
exception better, and also handle ANY exceptions that may bubble
up by deleting the autohold if that occurs.
Change-Id: I9d64995406e86cbad7536b85a3206fda7faac253
In case we lose the connection before fully updating all the nodes
associated with a node request, set the request attributes last.
Change-Id: Ib5099005ffb2990940672ce34623bc35b8903739
If our queue processing is slow, and we lose the ZooKeeper session
after a node request has been fulfilled, but before we actually accept
the nodes, we need to be aware of this and not try to use the nodes
given to us.
Also pass the request ID along in the event queue since the actual
request object can have its ID changed out from underneath us on a
resubmit. Compare this ID with the request ID, and also verify that
the actual request still exists.
Furthermore, when we lock nodes, let's make sure that they are actually
allocated to the request we are processing.
Change-Id: Id89f6542afcf3f5d4a0b392b5cb8cf21ec3f6865
We set precedence in our pipeline configs but we do not pass it through
to the nodepool NodeRequest priority, which means that check can starve
gate.
Change-Id: Id3fa6f9ad6bdf23bf3af43c48289c4b918ea04f1
Zuul defines the node states within model.py. The states in zk.py
are a holdover from copied code from the nodepool code base and
are not used. Let's just have a single place for the states.
Change-Id: I3f0cd40a04040baffbc04895049e7472ad30bf4b
Adds the 'autohold' client option, the scheduler implementation
of it, and a unit test for it.
The autohold is automatically removed from the in-memory data
structure once we've reached the number of requested runs of
the job.
Story: 2000905
Change-Id: Ieac0b5fee6801313fa23cce69520eb348735ad99
This code was only used in our nodepool integration tests, so remove
it and update our documentation.
Change-Id: I5698321992f58064683a772720e1349742d96d25
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Just pass through the zookeeper host list string unparsed since
that what we're going to expect from our config file.
Change-Id: Ife8fd97860ad35c793ef956adbb9d626569f60bf
We ended up using classes from model.py instead to better interact
with the rest of Zuul, so remove these.
Change-Id: I4fa4a06b27d9ef6cc7f7878f29a92aafd7ffe9d1
While we immediately lock a node given to us by nodepool, we delay
setting the node to 'in-use' until we actually request that the job
be launched so that if we end up canceling the job before it is
run, we might return the node unused to nodepool.
Change-Id: I2d2c0f9cdb4c199f2ed309e7b0cfc62e071037fa
Add a fake nodepool that immediately successfully fulfills all
requests, but actually uses the Nodepool ZooKeeper API.
Update the Zuul Nodepool facade to use the Nodepool ZooKeeper API.
Change-Id: If7859f0c6531439c3be38cc6ca6b699b3b5eade2
Jan Kubovy