zuul
/
nodepool
Code Issues Proposed changes

Remove Jenkins 

Remove files and fakes related to Jenkins. Since the 'targets'
config section was for mapping to Jenkins, this isn't needed either.

Change-Id: Ib5c615a95fcdce5234b3c63957171d77b8fbc65d
This commit is contained in:
David Shrewsbury 2017-03-12 10:21:03 -04:00
parent 28405cf1fa
commit 8c6461ebe4
37 changed files with 5 additions and 545 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
devstack/plugin.sh
View File

@ -184,11 +184,6 @@ zookeeper-servers:
- host: localhost
port: 2181
# Need to have at least one target for node allocations, but
# this does not need to be a jenkins target.
targets:
- name: dummy
cron:
cleanup: '*/1 * * * *'
check: '*/15 * * * *'

84
doc/source/configuration.rst
View File

@ -5,18 +5,11 @@ Configuration
Nodepool reads its secure configuration from ``/etc/nodepool/secure.conf``
by default. The secure file is a standard ini config file, with
one section for database, and another section for the jenkins
secrets for each target::
one section for the database.
[database]
dburi={dburi}
[jenkins "{target_name}"]
user={user}
apikey={apikey}
credentials={credentials}
url={url}
Following settings are available::
**required**
@ -29,36 +22,11 @@ Following settings are available::
dburi='mysql+pymysql://nodepool@localhost/nodepool'
**optional**
While it is possible to run Nodepool without any Jenkins targets,
if Jenkins is used, the `target_name` and `url` are required. The
`user`, `apikey` and `credentials` also may be needed depending on
the Jenkins security settings.
``target_name``
Name of the jenkins target. It needs to match with a target
specified in nodepool.yaml, in order to retrieve its settings.
``url``
Url to the Jenkins REST API.
``user``
Jenkins username.
``apikey``
API key generated by Jenkins (not the user password).
``credentials``
If provided, Nodepool will configure the Jenkins slave to use the Jenkins
credential identified by that ID, otherwise it will use the username and
ssh keys configured in the image.
Nodepool reads its configuration from ``/etc/nodepool/nodepool.yaml``
by default. The configuration file follows the standard YAML syntax
with a number of sections defined with top level keys. For example, a
full configuration file may have the ``diskimages``, ``labels``,
``providers``, and ``targets`` sections::
and ``providers`` sections::
diskimages:
...
@ -66,8 +34,6 @@ full configuration file may have the ``diskimages``, ``labels``,
...
providers:
...
targets:
...
The following sections are available. All are required unless
otherwise indicated.
@ -135,8 +101,7 @@ labels
Defines the types of nodes that should be created. Maps node types to
the images that are used to back them and the providers that are used
to supply them. Jobs should be written to run on nodes of a certain
label (so targets such as Jenkins don't need to know about what
providers or images are used to create them). Example::
label. Example::
labels:
- name: my-precise
@ -430,9 +395,8 @@ provider, the Nodepool image types are also defined (see
``ipv6-preferred``
If it is set to True, nodepool will try to find ipv6 in public net first
as the ip address for ssh connection to build snapshot images and create
jenkins slave definition. If ipv6 is not found or the key is not
specified or set to False, ipv4 address will be used.
as the ip address for the ssh connection. If ipv6 is not found or the key
is not specified or set to False, ipv4 address will be used.
``api-timeout`` (compatability)
Timeout for the OpenStack API calls client in seconds. Prefer setting
@ -533,41 +497,3 @@ Example configuration::
Arbitrary key/value metadata to store for this server using the Nova
metadata service. A maximum of five entries is allowed, and both keys and
values must be 255 characters or less.
.. _targets:
targets
-------
Lists the Jenkins masters to which Nodepool should attach nodes after
they are created. Nodes of each label will be evenly distributed
across all of the targets which are on-line::
targets:
- name: jenkins1
- name: jenkins2
**required**
``name``
Identifier for the system an instance is attached to.
**optional**
``rate``
In seconds. Default 1.0
``jenkins`` (dict)
``test-job`` (optional)
Setting this would cause a newly created instance to be in a TEST state.
The job name given will then be executed with the node name as a
parameter.
If the job succeeds, move the node into READY state and relabel it with
the appropriate label (from the image name).
If it fails, immediately delete the node.
If the job never runs, the node will eventually be cleaned up by the
periodic cleanup task.

12
nodepool/cmd/config_validator.py
View File

@ -92,17 +92,6 @@ class ConfigValidator:
}],
}
targets = {
'name': str,
'jenkins': {
'url': str,
'user': str,
'apikey': str,
'credentials-id': str,
'test-job': str
}
}
diskimages = {
'name': str,
'pause': bool,
@ -125,7 +114,6 @@ class ConfigValidator:
'cron': cron,
'providers': [providers],
'labels': [labels],
'targets': [targets],
'diskimages': [diskimages],
}

33
nodepool/config.py
View File

@ -82,11 +82,6 @@ class ProviderImage(ConfigValue):
return "<ProviderImage %s>" % self.name
class Target(ConfigValue):
def __repr__(self):
return "<Target %s>" % self.name
class Label(ConfigValue):
def __repr__(self):
return "<Label %s>" % self.name
@ -138,13 +133,11 @@ def loadConfig(config_path):
newconfig.db = None
newconfig.dburi = None
newconfig.providers = {}
newconfig.targets = {}
newconfig.labels = {}
newconfig.elementsdir = config.get('elements-dir')
newconfig.imagesdir = config.get('images-dir')
newconfig.dburi = None
newconfig.provider_managers = {}
newconfig.jenkins_managers = {}
newconfig.zookeeper_servers = {}
newconfig.diskimages = {}
newconfig.crons = {}
@ -277,19 +270,6 @@ def loadConfig(config_path):
p.name = provider['name']
l.providers[p.name] = p
for target in config.get('targets', []):
t = Target()
t.name = target['name']
newconfig.targets[t.name] = t
jenkins = target.get('jenkins', {})
t.online = True
t.rate = target.get('rate', 1.0)
t.jenkins_test_job = jenkins.get('test-job')
t.jenkins_url = None
t.jenkins_user = None
t.jenkins_apikey = None
t.jenkins_credentials_id = None
return newconfig
@ -299,19 +279,6 @@ def loadSecureConfig(config, secure_config_path):
config.dburi = secure.get('database', 'dburi')
for target in config.targets.values():
section_name = 'jenkins "%s"' % target.name
if secure.has_section(section_name):
target.jenkins_url = secure.get(section_name, 'url')
target.jenkins_user = secure.get(section_name, 'user')
target.jenkins_apikey = secure.get(section_name, 'apikey')
try:
target.jenkins_credentials_id = secure.get(
section_name, 'credentials')
except:
pass
def _cloudKwargsFromProvider(provider):
cloud_kwargs = {}

46
nodepool/fakeprovider.py
View File

@ -20,7 +20,6 @@ import threading
import time
import uuid
from jenkins import JenkinsException
import shade
import exceptions
@ -294,48 +293,3 @@ class FakeSSHClient(object):
def open_sftp(self):
return FakeSFTPClient()
class FakeJenkins(object):
def __init__(self, user):
self._nodes = {}
self.quiet = False
self.down = False
if user == 'quiet':
self.quiet = True
if user == 'down':
self.down = True
def node_exists(self, name):
return name in self._nodes
def create_node(self, name, **kw):
self._nodes[name] = kw
def delete_node(self, name):
del self._nodes[name]
def get_info(self):
if self.down:
raise JenkinsException("Jenkins is down")
d = {u'assignedLabels': [{}],
u'description': None,
u'jobs': [{u'color': u'red',
u'name': u'test-job',
u'url': u'https://jenkins.example.com/job/test-job/'}],
u'mode': u'NORMAL',
u'nodeDescription': u'the master Jenkins node',
u'nodeName': u'',
u'numExecutors': 1,
u'overallLoad': {},
u'primaryView': {u'name': u'Overview',
u'url': u'https://jenkins.example.com/'},
u'quietingDown': self.quiet,
u'slaveAgentPort': 8090,
u'unlabeledLoad': {},
u'useCrumbs': False,
u'useSecurity': True,
u'views': [
{u'name': u'test-view',
u'url': u'https://jenkins.example.com/view/test-view/'}]}
return d

137
nodepool/jenkins_manager.py
View File

@ -1,137 +0,0 @@
#!/usr/bin/env python
# Copyright (C) 2011-2013 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
#
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
import re
import myjenkins
import fakeprovider
from task_manager import Task, TaskManager
class CreateNodeTask(Task):
def main(self, jenkins):
if 'credentials_id' in self.args:
launcher_params = {'port': 22,
'credentialsId': self.args['credentials_id'],
'host': self.args['host']}
else:
launcher_params = {'port': 22,
'username': self.args['username'],
'privatekey': self.args['private_key'],
'host': self.args['host']}
args = dict(
name=self.args['name'],
numExecutors=self.args['executors'],
nodeDescription=self.args['description'],
remoteFS=self.args['root'],
exclusive=True,
launcher='hudson.plugins.sshslaves.SSHLauncher',
launcher_params=launcher_params)
if self.args['labels']:
args['labels'] = self.args['labels']
try:
jenkins.create_node(**args)
except myjenkins.JenkinsException as e:
if 'already exists' in str(e):
pass
else:
raise
class NodeExistsTask(Task):
def main(self, jenkins):
return jenkins.node_exists(self.args['name'])
class DeleteNodeTask(Task):
def main(self, jenkins):
return jenkins.delete_node(self.args['name'])
class GetNodeConfigTask(Task):
def main(self, jenkins):
return jenkins.get_node_config(self.args['name'])
class SetNodeConfigTask(Task):
def main(self, jenkins):
jenkins.reconfig_node(self.args['name'], self.args['config'])
class StartBuildTask(Task):
def main(self, jenkins):
jenkins.build_job(self.args['name'],
parameters=self.args['params'])
class GetInfoTask(Task):
def main(self, jenkins):
return jenkins.get_info()
class JenkinsManager(TaskManager):
log = logging.getLogger("nodepool.JenkinsManager")
def __init__(self, target):
super(JenkinsManager, self).__init__(None, target.name, target.rate)
self.target = target
self._client = self._getClient()
def _getClient(self):
if self.target.jenkins_apikey == 'fake':
return fakeprovider.FakeJenkins(self.target.jenkins_user)
return myjenkins.Jenkins(self.target.jenkins_url,
self.target.jenkins_user,
self.target.jenkins_apikey)
def createNode(self, name, host, description, executors, root, labels=[],
credentials_id=None, username=None, private_key=None):
args = dict(name=name, host=host, description=description,
labels=labels, executors=executors, root=root)
if credentials_id:
args['credentials_id'] = credentials_id
else:
args['username'] = username
args['private_key'] = private_key
return self.submitTask(CreateNodeTask(**args))
def nodeExists(self, name):
return self.submitTask(NodeExistsTask(name=name))
def deleteNode(self, name):
return self.submitTask(DeleteNodeTask(name=name))
LABEL_RE = re.compile(r'<label>(.*)</label>')
def relabelNode(self, name, labels):
config = self.submitTask(GetNodeConfigTask(name=name))
old = None
m = self.LABEL_RE.search(config)
if m:
old = m.group(1)
config = self.LABEL_RE.sub('<label>%s</label>' % ' '.join(labels),
config)
self.submitTask(SetNodeConfigTask(name=name, config=config))
return old
def startBuild(self, name, params):
self.submitTask(StartBuildTask(name=name, params=params))
def getInfo(self):
return self._client.get_info()

136
nodepool/myjenkins.py
View File

@ -1,136 +0,0 @@
#!/usr/bin/env python
# Copyright 2011-2013 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import jenkins
import json
import six.moves.urllib.parse as urlparse
import six.moves.urllib.request as urlrequest
from jenkins import JenkinsException, NODE_TYPE, CREATE_NODE
TOGGLE_OFFLINE = '/computer/%(name)s/toggleOffline?offlineMessage=%(msg)s'
CONFIG_NODE = '/computer/%(name)s/config.xml'
class Jenkins(jenkins.Jenkins):
def disable_node(self, name, msg=''):
'''
Disable a node
@param name: Jenkins node name
@type name: str
@param msg: Offline message
@type msg: str
'''
info = self.get_node_info(name)
if info['offline']:
return
self.jenkins_open(
urlrequest.Request(self.server + TOGGLE_OFFLINE % locals()))
def enable_node(self, name):
'''
Enable a node
@param name: Jenkins node name
@type name: str
'''
info = self.get_node_info(name)
if not info['offline']:
return
msg = ''
self.jenkins_open(
urlrequest.Request(self.server + TOGGLE_OFFLINE % locals()))
def get_node_config(self, name):
'''
Get the configuration for a node.
:param name: Jenkins node name, ``str``
'''
get_config_url = self.server + CONFIG_NODE % locals()
return self.jenkins_open(urlrequest.Request(get_config_url))
def reconfig_node(self, name, config_xml):
'''
Change the configuration for an existing node.
:param name: Jenkins node name, ``str``
:param config_xml: New XML configuration, ``str``
'''
headers = {'Content-Type': 'text/xml'}
reconfig_url = self.server + CONFIG_NODE % locals()
self.jenkins_open(
urlrequest.Request(reconfig_url, config_xml, headers))
def create_node(self, name, numExecutors=2, nodeDescription=None,
remoteFS='/var/lib/jenkins', labels=None, exclusive=False,
launcher='hudson.slaves.JNLPLauncher', launcher_params={}):
'''
@param name: name of node to create
@type name: str
@param numExecutors: number of executors for node
@type numExecutors: int
@param nodeDescription: Description of node
@type nodeDescription: str
@param remoteFS: Remote filesystem location to use
@type remoteFS: str
@param labels: Labels to associate with node
@type labels: str
@param exclusive: Use this node for tied jobs only
@type exclusive: boolean
@param launcher: The launch method for the slave
@type launcher: str
@param launcher_params: Additional parameters for the launcher
@type launcher_params: dict
'''
if self.node_exists(name):
raise JenkinsException('node[%s] already exists' % (name))
mode = 'NORMAL'
if exclusive:
mode = 'EXCLUSIVE'
#hudson.plugins.sshslaves.SSHLauncher
#hudson.slaves.CommandLauncher
#hudson.os.windows.ManagedWindowsServiceLauncher
launcher_params['stapler-class'] = launcher
inner_params = {
'name': name,
'nodeDescription': nodeDescription,
'numExecutors': numExecutors,
'remoteFS': remoteFS,
'labelString': labels,
'mode': mode,
'type': NODE_TYPE,
'retentionStrategy': {
'stapler-class': 'hudson.slaves.RetentionStrategy$Always'},
'nodeProperties': {'stapler-class-bag': 'true'},
'launcher': launcher_params
}
params = {
'name': name,
'type': NODE_TYPE,
'json': json.dumps(inner_params)
}
self.jenkins_open(urlrequest.Request(
self.server + CREATE_NODE % urlparse.urlencode(params)))
if not self.node_exists(name):
raise JenkinsException('create[%s] failed' % (name))

3
nodepool/tests/fixtures/config_validate/good.yaml vendored
View File

@ -65,9 +65,6 @@ providers:
user-home: /home/jenkins
private-key: /home/nodepool/.ssh/id_rsa
targets:
- name: zuul
diskimages:
- name: trusty
formats:

3
nodepool/tests/fixtures/config_validate/yaml_error.yaml vendored
View File

@ -60,9 +60,6 @@ providers:
username: jenkins
private-key: /home/nodepool/.ssh/id_rsa
targets:
- name: zuul
diskimages:
- name: trusty
elements:

7
nodepool/tests/fixtures/integration.yaml vendored
View File

@ -35,12 +35,5 @@ providers:
key: value
key2: value
targets:
- name: fake-target
jenkins:
url: https://jenkins.example.org/
user: fake
apikey: fake
diskimages:
- name: fake-image

3
nodepool/tests/fixtures/integration_occ.yaml vendored
View File

@ -31,8 +31,5 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image

3
nodepool/tests/fixtures/leaked_node.yaml vendored
View File

@ -38,9 +38,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node.yaml vendored
View File

@ -40,9 +40,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_az.yaml vendored
View File

@ -40,9 +40,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_cmd.yaml vendored
View File

@ -59,9 +59,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image1
- name: fake-image2

3
nodepool/tests/fixtures/node_disabled_label.yaml vendored
View File

@ -38,9 +38,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_diskimage_fail.yaml vendored
View File

@ -38,9 +38,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

2
nodepool/tests/fixtures/node_diskimage_only.yaml vendored
View File

@ -14,8 +14,6 @@ labels: []
providers: []
targets: []
diskimages:
- name: fake-image
formats:

3
nodepool/tests/fixtures/node_diskimage_pause.yaml vendored
View File

@ -45,9 +45,6 @@ providers:
- name: fake-image2
min-ram: 8192
targets:
- name: fake-target
diskimages:
- name: fake-image
pause: True

3
nodepool/tests/fixtures/node_image_upload_pause.yaml vendored
View File

@ -46,9 +46,6 @@ providers:
- name: fake-image2
min-ram: 8192
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_ipv6.yaml vendored
View File

@ -92,9 +92,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_label_provider.yaml vendored
View File

@ -57,9 +57,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_launch_retry.yaml vendored
View File

@ -39,9 +39,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_net_name.yaml vendored
View File

@ -40,9 +40,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_quota.yaml vendored
View File

@ -40,9 +40,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_two_image.yaml vendored
View File

@ -45,9 +45,6 @@ providers:
- name: fake-image2
min-ram: 8192
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_two_image_remove.yaml vendored
View File

@ -38,9 +38,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_two_provider.yaml vendored
View File

@ -58,9 +58,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_two_provider_remove.yaml vendored
View File

@ -51,9 +51,6 @@ providers:
rate: 0.0001
images: []
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_upload_fail.yaml vendored
View File

@ -59,9 +59,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_vhd.yaml vendored
View File

@ -39,9 +39,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

3
nodepool/tests/fixtures/node_vhd_and_qcow2.yaml vendored
View File

@ -60,9 +60,6 @@ providers:
key: value
key2: value
targets:
- name: fake-target
diskimages:
- name: fake-image
elements:

6
nodepool/tests/fixtures/secure.conf vendored
View File

@ -1,8 +1,2 @@
[database]
dburi={dburi}
[jenkins "fake-target"]
user=fake
apikey=fake
credentials=fake
url=http://fake-url

1
requirements.txt
View File

@ -1,7 +1,6 @@
pbr>=1.3
PyYAML
python-jenkins
paramiko>1.11.6,<2.0.0
python-daemon>=2.0.4,<2.1.0
extras

3
tools/fake-dib.yaml
View File

@ -36,6 +36,3 @@ providers:
- name: fake-dib-image
min-ram: 8192
diskimage: fake-dib-image
targets:
- name: fake-jenkins

6
tools/fake-secure.conf
View File

@ -1,8 +1,2 @@
[database]
dburi=mysql+pymysql://nodepool@localhost/nodepool
[jenkins "fake-target"]
user=fake
apikey=fake
credentials=fake
url=http://fake-url

3
tools/fake.yaml
View File

@ -46,6 +46,3 @@ providers:
min-ram: 8192
name-filter: 'Fake'
diskimage: fake-nodepool
targets:
- name: zuul