Commit Graph

122 Commits

Author SHA1 Message Date
Zuul 9764937c49 Merge "Add rackspaceauth as a nodepool dependency" 2024-03-07 18:09:28 +00:00
James E. Blair 1ac3f8bda4 Pin kazoo
To match Zuul, pin the version of kazoo we're using.

Change-Id: I104faa50c6a2fb4bfa7955c0ead34453a0505672
2024-03-06 15:01:28 -08:00
Clark Boylan 6820b309a1 Add rackspaceauth as a nodepool dependency
Rackspace has announced that MFA will be required starting on March 26,
2024. When MFA is enabled on an account you will no longer be able to
log in to rackspace using a username and password with
openstacksdk/openstackclient/etc as the APIs apparently don't support
negotiating the MFA token. Instead we can either use a rackspace
specific api_key or a keystone bearer token.

We opt for the rackspace specific api_key because it doesn't expire
like the bearer tokens do. But using the rackspace api_key does require
a keystoneauth1 plugin called `rackspaceauth` to be installed which this
change adds to nodepool.

This new dep is Apache2 licensed according to the License file in the
sdist. The new dep has minimal deps of its own and they are all alread
shared by the existing dep tree. Seems reasonable to install this small
lib in hopes that we can keep rackspace working with nodepool

As a final note the OpenDev team plans to test use of the api_key with
this library against a single rackspace region. It is possible this
won't work out of the box and we may need to make additional updates.
Unfortunately, it isn't easy to test this without talking directly to
rax so we opt for the lib install and testing via OpenDev.

Change-Id: Ibff32bb44e05413391dd7a320ba356f521bb30e8
2024-03-05 09:49:04 -08:00
Clark Boylan fe693eeb5e Exclude openstacksdk 2.0.0
This version includes a partial implementation of a migration to Proxy
objects for block limitations. Unfortunately that implemenation only
covered cinder v3 apis and not cinder v2 apis. In the case of cinder v2
apis it just breaks on us:

  2023-10-31 16:37:06,135 ERROR nodepool.PoolWorker.rax-iad-main: Error in PoolWorker:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.11/site-packages/nodepool/launcher.py", line 491, in run
      for chunk in self._assignHandlers():
    File "/usr/local/lib/python3.11/site-packages/nodepool/launcher.py", line 242, in _assignHandlers
      reasons_to_decline = rh.getDeclinedReasons()
                           ^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/nodepool/driver/__init__.py", line 600, in getDeclinedReasons
      elif not self.hasProviderQuota(self.request.node_types):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/nodepool/driver/statemachine.py", line 520, in hasProviderQuota
      cloud_quota = self.manager.estimatedNodepoolQuota()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/nodepool/driver/utils.py", line 312, in estimatedNodepoolQuota
      nodepool_quota = self.getProviderLimits()
                       ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/nodepool/driver/statemachine.py", line 785, in getProviderLimits
      return self.adapter.getQuotaLimits()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/nodepool/driver/openstack/adapter.py", line 500, in getQuotaLimits
      volume = self._client.get_volume_limits()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/openstack/cloud/_block_storage.py", line 277, in get_volume_limits
      return self.block_storage.get_limits(**params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: 'Proxy' object has no attribute 'get_limits'

We can exclude the broken version for now.

Change-Id: Ib3bac4c47db01a1c5025c07173053495fc493ce9
2023-10-31 09:44:48 -07:00
Ian Wienand 74bc753333 Update to DIB 3.27.0
The major reason for this is Fedora 37 support.

Change-Id: I9413f62e08795359b855c6169b4600a8a5aa7746
2023-03-21 03:41:57 +00:00
James E. Blair f7ed1eb1ea Fix openstack image deletion with newer sdk
Openstacksdk version 0.103.0 removed an informal API we were using
which accepted an abbreviated dictionary as input to the delete_image
method.

It now requires either a complete image object or just a name_or_id,
so we now pass in the id.

The sdk min version is increased since older versions have not been
tested with this.

Change-Id: I7df276ab76e9b8fc17612853b474fec414dae977
2022-12-13 15:48:16 -08:00
Clark Boylan 741c545896 Unpin openstacksdk
This unpins openstacksdk and has nodepool use the latest version. We do
this becaues 0.61.0 and older are not compatible with python 3.11.
0.99.0 and newer should be, but they may introduce compatibility issues
due to major rewrites in the SDK.

We know 0.99.0 through 0.102.0 do not work due improper handling of
nova api microversions when setting network information during server
creation. 0.103 fixes this and I've tested the fix with the proposed
openstack driver rewrite to use statemachine with openstacksdk 0.103.0
and it appears to function against rax and iweb. While it is possible
there are other issues say in image uplodaing the impact htere is lower
and we can always revert if necessary.

Change-Id: Iace27698a91aa2e4b5f169f8a2c1613e5dd37356
2022-11-30 13:42:24 -08:00
Ian Wienand d292376db2 Update DIB to 3.23.1
This point release just has a fix for installing the bootloader on
Rocky 9.

Change-Id: I418055eec2352e603e21010d91005135b920b61d
2022-08-22 08:34:17 +10:00
Ian Wienand 37676aa23c Bump dib to 3.23.0
This includes fixes for Rocky 8, new support for Rocky 9, updates to
openEuler 22.03 LTS and fixes for downloading Ubuntu images.

Change-Id: If5ef0d7c15a4a661042a6c89a9d175864645e0d9
2022-08-03 09:51:02 +10:00
Ian Wienand b5b17a8b03 dib: Bump to 3.22.0
This is to get fresh container images that contain dib 3.22.0.  The
main things of interest in this change are Fedora 36 support and some
fixes to long-standing issues with the Gentoo bootloader installation.

Change-Id: Id4fc7358aee20c1c38ebc179ebb0cdb63fe10c6e
2022-06-27 10:39:09 +10:00
Jeremy Stanley c971179937 Temporarily pin OpenStackSDK before 0.99
The OpenStack SDK/CLI team made an experimental "release candidate"
with 0.99.0, and OpenDev observed errors with the format of network
specifications in API calls. Pin to an earlier version for now, in
order to avoid this and any other as of yet undiscovered problems in
that version.

Change-Id: Icc9adcbfae7a37c335ce4586741cb27d6e0f5c66
2022-06-01 20:27:04 +00:00
Ian Wienand 46c9e01ab8 dib: bump to 3.21.0
This release has fixes for centos7 building and dhcp networking
bringup.

Change-Id: I9024adc559455bf165ff6e41eb2cb3473fcf18fe
2022-05-04 09:43:20 +10:00
Zuul 8072773464 Merge "Update openshift dependency" 2022-04-08 17:00:43 +00:00
Clark Boylan 6df7e5284a Update DIB to version 3.20.2
This newer version of DIB brings better btrfs support, containerfile
improvements, https ubuntu image downloads and RHEL/CentOS 9 EPEL
support.

Change-Id: I164cbfa7b71641288e880119b92515c505dafed3
2022-04-07 14:40:43 -07:00
Clark Boylan 8efabbf1ac Update openshift dependency
This updates the openshift client library dependency which also updates
the kubernetes client library. Do this to ensure we keep the delta with
upstream client library updates small.

Change-Id: I43e7e615f5a42b9d11114ea44949d711eec54504
2022-04-06 11:03:23 -07:00
James E. Blair b13feb7ae8 Add a mininum version for ibm-cos-sdk-core
We recently had a build where pip chose an old version of this on
arm64 for some reason that we do not fully understand.  To avoid
creating broken builds in the future, set a minimum version.

Change-Id: Ic745a10f4bdaa7a2bab3e0826cfa33cf70bc92fe
2022-03-19 09:47:04 -07:00
Zuul 3bdd185904 Merge "Add IBM Cloud VPC driver" 2022-03-15 16:23:38 +00:00
James E. Blair 50bc4cea49 Add IBM Cloud VPC driver
This is a driver for the IBM Cloud VPC service, which has a
new and distinct API.

Change-Id: I7de7297138f5f50380840e4eef43600f9a761181
2022-03-15 06:49:57 -07:00
Ian Wienand d8c8c35069 Update to DIB 3.19.1
This has a fix for networking on Rocky Linux and RHEL9 bootloader
fixups.

Change-Id: I4f3f84250c6f22e1ce92206c5dbbea8f208708d0
2022-02-28 07:48:15 +11:00
Ian Wienand 21336dbafa Update to DIB 3.19.0
This has build fixes for Rocky Linux, ARM64 bootloader and Gentoo

Change-Id: I7b5d96f96f9ae9434791ec2cc5b271997b427836
2022-02-22 21:19:49 +11:00
Ian Wienand dec0e15a03 Bump diskimage-builder to 3.18.0
The major feature of this release is support for Rocky Linux via the
containerfile element.

Change-Id: I7b0963f9a6ce10484202b85c50517bec18c9c396
2022-02-10 16:31:39 +11:00
Ian Wienand a971b7bd25 Bump DIB to 3.17.0
This latest release has better support for latest Fedora

Change-Id: I12fe3f1fd41af8c7a9bbd6ce05f5537536306243
2022-02-01 20:26:37 +11:00
James E. Blair 11506163a3 Stop running siblings jobs and pin SDK
The OpenStackSDK project is in the process of making a breaking
API change and has stopped running this job on master.  Without
an interest in adjusting Nodepool to match before the release,
it is no longer useful to run this job on this repo.  Stop running
it until the sdk project is ready to re-establish co-gating.

We also set a constraint to avoid installing SDK 1.0.0 since we
expect that to include the breaking API change.

Change-Id: If9f45e24a71f349a85e94150e6a4d9ee9672173b
2022-01-26 14:19:58 -08:00
Ian Wienand e2bd2eb296 Bump DIB to 3.16.0
The most important thing in this release is [1] which fixes bootloader
issues with CentOS and Fedora hosts (see also
Ic15487f61a8d5f4c0c8f1941815d9649ed730add for enhanced testing
relating to this in this repo).

[1] https://review.opendev.org/c/openstack/diskimage-builder/+/818851

Change-Id: I759de000bbd65ea1cc02c062186b417c000d2863
2021-12-03 19:10:29 +11:00
Zuul c681315fe0 Merge "Bump DIB to 3.15.2" 2021-11-17 20:52:46 +00:00
Ian Wienand 51c55ad261 Bump DIB to 3.15.2
This point release has a single fix that corrects the way
source-repositories references cached directories with git
(Iadb23454e29d8869e11407e1592007b0f0963e17).

Change-Id: I2d9e3b7949cf005afb8307453791188177b63e36
2021-11-17 08:05:22 +11:00
Zuul 8ed1acef33 Merge "Bump DIB to 3.15.1" 2021-11-11 22:59:08 +00:00
Ian Wienand 0d1f1b0f7a Bump DIB to 3.15.1
This has a fix for extracting tarballs when using the containerfile
element.

Change-Id: I380bd1571a72d6c8c55dcfed5a17d9c400ad04c9
2021-11-11 09:16:14 +11:00
Clark Boylan 1dbb4746d8 Fix boto deps
boto3 1.20.0 depends on botocore 1.23.0 which depends on
urllib3>=1.26.0. Unfortunately they don't record that properly in their
setup.cfg which allows them to install using urllib3 1.25.11 which is
valid in our old requirements. This then fails at runtime due to an
invalid import:

  https://github.com/boto/botocore/issues/2562

Fix this by bumping up urllib3 then bump boto3 to match.

Change-Id: I941c252f9a3bb07eca2c52b18a5f2d609394e0c4
2021-11-08 14:06:31 -08:00
Ian Wienand e75630d5b6 Bump DIB requirement to 3.15.0
This update is required to build centos 9-stream and Fedora 35, and
fixes a Gentoo bootloader issue.

Change-Id: Ia1c83b4b7803c4b094800e02f5122ec89c0614cc
2021-11-04 07:06:00 +11:00
Clark Boylan 169c7ccafa Fix openshift dep for arm64 docker image builds
We tell pip to prefer wheels if present when building arm64 images. We
do this because building wheels for some package installs is extremely
slow on qemu emulated arm64 image builds. Unforatunetly openshift 0.0.1
is the only openshift package version to have a wheel built for it. This
means that we install 0.0.1 which is not compatible with our current
nodepool driver.

This could be a problem for other installations with older openshift
libraries as well as pip won't auto update them. To fix this correctly
we set the lower bound for the openshift library to 0.11.2 which is what
we are using on x86 and is expected to work.

Change-Id: Ida4a27b66fd875166dce2880c7a3c8e0e8aafe94
2021-11-02 11:26:01 -07:00
Ian Wienand d6ab8ac2cc Require DIB 3.13.0
The important change here relates to mounting /sys as RO in chroots
during build, which signals to things such as udev/systemd that you're
in a container and avoids some packages hanging when they are
installed under docker.

Change-Id: I88ffe04ad3b2f5aa938879126093efa0b1bfc6e3
2021-07-23 14:08:39 +10:00
Ian Wienand 0b61858497 Update dib to 3.12.0
Trigger a rebuild of nodepool images with DIB 3.12.0.  The main reason
is for people to use the containerfile element to build later Fedora
releases.

Change-Id: Iaeab9f27296c4432071fd2d8649b78b81efe656b
2021-06-09 07:14:09 +10:00
Ian Wienand 341bf727d5 Bump dib to 3.11.0
This includes fixes for Debian Bullseye image creation

Change-Id: Idaa1b68640890c7762ddf7b7c29c916acf11b9fb
2021-04-30 21:25:16 +10:00
Ian Wienand 84a780f669 Require dib 3.10.0
This version enables building Debian Bullseye images

Change-Id: Ie12bf4c0ace0731b2e8f85f210d43b04a91585d6
2021-04-20 08:16:19 +10:00
Ian Wienand 9c83e8a95e Require dib 3.9.0
Remember how Ib36483bbed95a04fb6a0e656b1890138c8002203 had some fixes
for ARM64 building? ... well this has more to fix the bootloader
installation :)

Change-Id: I4ce8f4af8c7cdd4610a723ccd8379982c9ad4cc3
2021-04-08 19:12:22 +10:00
Ian Wienand 96f58b8379 Bump dib requirement to 3.8.0
This includes some fixes for ARM64 builds, particularly for grub
installation on centos-8-stream.

Change-Id: Ib36483bbed95a04fb6a0e656b1890138c8002203
2021-03-31 19:28:27 +11:00
Ian Wienand bd6dea8f97 Require diskimage-builder 3.6.0
Amongst other things, this has fixes for building Gentoo images which
we would like in the production containers.

Change-Id: Icae1c2964cac875ea0ba2bea19f780acae12a612
2021-01-21 09:35:42 +11:00
Clark Boylan e7f831c34e
Bump openshift dep
The openshift library has been completely redesigned with recent
releases so bump the dep and adapt to the new api. The update is
necessary in order to fix a urllib3 version conflict [1].

[1] Trace:
ERROR: nodepool 3.14.1.dev3 has requirement urllib3<1.26,>=1.25.4, but you'll have urllib3 1.24 which is incompatible.
ERROR: kubernetes 8.0.2 has requirement urllib3>=1.24.2, but you'll have urllib3 1.24 which is incompatible.
ERROR: botocore 1.19.30 has requirement urllib3<1.27,>=1.25.4; python_version != "3.4", but you'll have urllib3 1.24 which is incompatible.

Change-Id: Ia4d09fd0a4a49d644bb575b74184de930c62ce89
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
Story: 2008427
Task: 41373
2021-01-11 17:26:31 +01:00
Ian Wienand 31b72548fd Require dib 3.5.0
This is the latest DIB release with CentOS fixes; require it to
rebuild docker images.

Change-Id: I554ddc4979c3ab9a2635b304c075d7842f43b94b
2020-12-17 11:23:40 +11:00
Ian Wienand 36666fbb2f Bump diskimage-builder requirement to 3.4.0
The most important fixes in this DIB are fixes for CentOS builds
(regular and stream).

Change-Id: I3b10aee8f93d30c00f57ece1a2a619a4d75c488c
2020-11-24 14:31:34 +11:00
Albin Vass 2eba3a4c26 Fix urllib3 dependency
requests 2.24.0 requires urllib3 < 1.26

Change-Id: I16bd4e1f41cb25114cdedd2e0b8068908d9acc99
2020-11-10 22:26:59 +00:00
Daniel Pawlik 7e5c511a37 Set mininum urllib3 version
There is a problem on executing unit tests: "ImportError: cannot import
name 'IPV6_ADDRZ_RE'". To avoid such error, the best way is to set minimum
version of urllib3 [1].

[1] https://github.com/boto/botocore/issues/2186#issuecomment-712467507

Change-Id: Ie1eaee065959068c3967e0c22cfda07571090356
2020-10-21 14:21:33 +02:00
Clark Boylan 02ddf7fc04 Cap azure dep versions due to breaking changes
Azure has made breaking changes to how authentication/credential objects
work as well as exceptions for requests. Unfortunately the azure.common
code base which we use to load up configs from disk to be used as
credentials doesn't seem to have been updated along with the other
tools.

Cap the other tools so that our use of azure.common remains happy.

Change-Id: Ibdc50fbd83b58de756d5cc8ad1009aa3f91fdbcd
2020-09-18 09:21:19 -07:00
Ian Wienand 05c300863c Bump dib requirement to 3.2.1
diskimage-builder 3.2.1 incorporates an important change
Ifb18c4243aa08055d98602aa4fdb30b16dae061d that reverts the inadvertent
removal of git from the base images, which causes all sorts of
interesting issues with zuul-jobs which wants git before even
package-management is setup on hosts.

Change-Id: Ide4b617f419b57ca8be53bc36b1c506fd399d3e6
2020-08-21 08:04:44 +10:00
James E. Blair 898bfe1c6f Require kazoo 2.8.0
This has a fix required for using TLS.

Change-Id: I6d29ca3bf31b1f50cebc245d3db2095ab65921c4
2020-07-15 13:49:46 -07:00
Graham Hayes c1a914fa4a Implement an Azure driver
This change adds an Azure driver.

Supports:
    * Public IPv4 address per VM
    * Private IPv6 address per VM (optional, and not useful yet)
    * Standard Flavors
    * Resource Tagging (for billing / cleanup)

Change-Id: Ief0f8574832df69db472d8704ea3710bc6ca5c59
Co-authored-by: Tristan Cacqueray <tdecacqu@redhat.com>
Co-authored-by: Tobias Henkel <tobias.henkel@bmw.de>
Signed-off-by: Graham Hayes <gr@ham.ie>
2020-06-15 19:57:11 +01:00
Ian Wienand e998607471 Update dib to 2.36.0
The most important change here is fixes for building SuSE with
pip-and-virtualenv; something that has been failing for some time.

Change-Id: I90328693f3ad45f44bec72fb9e72f45ac3be6790
2020-04-28 10:31:38 +10:00
Ian Wienand 54d1d31814 Update dib dep to 2.35.0
This includes support for Fedora 31 and a new python-stow element

Change-Id: Ic521e8cba6d362df53bc24997f06600b30f44e32
2020-03-31 09:49:56 +11:00
Jeremy Stanley 0c9b96bf62 Declare support for Python3.8
Replace our tox-py36 job with tox-py38, extend the list of trove
classifiers for Python versions in package metadata, and replace the
"py35" in the tox.ini envlist with just "py3" so that folks running
`tox` unqualified on their systems will use whatever python3
interpreter they have on hand (odds are it's in our supported range
these days). Also uncap python-daemon so we use a version compatible
with Python >=3.8.

Change-Id: Ic464eefbd90ffaa2ef9079a23e7b15c167de3103
2020-03-11 20:17:45 +00:00