Commit Graph

145 Commits

Author SHA1 Message Date
Clark Boylan 5702331087 Move nodepool functests to podman
Now that nodepool images are on quay.io we don't get speculative
container image testing with docker. The reason for this is docker only
knows how to lookup images hosted by docker.io in mirrors which
specualtive container image testing relies on. Since the images are
hosted on quay.io instead of docker.io we lose this functionality.

Address this by switching to podman and podman-compose which does
understand how to fetch images with mirrors from any location.

Depends-On: https://review.opendev.org/c/zuul/zuul/+/687135
Change-Id: I1a510a9b68a2f01098f3c099a129d6d268b422d9
2023-05-30 15:35:47 -07:00
James E. Blair 1baa3193f4 Output rate information in load-test.py
This outputs the overall request rate in load-test.py.  This can
help when trying to optimize driver performance.

Change-Id: I30f3d5fbef4f2de7a2ec88e9a767c00d99027863
2023-02-28 06:51:37 +00:00
Benjamin Schanzel 94719265d8 Adapt load-test to new zk structure
The zk handling and structure was reworked some time ago to match that
of Zuul

Change-Id: Ie967ce08f2467b1993291d48f3cc80e410e8fda7
2023-01-10 10:30:14 -08:00
Jeremy Stanley dfdf9d3684 Run docker-compose up test setup script with env
When not using a ROOTCMD when running the test-setup-docker.sh the
script will fail with the following error message:

    ++ id -u
    + USER_ID=1000 docker-compose up -d
    ./test-setup-docker.sh: line 50: USER_ID=1000: command not found

Due to Bash's simple command expansion[0] variable assignments will
be interpreted as the command when no ROOTCMD is given. To work
around this we use default ROOTCMD to the `env` command.

[0]: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Simple-Command-Expansion

Change-Id: I3e4dd844efae8971d9ba83060a7134f1f1f0e770
Co-Authored-By: Simon Westphahl <simon.westphahl@bmw.de>
2022-08-23 12:38:17 +00:00
Jeremy Stanley d23662e50f Update unit test container setup and instructions
Update the TESTING.rst file to reflect that just installing and
starting zookeeperd is no longer sufficient now that we require TLS
and auth for the connection. Suggest the container-based setup
script instead. Also improve that script to allow it to be invoked
as a normal user, with root command escalation tool choice (e.g.
"sudo") supplied through a ROOTCMD environment variable, so that
things created inside the git worktree like the CA don't end up
root-owned.

Change-Id: Ia00f5514ba0bd97b2ef17fb798b7c2dc68d9a55e
2022-08-05 18:27:46 +00:00
Clark Boylan bde9bdb975 Update to python3.10
This adds python 3.10 testing and updates our docker images to
python3.10.

On the docker image side of things we use opendev's python images which
are based on Debian then have python compiled on top of that external to
Debian python packaging. Debian bullseye ships with python3.9 but our
images come with python3.10. What this means is that we cannot rely on
external wheel caches for Debian bullseye + python3.9 here as we've got
the wrong version of python3.10. The good news is that all of the
dependencies that have historically given us trouble on arm64 in
particular already ship python3.10 aarch64/arm64 wheels. This means we
can rely on pypi as is. This is probably better for us anyway as it
decouples us from relying on additional external resources.

We also update a number of jobs to use nodeset: ubuntu-jammy as this
ubuntu version defaults to python 3.10.

Change-Id: I7fb585bc5ccc52803eea107e76dddf5e9fde8646
2022-07-01 12:05:26 -07:00
James E. Blair 7bbdfdc9fd Update ZooKeeper class connection methods
This updates the ZooKeeper class to inherit from ZooKeeperBase
and utilize its connection methods.

It also moves the connection loss detection used by the builder
to be more localized and removes unused methods.

Change-Id: I6c9dbe17976560bc024f74cd31bdb6305d51168d
2022-06-29 07:46:34 -07:00
James E. Blair 10df93540f Use Zuul-style ZooKeeper connections
We have made many improvements to connection handling in Zuul.
Bring those back to Nodepool by copying over the zuul/zk directory
which has our base ZK connection classes.

This will enable us to bring other Zuul classes over, such as the
component registry.

The existing connection-related code is removed and the remaining
model-style code is moved to nodepool.zk.zookeeper.  Almost every
file imported the model as nodepool.zk, so import adjustments are
made to compensate while keeping the code more or less as-is.

Change-Id: I9f793d7bbad573cb881dfcfdf11e3013e0f8e4a3
2022-05-23 07:40:20 -07:00
Ian Wienand 9a3fb29816 functional-test : ensure glean started
We use glean in the gate tests to enable the network.  Because we are
in a DHCP environment with the devstack environment, it's possible
that glean didn't start but we still manage to get an address anyway
via autoconfiguration.  But the point of having glean is that in
clouds without DHCP things work too.  This is most common with new
distributions, where glean matching may not have updated to configure
things fully.

We could replicate a non-DHCP environment, at some considerable
initial cost of figuring out how to setup devstack to do that and then
ongoing costs of having a fairly bespoke environment.  A simpler path
is to just check the glean services and make sure they started
correctly.

Change-Id: Idb8d67cd2ba6e8c5d7157177e9cfd6be7b99cacd
2022-03-07 14:54:27 +11:00
Ian Wienand e7822cf89a nodepool-zuul-functional: remove
As described in the dependent change, the testing done here is better
done by the quickstart jobs these days.  The dependent change has
removed the tox environment this calls in Zuul.  This removes the job
definiton and related files from nodepool.

Change-Id: I17e1002012e9ac6abc434454af989f1da1c379b7
Depends-On: https://review.opendev.org/c/zuul/zuul/+/826772
2022-01-28 08:20:08 +11:00
Ian Wienand b9977b16f5 functional test: check DIB kernel flags
As noted inline, check the kernel flags on booted images to increase
confidence the bootloader is making generic images.

Change-Id: Ic15487f61a8d5f4c0c8f1941815d9649ed730add
2021-11-25 11:41:09 +11:00
Ian Wienand cce7dbc669 Use bullseye podman in container
After updating images to bullseye
(I21cfbd3935e48be4b92591ea36c7eed301230753) we can use the native
podman packages.  These are slightly older, but should be fine for the
intended usages.

Change-Id: Ica62392ebf4a665a04cd65458dda9e0a7545ccc8
2021-10-28 08:35:19 +11:00
Ian Wienand f5d015c883 Update Docker and bindep for Bullseye base images
Similar to Zuul (I71182e9d3e6e930977a9f983b37743ee3300ec91), the base
images have updated to Bullseye.

This updates various things to get a building Bullseye image.

We have upgrade to 3.9-based images here because OpenDev builds ARM64
wheels for a bullseye+arm64 combo, which we use to speed up the ARM64
cross-build (we do not have any repository of <3.7|3.8>+bullseye ARM64
wheels, so it makes it difficult to use these combos as the
cross-build can take a very long time)

Depends-On: https://review.opendev.org/c/openstack/diskimage-builder/+/806318
Change-Id: I21cfbd3935e48be4b92591ea36c7eed301230753
2021-10-26 11:11:19 +11:00
Ian Wienand a07bb0a0ae Install podman in container for container-based builds
This installs podman inside the nodepool container, which is used by
the dependent change in DIB to extract initial chroot environments
from upstream containers.  This eliminates the need to run non-native
tools on build hosts (rpm/zypper on Ubuntu, etc.).

As noted in the config, podman defaults to assuming systemd is
installed and using various systemd interfaces.

Additionally, we map the a volume into the container which allows
nested podman to do what it needs to do.

Needed-By: https://review.opendev.org/700083
Change-Id: I6722aa2b32db57e099dae4417955a8a2cd28847e
2021-05-05 10:34:42 +10:00
Tobias Henkel c1e5db76d7
Add simple load testing script
This adds a simple load testing script that basically maintains a
stable number of pending node requests. Combined with max-ready age
this can be easily used for load testing instance creations and
deletions.

Change-Id: I2f754e88fdc541914f929511c713a43eb910a344
2021-04-07 17:22:46 +02:00
James E. Blair d921fc2622 Add test-setup-docker.sh
This is intended as an aid for developers (since we have moved the
ZK setup which was in test-setup.sh to a playbook for tox jobs).

Change-Id: I9ca03831a74928ec6875c5f6668cfcfcdedb37fd
2021-02-24 17:21:01 -08:00
James E. Blair 4c5fa46540 Require TLS
Require TLS Zookeeper connections before making the 4.0 release.

Change-Id: I69acdcec0deddfdd191f094f13627ec1618142af
Depends-On: https://review.opendev.org/776696
2021-02-19 18:42:33 +00:00
Clark Boylan 2e25ea65fe Use rax.dfw mirror for arm64 wheels
We had been using the linaro mirror to get arm64 python wheels.
Unfortunately this mirror has been having some reliability issues. The
wheels themselves are served from all our mirrors which means we can
switch to rax.dfw which should be more reliable.

Change-Id: I3da953fe49d8e4600d4835224a33ab558af88a06
2020-09-17 12:28:51 -07:00
Clark Boylan ed42e5535b Build arm64 images using arm64 wheels from openstack if available
We updated python-base and python-builder to include arm64 images in
support of nodepool's arm64 python-builder image. In doing so we have
discovered a number of issues, but the biggest is slowness of building
python packages in an emulated environment.

In order to speed up package builds we consume the OpenDev linaro
cloud arm64 wheel cache. This doesn't have wheels for every package we
need, but for the things that it does have it will speed up our builds.

One of the risks with this setup is that we're relying on wheels built
for openstack on arm64 and those follow openstack's contraints. In order
to mitigate this risk we set pip install's --prefer-binary flag in the
pip.conf. This means that if openstack's constraints lag what is
availale on pypi we should use the existing wheels as long as they are
valid version according to requirements rather than trying to build from
sdist.

Co-Authored-By: James E. Blair <corvus@inaugust.com>
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I3b358721eebbceafc12daf9d706306634048b196
2020-09-03 14:02:21 -07:00
Ian Wienand 043a4258fc Dockerfile: use local openstack-ci-core PPA key
We see timeouts trying to get this key fairly frequently in the gate.
Store it locally and use that in the container build.

Change-Id: Ifd706849f1fad88c8ec4afc79090df4afb88abb4
2020-04-30 17:32:02 +10:00
James E. Blair 0c07364c5b Stop checking user_data in func test
python-novaclient has stopped outputting this information:

03dca4bc82

Change-Id: I71dac1f6c74299ddbc08ddd5092b23c68e1a6ec7
2020-04-14 16:03:53 -07:00
Monty Taylor 6da36e8a3e Revert abitrary uid support
This code was already reverted in the zuul images, it doesn't
actually provide the value is claims to add and it breaks the
running under podman.

Revert "Dockerfile: add support for arbritary uid"

This reverts commit da2701e0b1.

Revert "Dockerfile: add user to shadow file too"

This reverts commit 747e957263.

Change-Id: Iff606c65c6a3223f13d963d90455fa895193cce8
2020-03-16 14:20:42 -07:00
Ian Wienand 809cada20a Functional tests - use common verification script
There is a lot of logic in the check.sh script of the openstack
functional tests.  Extract into a single location in /tools and call
it from the install and container tests.

Change-Id: Ib5728f5cee917c73d0da276d36da5776dee279fc
2019-12-16 12:37:19 +11:00
Ian Wienand 747e957263 Dockerfile: add user to shadow file too
Without an entry in the shadow file, this user can't use sudo with the
following error:

 account validation failure, is your account locked

(which I include here for future googling because it's pretty obscure,
you have to have this odd situation, or a pretty broken PAM to see it).

The "nodepool" user (10001) is in the root group, which is why the
uid_entrypoint script can update the /etc/passwd file.  We need to
change the ownership of the /etc/shadow file for this to work.  It
feels a bit weird, but there's no password to actually guess anyway.

Change-Id: I8846757edffe31f96df58999d05727910c9fca43
2019-12-05 10:27:22 +11:00
Ian Wienand 9367cf8ed8 Add a dib-cmd option for diskimages
This change allows you to specify a dib-cmd parameter for disk images,
which overrides the default call to "disk-image-create".  This allows
you to essentially decide the disk-image-create binary to be called
for each disk image configured.

It is inspired by a couple of things:

The "--fake" argument to nodepool-builder has always been a bit of a
wart; a case of testing-only functionality leaking across into the
production code.  It would be clearer if the tests used exposed
methods to configure themselves to use the fake builder.

Because disk-image-create is called from the $PATH, it makes it more
difficult to use nodepool from a virtualenv.  You can not just run
"nodepool-builder"; you have to ". activate" the virtualenv before
running the daemon so that the path is set to find the virtualenv
disk-image-create.

In addressing activation issues by automatically choosing the
in-virtualenv binary in Ie0e24fa67b948a294aa46f8164b077c8670b4025, it
was pointed out that others are already using wrappers in various ways
where preferring the co-installed virtualenv version would break.

With this, such users can ensure they call the "disk-image-create"
binary they want.  We can then make a change to prefer the
co-installed version without fear of breaking.

In theory, there's no reason why a totally separate
"/custom/venv/bin/disk-image-create" would not be valid if you
required a customised dib for some reason for just one image.  This is
not currently possible, even modulo PATH hacks, etc., all images will
use the same binary to build.  It is for this flexibility I think this
is best at the diskimage level, rather than as, say a global setting
for the whole builder instance.

Thus add a dib-cmd option for diskimages.  In the testing case, this
points to the fake-image-create script, and the --fake command-line
option and related bits are removed.

It should have no backwards compatibility effects; documentation and a
release note is added.

Change-Id: I6677e11823df72f8c69973c83039a987b67eb2af
2019-08-22 10:09:00 +10:00
Sagi Shnaidman 635ff11055 Fix nodepool container failure
Nodepool container fails with error message:
whoami: extra operand ‘/dev/null’
Try 'whoami --help' for more information.

Change-Id: I7ef5b6527eb08d00b9b27e37b5d5b5dce69bb4ef
2019-07-22 15:07:38 +00:00
James E. Blair f0b4930ee5 Remove devstack plugin functional test jobs
The dependent commits mark the point at which all external usage
of these jobs has been replaced with the new jobs, so these can
be removed.

Change-Id: Ie51f9c177668190a55563fca944b825253d90f3f
Depends-On: https://review.opendev.org/667225
Depends-On: https://review.opendev.org/667992
Depends-On: https://review.opendev.org/667993
2019-06-27 17:21:52 +00:00
Zuul 3412764a98 Merge "Add Debian Buster boot tests" 2019-06-12 20:40:31 +00:00
Tristan Cacqueray da2701e0b1 Dockerfile: add support for arbritary uid
This change adds the uid_entrypoint script to support arbitrary uid
as documented in:
https://github.com/RHsyseng/container-rhel-examples/tree/master/starter-arbitrary-uid

Change-Id: I857308fa1f2363d2b4bc257e86963c36ce1d49e2
2019-04-10 12:20:21 +00:00
Ian Wienand 4379810a07 Add Debian Buster boot tests
Infra has a mirror for Debian Buster now, add boot tests

Depends-On: https://review.openstack.org/649496
Change-Id: Ib1567b2576631c078fe11d0f250aeb4e6f9fa0b3
2019-04-04 10:48:53 +11:00
Sagi Shnaidman d5027ff6a9 Support userdata for instances in openstack
Use "userdata" from Nova API to pass cloud-init config to nova
instances in openstack.

Change-Id: I1c6a1cbc5377d268901210631a376ca26f4887d8
2019-01-22 19:14:52 +02:00
Tristan Cacqueray 7fd6411dac print-zk: convert zookeeper_servers to a list
The print-zk script needs to convert the dictionary iterator to a proper list.

Change-Id: I786799ff881c361071fa06c78541c617fc7461a9
2019-01-10 02:32:09 +00:00
Zuul 2427bfb277 Merge "Add Fedora 29 testing" 2019-01-09 05:54:07 +00:00
Zuul f116826d2b Merge "Fix print-zk tool for python3" 2018-12-01 20:44:29 +00:00
Zuul 40162102a7 Merge "Only setup zNode caches in launcher" 2018-11-29 09:59:41 +00:00
Tobias Henkel 9d77f05d8e
Only setup zNode caches in launcher
We currently only need to setup the zNode caches in the
launcher. Within the commandline client and the builders this is just
unneccessary work.

Change-Id: I03aa2a11b75cab3932e4b45c5e964811a7e0b3d4
2018-11-26 20:13:39 +01:00
Ian Wienand 4422d53529 Add Fedora 29 testing
Replace Fedora 28 with Fedora 29 functional testing.

Note this changes our Red Hat platforms to use NetworkManager for
interface configuration, rather than legacy scripts.  Fedora 29 has
split the legacy scripts into a new package and it marked for future
removal.  NetworkManager is the default on Centos 7 and will also be
on Centos 8, so it makes sense to use it there too.

Depends-On: https://review.openstack.org/619120
Change-Id: I640838c68a05f3b22683c1e90279725a77678526
2018-11-23 17:49:17 +11:00
Tobias Henkel cacedff80e
Fix print-zk tool for python3
The print-zk tool still contained python2 style prints.

Change-Id: Ief9cb9e2c76cf682daf34b57a754d4d26446fdc0
2018-11-22 07:10:11 +01:00
Ian Wienand 7679b2c14b Update devstack test to Fedora 28
Running a bit behind on this transition ... s/27/28/ to update to
Fedora 28.  This is the default in dib now.

Change-Id: I648ab9d9ba4bba7323c432c65f3ef056703f4303
2018-10-31 11:30:00 +11:00
Clark Boylan 674d6d88b4 Run test zookeeper on top of tmpfs
We have been running into what appear to be zookeeper performance issues
causing tests to fail. Run the zookeeper on a tmpfs to reduce the impact
of iops to disk.

Other alternatives include using something like eatmydata to make writes
and syncs fast but unsafe.

Change-Id: Iea5e44af6844281c7f2078da57da9f13691e2642
2018-10-24 08:36:18 -07:00
Ian Wienand 7015bd9af4 Add instance boot properties
This allows us to set parameters for server boot on various images.
This is the equivalent of the "--property" flag when using "openstack
server create".  Various tools on the booted servers can then query
the config-drive metadata to get this value.

Needed-By: https://review.openstack.org/604193/

Change-Id: I99c1980f089aa2971ba728b77adfc6f4200e0b77
2018-09-21 16:29:16 +10:00
Zuul d803ffb717 Merge "Use detail listing in integration testing" 2018-07-23 16:00:34 +00:00
Clark Boylan 218805a20d Use detail listing in integration testing
This helps confirm that all fields are populated properly when reviewing
changes.

Change-Id: Ieb4876bcbf01c03189dd176239e2644256b5306e
2018-07-20 16:50:06 -07:00
David Shrewsbury bade82d446 Fix plugin and examples for using openstacksdk
These appear to have been missed in: https://review.openstack.org/572829

Change-Id: I5c008c369b3789c3ae79ce89726194ab715767a9
2018-07-17 15:02:59 -04:00
Markos Chandras 06bd892248 Add opensuse-150 to nodepool dsvm
openSUSE Leap 15 is the latest version of openSUSE, bring an image
online to validate we can properly build it.

Depends-On: https://review.openstack.org/#/c/572424/
Change-Id: Ib0f48d9788aafd763e857c2d33784c4f75af4c17
2018-06-06 11:13:08 +01:00
Paul Belanger 35d589296c
Remove debian-jessie from nodepool dsvm testing
Now that debian-stretch is working as expected, we can remove
debian-jessie.

Change-Id: If897757023772bb4549e40e7fcd048998175fb5b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-05-04 12:16:34 -04:00
Ian Wienand d187906839 Reduce strictness of growroot test
We are seeing consistent failures where the Trusty root partition is
64MiB shorter than we expect.  Unfortunately we don't currently have a
concrete explanation for this but rounding due to alignment is
suspected.

Reduce the expected size to something bigger than the images, but not
so close to 5GiB.  Also add a more useful failure message; currently
you have to dig back through the logs to see where it went wrong.

Change-Id: Iba1fafdb1fe0f3c1b751772af939f079c429fcf3
2018-04-11 06:51:26 +10:00
Ian Wienand 899fb6e368 Add a backoff for failed builds
If a build has a systematic failure, we currently just let nodepool
run, looping builds until we hit the overall job timeout (1.5 hours).
This adds a count of output log files; if we see three failed builds,
then assume the problem won't get better and fail early.

Change-Id: Id7e163b4937dd57cc8afbf72ed795f73b46a05b1
2018-04-04 14:05:41 +10:00
Ian Wienand 1ad32c2509 Add some more fudge factor for growroot test
It seems that 2MiB of fudge isn't quite enough; Trusty has been seen
to round it's root partition size down and we miss it.  Increase the
fudge factor.

Change-Id: I26e3bc7b5f68ea6642b8b57119fbd286688d593e
2018-04-04 06:12:32 +10:00
Ian Wienand ee78684521
Test growroot in boot tests
Test that we see the root partition grow.

Increase the root disk size to 5gb, and check that the booted vm has
grown the disk to at least that.  Add disk size tracking so we can
more clearly see what's being built into the images.

Change-Id: I377beffc4896e03f0c2d01c0061c5f8652e8b1d1
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-02 13:38:53 -04:00