Our current approach to enforce the disk limit per job can be very
expensive by running 'du' in a loop. When having many repos in the
cache and many running jobs this can poison the cache and induce a
large amount of IO load. This can influence overall performance
especially if zuul is running on a shared storage like ceph.
Change-Id: Ic03168e30e0cba4a4adb42eebf4709ceba0d8c3e
There are a couple places where we appear to leak tempfile and tempdirs
because we are creating dirs/files in the tests then not deleting them
when we are done. Address this by using the nested tempfile fixture
which will nest all tempfiles and tempdirs in a tempdir that the fixture
cleans up when the test is completed.
Change-Id: I2818ed48823c544bb1be6b4e1e58f78109197bc1
Depending on your filesystem and kernel you may not actually sync things
to disk quickly enough in the tests to cause the disk accountant tests
to pass reliably. Update the tests to sync to disk explicitly so that
the tests are reliable.
This was noticed as being a problem on btrfs /tmp on suse tumbleweed.
Change-Id: I38fce053ee878751540b4c9d4b05863dc5588de8
When running with the test root on tmpfs the test_cache_hard_links
fails because in this case du reports 0 as usage for the link instead
of 1 block(s). Adapt the test for that.
Also honor the ZUUL_TEST_ROOT variable if it is present.
Change-Id: Ie5789299de4b23161d80e5375f8a94be109b67f4
If one of the assertions hit the disk accountant thread is not
stopped. Due to this all further tests in the same batch are broken
during shutdown.
Change-Id: I9534c206bbb2e20a2bc44eba5317d136ed422957
If a job were to be pointed at an abnormally large git repository (or
a maliciously large one), a clone would fill the disk. Or anything else
that might happen that writes data onto the executor disk.
We run a single thread periodically running du on the root of all jobs
on this executor. This is called the DiskAccountant.
We set a config item per executor of the limit per job. This won't
actually save a server from a full disk if many thousands of concurrent
changes are submitted, but this will prevent any accidental filling of
the disk, and make malicious disk filling much harder.
We also ignore hard links from the merge root, which will exempt bits
cloned from the merge root from disk accounting.
Change-Id: I415e5930cc3ebe2c7e1a84316e78578d6b9ecf30
Story: 2000879
Task: 3504
Tobias Henkel