As a first step towards minimizing code duplication between the
various upload-logs roles move the uplaod modules into a common role
upload-logs-base. For easier review common code will be consolidated
in a followup change.
The google and s3 variant missed the unicode fix that swift log upload
received. Add this to make to make the test cases work with the same
fixtures.
Change-Id: I2d4474ae1023c3f3e03faaa5b888e705ee2ed0bc
We are facing some issues where the log upload to swift fails, but the
role is always succeeding. To get some more information about the
upload failures, we let the upload() method return those to the Ansible
module and provide them in the module's JSON result.
The equivalent change in the test-upload-logs-swift [1] role is
validated in [2].
[1] https://review.opendev.org/#/c/735503/1
[2] https://review.opendev.org/#/c/737441/
Change-Id: Ie0d4ea2f3365600eae0e572e4c0790b131d3b13e
This reverts commit acde44818d and
the testing part of b3f417a6e6.
We'd like to obtain more consensus on the download script before
we commit to this. In particular, the new zuul manifest file may
make it possible to do this without adding the feature to the
log upload roles.
Change-Id: I959c44b4dac6cad6d1b3d82ba6bc0949c9c759ff
After we have determinted the root URL, create a download script from
a given jinja2 template. This is added to the file list at the root
and uploaded with the other files.
Generated index files are given a new flag so they can be
differentiated.
This is an impelementation of
Iea8abd4cd71ece26b51335642f73bd2e544c42dd for the swift-upload role.
Change-Id: I98c80f657f38c5e1ed5f28e5d36988a3429ad1f8
When retrieving gzipped files out of swift with gzip encoding set some
swift implementations return a decompressed version of the file if your
client cannot accept gzip encoding. This causes problems when the file
you want is actually compressed like a .tar.gz. Instead we avoid setting
the encoding type of these files forcing swift to give them back as is.
This change should only be approved after confirming its parent is
tested and working.
Change-Id: Ibcf478b572ba84273732e0ede17bf92bddd8c36f
By doing this, we're not constrained about where to run the uploader
while still providing some useful testing in dry-run mode.
Change-Id: Ie4888606a8ca4ffe2eb99ddbbcd9d5cee8ceec44
We've discovered that rackspace swift seems to always want to gzip
encode files when clients request their contents. When our files are
deflate encoded this results in files that are first deflate encoded
then gzip encoded. Not all browers or layer 7 firewalls can handle this
(despite being perfectly valid according to the HTTP RFCs). We'll use
gzip to see if that causes rackspace to not double encode the files.
To do this with memory efficienty we vendor a tool from pypi called
gzip-stream which allows us to read chunks of the compressed data at a
time without loading the entire file into memory or writing multiple
gzip headers in a single file.
Change-Id: I9483cfdbd8e7d0683eeb24d28dd6d8b0c0e772fa
The href url paths need to have quoted filenames to handle cases where
filenames have special characters like : in them.
Change-Id: I0bc0de8d27c6e45c4a6b8841985b8265f0219df2
We have been getting HTTP 401 unauthorized errors at the rate of about
once a day when trying to get containers in the swift logs role.
Manually getting and posting objects to the same container after the
jobs fail seems to work so this appears to be a transient failure.
Attempt to workaround this by retrying the container get calls several
times.
Change-Id: Ia7395ffa0b120fbbecde0c9bb6e8583078167143
The old code will log swift upload tracebacks via logging.exception()
which doesn't seem to bubble back up into ansible's logging. We address
this by using traceback.format_exc() to format an exception traceback
string which we pass to ansible's module.fail_json().
Change-Id: I524bd0d5a9529011cffb6d09866b22b2c97fab7d
There may be broken symlinks within the log directories, those fail with
an error when os.stat() is executed on them. So if/else is replaced with
try/except while TypeError used to catch when self.full_path is None
Change-Id: Iffee97760a39fa4f7760bd67fb63c5f0905064bd
Import keystoneauth1.exceptions to access the exceptions. HttpError
also lives under "exceptions.http.", so update that reference.
Change-Id: I4afe8c9fc8239a31d62a2a1d09794211b5066472
This adds a generic retry handler that takes a callable, and uses
that not only for the exsting retries around the POST call, but
also the actions when creating the container.
Change-Id: I910b8e886f107d4fe38a9334ba836f010f92557c
If we get an unexpected exception, it's helpful to know the
cloud and region involved. It's also nice to return that information
for maybe better readability.
Change-Id: I1c589744103512d981e64e1a3f9506d40e1bf4cf
The CDN setup that rackspace has requires that the
Access-Control-Allow-Origin header be added to each object uploaded.
Change-Id: I0f6af613e2ebcf9cfe835dc8018a73922f9f0ed5
When creating containers, add a CORS header which allows access
from any host, so that the zuul web app can fetch.
Change-Id: I013265643a8fcb4cff001375136c2c37958fd97a
Rackspace only provides unauthenticated access to object storage
via CDN in a non-openstack-standard way, so we need to do some
extra work to support that.
This also adds a helper script for testing which deletes a container
(since in order to do so, you may need to delete all the contents
first).
Some commented-out debug configuration lines are added for the
convenience of future developers.
Change-Id: I3d1fce824fb40136048f0988939d22f755236a59
This converts the Indexer class from something that strictly generates
index.html files for folders to a more generic class that will be able
to hold other types of transformations we might like to do on the
total collections of uploaded files.
The index.html specific arguments are moved into make_indexes() and
the two helper functions that should not be called externally are
renamed private.
Change-Id: I388042ffb6a74c3200d92fb3a084369fcf2cf3a9
Add a function to the FileList context manager to get a temporary
directory; keep track and remove these on exit. Use this in the index
creation.
Change-Id: I9d9220ad70ce191af02ae0331c98eafe487d96d4
We currently try to skip size formatting for folders. However we
compare with a bogix mimetype so the check is false in every case.
Further folders typically have a size of 512 bytes or 4k. In both
cases we don't really need to skip the size formatting so instead of
fixing the check just skip it and do the size formatting
unconditionally.
Change-Id: I7ef021381bb56acf4b22551cc5d5613470fd6d08
Moving this reference to the file_list into the class encapsulates
things better when the Indexer class becomes a more a collection of
tools to modify a FileList before upload.
Change-Id: I2bedee35ce178df40c15d5867edf560a62232c57
The FileList is dynamic and needs to be able to keep track of things
to cleanup. For example, it has index files added to it from
temporary files which should be removed up when we're finished with
the list. In a future change we propose a similar addition of a
download script for logs which should also be managed.
Turn the FileList into a context manager. Modify the index generation
to not create a new FileList, but just replace the internal list. Use
this for the life-span of the upload by wrapping the relevant parts in
a "with:" statement.
Change-Id: I7135bf5a55d133ce146e9aa84f00041fc8125cbc
With URLs that may include any number of directory levels before
even the prefix of our upload, it's difficult (though not impossible)
to upload the icons to a fixed location in the container and
reference that location. A more self-contained approach is just
to embed the icon data directly into the HTML. That is what is
done here.
Change-Id: I12342aa479bac41eb3b401d1e92689a56b3c2a2b
Tobias Henkel