As a first step towards minimizing code duplication between the
various upload-logs roles move the uplaod modules into a common role
upload-logs-base. For easier review common code will be consolidated
in a followup change.
The google and s3 variant missed the unicode fix that swift log upload
received. Add this to make to make the test cases work with the same
fixtures.
Change-Id: I2d4474ae1023c3f3e03faaa5b888e705ee2ed0bc
We are facing some issues where the log upload to swift fails, but the
role is always succeeding. To get some more information about the
upload failures, we let the upload() method return those to the Ansible
module and provide them in the module's JSON result.
The equivalent change in the test-upload-logs-swift [1] role is
validated in [2].
[1] https://review.opendev.org/#/c/735503/1
[2] https://review.opendev.org/#/c/737441/
Change-Id: Ie0d4ea2f3365600eae0e572e4c0790b131d3b13e
Everything's better with some unicode sprinkled in. Add a unicode
filename to keep unit testing on its toes.
Note this is duplicated across the test role too.
Change-Id: Iaefe9bea2c1a10d440ef75df3acd71fdd9a4157e
These aren't all getting cleaned up, which winds up breaking
the second runs. Instead of doing addCleanup with a method that
does the loop again, which can fail in the middle and not
clean up subsequent files, add an individual cleanup when we
add the symlink. This results in all of the symlinks consistently
being cleaned.
Change-Id: Id5a5b09c830ad2ad3bb0b77fb9dbdc494c629824
The href url paths need to have quoted filenames to handle cases where
filenames have special characters like : in them.
Change-Id: I0bc0de8d27c6e45c4a6b8841985b8265f0219df2
There may be broken symlinks within the log directories, those fail with
an error when os.stat() is executed on them. So if/else is replaced with
try/except while TypeError used to catch when self.full_path is None
Change-Id: Iffee97760a39fa4f7760bd67fb63c5f0905064bd
This converts the Indexer class from something that strictly generates
index.html files for folders to a more generic class that will be able
to hold other types of transformations we might like to do on the
total collections of uploaded files.
The index.html specific arguments are moved into make_indexes() and
the two helper functions that should not be called externally are
renamed private.
Change-Id: I388042ffb6a74c3200d92fb3a084369fcf2cf3a9
Moving this reference to the file_list into the class encapsulates
things better when the Indexer class becomes a more a collection of
tools to modify a FileList before upload.
Change-Id: I2bedee35ce178df40c15d5867edf560a62232c57
The FileList is dynamic and needs to be able to keep track of things
to cleanup. For example, it has index files added to it from
temporary files which should be removed up when we're finished with
the list. In a future change we propose a similar addition of a
download script for logs which should also be managed.
Turn the FileList into a context manager. Modify the index generation
to not create a new FileList, but just replace the internal list. Use
this for the life-span of the upload by wrapping the relevant parts in
a "with:" statement.
Change-Id: I7135bf5a55d133ce146e9aa84f00041fc8125cbc
Tobias Henkel