The isAlive() method of threading.Thread has been removed in Python 3.9.
The is_alive() method is available on Python 2.6+.
See https://bugs.python.org/issue37804
Change-Id: I951b1ae331c3101722fe34babf81d6f82d838380
When running test cases in the debugger it spits out resource warnings
about non-closed streams [1]. Explicitly closing the streams of the
subprocesses and log streamer fixes this warning. Maybe this even
solves the memory leak we're currently seeing in the executors.
[1] Trace:
/zuul/executor/server.py:1894: ResourceWarning: unclosed file <_io.BufferedReader name=60>
self.proc = None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/zuul/executor/server.py:124: ResourceWarning: unclosed file <_io.BufferedReader name=11>
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Change-Id: I65f191dc2e50f9c84f5bf6a3967d768d7ebe6b04
Now that we have a finger gateway, we no longer need to start the
executor as root so that the finger streamer on the executor can
bind to port 79 (default port for the finger streamer is changed
from 79 to 7900). Remove that requirement.
Change-Id: I6df685044c4ce81fd263043adba832609da100af
This adds the zuul-fingergw app that should be run as root (so that
it can connect to the standard finger port 79), but changes user privs
immediately after binding that port.
Common streaming functions have been moved to streamer_utils.py to
be shared among modules.
Support for CommandSocket has been included.
Change-Id: Ia35492fe951e7b9367eeab0b145d96189d72c364
In an attempt to maybe get more information as to why the finger
daemon seems to be disappearing on us, try using threads instead
of forks. Alas, there is no spoon.
Change-Id: I473d874037fb81b940bfe71e5d6d1a17b131635f
Add some exception handling around our internal code and some
logging so we can see when exceptions happen.
Change-Id: Iedd4f4df7f7ac9f9ec6b912131d0bf717f63fe58
zuul now provides socket-based console streaming, which is super cool.
In order to have jenkins parity with web streaming, we need to provide a
websocket (javascript in browsers can't really connect to random ports
on servers)
After surveying the existing python websocket options, basically all of
them are based around twisted, eventlet, gevent or asyncio. It's not
just a thing we can easily deal with from our current webob/paste
structure, because it is a change to the fundamental HTTP handling.
While we could write our own websocket server implementation that was
threaded like the rest of zuul, that's a pretty giant amount of work.
Instead, we can run an async-based server that's just for the
websockets, so that we're not all of a sudden putting async code into
the rest of zuul and winding up frankensteined. Since this is new code,
using asyncio and python3 seems like an excellent starting place.
aiohttp supports running a websocket server in a thread. It also
supports doing other HTTP/REST calls, so by going aiohttp we can set
ourselves up for a single answer for the HTTP tier.
In order to keep us from being an open socket relay, we'll expect two
parameters as the first message on the websocket - what's the zuul build
uuid, and what log file do we want to stream. (the second thing,
multiple log files, isn't supported yet by the rest of zuul, but one can
imagine a future where we'd like to support that too, so it's in the
protocol) The websocket server will then ask zuul over gearman for the
IP and port associated with the build and logfile and will start
streaming it to the socket.
Ultimately we'll want the status page to make links of the form:
/console.html?uuid=<uuid>&logfile=console.log
and we'll want to have apache map the websocket server to something like
/console.
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
Change-Id: Idd0d3f9259e81fa9a60d7540664ce8d5ad2c298f
The current streaming code will never terminate the streaming (unless
the client initiates the termination). Let's change it so that we stop
streaming when the log file is removed (job finishes). Also, no need to
check for log truncation since that doesn't happen for these logs.
Change-Id: I73f04ffe998e88b89aa660be05e645d3defb87cc
ansible-playbook uses basicConfig at the top of the command, which
essentially takes over everything if you send things to log_path.
Instead, go ahead and log things ourselves so that our log lines can all
look consistent (and so that we can easily html-ify) We pass in the
location via an environment variable since we're not using the log_dir
option in ansible.cfg.
As of this patch, there will be less output. A follow up will add back
the missing callback plugin bits.
Change-Id: Ic3ff57bba7a3a23dc5d0055e8e9888f24641f7d5
This lets us protect against long commands (as before), as well
as set a 10 second timeout for a client to send a job uuid for
log streaming.
This should fix a potential hang in the test runner where a
request handler process would stick around forever.
Change-Id: I8c1f21c349e72892237b3b93d867ebbd41f77b0a
For some reason, it can (rarely) happen that the socket used by the log
streamer is already closed when we are in server_close(). Since
our intent is to close it anyway, do not raise an exception on an
already closed socket.
Change-Id: I5828f6351341bac5e55ae2211e8a3b3f94032899
This will be started along side the the executor process, similar
to how the scheduler and gearman server are linked.
This will require starting the process as the root user so that we
can grab the finger port properly. The process listening on the
finger port will drop its privileges to the designated user after
grabbing the socket. The executor will also drop its privileges
to the same user after starting the log streamer.
Change-Id: Ib52585cafbd073ccdb7f87432888ce15c7a66f67
Miro Hrončok