Since jobs are no longer identified by name but by UUID we also need to
reference job dependencies in the MQTT payload by UUID.
For backward-compatibility we'll keep the old "dependencies" field and
add a new "job_dependencies" mapping with the job names and UUIDs.
Change-Id: Ib74b11faf72602e1708ea6364cc4a1000e3f0d3b
The characters '+' and '#' have a special meaning (wildcards) and are
not allowed when publishing messages.
ERROR zuul.MQTTConnection: Could not publish message to topic 'foobar/zuul/c++-test' via mqtt
Traceback (most recent call last):
File "/opt/zuul/lib/python3.11/site-packages/zuul/driver/mqtt/mqttconnection.py", line 97, in publish
self.client.publish(topic, payload=json.dumps(message), qos=qos)
File "/opt/zuul/lib/python3.11/site-packages/paho/mqtt/client.py", line 1233, in publish
raise ValueError('Publish topic cannot contain wildcards.')
ValueError: Publish topic cannot contain wildcards.
Change-Id: Iad2ad551151284910de076cec15b3ac6b1cbda52
This endpoint runs an optimized query for returning information
suitable for displaying a graph of build times.
This includes a schema migration to add some indexes to aid
the query.
Change-Id: I56e8422a599c1ee51216f26fcae5a39013066e6b
This drops the (project, change) index in favor of several single-column
indexes that the dbms can combine as needed to quickly seek zuul_refs in queries.
Change-Id: I50ec1552e24c5aa634538ebb181da907898f1284
This refactors the sql connection to accomodate multiple
simulataneous changes in a buildset.
The change information is removed from the buildset table and
placed in a ref table. Buildsets are associated with refs
many-to-many via the zuul_buildset_ref table. Builds are also
associated with refs, many-to-one, so that we can support
multiple builds with the same job name in a buildset, but we
still know which change they are for.
In order to maintain a unique index in the new zuul_ref table (so that
we only have one entry for a given ref-like object (change, branch,
tag, ref)) we need to shorten the sha fields to 40 characters (to
accomodate mysql's index size limit) and also avoid nulls (to
accomodate postgres's inability to use null-safe comparison operators
on indexes). So that we can continue to use change=None,
patchset=None, etc, values in Python, we add a sqlalchemy
TypeDectorator to coerce None to and from null-safe values such as 0
or the empty string.
Some previous schema migration tests inserted data with null projects,
which should never have actually happened, so these tests are updated
to be more realistic since the new data migration requires non-null
project fields.
The migration itself has been tested with a data set consisting of
about 3 million buildsets with 22 million builds. The runtime on one
ssd-based test system in mysql is about 22 minutes and in postgres
about 8 minutes.
Change-Id: I21f3f3dfc8f93a23744856e5b82b3c948c118dc2
The artifact, provides, and build_event tables all have foreign key
references to the build table. In MySQL this causes an automatic
index to be created, but that does not appear to be the case in
Postgres. Without these indexes, many of the queries we do can be
quite costly.
To address this, explicitly create these indexes. Under MySQL, this
will simply rename the existing indexes, so this should be safe
and effective for both systems.
Change-Id: I16223fba75c1295480431d03ac59f72a5281b498
This addresses SqlAlchemy's removed in 2.0 warnings. Now that SqlAlchemy
2.0 has released we can see that we are not compatible yet. A good first
step in adding compatibility is fixing warnings in 1.4.
In particular there are four types of warning we fix here:
1. Using raw strings in conn.execute() calls. We need to use the text()
construct instead.
2. Passing a list of items to select when doing select queries. Instead
we need to pass things as normal posargs.
3. Accessing row result items as if the row is a dict. THis is not
longer possible without first going through the row._mapping system.
Instead we can access items as normal object attributes.
4. You must now use sqlalchemy.inspect() on a connectable to create an
Inspector object rather than instantiating it directly.
Finally we set up alembic's engine creation to run with future 2.0
behavior now that the warnings are cleared up. This appears to have
already been done for the main zuul application.
Change-Id: I5475e39bd93d71cd1106ec6d3a5423ea2dd51859
When a build is paused or resumed, we now store this information on the
build together with the event time. Instead of additional attributes for
each timestamp, we add an "event" list attribute to the build which can
also be used for other events in the future.
The events are stored in the SQL database and added to the MQTT payload
so the information can be used by the zuul-web UI (e.g. in the "build
times" gantt chart) or provided to external services.
Change-Id: I789b4f69faf96e3b8fd090a2e389df3bb9efd602
This adds an option to include result data from a job in the MQTT
reporter. It is off by default since it may be quite large for
some jobs.
Change-Id: I802adee834b60256abd054eda2db834f8db82650
These two tests exercise the LookupError code paths in the branch
cache a bit more. In particular, we now have a test that verifies
that a failure in fetching will result in a retry later.
Change-Id: Ia83b0233ccc31f018a61093d02fd1ce68547f9c0
This corrects two shortcomings in the database handling:
1) If we are unable to create a build or buildset and a later operation
attempts to update that build or buildset, it will likely fail, possibly
aborting the pipeline processing run.
2) If a transient db error occurs, we may miss reporting data to the db.
To correct these, this change does the following:
1) Creates missing builds or buildsets at any point we try to update them.
2) Wraps every write operation in a retry loop which attempts to write to
the database 3 times with a 5 second delay. The retry loop is just
outside the transaction block, so the entire transaction will have been
aborted and we will start again.
3) If the retry loop fails, we log the exception but do not raise it
to the level of the pipeline processor.
Change-Id: I364010fada8cbdb160fc41c5ef5e25576a654b90
Add two columns to the buildset table in the database: the timestamp
of the start of the first build and the end of the last build. These
are calculated from the builds in the webui buildset page, but they
are not available in the buildset listing without performing
a table join on the server side.
To keep the buildset query simple and fast, this adds the columns to
the buildset table (which is a minor data duplication).
Return the new values in the rest api.
Change-Id: Ie162e414ed5cf09e9dc8f5c92e07b80934592fdf
Currently, NODE_FAILURE results are not reported via SQL in case the
node request failed. Tis is because those results are directly
evaluated in the pipeline manager before the build is even started.
Thus, there are no build result events sent by the executor and the
"normal" build result event handling is skipped for those builds.
As those build results are not stored in the database they are also not
visible in the UI. Thus, there could be cases where a buildset failed
because of a NODE_FAILURE, but all builds that are shown were
successful.
To fix this, we could directly call the SQL reporter when the
NODE_FAILURE is evaluated in the pipeline manager.
Also adapt the reportBuildEnd() method in the sql reporter so that the
build entry is created in case its not present. This could be the case
if the build started event was not processed or did not happen at all
(e.g. for the NODE_FAILURE results or any result that is created via a
"fake build" directly in the pipeline manager).
Change-Id: I2603a7ccf26a41e6747c9276cb37c9b0fd668f75
Currently, the MySQL tests are hanging whenever an assertion fails in
the check_results() method. Cancelling the test run doesn't show the
assertion error, but it reveals that the test is stuck in the cleanup()
method of the MySQLSchemaFixture in tests/base.py [1].
Using a context manager for the database connection in check_results()
seems to solve this issue. I assume that the test is stuck because the
database connection from the test case is still open when the cleanup()
methods tries to drop the database. Because of the assertion error the
test exists early and the connection is never closed. Using a context
manager takes care of this.
As an additional safeguard, use a read_timeout of 30s for the database
connection in the cleanup() method. Adding the read_timeout also
"solves" the problem and shows the assertion error at the end of the
test, but we would always have to wait 5s. However, keeping this might
still be useful in case other tests also don't use a context manager for
the database connection.
[1]:
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/lib/python3.6/unittest/__main__.py", line 18, in <module>
main(module=None)
File "/usr/lib/python3.6/unittest/main.py", line 95, in __init__
self.runTests()
File "/usr/lib/python3.6/unittest/main.py", line 256, in runTests
self.result = testRunner.run(self.test)
File "/usr/lib/python3.6/unittest/runner.py", line 176, in run
test(result)
File "/usr/lib/python3.6/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.6/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.6/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.6/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.6/unittest/case.py", line 653, in __call__
return self.run(*args, **kwds)
File "/usr/local/lib/python3.6/dist-packages/testtools/testcase.py", line 654, in run
return run_test.run(result)
File "/usr/local/lib/python3.6/dist-packages/testtools/runtest.py", line 80, in run
return self._run_one(actual_result)
File "/usr/local/lib/python3.6/dist-packages/testtools/runtest.py", line 94, in _run_one
return self._run_prepared_result(ExtendedToOriginalDecorator(result))
File "/usr/local/lib/python3.6/dist-packages/testtools/runtest.py", line 119, in _run_prepared_result
raise e
File "/usr/local/lib/python3.6/dist-packages/testtools/runtest.py", line 193, in _run_user
return fn(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/fixtures/fixture.py", line 125, in cleanUp
return self._cleanups(raise_errors=raise_first)
File "/usr/local/lib/python3.6/dist-packages/fixtures/callmany.py", line 83, in __call__
cleanup(*args, **kwargs)
File "/workspace/tests/base.py", line 3970, in cleanup
cur.execute("drop database %s" % self.name)
File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 548, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 775, in _read_query_result
result.read()
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1156, in read
first_packet = self.connection._read_packet()
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 692, in _read_packet
packet_header = self._read_bytes(4)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 732, in _read_bytes
data = self._rfile.read(num_bytes)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt
Change-Id: I2dc6cc89dfbaf3cab1c05af2983dd4b8538a8b09
The overall duration is from a user (developer) point of view, how much
time it takes from the trigger of the build (e.g. a push, a comment,
etc.), till the last build is finished.
It takes into account also the time spent in waiting in queue, launching
nodes, preparing the nodes, etc.
Technically it measures between the event timestamp and the end time of
the last build in the build set.
This duration reflects the user experience of how much time the user needs
to wait.
Change-Id: I253d023146c696d0372197e599e0df3c217ef344
In 4.0 we deprecated connections using the "sql" driver in favor of
using the new "database" config file section. Remove the backwards
compatible handling of that so that "sql" connections or lack of
"database" section report an error.
Change-Id: I7e592cf5ff63f73f487e41bb6e3e4a4ae523e3fc
These tests are storing the reported index on the fake elasticsearch
backend which is a different instance for each scheduler. Thus,
depending on which scheduler reports the item, the assertions in
these tests might pass or fail.
Change-Id: Ia5206d9c6097392ceff1abe6a911aedab22f2709
We missed some cases where builds might be aborted and the results
not reported to the database. This updates the test framework to
assert that tests end with no open builds or buildsets in the
database.
To fix the actual issues, we need to report some build completions
in the scheduler instead of the pipeline manager. So to do that,
we grab a SQL reporter object when initializing the scheduler
(and we therefore no longer need to do so when initializing the
pipeline manager). The SQL reporter isn't like the rest of the
reporters -- it isn't pipeline specific, so a single global instance
is fine.
Finally, initializing the SQL reporter during scheduler init had
some conflicts with a unit test which tested that the merger could
load "source-only" connections. That test actually verified that
the *scheduler* loaded source-only connections. So to correct this,
it now verifies that the executor (which has a merger and is under
the same constraints as the merger for this purpose) can do so. We
no longer need the source_only flag in tests.
Change-Id: I1a983dcc9f4e5282c11af23813a4ca1c0f8e9d9d
Now that the Zuul web server and database are both required, provide
a consistent user experience by always reporting the build page.
This means that success-url and failure-url are no longer useful,
so remove them.
Update the quick-start to reflect that the build page is always
reported.
Change-Id: I4ff108df3917c9b6f44e2f5b0ccc4a7adbda1677
This moves some functions of the SQL reporter into the pipeline
manager, so that builds and buildsets are always recorded in the
database when started and when completed. The 'final' flag is
used to indicate whether a build or buildset result is user-visible
or not.
Change-Id: I053e195d120ecbb2fd89cf7e1e9fc7eccc9dcd2f
Freze Zuul job variables when starting a build so that jinja
templates can not be used to expose secrets. The values will be
frozen by running a playbook with set_fact, and that playbook
will run without access to secrets. After the playbook
completes, the frozen variables are read from and then removed
from the fact cache. They are then supplied as normal inventory
variables for any trusted playbooks or playbooks with secrets.
The regular un-frozen variables are used for all other untrusted
playbooks.
Extra-vars are now only used to establish precedence among all
Zuul job variables. They are no longer passed to Ansible with
the "-e" command line option, as that level of precedence could
also be used to obtain secrets.
Much of this work is accomplished by "squashing" all of the Zuul
job, host, group, and extra variables into a flat structure for
each host in the inventory. This means that much of the variable
precedence is now handled by Zuul, which then gives Ansible
variables as host vars. The actual inventory files will be much
more verbose now, since each host will have a copy of every "all"
value. But this allows the freezing process to be much simpler.
When writing the inventory for the setup playbook, we now use the
!unsafe YAML tag which is understood by Ansible to indicate that
it should not perform jinja templating on variables. This may
help to avoid any mischief with templated variables since they
have not yet been frozen.
Also, be more strict about what characters are allowed in ansible
variable names. We already checked job variables, but we didn't
verify that secret names/aliases met the ansible variable
requirements. A check is added for that (and a unit test that
relied on the erroneous behavior is updated).
Story: 2008664
Story: 2008682
Change-Id: I04d8b822fda6628e87a4a57dc368f20d84ae5ea9
Since the MQTT reporter can be used to emit start or enqueue events,
it may be useful to match enqueue and dequeue events. That could
be done with tenant+pipeline+change+patchset, but we also have a
UUID for queue items, so to make it simpler for MQTT consumers,
let's expose that.
Change-Id: Iff88bcfd73e00f292e0cc947f548582a276a7975
Now that the SQL database is required, fail to start if the dburi has
an error (like an incorrect module specification), and wait forever
for a connection to the database before proceeding.
This can be especially helpful in container environments where starting
Zuul may race starting a SQL database.
A test which verified that Zuul would start despite problems with the
SQL connection is removed since that is no longer the desired behavior.
Change-Id: Iae8ea420297f6264ae1d265b22b96d81f1df9a12
This reverts commit ddb7259f0d.
This failed earlier because of differing import paths between the
unit tests and actually running the service. We have removed the
metaclass magic which led to that and are now more explicit with
our yaml loading/dumping.
Change-Id: Ifec567846acebe7b9e71c2629edf38f36d271cbf
Revert "Decrypt project ssh keys in executors"
This reverts commit 77bde6f765.
Revert "Decrypt secrets on the executors"
This reverts commit fbb17e1f35.
Revert "Support serializing encrypted secret objects"
This reverts commit 707570e46b.
We observed an error parsing the encrypted/pkcs1 yaml on the executors
in production on OpenDev. We have not found the cause yet; revert this
until we identify it.
Change-Id: Icc751c54a376e68fd5d9e29dbbb67ed7aa6d67c5
Rather than decrypting secrets on the scheduler and sending them
to the executors unencrypted, now that the private keys are in ZK
and the executors have access to them, we can defer decryption to
the executors. This means that when we move the build requests
from gearman to ZK, we avoid storing decrypted secrets in ZK.
We accomplish this by serializing the entire secret (parts or all
of which may be encrypted or plaintext) to YAML in the scheduler
and deserializing the YAML into a Secret object on the executor.
We do this because we already have support for indicating an
encrypted value via custom YAML tags.
This means that the build request (which is currently transmitted
via gearman and soon to be via ZK) serializes the rest of the job
to JSON. This means we're storing a serialized-to-YAML secret as
a scalar value in a serialized-to-JSON data structure. There's
nothing technically wrong with this, and it is the minimal version
of this change, however it's slightly unusual and may result in
a little extra work. We may want to consider serializing the
entire job request as YAML instead.
Change-Id: I6d94c1d8da8b68e5fb60c27e73039155a02fb485
The MQTT reporter now includes artifacts for completed builds. Systems
which watch for MQTT events can now directly consume those artifacts without
the intermediate step of looking them up via the API.
Change-Id: I9df9e1dfd6854518c110dd65d4f89dea449c6fc0
Currently if two triggers of the same connection type need to trigger on
different events it's not possible to do so since the events are never
filtered on which connection they came from.
For example with the following setup where gerrit-org-1 only wants to
trigger on changes to 'master' and gerrit-org-2 only wants to trigger on
changes to 'develop' they will instead both trigger on 'master' and
'develop'since the events are never filtered on which connection they
came from.
- pipeline:
name: check
trigger:
gerrit-org-1:
- event: patchset-created
branch: 'master'
gerrit-org-2:
- event: patchset-created
branch: 'develop'
Change-Id: Ia0476d71dee59c8b80db7630ac7a524bce87e6f9
On the way towards a fully scale out scheduler we need to move the
times database from the local filesystem into the SQL
database. Therefore we need to make at least one SQL connection
mandatory.
SQL reporters are required (an implied sql reporter is added to
every pipeline, explicit sql reporters ignored)
Change-Id: I30723f9b320b9f2937cc1d7ff3267519161bc380
Depends-On: https://review.opendev.org/621479
Story: 2007192
Task: 38329
The timestamp field helps to prepare visualization in
Kibana.
The duration field was set to integer in Kibana object, but
the value was string, so the Kibana was doing incorrect visualizations.
Also this commit provides fix for rst block in elasticserarch driver
document.
Change-Id: I92a034d78f9193476eccecc7efb4a818d4b4a658
It has the capability to index build and buildset results.
With the help of tools like Kibana, advanced analytics
dashboard could be built on top of the Zuul Elasticsearch
index.
Optionally job's variables and zuul_return data can be
exported along with build results under the job_vars and
job_returned_vars fields.
Change-Id: I5315483c55c10de63a3cd995ef681d0b64b98513
Each scheduler in tests needs its own connection registries.
This change only affects tests.
Change-Id: I2ad188cf5a72c46f7486c23ab653ff574123308b
Story: 2007192
The boolean "held" attribute is set to True if a build triggered
a autohold request, and its nodeset was held.
Allow filtering builds by "held" status.
Change-Id: I6517764c534f3ba8112094177aefbaa144abafae
Currently the mqtt reporter uses the report url as log_url. This is
fine as long as report-build-page is disabled. As soon as
report-build-page is enabled on a tenant it reports the url to the
result page of the build. As mqtt is meant to be consumed by machines
this breaks e.g. log post processing.
Fix this by reporting the real log url as log_url and add the field
web_url for use cases where really the human url is required.
This fixes also a wrong indentation in the mqtt driver documentation,
resulting in all buildset.builds.* attributes being listed as buildset.*
attributes.
Change-Id: I91ce93a7000ddd0d70ce504b70742262d8239a8f
Since we added those to the MQTT reporter, we should also store them in
the SQL database. They are stored in the zuul_build table and can be
identified via the new "final" column which is set to False for those
builds (and True for all others).
The final flag can also be used in the API to specifically filter for
those builds or remove them from the result. By default, no builds are
filtered out.
The buildset API includes these retried builds under a dedicated
'retry_builds' key to not mix them up with the final builds. Thus, the
JSON format is equally to the one the MQTT reporter uses.
For the migration of the SQL database, all existing builds will be set
to final.
We could also provide a filter mechanism via the web UI, but that should
be part of a separate change (e.g. add a checkbox next to the search
filter saying "Show retried builds").
Change-Id: I5960df8a997c7ab81a07b9bd8631c14dbe22b8ab
To improve consistency and remove `self.sched` and
use `self.scheds.first.sched` in all tests.
This change only touches tests.
Change-Id: I0a8c3f8ad634d1d88f3c68a11e4d5587993f4a0d
Story: 2007192
As a preparation for scale-out-scheduler the scheduler in tests
were extracted in order to start multiple instances in a previous
change.
This change continues on by introducing a manager to create
additional scheduler instances and the ability to call certain
methods on some or all of those instances.
This change only touches tests.
Change-Id: Ia05a7221f19bad97de1176239c075b8fc9dab7e5
Story: 2007192
Zuul