This was deprecated quite some time ago and we should remove it as
part of the next major release.
Also remove a very old Zuul v1 layout.yaml from the test fixtures.
Change-Id: I40030840b71e95f813f028ff31bc3e9b3eac4d6a
When we decide whether it's time to request nodes for a job which
requires something from another build, if the job providing the
requirement has failed, we currently say that our job is ready.
This will cause us to submit a node request which we will never
use.
That's because the act of checking whether the requirement is ready
has the side effect of marking our job as failed (since our
requirement failed). So the next time we go through the loop,
we'll see that failure and ignore the job from then on (never
accepting or returning the requested nodeset).
This can lead to a leak of nodes due to idle node requests.
This situation is not detected in the tests because we used empty
nodesets for these tests, and so no node requests were made for them.
To correct this, use non-empty nodesets in the relevant tests and
also indicate that the requirements are not ready in the case that
the providing job failed. This will cause us to skip requesting nodes
in the first iteration of the loop, then the resulting failure state
will avoid that in subsequent iterations.
Change-Id: Ib6e7d81f2c7129b78cdba3957c9f5b46939004db
On the way towards a fully scale out scheduler we need to move the
times database from the local filesystem into the SQL
database. Therefore we need to make at least one SQL connection
mandatory.
SQL reporters are required (an implied sql reporter is added to
every pipeline, explicit sql reporters ignored)
Change-Id: I30723f9b320b9f2937cc1d7ff3267519161bc380
Depends-On: https://review.opendev.org/621479
Story: 2007192
Task: 38329
If change B has a job J1 which requires something provided by a
dependent change A, and there is another job J2 on change B which
has a dependency on J1, then if the providing job on change A fails,
it is possible for change B to get stuck in the queue waiting for
J2 to start.
This is because when Zuul detects the requirement failure of J1,
it sets the result of J1 immediately, but did not call the 'setResult'
method which is responsible for marking child jobs as skipped. This
change adds that call which will then change the result of J2 to
skipped when J1 fails the requirement check.
This change also updates a test fixture to describe this case. It
is covered by test_provides_requires_check_old_failure. Several
other test cases are updated merely to accomodate the new job since
they share the fixture.
Change-Id: Ic541cb3ff6f811ca3a5a543c04321714ba965d24
Story: 2006988
Task: 37752
When requiring more than one artifact zuul runs into an sql exception
[1] which bubbles up to the run_handler. This effectively blocks all
operations of zuul until the change that triggers this bug is
dequeued. Fix this by correctly filtering the sql results.
[1] Trace
2019-04-04 17:15:01,158 ERROR zuul.Scheduler: Exception in run handler:
Traceback (most recent call last):
File "/opt/zuul/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/opt/zuul/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
psycopg2.ProgrammingError: operator does not exist: character varying = record
LINE 3: ...bc89ecf79fd84fc7c' AND zuul_provides.name = ('pro...
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Change-Id: I7f95e1376b7a1f46a5b4ef5242c777e16ceca451
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
The "current" provides/requires test (which is supposed to test
live changes) was inadvertently following the SQL code path in
some cases. The test has been updated to include a third job
which continues running until the end of the test to ensure that
the changes remain in the pipeline the entire time.
It, and the SQL version, have been extended with one more change
to illustrate a bug where artifact data were being included twice.
This was because the data collection method has two recursion points:
a) if it is called on a non-live change and it finds a live version
of the same change, it recursively calls itself on the non-live
version of the same change. Essentially, it "switches" to that
version of the change and continues.
b) at the end of normal processing of a change, it recurses "up"
the queue to the next change ahead.
The (a) case did not return in some cases, so in those cases, it
would "switch" to a change, add its data, recurse up the queue,
then, after unrolling back to the point of the switch, would then
recurse up the queue again in case (b). The solution is to avoid
recursing up the queue in the (a) case and allow it only in the (b)
case -- that is, we only walk up the queue of dependent changes for
the original change.
Finally, the artifacts were being returned in reverse order -- it
is more useful for them to be returned such that the latest version
of a given artifact is returned last so that competing versions
are overwritten in the correct order.
Change-Id: I41ac336c1bc776609645e3662003f2df076dd1d5
To avoid querying the database repeatedly, we cache the results
of the query for completed jobs which provide artifacts needed
for a running job. We only cached one such result per queue item,
so if a queue item had two jobs with differing requirements, then
we would only use the results associated with the first.
This change updates the cache to be a dictionary keyed by the
requirements used in the search.
Change-Id: Ic707013777303e696cf76120308724ec501979b2
Adds support for expressing artifact dependencies between jobs
which may run in different projects.
Change-Id: If8cce8750d296d607841800e4bbf688a24c40e08
James E. Blair