Validate taskgraph parameters using a schema.
Previously, parameters were verified using handwritten comparison to a sample set of parameters.
Switch to using an explicit schema instead.
Differential Revision: https://phabricator.services.mozilla.com/D23756
--HG--
extra : moz-landing-system : lando
When a cron task depends on tasks from the on-push build, wait for the on-push
decision task to complete, and fail if it doesn't complete succesfully.
Differential Revision: https://phabricator.services.mozilla.com/D29429
--HG--
extra : moz-landing-system : lando
This allows us to fix a regression where -sw tasks were scheduled on autoland/inbound.
Differential Revision: https://phabricator.services.mozilla.com/D29259
--HG--
extra : moz-landing-system : lando
This also adds an optimization for the case where there is only one result
(Which is common for actions where `times` defaults to 1)
Differential Revision: https://phabricator.services.mozilla.com/D28889
--HG--
extra : moz-landing-system : lando
Some groups of tasks need to share the same profile data. For example,
Android PGO builds and Android Nightly builds both use the
generate-profile-android-api-16/pgo task for profile data. Previously
this was done with a text substitution, but this is a bit hacky and
doesn't easily scale with different build types.
Allowing use_pgo to be a string means we can just directly point to the
generate-profile task that contains the profile data to be used in a PGO
build.
Differential Revision: https://phabricator.services.mozilla.com/D29247
--HG--
extra : moz-landing-system : lando
This is not strictly required in mozilla-central, as `mach` sets
the encoding of the output to UTF-8.
Differential Revision: https://phabricator.services.mozilla.com/D28861
--HG--
extra : moz-landing-system : lando
These artifacts are "internal" because we can't redistribute them, but
downloading them into an interactive task is not redistribution.
Differential Revision: https://phabricator.services.mozilla.com/D28656
--HG--
extra : moz-landing-system : lando
Bug 1474897 changed things such that Windows builds ended up in the
linux/macosx branch. That still works somehow, but ends up breaking when
wrapping with run-task. This change restores the originally intended
command line.
Differential Revision: https://phabricator.services.mozilla.com/D28017
--HG--
extra : moz-landing-system : lando
On windows, the generated path will be close to the path length limits, which
causes `mach try` to fail.
Differential Revision: https://phabricator.services.mozilla.com/D28554
--HG--
extra : moz-landing-system : lando
This officially makes 'moztest.resolve' the source of truth when it comes to
suite names. It aligns that file with the names used in both the
desktop_unittest and android_emulator_unittest scripts.
Differential Revision: https://phabricator.services.mozilla.com/D27555
--HG--
extra : moz-landing-system : lando
Currently we have the concept of a "suite" and a "flavour" in our task
configuration. Typically, the "suite" refers to the high-level test harness
like "mochitest" or "reftest", whereas the flavour is more specific, e.g
"browser-chrome-instrumentation" or "crashtest". However the line between suite
and flavour is not applied with any semblance of consistency which results in
inconsistent naming throughout the tree.
This patch gets rid of the concept of "flavours" entirely (at least when it
comes to task configuration). A suite is a type of test run, for example:
- mochitest-plain
- mochitest-devtools-chrome
- mochitest-browser-chrome-instrumentation
- jsreftest
- reftest
- firefox-ui-functional-remote
etc
There is no confusion here between suites and flavours because flavours don't
exist. However, there are a couple of places where we *do* need to know what
"test harness" is used to run a suite. These cases are:
1. For SCHEDULES moz.build rules
2. For the desktop_unittest.py mozharness script which takes arguments like
--mochitest-suite=browser (this is not a compelling use of this information
and should be refactored to work more like the android_emulator_unittest.py
script)
So to get this information, this patch introduces a new concept of a "category"
which is the overall "test harness" that runs the suite. For many suites, the
"category" is identical to the suite name. Unlike flavours, "categories" have
no bearing on how we call or refer to the suite.
Differential Revision: https://phabricator.services.mozilla.com/D27554
--HG--
extra : moz-landing-system : lando
We are starting to spin off more and more "variants" of test suites. These are
usually just duplicates of our pre-existing tasks, except with an additional
pref set.
Currently there are two variants (serviceworker-e10s and socketprocess-e10s),
but a third will be added soon (fission). This change ensures we handle these
types of requests in a consistent and well defined manner. It also splits tasks
in a loop, so we don't accidentally risk combinatorial explosion.
Variants should typically be reserved for very large changes that will impact
the entire codebase (think e10s).
Differential Revision: https://phabricator.services.mozilla.com/D28061
--HG--
extra : moz-landing-system : lando
We only want to append the action TASK_ID to the task dependencies when the taskGroupId doesn't match, otherwise we hit dup dependency errors.
Differential Revision: https://phabricator.services.mozilla.com/D27994
--HG--
extra : moz-landing-system : lando
this change adds support for parallel gcp builds for the following linux build configurations:
- linux(32)
- opt
- debug
- shippable
- linux64
- opt
- debug
- shippable
implementation notes:
- this patch mostly mirrors the equivalent windows-on-gcp patch at: https://phabricator.services.mozilla.com/D24865
- gcp builds are triggered with a treeherder tier 3 flag so that they are only displayed in the treeherder ui when the user has a tier 3 flag set.
- gcp builds use a th build symbol of "Bg" to make them easy to differentiate from ec2 builds in the treeherder ui.
- gcp builds use a perfherder "gcp" flag to make them easier to differentiate from ec2 builds in the perfherder ui.
- gcp builds on linux for all scm levels are built on the only available gcp linux worker type (at the time of this change): gce/gecko-1-b-linux-32
Differential Revision: https://phabricator.services.mozilla.com/D26490
--HG--
extra : moz-landing-system : lando
Many tasks (release tasks and cached tasks, in particular) should be re-run rather
than retriggered. Instead, make the `retrigger-multiple` action re-run them instead.
Differential Revision: https://phabricator.services.mozilla.com/D27208
--HG--
extra : moz-landing-system : lando
In Bug 1519599, treeherder switched to using add-new-jobs to retrigger jobs, since
there wasn't an action to retrigger multiple jobs. This prevents us from adding logic
to rerun some jobs instead of retriggering them.
This adds a new action that takes input like `add-new-jobs`, but that we can add logic
to handle rerun vs. retrigger in. Additionally, the input it takes is designed
to make Bug 1521032 easier to implement.
Differential Revision: https://phabricator.services.mozilla.com/D27207
--HG--
extra : moz-landing-system : lando
Many tasks (release tasks and cached tasks, in particular) should be re-run rather
than retriggered. Disable retrigger action for those tasks by default.
Differential Revision: https://phabricator.services.mozilla.com/D27206
--HG--
extra : moz-landing-system : lando
In order to prevent retriggers for release tasks, we will cause the `retrigger`
action to rerun instead, so move the cdoe togehter.
Differential Revision: https://phabricator.services.mozilla.com/D27205
--HG--
extra : moz-landing-system : lando
Changes:
- added windows10-aarch64 to the filter for fuzzy, to require `--full` in order to trigger jobs
- return False for any test tasks that contain windows10-aarch64 to prevent users using old try syntax from overwhelming the limited number of hardware
Differential Revision: https://phabricator.services.mozilla.com/D27590
--HG--
extra : moz-landing-system : lando
Since e10s is the default configuration, we shouldn't explicitly mark things
with the "-e10s" suffix. Instead we should mark things that *don't* run with
'e10s. This patch removes '-e10s' from all treeherder group symbols and task
labels, adds the "-1proc" suffix to tasks that are non-e10s.
Differential Revision: https://phabricator.services.mozilla.com/D25958
--HG--
extra : moz-landing-system : lando
Much of this was already reviewed in D21473 (my test change where I developed the payload modifications and that pointed tests at my test queue).
This change keeps the payload changes from D21473, but points at the new 'real' queues we'll be using.
Differential Revision: https://phabricator.services.mozilla.com/D25009
--HG--
extra : moz-landing-system : lando
ensure high_value_tasks has a default value when we fail to get data from treeherder/seta.
Differential Revision: https://phabricator.services.mozilla.com/D27445
--HG--
extra : moz-landing-system : lando
Now that release promotion is using a hook, all the code for non-hook actions
can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D27204
--HG--
extra : moz-landing-system : lando
So, instead of fetches['by-test-platform']['fetch'], we have
fetches['fetch']['by-test-platform'].
Differential Revision: https://phabricator.services.mozilla.com/D27227
--HG--
extra : moz-landing-system : lando
Now that we have added the necessary scopes to `ci-configuration`,
we can add the in-tree scopes to give tasks access to the
`hgmointernal` config Taskcluster secret.
Differential Revision: https://phabricator.services.mozilla.com/D25001
--HG--
extra : moz-landing-system : lando
I've refactored the artifact map generation slightly to make the list
of platforms more flexible, and also to let us have the previous name for
win64_aarch64_info.txt.
Differential Revision: https://phabricator.services.mozilla.com/D27049
--HG--
extra : moz-landing-system : lando
