Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
If we are generating only a part of the graph, to given kind, don't fail if a
build is packaging tests and there is no corresponding test task, as the tests
may not have been generated.
Differential Revision: https://phabricator.services.mozilla.com/D82097
This causes actions to fail on graphs with large parameters, such as
release graphs and try pushes.
Differential Revision: https://phabricator.services.mozilla.com/D37499
--HG--
extra : moz-landing-system : lando
Previously, this code looked parameters under `gecko.v2`, but that doesn't work
for projects using the out-of-tree taskgraph code, or Thunderbird. This moves
the parameter loading slightly later to vary the index used based on trust domain.
Differential Revision: https://phabricator.services.mozilla.com/D19028
--HG--
extra : moz-landing-system : lando
This allows the target tasks to include cached tasks, which by default, get
optimized out of the graph on on-`try` branches.
Differential Revision: https://phabricator.services.mozilla.com/D15278
--HG--
extra : moz-landing-system : lando
Eventually, workers will provide these variables directly
(https://bugzilla.mozilla.org/show_bug.cgi?id=1460015). But for now, this
ensures that TASKCLUSTER_ROOT_URL is set everywhere, and TASKCLUSTER_PROXY_URL
is set wherever the proxy is active.
The setup for the mach commands defaults to https://taskcluster.net for user
convenience. When the production instance's URL changes, we can simply change
that default.
This changes the docker build process propagate TASKCLUSTER_ROOT_URL into the
docker images where necessary (using %ARG), specifically to create URLs for
debian repo paths.
--HG--
extra : rebase_source : 0626ebdb66a9f4078cb75ab71be256c334297363
Eventually, workers will provide these variables directly
(https://bugzilla.mozilla.org/show_bug.cgi?id=1460015). But for now, this
ensures that TASKCLUSTER_ROOT_URL is set everywhere, and TASKCLUSTER_PROXY_URL
is set wherever the proxy is active.
The setup for the mach commands defaults to https://taskcluster.net for user
convenience. When the production instance's URL changes, we can simply change
that default.
This changes the docker build process to propagate TASKCLUSTER_ROOT_URL into
the docker images, and for good measure includes some code to use that value to
generate debian repo paths.
Differential Revision: https://phabricator.services.mozilla.com/D14196
--HG--
extra : moz-landing-system : lando
`mach artifact toolchain` gets task definitions from taskgraph, to get the
index path to find the artifacts at. Now that these index paths depend on the
digests of fetch tasks, those kinds need to be loaded as well. This adds a
supported API to get task definitions for a given kind, which loads all the
kind dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D12161
--HG--
extra : moz-landing-system : lando
For kind=hook, the spec doesn't include this value as it's untrustworthy.
For kind=task, it's still untrustworthy, but there is no privilege escalation
so that's not important. Still, it dramatically expands the size of the task
definition.
MozReview-Commit-ID: 6scQ2ZwxP10
--HG--
extra : rebase_source : 4dc34390a510091ddc26023755992995fe358e47
The latest python-zstandard uses a newer zstandard that is faster.
It also has wheels available, which means installation doesn't require
Python development headers, etc.
MozReview-Commit-ID: 5gRq81KYmX4
--HG--
extra : rebase_source : 96ccc64e9707c6b4815c1bfa5c1a98b9a428b387
Now that `mach taskcluster-build-image` can, we can avoid all the manual
handling based on curl and jq in the image builder.
An additional advantage on relying on `mach taskcluster-build-image`
doing more is that less changes to the build-image.sh script will be
necessary, and thus less updates of the image builder docker image.
--HG--
extra : rebase_source : dd174d60675e41e4391894f28235c674c1840829
The zstd command we spawn, if available at all, might be the wrong
version: zstd changed its stream format in an incompatible way at some
point, and the version shipped in e.g. Ubuntu 16.04 uses the old format,
while the version taskcluster relies on uses the new format.
Relying on gps's zstandard library allows to ensure we use the right
version. Another advantage is that we can trivially pip install it in a
virtualenv if it isn't available on the system running the command.
If we're ridding ourselves of the subprocess spawning for zstd, we might
as well cover curl as well. Especially considering the error handling
when subprocesses are involved is not trivial, such that the current
error handling code is actually broken and leads to dead-lock
conditions, when, for example, curl is still waiting for the python side
to read data, but the python side is not reading data anymore because
an exception was thrown in the tar reading loop.
--HG--
extra : rebase_source : 054c37cfaa68bf475b37545ebaa99144584b93d4
Ideally, we'd simply use the --build-arg docker argument along with ARG
in the Dockerfile, but that's only supported from Docker API 1.21, and
we're stuck on 1.18 for the moment.
So we add another hack to how we handle the Dockerfile, by adding a
commented syntax that allows to declare arguments to the Dockerfile.
The arguments can be defined in the docker images kind.yml file through
the `args` keyword. Under the hood, they are passed down to the docker
image task through the environment. The mach taskcluster-build-image
command then uses the corresponding values from the environment to
generate a "preprocessed" Dockerfile for its context.
--HG--
extra : rebase_source : 26a43dd680c1ab97b1a4689a23c55594a3b21b67
Initially this will skip toolchain task optimizations, which avoids hashing
local directory contents and speeds up taskgraph generation by about 25%.
MozReview-Commit-ID: B4LB5BV86nw
--HG--
extra : rebase_source : c41e4d838e8920b865cd62bb8de38e64b85b2d84
