Now that we have added the necessary scopes to `ci-configuration`,
we can add the in-tree scopes to give tasks access to the
`hgmointernal` config Taskcluster secret.
Differential Revision: https://phabricator.services.mozilla.com/D25001
--HG--
extra : moz-landing-system : lando
This implements support for adding generic-worker caches. As a consequence this
also turns on caching for the gecko checkout if present.
Differential Revision: https://phabricator.services.mozilla.com/D17690
--HG--
extra : moz-landing-system : lando
We add caches at various places in common.py. This consolidates the logic into
a re-useable function. This is in preparation for adding generic-worker cache
support.
This also adds a test. The test is not terribly useful, but I've been looking
for an excuse to lay some groundwork for further tests in the 'job' submodule.
This will do.
Differential Revision: https://phabricator.services.mozilla.com/D17689
--HG--
extra : moz-landing-system : lando
This implements support for adding generic-worker caches. As a consequence this
also turns on caching for the gecko checkout if present.
Differential Revision: https://phabricator.services.mozilla.com/D17690
--HG--
extra : moz-landing-system : lando
The hosts don't have enough disk space to cache mozilla-central.
Depends on D17689
Differential Revision: https://phabricator.services.mozilla.com/D18738
--HG--
extra : moz-landing-system : lando
We add caches at various places in common.py. This consolidates the logic into
a re-useable function. This is in preparation for adding generic-worker cache
support.
This also adds a test. The test is not terribly useful, but I've been looking
for an excuse to lay some groundwork for further tests in the 'job' submodule.
This will do.
Differential Revision: https://phabricator.services.mozilla.com/D17689
--HG--
extra : moz-landing-system : lando
This enables Windows generic-worker based tasks to use the run-task script.
Differential Revision: https://phabricator.services.mozilla.com/D10758
--HG--
extra : moz-landing-system : lando
This enables Windows generic-worker based tasks to use the run-task script.
MozReview-Commit-ID: C07FANaYzf7
Differential Revision: https://phabricator.services.mozilla.com/D10758
--HG--
extra : moz-landing-system : lando
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
This will get the js-bench tasks to run on physical hardware instead of AWS.
MozReview-Commit-ID: 44XavXAwxxn
--HG--
extra : rebase_source : ae1ba4c7f90b3a8526511a3f3c1dff913a334619
This adds an optional 'workdir' key to all job schemas. It still defaults to
/builds/worker, but can be overriden by individual tasks or schema
implementations.
MozReview-Commit-ID: LY20xfBhbCP
--HG--
extra : rebase_source : 7ac76ebf55d33d30c2aad73484421c6b4002cd33
Extends support of the use-artifacts key to native-engine based tasks.
MozReview-Commit-ID: FJILoyD5XVZ
--HG--
extra : rebase_source : 0cf8bf63f73d0fbb634f6b437bcc9bcce7821900
support_vcs_checkout() always sets the environment variables that
were set by this deleted function. In addition, support_vcs_checkout()
also adds caches and scopes - at least for docker-worker and
docker-engine. For generic-worker - which was used in all call sites
of docker_worker_add_gecko_vcs_env_vars() (yes, the "docker_worker"
bit of the name was completely wrong - probably a legacy holdover) -
support_vcs_checkout() was *almost* exactly equivalent to
docker_worker_add_gecko_vcs_env_vars(). The only difference is that
support_vcs_checkout() adds the
secrets:get:project/taskcluster/gecko/hgfingerprint scope in addition
to setting the environment variables.
MozReview-Commit-ID: 8fl3u9be5fT
--HG--
extra : rebase_source : 0eec2f143f903a3fcc5502b60026f5d8061100ea
This functionality was implemented at least 3 times. Let's consolidate
it to a central function.
Returning multiple command strings is kind of funky. I preserved
existing behavior and mozharness jobs are the only ones printing the
forensic logging. We should probably move this logging into
robustcheckout so we don't need to involve taskgraph with this. But
that can be deferred to another day.
MozReview-Commit-ID: I2LglJvfI6
--HG--
extra : rebase_source : 7cb413694aee4e46a6522febe9daa4b73b5307ca
extra : source : 096d7d374af427ee950c7a550878781eebad4135
This adds a 'use-artifacts' key to the run_task schema. Tasks can specify artifacts to download like this:
run:
using: run-task
use_artifacts:
build:
- target.tar.bz2
- target.common.tests.zip
- target.mochitest.tests.zip
This will cause the run-task script to download those three artifacts from the task's 'build' dependency.
If the task doesn't have a 'build' dependency, taskgraph generation will error. The artifacts will be
downloaded into $USE_ARTIFACT_PATH. It is up to the task to do whatever extracting/setup may be required.
E.g this setup could go in the task's command.
At this time, only 'run-task' tasks using docker-worker are supported.
MozReview-Commit-ID: 3f02oCys62i
--HG--
extra : rebase_source : e8a85040e45042b537d4119334c4a8b7280b295c
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
`run-task` is taught a --sparse-profile argument to be passed down
to `hg robustcheckout` for the main source checkout. It does what
you expect: performs a sparse checkout using the named profile.
The Taskgraph YAML for run-task is taught a "sparse-profile"
property to define the sparse profile. When defined, --sparse-profile
will be passed down to `run-task` and the cache name will be updated
to reflect the use of sparse checkout.
Our cache checking transform is updated to audit for the use of
--sparse-profile without the corresponding "-sparse" cache name
variation.
The reason we need a distinct cache name for sparse is because
clients that aren't sparse aware will be unable to read checkouts
that are sparse. By forcing sparse and non-sparse into different
cache pools, we avoid compatibility issues.
In the ideal world, we probably support sparse profiles on all the
VCS checkouts that `run-task` supports (e.g. --tools-checkout).
Perfect is the enemy of done. All of this is defined in-tree and
it is easy enough to change atomically.
MozReview-Commit-ID: 79k7Vul0hHO
--HG--
extra : rebase_source : babe9b42e2796c2341bffc6ecfe829f4daff9e0f
Previously, we conditionally added caches to a task if the current
parameters warranted it.
In order to audit that all caches fulfill basic requirements, we need
to have unconditional knowledge of all caches.
This commit introduces an optional key on each cache entry stating
whether it should be skipped in "untrusted" environments. When we
convert a task definition to a worker payload, we filter out these
caches if necessary.
This change uncovered an inconsistency with filtering caches. In
one location we filtered on the source repo name. In others, we
filtered on the SCM level.
Setting the caches in the spidermonkey kind also changed slightly
to ensure we're not overwriting existing caches. I don't think this
has any behavior changes. But the new method is more correct.
MozReview-Commit-ID: 1crpdWHqQ68
--HG--
extra : rebase_source : 69469014dcc92d4de98d5e887b10542af00e05be
~ in paths is mostly a shell-ism. Let's not use it.
The real reason for this is it interfes with upcoming commits that
audit cache and volume paths.
MozReview-Commit-ID: AhjMwg5gexx
--HG--
extra : rebase_source : dd997a5d9c1123beecaa14bc5458a9338b3e07db
This corrects a bug introduced in 265873cf1388.
MozReview-Commit-ID: LkZlTVAM17E
--HG--
extra : rebase_source : bf442e620abd6b47adee3d4c56e6f0c19964aea1
We added this versioning as a mechanism to create clean breaks when
version control mechanisms change in significant ways. Now that
run-task's content influences cache names and this cache name
pattern must be used with run-task, we no longer need this
versioning and can rely on run-task's content and cache requirements
to switch to new caches.
MozReview-Commit-ID: KJ7IF7UzDsg
--HG--
extra : rebase_source : 9d562ec3977fea7920039a66be13cfae8980475e
run-task just grew features to aid with cache validation.
Attempts by run-task to use caches not under its control will fail.
So, we add a transform that audits for and ensures that certain
caches are only being used with run-task. This will help catch
stragglers attempting to use e.g. the legacy VCS checkouts or
tooltool caches without run-task. Fortunately, there are no
violations for this policy. Yay!
MozReview-Commit-ID: LBCmDUdgcuM
--HG--
extra : rebase_source : 9a290470088a8d55693dfbadc5186a018d3669b6
Caches shared across levels scare me, even if readers are purported to
perform content verification. We shouldn't take any risks with released
Firefox builds being contaminated by e.g. Try tasks.
Also, the old cache name interferes with my desire to make cache
names dynamic. This requires dynamic scopes. We already have
have level-{{level}}-* scopes for caches. So having all caches
prefixed with this makes things flexible.
MozReview-Commit-ID: LsrcxIYoEh1
--HG--
extra : rebase_source : dfe97f92a726059200ed79afe215ef2cf1fd7bf1
This pattern is repeated a number of times. Let's consolidate it.
A few places set relengapi-proxy=False. AFAICT this was a no-op because
the default value in the worker schema is False. So, these assignments
were dropped.
MozReview-Commit-ID: EoNPBhaMT2J
--HG--
extra : rebase_source : 72b9c6e4b42369720a19bab42c9ea40c942b5494
Land date changes to support windows nightlies onto central
MozReview-Commit-ID: BwBvzr46d20
--HG--
extra : rebase_source : a0f0390f3dcef510f653a5d35963d466befa5ee7
