All the workers that had hg under /tools/python27-mercurial are gone.
This also fixes run-task for workers running > 10.14.
Differential Revision: https://phabricator.services.mozilla.com/D92867
All the workers that had hg under /tools/python27-mercurial are gone.
This also fixes run-task for workers running > 10.14.
Differential Revision: https://phabricator.services.mozilla.com/D92867
In Bug 1466660, we started deleting the fetches after a task had run, to avoid
interference between tasks. It turns out the only tasks this was for were the
`source-test-jsshell` tasks, which were changed to use an absolute directory in
Bug 1465181. However, since Bug 1568460 we've always used a per-task directory
for fetches, so can remove the work-around of removing fethes.
Differential Revision: https://phabricator.services.mozilla.com/D86670
In Bug 1466660, we started deleting the fetches after a task had run, to avoid
interference between tasks. It turns out the only tasks this was for were the
`source-test-jsshell` tasks, which were changed to use an absolute directory in
Bug 1465181. However, since Bug 1568460 we've always used a per-task directory
for fetches, so can remove the work-around of removing fethes.
Differential Revision: https://phabricator.services.mozilla.com/D86670
In Bug 1466660, we started deleting the fetches after a task had run, to avoid
interference between tasks. It turns out the only tasks this was for were the
`source-test-jsshell` tasks, which were changed to use an absolute directory in
Bug 1465181. However, since Bug 1568460 we've always used a per-task directory
for fetches, so can remove the work-around of removing fethes.
Differential Revision: https://phabricator.services.mozilla.com/D86670
I've left the monitor disabled for now, so that we can have a smaller pushes for enabling and disabling it if needed. It should allow more fine grained control.
We may also want to include extracting the monitor tool from a github version instead, and also removing the assumption and it being forked from the parent, so that it's instead given a process ID to treat as the parent it should watch.
Differential Revision: https://phabricator.services.mozilla.com/D84374
The hard limit prevents program to set a value higher than the soft
limit for themselves, and it turns out Firefox does that.
Also, keep the soft limit to its original value if it was already less
than 1024.
Differential Revision: https://phabricator.services.mozilla.com/D63623
--HG--
extra : moz-landing-system : lando
while allowing it to be tweakable without editing run-task each time
(rebuilding all docker images as a consequence)
Differential Revision: https://phabricator.services.mozilla.com/D62701
--HG--
extra : moz-landing-system : lando
Previously we inspected the `TASKCLUSTER_WORKER_GROUP` environment variable,
which now only returns the cloud provider of the worker. This commit teaches
`run-task` to instead use the `TASKCLUSTER_WORKER_LOCATION` to gather
information about the location of the worker. We also use the extra data
about the cloud provider for the worker to construct a key for use in the
config, in the form `cloudprovider/region`, so GCP hgweb mirrors can be
amended to the `hgmointernal` config when they are ready.
While we're here we make the error handling for a missing environment
variable slightly nicer.
Differential Revision: https://phabricator.services.mozilla.com/D53209
--HG--
extra : moz-landing-system : lando
Previously we inspected the `TASKCLUSTER_WORKER_GROUP` environment variable,
which now only returns the cloud provider of the worker. This commit teaches
`run-task` to instead use the `TASKCLUSTER_WORKER_LOCATION` to gather
information about the location of the worker. We also use the extra data
about the cloud provider for the worker to construct a key for use in the
config, in the form `cloudprovider/region`, so GCP hgweb mirrors can be
amended to the `hgmointernal` config when they are ready.
While we're here we make the error handling for a missing environment
variable slightly nicer.
Differential Revision: https://phabricator.services.mozilla.com/D53209
--HG--
extra : moz-landing-system : lando
Check the return code of write() to determine if the output line has actually been written completely; if not, write the remainder.
Tests suggest that incomplete writes are possible when the buffer exceeds a few thousand bytes. Very long log lines are unusual but this is important for cases like reftest failures where an encoded screenshot is dumped to the log.
Differential Revision: https://phabricator.services.mozilla.com/D48218
--HG--
extra : moz-landing-system : lando
For some tasks, the workdir known to the decision task doesn't actually match
the workdir used in the task, which makes MOZ_FETCHES_DIR wrong when the
decision task derives it from the workdir.
On other tasks, MOZ_FETCHES_DIR is set to a relative directory, which
may work in some places where MOZ_FETCHES_DIR is used, but not in
others, that happen to be executed from a different directory.
To solve both problems, we set MOZ_FETCHES_DIR as a relative directory
everywhere, and we make run-task normalize it to an absolute path
before executing the task.
Differential Revision: https://phabricator.services.mozilla.com/D39152
--HG--
extra : moz-landing-system : lando
For some tasks, the workdir known to the decision task doesn't actually match
the workdir used in the task, which makes MOZ_FETCHES_DIR wrong when the
decision task derives it from the workdir.
On other tasks, MOZ_FETCHES_DIR is set to a relative directory, which
may work in some places where MOZ_FETCHES_DIR is used, but not in
others, that happen to be executed from a different directory.
To solve both problems, we set MOZ_FETCHES_DIR as a relative directory
everywhere, and we make run-task normalize it to an absolute path
before executing the task.
Differential Revision: https://phabricator.services.mozilla.com/D39152
--HG--
extra : moz-landing-system : lando
Currently, all things running via run-task don't really care that the
current directory is set to /. However, on generic-worker, many things
assume the current directory is the task directory, which varies by
task, and wrapping them with run-task fails because it resets the
current directory.
Differential Revision: https://phabricator.services.mozilla.com/D28018
--HG--
extra : moz-landing-system : lando
With tasks able to access the hgmointernal config from a Taskcluster
secret, we can now add functionality to `run-task` to support checking
out from the private hg service. Here we add add a `resolve_checkout_url`
function which takes the base/head repository URLs and determines
whether we should clone from the public or private service, returning
the resolved URL. The function pulls down the secret and checks that
the region the task is executing in is in the set of supported regions.
Then we generate a random number and default to the public service if
the number is lower than our "rate". If all the above conditions are
met, we replace `hg.mozilla.org` with the resolved domain name for the
given region.
We add a call to this function to `collect_vcs_options`, and skip
resolving the private URL if we aren't performing a checkout from
within `run-task`.
Differential Revision: https://phabricator.services.mozilla.com/D25002
--HG--
extra : moz-landing-system : lando
The shebang at the top of fetch-content doesn't work on macOS because
the path to python3 is not /usr/bin. Using /usr/bin/env doesn't work
properly on all platforms either so instead we invoke the script using
the currently running python executable.
Differential Revision: https://phabricator.services.mozilla.com/D19744
--HG--
extra : source : f6413e576a2169855f766085704570c9fc851ee2
The shebang at the top of fetch-content doesn't work on macOS because
the path to python3 is not /usr/bin. Using /usr/bin/env doesn't work
properly on all platforms either so instead we invoke the script using
the currently running python executable.
Differential Revision: https://phabricator.services.mozilla.com/D19744
--HG--
extra : moz-landing-system : lando
We create a minimal wrapper function to call collect_vcs_options()
and vcs_checkout().
We could consolidate this logic into vcs_checkout(). But I don't have
strong feelings about doing that.
Differential Revision: https://phabricator.services.mozilla.com/D13821
--HG--
extra : moz-landing-system : lando
One step closer to making all state gathering and normalization in one
place.
Differential Revision: https://phabricator.services.mozilla.com/D13819
--HG--
extra : moz-landing-system : lando
This makes behavior consistent across all VCS checkouts. I'm still not
a fan of using environment variables here. But at least this gets us
1 step closer to being able to plug alternate logic in without having
to update use of environment variables outside a single function.
Differential Revision: https://phabricator.services.mozilla.com/D13816
--HG--
extra : moz-landing-system : lando
We currently manage VCS checkout arguments as one-offs for each
project. This isn't scalable and results in a bit of copy-pasta.
Let's make the VCS checkout arguments generic so we can have the
same control for all repositories.
This commit focuses on consolidating the existing argument parser
code. It stops short of further unification, which will be done in
subsequent commits.
Differential Revision: https://phabricator.services.mozilla.com/D13815
--HG--
extra : moz-landing-system : lando
This code was used by mozharness jobs to check out the tools repo.
However, those jobs aren't actually using the repository.
Differential Revision: https://phabricator.services.mozilla.com/D13855
--HG--
extra : moz-landing-system : lando
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "gecko"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
We create a minimal wrapper function to call collect_vcs_options()
and vcs_checkout().
We could consolidate this logic into vcs_checkout(). But I don't have
strong feelings about doing that.
Differential Revision: https://phabricator.services.mozilla.com/D13821
--HG--
extra : moz-landing-system : lando
One step closer to making all state gathering and normalization in one
place.
Differential Revision: https://phabricator.services.mozilla.com/D13819
--HG--
extra : moz-landing-system : lando
This makes behavior consistent across all VCS checkouts. I'm still not
a fan of using environment variables here. But at least this gets us
1 step closer to being able to plug alternate logic in without having
to update use of environment variables outside a single function.
Differential Revision: https://phabricator.services.mozilla.com/D13816
--HG--
extra : moz-landing-system : lando
We currently manage VCS checkout arguments as one-offs for each
project. This isn't scalable and results in a bit of copy-pasta.
Let's make the VCS checkout arguments generic so we can have the
same control for all repositories.
This commit focuses on consolidating the existing argument parser
code. It stops short of further unification, which will be done in
subsequent commits.
Differential Revision: https://phabricator.services.mozilla.com/D13815
--HG--
extra : moz-landing-system : lando
This code was used by mozharness jobs to check out the tools repo.
However, those jobs aren't actually using the repository.
Differential Revision: https://phabricator.services.mozilla.com/D13855
--HG--
extra : moz-landing-system : lando
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "gecko"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
