As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
This adds a `mach wpt-fission-regressions` command that uses the wpt
expectation data to look for tests which have a worse result in
fission. With the `--all-json=<path>` argument it will output a JSON
file containing details of all the regressions. With the
`--untriaged=<path>` argument it will output a file containing a list
of failures that have not yet been triaged.
It also adds a try job to produce those files as artifacts whenever
wpt metadata is changed.
The actual implementation is based on reading the wpt expectation data
with sample run_info values corresponding to the configurations in
which we have fission enabled, but with the "fission" property set to
False (to get a baseline result) and True (to get a with-fission
result) and then comparing the resulting expectations.
The implemenation is pretty suboptimal performance wise since we end
up reading the metadata once per configuration i.e. 6 times, and this
is slow. It could be optimised by using the conditional metadata
backend, reading it once, and then evaluating per
configuration. However that would require a little more work and the
presumption is that this will be shortlived until fission becomes the
default configuration.
Differential Revision: https://phabricator.services.mozilla.com/D106954
This replicates the logic in
testing/web-platform/tests/tools/wptrunner/wptrunner/environment.py to
ensure that all the configured URL base paths are added to the
wptserve router configuration when starting via mach wpt-serve.
Differential Revision: https://phabricator.services.mozilla.com/D96723
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Lack of support for specialPowers is often cited as a reason for
developers to write mochitests rather than wpt. Although using
specialPowers means that the tests can't be upstreamed, having it
available in gecko-only tests should reduce the need to write
mochitests and potentially means that a patch which previously would
be written with just mochitests would instead be written with just
wpt, a fraction of which would be unsuitable for sharing.
Nevertheless a gecko-only specialPowers-using test should be
considered the option of last resort when cross-browser APIs such as
testdriver aren't sufficient.
This patch installs the specialPowers extension when the path to the
XPI is provided via a --specialpowers-path command line argument. To
ensure this only happens for gecko-only tests, the url_base is added
to the Test object, and the firefox settings() method is updated to
contain a specialpowers setting, which uses the url_base of the test
to determine if this is a gecko-only test.
Because the extension has to be enabled using marionette, we also need
to route this setting through the to executor, along with the provided
XPI path.
Differential Revision: https://phabricator.services.mozilla.com/D92035
Lack of support for specialPowers is often cited as a reason for
developers to write mochitests rather than wpt. Although using
specialPowers means that the tests can't be upstreamed, having it
available in gecko-only tests should reduce the need to write
mochitests and potentially means that a patch which previously would
be written with just mochitests would instead be written with just
wpt, a fraction of which would be unsuitable for sharing.
Nevertheless a gecko-only specialPowers-using test should be
considered the option of last resort when cross-browser APIs such as
testdriver aren't sufficient.
This patch installs the specialPowers extension when the path to the
XPI is provided via a --specialpowers-path command line argument. To
ensure this only happens for gecko-only tests, the url_base is added
to the Test object, and the firefox settings() method is updated to
contain a specialpowers setting, which uses the url_base of the test
to determine if this is a gecko-only test.
Because the extension has to be enabled using marionette, we also need
to route this setting through the to executor, along with the provided
XPI path.
Differential Revision: https://phabricator.services.mozilla.com/D92035
Lack of support for specialPowers is often cited as a reason for
developers to write mochitests rather than wpt. Although using
specialPowers means that the tests can't be upstreamed, having it
available in gecko-only tests should reduce the need to write
mochitests and potentially means that a patch which previously would
be written with just mochitests would instead be written with just
wpt, a fraction of which would be unsuitable for sharing.
Nevertheless a gecko-only specialPowers-using test should be
considered the option of last resort when cross-browser APIs such as
testdriver aren't sufficient.
This patch installs the specialPowers extension when the path to the
XPI is provided via a --specialpowers-path command line argument. To
ensure this only happens for gecko-only tests, the url_base is added
to the Test object, and the firefox settings() method is updated to
contain a specialpowers setting, which uses the url_base of the test
to determine if this is a gecko-only test.
Because the extension has to be enabled using marionette, we also need
to route this setting through the to executor, along with the provided
XPI path.
Differential Revision: https://phabricator.services.mozilla.com/D92035
This also enables py3 linting for testing/web-platform. The external testing/web-platform/tests is excluded from linting.
Differential Revision: https://phabricator.services.mozilla.com/D90744
This utilises the fact that we can now specify the manifest
independently of the metadata path, which wasn't possible before. We
need to edit the default (i.e. Firefox) configuration to update the
metadata path (only) to point at something product-specific so we
don't end up using fx metadata for everything.
Differential Revision: https://phabricator.services.mozilla.com/D89461
wpt previously required Ahem as a system font. That requirement is now
relaxed and installing fonts is only required in order to see bugs
that result from differences between the system font loader and the
webfont loader. Since installing fonts can cause problems on some
systems, make this possible to disable.
Differential Revision: https://phabricator.services.mozilla.com/D61801
--HG--
extra : moz-landing-system : lando
This command takes a list of test ids and returns the list of test
files that correspond to those test ids. The mapping is one-to-many
between files and test ids i.e. a single file can generate many tests.
Differential Revision: https://phabricator.services.mozilla.com/D56192
--HG--
extra : moz-landing-system : lando
Gecko was trying to pass formatter options in by default, which
doesn't work if the user overrides the formatter. Instead pass in the
default options explicitly using the designed mechanism, which
previously wasn't exposed in wpt.
Differential Revision: https://phabricator.services.mozilla.com/D47117
--HG--
extra : moz-landing-system : lando
This command is inteded to be usable as a git mergetool for the
specific case of merging ini files in a way that should always
succeed, produces reasonably likely output, but isn't reliably
correct.
The main use case is for the sync bot where we update metadata on
branches and experience conflicts when we also have changes on
master. We don't necessarily need to resolve these perfectly but we do
need to provide a resolution automatically since otherwise it blocks
syncs. Since the ini parser is in-tree we want to make this an in-tree
command that the sync can use to provide the resolution.
The general strategy is that we want to prefer the "new" metadata
where possible. This won't always be correct e.g. if a test got fixed
on master and simultaneously got edited to go from TIMEOUT to FAIL in
a browser without the fix. But it's not a bad approximation (and
generally a human will struggle to do the merge better by hand, so we
have to assume later try jobs will fix things).
Differential Revision: https://phabricator.services.mozilla.com/D40832
--HG--
extra : moz-landing-system : lando
Various cleanup:
- remove extraneous calls to grant_runtime_permissions
- remove unused legacy jimdb support code
- remove "This may take a while" logging
- emphasize x86/x86_64 capabilities of emulator
Differential Revision: https://phabricator.services.mozilla.com/D40302
--HG--
extra : moz-landing-system : lando
Use the geckoview TestRunnerActivity, org.mozilla.geckoview.test, by default
for all types of mochitests, reftests, and web-platform tests. TRA is already
the default for gtest and geckoview-junit. Fennec, based on ANDROID_PACKAGE_NAME,
remains the default for robocop and marionette-test and I have no plans to
change those. There is a related issue for xpcshell-test -- not the package
name, but the default apk -- but I am reluctant to handle that until bug 1553225
is resolved.
Differential Revision: https://phabricator.services.mozilla.com/D35479
--HG--
extra : moz-landing-system : lando
./mach wpt-unittest command will run tox in testing/web-platform/tests/tools
and testing/web-platform/tests/tools/wptrunner. It does some hacking to update
the requirements.txt files so that we use the in-tree versions of packages
rather than updating from pypi; this involves temporarily rewriting the files
since I couldn't figure out a good way to get tox to install something different.
Differential Revision: https://phabricator.services.mozilla.com/D33811
--HG--
extra : moz-landing-system : lando
This avoids a mismatch in the available arguments between upstream pip and
the version vendored into m-c at the cost of the code being more liable
to break when upstream changes.
Differential Revision: https://phabricator.services.mozilla.com/D32991
--HG--
extra : moz-landing-system : lando
This parses the metadata files without evaluating the conditional
expressions, producing a JSON summary file in which each possible
condition is represented as a string.
Not all data from the metadata files appears in the summary, only
exceptional cases likely to be of interest to engineers and unlikely
to be on wpt.fyi e.g. tests that have lsan leaks or tests that crash,
not just tests that fail. This is intended to keep the filesize down.
Depends on D24177
Depends on D24177
Differential Revision: https://phabricator.services.mozilla.com/D24178
--HG--
extra : moz-landing-system : lando
Add addtest support for per-suite arguments and multiple files. Also
support opening the created test in an editor. This allowed supporting
the wpt suite and replaces `mach wpt-create`.
# Create a wpt test
./mach addtest testing/web-platform/tests/accelerometer/test.html
# Create a wpt reftest
./mach addtest --suite wpt-reftesttesting/web-platform/tests/css/example.html --ref example-ref.html
The files created will be opened in the default editor if --editor is
supplied or a specified editor if the argument is given a value.
Differential Revision: https://phabricator.services.mozilla.com/D26339
--HG--
extra : moz-landing-system : lando
This parses the metadata files without evaluating the conditional
expressions, producing a JSON summary file in which each possible
condition is represented as a string.
Not all data from the metadata files appears in the summary, only
exceptional cases likely to be of interest to engineers and unlikely
to be on wpt.fyi e.g. tests that have lsan leaks or tests that crash,
not just tests that fail. This is intended to keep the filesize down.
Depends on D24177
Differential Revision: https://phabricator.services.mozilla.com/D24178
--HG--
extra : moz-landing-system : lando
