forked from mirrors/gecko-dev
348 lines
16 KiB
ReStructuredText
348 lines
16 KiB
ReStructuredText
Automated Testing
|
|
=================
|
|
|
|
You've just written a feature and (hopefully!) want to test it. Or you've
|
|
decided that an existing feature doesn't have enough tests and want to contribute
|
|
some. But where do you start? You've looked around and found references to things
|
|
like "xpcshell" or "web-platform-tests" or "talos". What code, features or
|
|
platforms do they all test? Where do their feature sets overlap? In short, where
|
|
should your new tests go? This document is a starting point for those who want
|
|
to start to learn about Mozilla's automated testing tools and procedures. Below
|
|
you'll find a short summary of each framework we use, and some questions to help
|
|
you pick the framework(s) you need for your purposes.
|
|
|
|
If you still have questions, ask on `Matrix <https://wiki.mozilla.org/Matrix>`__
|
|
or on the relevant bug.
|
|
|
|
Firefox Production
|
|
------------------
|
|
These tests are found within the `mozilla-central <https://hg.mozilla.org/mozilla-central>`__
|
|
tree, along with the product code.
|
|
|
|
They are run when a changeset is pushed
|
|
to `mozilla-central <https://hg.mozilla.org/mozilla-central>`__,
|
|
`autoland <https://hg.mozilla.org/integration/autoland/>`__, or
|
|
`try </tools/try/index.html>`_, with the results showing up on
|
|
`Treeherder <https://treeherder.mozilla.org/>`__. Not all tests will be run on
|
|
every changeset; alogrithms are put in place to run the most likely failures,
|
|
with all tests being run on a regular basis.
|
|
|
|
They can also be run on local builds.
|
|
Note: Most of the mobile tests run on emulators, but some of the tests
|
|
(notably, performance tests) run on hardware devices.
|
|
We try to avoid running mobile tests on hardware devices unnecessarily.
|
|
In Treeherder, tests with names that start with "hw-" run on hardware.
|
|
|
|
Linting
|
|
~~~~~~~
|
|
|
|
Lint tests help to ensure better quality, less error-prone code by
|
|
analysing the code with a linter.
|
|
|
|
|
|
.. csv-table:: Linters
|
|
:header-rows: 1
|
|
|
|
"Treeherder Symbol", "Name", "Platforms", "What is Tested"
|
|
"``ES``", "`ESLint </code-quality/lint/linters/eslint.html>`__", "All", "JavaScript is analyzed for correctness."
|
|
"``ES-build``", "`eslint-build </code-quality/lint/linters/eslint.html#eslint-build-es-b>`_", "All", "Extended javascript analysis that uses build artifacts."
|
|
"``mocha(EPM)``", "`ESLint-plugin-mozilla </code-quality/lint/linters/eslint-plugin-mozilla.html>`__", "Desktop", "The ESLint plugin rules."
|
|
"``f8``", "`flake8 </code-quality/lint/linters/flake8.html>`__", "All", "Python analyzed for style and correctness."
|
|
"``stylelint``", "`Stylelint </code-quality/lint/linters/stylelint.html>`__", "All", "CSS is analyzed for correctness."
|
|
"``W``", "`wpt lint </web-platform/index.html>`__", "Desktop", "web-platform-tests analyzed for style and manifest correctness"
|
|
"``WR(tidy)``", "`WebRender servo-tidy </testing/webrender/index.html>`__", "Desktop", "Code in gfx/wr is run through servo-tidy."
|
|
"``A``", "`Spotless </code-quality/lint/linters/android-format.html>`_", "Android", "Java is analyzed for style and correctness."
|
|
|
|
.. _Functional_testing:
|
|
|
|
Functional testing
|
|
~~~~~~~~~~~~~~~~~~
|
|
|
|
.. csv-table:: Automated Test Suites
|
|
:header-rows: 2
|
|
|
|
"Treeherder Symbol", "Name", "Platform", "Process", "Environment", "", "Privilege", "What is Tested"
|
|
"", "", "", "", "Shell", "Browser Profile", "",
|
|
"``R(J)``", "JS Reftest", "Desktop", "N/A", "JSShell", "N/A", "N/A", "The JavaScript engine's implementation of the JavaScript language."
|
|
"``R(C)``", "`Crashtest </web-platform/index.html>`__", "All", "Child", "Content", "Yes", "Low", "That pages load without crashing, asserting, or leaking."
|
|
"``R(R)``", "`Reftest </web-platform/index.html>`__", "All", "Child", "Content", "Yes", "Low", "That pages are rendered (and thus also layed out) correctly."
|
|
"``GTest``", "`GTest </gtest/index.html>`__", "All", "N/A", "Terminal", "N/A", "N/A", "Code that is not exposed to JavaScript."
|
|
"``X``", "`xpcshell </testing/xpcshell/index.html>`__", "All", "Parent, Allow", "XPCShell", "Allow", "High", "Low-level code exposed to JavaScript, such as XPCOM components."
|
|
"``M(a11y)``", "Accessibility (mochitest-a11y)", "Desktop", "Child", "Content", "Yes", "?", "Accessibility interfaces."
|
|
"``M(1), M(2), M(...)``", "`Mochitest plain </testing/mochitest-plain/index.html>`__", "All", "Child", "Content", "Yes", "Low, Allow", "Features exposed to JavaScript in web content, like DOM and other Web APIs, where the APIs do not require elevated permissions to test."
|
|
"``M(c1/c2/c3)``", "`Mochitest chrome </testing/chrome-tests/index.html>`__", "All", "Child, Allow", "Content", "Yes", "High", "Code requiring UI or JavaScript interactions with privileged objects."
|
|
"``M(bc)``", "`Mochitest browser-chrome </testing/mochitest-plain/index.html>`__", "All", "Parent, Allow", "Browser", "Yes", "High", "How the browser UI interacts with itself and with content."
|
|
"``M(remote)``", "Mochitest Remote Protocol", "All", "Parent, Allow", "Browser", "Yes", "High", "Firefox Remote Protocol (Implements parts of Chrome dev-tools protocol). Based on Mochitest browser-chrome."
|
|
"``SM(...), SM(pkg)``", "`SpiderMonkey automation <https://wiki.mozilla.org/Javascript:Automation_Builds>`__", "Desktop", "N/A", "JSShell", "N/A", "Low", "SpiderMonkey engine shell tests and JSAPI tests."
|
|
"``W``", "`web-platform-tests </web-platform/index.html>`__", "Desktop", "Child", "Content", "Yes", "Low", "Standardized features exposed to ECMAScript in web content; tests are shared with other vendors."
|
|
"``Wr``", "`web-platform-tests </web-platform/writing-tests/reftests.html>`__", "All", "Child", "Content", "Yes", "Low", "Layout and graphic correctness for standardized features; tests are shared with other vendors."
|
|
"``Mn``", "`Marionette </testing/marionette/Testing.html>`__", "Desktop", "?", "Content, Browser", "?", "High", "Large out-of-process function integration tests and tests that do communication with multiple remote Gecko processes."
|
|
"``Fxfn``", "`Firefox UI Tests </remote/Testing.html#puppeteer-tests>`__", "Desktop", "?", "Content, Browser", "Yes", "High", "Integration tests with a focus on the user interface and localization."
|
|
"``tt(c)``", "`telemetry-tests-client </toolkit/components/telemetry/internals/tests.html>`__", "Desktop", "N/A", "Content, Browser", "Yes", "High", "Integration tests for the Firefox Telemetry client."
|
|
"``TV``", "`Test Verification (test-verify) </testing/test-verification/index.html>`__", "All", "Depends on test harness", "?", "?", "?", "Uses other test harnesses - mochitest, reftest, xpcshell - to perform extra testing on new/modified tests."
|
|
"``TVw``", "`Test Verification for wpt (test-verify-wpt) </testing/test-verification/index.html>`__", "Desktop", "Child", "?", "?", "?", "Uses wpt test harnesses to perform extra testing on new/modified web-platform tests."
|
|
"``WR(wrench)``", "`WebRender standalone tests </testing/webrender/index.html>`__", "All", "N/A", "Terminal", "N/A", "N/A", "WebRender rust code (as a standalone module, with Gecko integration)."
|
|
|
|
Note: there are preference-based variations of the previous testing suites.
|
|
For example, mochitests on Treeherder can have ``gli``, ``swr``, ``spi``,
|
|
``nofis``, ``a11y-checks``, ``spi-nw-1proc``, and many others. Another
|
|
example is GTest, which can use ``GTest-1proc``. To learn more about
|
|
these variations, you can mouse hover over these items to read a
|
|
description of what these abbreviations mean.
|
|
|
|
.. _Table_key:
|
|
|
|
Table key
|
|
^^^^^^^^^
|
|
|
|
Symbol
|
|
Abbreviation for the test suite used by
|
|
`Treeherder <https://treeherder.mozilla.org/>`__. The first letter
|
|
generally indicates which of the test harnesses is used to execute
|
|
the test. The letter in parentheses identifies the actual test suite.
|
|
Name
|
|
Common name used when referring to the test suite.
|
|
File type
|
|
When adding a new test, you will generally create a file of this type
|
|
in the source tree and then declare it in a manifest or makefile.
|
|
Platform
|
|
Most test suites are supported only on a subset of the available
|
|
plaforms and operating systems. Unless otherwise noted:
|
|
|
|
- **Desktop** tests run on Windows, Mac OS X, and Linux.
|
|
- **Mobile** tests run on Android emulators or remotely on Android
|
|
devices.
|
|
|
|
Process
|
|
- When **Parent** is indicated, the test file will always run in the
|
|
parent process, even when the browser is running in Electrolysis
|
|
(e10s) mode.
|
|
- When **Child** is indicated, the test file will run in the child
|
|
process when the browser is running in Electrolysis (e10s) mode.
|
|
- The **Allow** label indicates that the test has access to
|
|
mechanisms to run code in the other process.
|
|
|
|
Environment
|
|
- The **JSShell** and **XPCShell** environments are limited
|
|
JavaScript execution environments with no windows or user
|
|
interface (note however that XPCShell tests on Android are run
|
|
within a browser window.)
|
|
- The **Content** indication means that the test is run inside a
|
|
content page loaded by a browser window.
|
|
- The **Browser** indication means that the test is loaded in the
|
|
JavaScript context of a browser XUL window.
|
|
- The **Browser Profile** column indicates whether a browser profile
|
|
is loaded when the test starts. The **Allow** label means that the
|
|
test can optionally load a profile using a special command.
|
|
|
|
Privilege
|
|
Indicates whether the tests normally run with low (content) or high
|
|
(chrome) JavaScript privileges. The **Allow** label means that the
|
|
test can optionally run code in a privileged environment using a
|
|
special command.
|
|
|
|
.. _Performance_testing:
|
|
|
|
Performance testing
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
There are many test harnesses used to test performance.
|
|
`For more information on the various performance harnesses,
|
|
check out the perf docs. </testing/perfdocs>`_
|
|
|
|
|
|
.. _So_which_should_I_use:
|
|
|
|
So which should I use?
|
|
----------------------
|
|
|
|
Generally, you should pick the lowest-level framework that you can. If
|
|
you are testing JavaScript but don't need a window, use XPCShell or even
|
|
JSShell. If you're testing page layout, try to use
|
|
`web-platform-test reftest.
|
|
<https://web-platform-tests.org/writing-tests/reftests.html>`_
|
|
The advantage in lower level testing is that you don't drag in a lot of
|
|
other components that might have their own problems, so you can home in
|
|
quickly on any bugs in what you are specifically testing.
|
|
|
|
Here's a series of questions to ask about your work when you want to
|
|
write some tests.
|
|
|
|
.. _Is_it_low-level_code:
|
|
|
|
Is it low-level code?
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If the functionality is exposed to JavaScript, and you don't need a
|
|
window, consider `XPCShell </testing/xpcshell/index.html>`__. If not,
|
|
you'll probably have to use `GTest </gtest/index.html>`__, which can
|
|
test pretty much anything. In general, this should be your
|
|
last option for a new test, unless you have to test something that is
|
|
not exposed to JavaScript.
|
|
|
|
.. _Does_it_cause_a_crash:
|
|
|
|
Does it cause a crash?
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If you've found pages that crash Firefox, add a
|
|
`crashtest </web-platform/index.html>`__ to
|
|
make sure future versions don't experience this crash (assertion or
|
|
leak) again. Note that this may lead to more tests once the core
|
|
problem is found.
|
|
|
|
.. _Is_it_a_layoutgraphics_feature:
|
|
|
|
Is it a layout/graphics feature?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
`Reftest </layout/Reftest.html#writing-tests>`__ is your best bet, if possible.
|
|
|
|
.. _Do_you_need_to_verify_performance:
|
|
|
|
Do you need to verify performance?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
`Use an appropriate performance test suite from this list </testing/perfdocs>`_.
|
|
|
|
.. _Are_you_testing_UI_features:
|
|
|
|
Are you testing UI features?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Try one of the flavors of
|
|
`mochitest </testing/mochitest-plain/index.html>`__, or
|
|
`Marionette </docs/Marionette>`__ if the application also needs to be
|
|
restarted, or tested with localized builds.
|
|
|
|
.. _Are_you_testing_MobileAndroid:
|
|
|
|
Are you testing Mobile/Android?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If you are testing GeckoView, you will need to need to use
|
|
`JUnit integration tests
|
|
</mobile/android/geckoview/contributor/junit.html#testing-overview>`__.
|
|
|
|
There are some specific features that
|
|
`Mochitest </testing/mochitest-plain/index.html>`__ or
|
|
`Reftest </layout/Reftest.html>`__ can cover. Browser-chrome tests do not run on
|
|
Android. If you want to test performance, `Raptor </testing/perfdocs/raptor.html>`__ will
|
|
be a good choice.
|
|
|
|
|
|
.. _Are_you_doing_none_of_the_above:
|
|
|
|
Are you doing none of the above?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
- To get your tests running in continuous integration, try
|
|
`web-platform-tests </web-platform/index.html>`_, or
|
|
`Mochitest </testing/mochitest-plain/index.html>`__, or,
|
|
if higher privileges are required, try
|
|
`Mochitest browser chrome tests </testing/mochitest-plain/index.html>`__.
|
|
- For Desktop Firefox, or if you just want to see the future of Gecko
|
|
testing, look into the on-going
|
|
`Marionette </testing/marionette/Testing.html#harness-tests>`__ project.
|
|
|
|
.. _Need_to_get_more_data_out_of_your_tests:
|
|
|
|
Need to get more data out of your tests?
|
|
----------------------------------------
|
|
|
|
Most test jobs now expose an environment variable named
|
|
``$MOZ_UPLOAD_DIR``. If this variable is set during automated test runs,
|
|
you can drop additional files into this directory, and they will be
|
|
uploaded to a web server when the test finishes. The URLs to retrieve
|
|
the files will be output in the test log.
|
|
|
|
.. _Need_to_set_preferences_for_test-suites:
|
|
|
|
Need to set preferences for test-suites?
|
|
----------------------------------------
|
|
|
|
First ask yourself if these prefs need to be enabled for all tests or
|
|
just a subset of tests (e.g to enable a feature).
|
|
|
|
.. _Setting_prefs_that_only_apply_to_certain_tests:
|
|
|
|
Setting prefs that only apply to certain tests
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If the answer is the latter, try to set the pref as local to the tests
|
|
that need it as possible. Here are some options:
|
|
|
|
- If the test runs in chrome scope (e.g mochitest chrome or
|
|
browser-chrome), you can use
|
|
`Services.prefs
|
|
<https://searchfox.org/mozilla-central/source/modules/libpref/nsIPrefBranch.idl>`__
|
|
to set the prefs in your test's setup function. Be sure to reset the
|
|
pref back to its original value during teardown!
|
|
|
|
- Mochitest plain tests can use
|
|
`SpecialPowers
|
|
<https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest/SpecialPowers>`__
|
|
to set prefs.
|
|
|
|
- All variants of mochitest can set prefs in their manifests. For
|
|
example, to set a pref for all tests in a manifest:
|
|
|
|
::
|
|
|
|
[DEFAULT]
|
|
prefs =
|
|
my.awesome.pref=foo,
|
|
my.other.awesome.pref=bar,
|
|
[test_foo.js]
|
|
[test_bar.js]
|
|
|
|
- All variants of reftest can also set prefs in their
|
|
`manifests </layout/Reftest.html>`__.
|
|
|
|
- All variants of web-platform-tests can also `set prefs in their
|
|
manifests </web-platform/index.html#enabling-prefs>`__.
|
|
|
|
.. _Setting_prefs_that_apply_to_the_entire_suite:
|
|
|
|
Setting prefs that apply to the entire suite
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Most test suites define prefs in user.js files that live under
|
|
`testing/profiles
|
|
<https://searchfox.org/mozilla-central/source/testing/profiles>`__.
|
|
Each directory is a profile that contains a ``user.js`` file with a
|
|
number of prefs defined in it. Test suites will then merge one or more
|
|
of these basic profiles into their own profile at runtime. To see which
|
|
profiles apply to which test suites, you can inspect
|
|
`testing/profiles/profiles.json
|
|
<https://searchfox.org/mozilla-central/source/testing/profiles/profiles.json>`__.
|
|
Profiles at the beginning of the list get overridden by profiles at the
|
|
end of the list.
|
|
|
|
Because this system makes it hard to get an overall view of which
|
|
profiles are set for any given test suite, a handy ``profile`` utility
|
|
was created:
|
|
|
|
::
|
|
|
|
$ cd testing/profiles
|
|
$ ./profile -- --help
|
|
usage: profile [-h] {diff,sort,show,rm} ...
|
|
$ ./profile show mochitest # prints all prefs that will be set in mochitest
|
|
$ ./profile diff mochitest reftest # prints differences between the mochitest and reftest suites
|
|
|
|
.. container:: blockIndicator note
|
|
|
|
**Note:** JS engine tests do not use testing/profiles yet, instead
|
|
`set prefs
|
|
here <https://searchfox.org/mozilla-central/source/js/src/tests/user.js>`__.
|
|
|
|
Adding New Context to Skip Conditions
|
|
-------------------------------------
|
|
|
|
Often when standing up new test configurations, it's necessary to add new keys
|
|
that can be used in ``skip-if`` annotations.
|
|
|
|
.. toctree::
|
|
|
|
manifest-sandbox
|