mirror of
https://github.com/mozilla/gecko-dev.git
synced 2025-11-04 10:18:41 +02:00
7b6a1139ad
Fix some typos in new directories Differential Revision: https://phabricator.services.mozilla.com/D165660
154 lines
5.5 KiB
ReStructuredText
154 lines
5.5 KiB
ReStructuredText
Developing in mozperftest
|
|
=========================
|
|
|
|
Architecture overview
|
|
---------------------
|
|
|
|
`mozperftest` implements a mach command that is a thin wrapper on the
|
|
top of `runner.py`, which allows us to run the tool without having to go through
|
|
a mach call. Command arguments are prepared in `argparser.py` and then made
|
|
available for the runner.
|
|
|
|
The runner creates a `MachEnvironment` instance (see `environment.py`) and a
|
|
`Metadata` instance (see `metadata.py`). These two objects are shared during the
|
|
whole test and used to share data across all parts.
|
|
|
|
The runner then calls `MachEnvironment.run`, which is in charge of running the test.
|
|
The `MachEnvironment` instance runs a sequence of **layers**.
|
|
|
|
Layers are classes responsible of one single aspect of a performance test. They
|
|
are organized in three categories:
|
|
|
|
- **system**: anything that sets up and tears down some resources or services
|
|
on the system. Existing system layers: **android**, **proxy**
|
|
- **test**: layers that are in charge of running a test to collect metrics.
|
|
Existing test layers: **browsertime** and **androidlog**
|
|
- **metrics**: all layers that process the metrics to turn them into usable
|
|
metrics. Existing system layers: **perfherder** and **console**
|
|
|
|
The MachEnvironment instance collects a series of layers for each category and
|
|
runs them sequentially.
|
|
|
|
The goal of this organization is to allow adding new performance tests runners
|
|
that will be based on a specific combination of layers. To avoid messy code,
|
|
we need to make sure that each layer represents a single aspect of the process
|
|
and that is completely independent from other layers (besides sharing the data
|
|
through the common environment.)
|
|
|
|
For instance, we could use `perftest` to run a C++ benchmark by implementing a
|
|
new **test** layer.
|
|
|
|
|
|
Layer
|
|
-----
|
|
|
|
A layer is a class that inherits from `mozperftest.layers.Layer` and implements
|
|
a few methods and class variables.
|
|
|
|
List of methods and variables:
|
|
|
|
- `name`: name of the layer (class variable, mandatory)
|
|
- `activated`: boolean to activate by default the layer (class variable, False)
|
|
- `user_exception`: will trigger the `on_exception` hook when an exception occurs
|
|
- `arguments`: dict containing arguments. Each argument is following
|
|
the `argparser` standard
|
|
- `run(self, medatata)`: called to execute the layer
|
|
- `setup(self)`: called when the layer is about to be executed
|
|
- `teardown(self)`: called when the layer is exiting
|
|
|
|
Example::
|
|
|
|
class EmailSender(Layer):
|
|
"""Sends an email with the results
|
|
"""
|
|
name = "email"
|
|
activated = False
|
|
|
|
arguments = {
|
|
"recipient": {
|
|
"type": str,
|
|
"default": "tarek@mozilla.com",
|
|
"help": "Recipient",
|
|
},
|
|
}
|
|
|
|
def setup(self):
|
|
self.server = smtplib.SMTP(smtp_server,port)
|
|
|
|
def teardown(self):
|
|
self.server.quit()
|
|
|
|
def __call__(self, metadata):
|
|
self.server.send_email(self.get_arg("recipient"), metadata.results())
|
|
|
|
|
|
It can then be added to one of the top functions that are used to create a list
|
|
of layers for each category:
|
|
|
|
- **mozperftest.metrics.pick_metrics** for the metrics category
|
|
- **mozperftest.system.pick_system** for the system category
|
|
- **mozperftest.test.pick_browser** for the test category
|
|
|
|
And also added in each `get_layers` function in each of those category.
|
|
The `get_layers` functions are invoked when building the argument parser.
|
|
|
|
In our example, adding the `EmailSender` layer will add two new options:
|
|
|
|
- **--email** a flag to activate the layer
|
|
- **--email-recipient**
|
|
|
|
|
|
Important layers
|
|
----------------
|
|
|
|
**mozperftest** can be used to run performance tests against browsers using the
|
|
**browsertime** test layer. It leverages the `browsertime.js
|
|
<https://www.sitespeed.io/documentation/browsertime/>`_ framework and provides
|
|
a full integration into Mozilla's build and CI systems.
|
|
|
|
Browsertime uses the selenium webdriver client to drive the browser, and
|
|
provides some metrics to measure performance during a user journey.
|
|
|
|
|
|
Coding style
|
|
------------
|
|
|
|
For the coding style, we want to:
|
|
|
|
- Follow `PEP 257 <https://www.python.org/dev/peps/pep-0257/>`_ for docstrings
|
|
- Avoid complexity as much as possible
|
|
- Use modern Python 3 code (for instance `pathlib` instead of `os.path`)
|
|
- Avoid dependencies on Mozilla build projects and frameworks as much as possible
|
|
(mozharness, mozbuild, etc), or make sure they are isolated and documented
|
|
|
|
|
|
Landing patches
|
|
---------------
|
|
|
|
.. warning::
|
|
|
|
It is mandatory for each patch to have a test. Any change without a test
|
|
will be rejected.
|
|
|
|
Before landing a patch for mozperftest, make sure you run `perftest-test`::
|
|
|
|
% ./mach perftest-test
|
|
=> black [OK]
|
|
=> flake8 [OK]
|
|
=> remove old coverage data [OK]
|
|
=> running tests [OK]
|
|
=> coverage
|
|
Name Stmts Miss Cover Missing
|
|
------------------------------------------------------------------------------------------
|
|
mozperftest/metrics/notebook/analyzer.py 29 20 31% 26-36, 39-42, 45-51
|
|
...
|
|
mozperftest/system/proxy.py 37 0 100%
|
|
------------------------------------------------------------------------------------------
|
|
TOTAL 1614 240 85%
|
|
|
|
[OK]
|
|
|
|
The command will run `black`, `flake8` and also make sure that the test coverage has not regressed.
|
|
|
|
You can use the `-s` option to bypass flake8/black to speed up your workflow, but make
|
|
sure you do a full tests run. You can also pass the name of one single test module.
|