What are we doing:
- Resolving a few bugs/user requests
Issues being addressed:
- Resolved issue where if the WPT_key.txt file is not available locally it does not affect running ./mach perftest-test
- Added section to WPT where we display the amount of tests we have remaining
- Altered the request_with_timeout function, to better handle requests
Differential Revision: https://phabricator.services.mozilla.com/D155268
The older versions don't have prebuilt wheels for python 3.9 and newer.
Unfortunately, the latest versions don't support python 3.7 and older,
so keep the older versions for those.
Differential Revision: https://phabricator.services.mozilla.com/D152246
A week ago we received a notification that we had a test that the WPT chrome tests were perma failing on [[ https://bugzilla.mozilla.org/show_bug.cgi?id=1773621 | bugzilla ]]
After going through the fail logs I realized it was because of website "panda.tv" directing to a unable to connect page message, after some digging it was not returning proper data because panda.tv has not been a company since March 2019(bankruptcy filing), why this only is causing an issue now I believe is because of some kind of update from WPT as I can see a noticeable UI difference on the test results page from before and after the failures started.
My resolution was to remove Panda.tv from our test list and that seems to have resolved the issue.
I also updated the error message to display which website is causing the issue so that if this happens again I don't need to go through each and every webpagetest result to know which of the 40 websites are having an issue.
Differential Revision: https://phabricator.services.mozilla.com/D149642
The patch aims to improve the `_should_install()` method for installing browsertime in mozperftest.
Here, the same approach used in `testing/raptor/mach_commands.py` is used to address potential issues (e.g. KeyError)
that one may encounter when trying to install from the `package.json` through this approach.
Differential Revision: https://phabricator.services.mozilla.com/D148565
This next patch in the series utilizes the same login-logic in Mozperftest and makes it available to the `pageload_test` method so that we can now automate the logging into of accounts during perftest recordings.
Additional logic is also added to account for if the site requires login, if we are running on CI or locally (and if on CI, accounting for the SCM level), and removal of the verbose flags so secrets do not leak.
Differential Revision: https://phabricator.services.mozilla.com/D147775
* There is no `--duration` argument, but there //is// `--durations`
* `pytest` behaves better when called via its main entry points (either
`bin/pytest`, or `-m pytest`)
Differential Revision: https://phabricator.services.mozilla.com/D143609
All commands declaring a virtualenv will have them activated before the
command executes. Removes all now-redundant manual activations of
declared virtualenvs.
Commands that don't declare a virtualenv will still implicitly be
associated with the "common" virtualenv, but unlike explicit
virtualenv declarations it'll have to be activated manually, just
like it was before this patch.
To smooth the migration with existing usages, virtualenv activation
behaviour was changed slightly: if attempting to activate a new
virtualenv, but the source venv is already command venv, then raise an
exception. (In the future, we should improve testability of
virtualenv scaffolding logic so that tests can be added for this
sort of thing.) This did cause some issues with some tests, which
will be solved more cleanly with bug 1724273. In the meantime,
minimal modifications were made to failing tests to keep them green:
* `test_command_line.py` was activating the `common` virtualenv so
that it could install `mozproxy`, and use its CLI. Instead, I
modified the test to use `mozproxy` using the "module" interface
(`python -m mozproxy ...`). At that point, `MozbuildObject` was
unnecessary and usages were replaced with simpler variants.
* `test_vendor.py` needed its explicit `activate_virtualenv()` call
patched out. It still needs to use a virtualenv's Python
executable, but due to `sys.executable` now being kept up-to-date
as of bug 1717051, it could be used directly.
Differential Revision: https://phabricator.services.mozilla.com/D122892
`coverage` has native code, so the vendored version was only used as
source code from which the actual package could be built.
Since its always used in a context where we can `pip install` over the
network, let's do that. This cleans up our tree a bit and allows us to
leverage the `coverage` wheels.
Differential Revision: https://phabricator.services.mozilla.com/D138816
This patch adds the --proxy-perftest-page option. With this option, we'll be able to specify which test pages we want to record rather than modifying the pageload_sites.json. Note that the login fields will not be taken into consideration with this option.
You can use the flag as follows (seperate multiple pages with a comma): --proxy-perftest-page microsft,linkedin,netflix
Furthermore, some changes were made to strengthen the conditions around when a login site can be tested (only when RAPTOR_LOGINS is defined locally, or if we are in CI).
Differential Revision: https://phabricator.services.mozilla.com/D137468
Though _most_ of the old paths that used to be defined in
`mozperftest/runner.py` exist in `common_virtualenv_packages.txt`,
"xpcshell" was not because it's put in a different location depending on
whether the source directory is sourced from the "target.xpcshell.tests"
artifact or from VCS directly.
As part of this change, I've verified that all of the other
path changes in D132503 are fulfilled by the "mach" site.
Differential Revision: https://phabricator.services.mozilla.com/D133215
When activating a command virtualenv, it rightfully complains if it
isn't already running within an active Mach site - the abstraction has
restrictions to act as guardrails around the risky business that is
`sys.path` patching.
Replace `mozperftest/runner.py`'s ad-hoc `sys.path` initializing with
`MachSiteManager`'s `activate()` instead.
Removes the calls to `_setup_path()` where the `sys.path` should
already be set-up.
Differential Revision: https://phabricator.services.mozilla.com/D132503
Build and run the Mach virtualenv from a `state_dir` that is
"specific-to-topsrcdir".
As part of this, move `get_state_dir()` to `mach` so that it's usable
before `sys.path` entries are fully set up.
Differential Revision: https://phabricator.services.mozilla.com/D130383
As `_run_pip()` is being removed from `VirtualenvManager` in an upcoming
patch, its usages need to be removed. Besides, they're using an
"internal" function, which is a bit of a smell.
Note that this _could_ have been solved by exposing a public `run_pip()`
function. However, I felt like that was worse because:
* Friction here is good as we try to migrate the codebase to embrace the
"requirements definition file" technique to install dependencies.
* There could be confusion about the relationship between
`install_pip_package()` (only works if venv already activated)
and `_run_pip()`, which works "in general".
Differential Revision: https://phabricator.services.mozilla.com/D130120
The logic to handle virtualenv requirements definitions now need to be
able to import `packaging` and `pyparsing` in order to parse the pip
requirement specifier notation.
Differential Revision: https://phabricator.services.mozilla.com/D130288
Now that are prioritizing system over virtualenv site-packages, the
system `pip` is sometimes being used instead.
This is causing issues when the system pip is set up in a
distro-specific way, such as when "debundled":
https://github.com/pypa/pip/blob/9.0.1/pip/_vendor/__init__.py#L53-L61
However, if we vendor `pip`, `setuptools` and `wheel`, and ensure that
they're prioritized in the `sys.path` before anything is imported from
the system, then we can ensure that we're using a modern `pip` _and_
sidestep system-specific pip weirdness.
Note that `pip-compile`'s `--allow-unsafe` flag is not as dangerous as
it sounds.
There's confusion among maintainers about its origin:
https://github.com/jazzband/pip-tools/issues/522
Additionally, it's going to be enabled by default in a future
`pip-tools` release. So, it's not scary for us to embrace here.
Also, heads up that the "pip outdated warning" no longer needs
to be manually silenced, since pip avoids that code path when
not running from an "installed" context.
Differential Revision: https://phabricator.services.mozilla.com/D127182
The `pytest` dependency is going to be moved from `common` to the
`python-test` environment. Shift all existing `pytest` usages to
the virtualenv accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D126287
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.
Differential Revision: https://phabricator.services.mozilla.com/D118058
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.
Differential Revision: https://phabricator.services.mozilla.com/D118058
