In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
This resolves a long-standing issue in development where `mach artifact` (and therefore `mach bootstrap`) would fail unpredictably if you had dirty, but ignored, files in your checkout. Resolving this problem often required unwieldy `hg purge`/`git ignore` incantations that are easy to get wrong.
This patch addresses the problem by doing what we "should" have been doing all along, and consulting the VCS to list tracked files rather than listing EVERY file on disk and applying heuristics to determine whether they should be included in the hash.
Differential Revision: https://phabricator.services.mozilla.com/D86780
Now you can pass the `virtualenv_name` kwarg to the `Command` decorator which will configure the `_virtualenv_manager` accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D86256
This check is unsound; `virtualenv` binaries are apparently not guaranteed to have the same file size as the `python` binaries used to create those `virtualenv`s, at least not with our current vendored version of the `virtualenv` library on macOS. This is trivially reproducible on my own Macbook:
```
rickystewart-a5lvdq:mozilla-unified rickystewart$ rm -rf obj-x86_64-apple-darwin19.5.0/
rickystewart-a5lvdq:mozilla-unified rickystewart$ ./mach configure
...
rickystewart-a5lvdq:mozilla-unified rickystewart$ python3 -c 'import os; print(os.path.getsize("obj-x86_64-apple-darwin19.5.0/_virtualenvs/init_py3/bin/python"))'
16644 # <- ACTUAL VIRTUALENV SIZE
rickystewart-a5lvdq:mozilla-unified rickystewart$ python3 -c 'import os; print(os.path.getsize("/usr/local/opt/python/bin/python3.7"))'
17704 # <- SIZE OF THE PYTHON USED TO CREATE THE VIRTUALENV
```
Concretely, this was causing unit tests to be very aggressive about deleting the parent `init_py3` `virtualenv` repeatedly in unit tests, resulting in failures. The removal of this check fixes the issue.
Differential Revision: https://phabricator.services.mozilla.com/D86872
Now you can pass the `virtualenv_name` kwarg to the `Command` decorator which will configure the `_virtualenv_manager` accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D86256
To check if pipenv is installed, the existing logic looked to see if the "pipenv" binary existed. However, on Windows, this binary is named "pipenv.exe".
Differential Revision: https://phabricator.services.mozilla.com/D86680
Pipenv enthusiastically checks pyenv for possible Python installations, and it seems to always want to use the most
up-to-date version possible. This can lead to issues if the version of python used for the regular venvs is older
than another version of python that exists on the machine (such as if the system python is 3.8.2, but pyenv has
3.8.3 installed).
This patch addresses this by scoping pipenv to match the currently-running version of python (if major versions line
up).
Differential Revision: https://phabricator.services.mozilla.com/D86448
Without this patch, the last "path" in this list will always be the empty string due to how the `-z` option to `git` works. This mirrors what we already do in the `get_files_in_working_directory` implementation for `hg`.
Differential Revision: https://phabricator.services.mozilla.com/D86752
There's a Windows Defender CLI (`Get-MpComputerStatus`) available, but MozillaBuild
can't (easily) access PowerShell. So, instead, we find Windows Defender status and path exclusions
by checking the registry at HKLM\SOFTWARE\Microsoft\Windows Defender.
Determining if Windows Defender is "on" or not is a surprisingly vague task.
Not only does it encompass a lot of components (of which Ricky and I believe "Real-time protection"
is the part slowing down the build), but there's (at least?) two different kinds of "disabled"
states that it can be in:
* If "disabled" via Settings, it will turn itself back on after a reboot
* If an antivirus is installed, it will turn itself off permanently
Unfortunately, disabling "Real-time protection" in Settings doesn't affect any registry keys, but
I'm opting to ignore this because I doubt many users are manually disabling this protection before
each build. The repercussion of this is that users may be incorrectly warned "your Firefox
directory isn't excluded from Windows Defender!".
Focusing on the antivirus use case and querying the registry I found that:
* Querying the `IsServiceRunning` value wasn't sufficient because new Windows installations didn't have that key
* Querying the `DisableRealtimeMonitoring` value was inconsistent - it can be missing, and not always because an antivirus removed it
* Querying the `DisableAntiVirus` value doesn't _sound_ accurate (we care about Real-time Protection), but it's consistently "off" for non-AV machines, and "on" for machines with an AV installed. So, this is our winner!
TL;DR: there may be some "false positive" warnings about excluding the Firefox srcdir, but they're
accurate for my test cases and workaround-able (just add the exclusion to Windows Defender).
Also, this patch updates our Windows Defender docs to make them:
* More easily link-able
* Have direct advice to resolve the issue
Differential Revision: https://phabricator.services.mozilla.com/D85952
I wrote this patch because I noticed that the `.pth` files in my `objdir` `virtualenv`s were extremely repetitive, containing multiple references to the same directories. This happens because we [append](https://searchfox.org/mozilla-central/rev/03794edd6edcc3fc1e222de966cb27256ce08998/python/mozbuild/mozbuild/virtualenv.py#366) to the in-`virtualenv` `.pth` files when calling `populate()`, but we don't ever clean up the old ones, meaning that whenever we determine that the `virtualenv`s are out of date and need to be recreated, we actually leave A LOT of state lying around on-disk that is going to go on to impact further uses of the `virtualenv`. Concretely, how this manifested is that when I erroneously removed an entry from `virtualenv_packages.txt`, the build actually succeeded because that entry was still in the `.pth` file in the `virtualenv`; instead of "creating" a new `virtualenv` with the correct `.pth` files, it just appended the new `.pth` data to the old, stale data.
I've chosen to address this by completely deleting the entire `virtualenv` when we try to re-create it. Another way you might solve this problem is by doing a `find $VIRTUALENV -name '*.pth' | xargs rm` before doing the `virtualenv` re-creation, but I'm suggesting we do it this way because we have had a long history of difficulty with `virtualenv` persistence. Bug 1628498 is an obvious example; note that we would never have encountered that bug if we always unconditionally deleted the `virtualenv` before creating a new one, as in this patch. A patch that is laser-targeted at handling the issue with `.pth` files might be fine for now but this is more foolproof and future-proof.
Differential Revision: https://phabricator.services.mozilla.com/D85636
To ensure that a python 3 virtualenv exists for tests, python-test will ensure that it's created (regardless of which version of python is currently in use).
However, the existing logic was incorrectly creating an "extra py3 virtualenv", even if currently running python 3 and having already created a "first" py3 virtualenv
Differential Revision: https://phabricator.services.mozilla.com/D86269
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
When subprocessing to get the venv from pipenv with PyCharm, the returned path has a command sequence to reset the terminal color.
This command sequence is unexpected, and causes the returned path to be incorrect.
The root issue is that colorama mistakenly believes that, if it's ever running underneath PyCharm, it's probably attached to a tty.
However, this assumption isn't true if PyCharm is running a script which subprocesses out a colorama-enabled script.
This issue has already been reported here: https://github.com/tartley/colorama/issues/263
To workaround this issue, we clear the "PYCHARM_HOSTED" environment variable when we invoke pipenv, which works around the colorama logic.
Differential Revision: https://phabricator.services.mozilla.com/D86242
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
The resource file is always generated so being able to configure its name
is not useful. On the other hand, the way things are currently implemented,
the lack of RESFILE also makes RCFILE ignored, which we fix at the same
time.
And remove a spurious RESFILE in widget/windows/moz.build, where no binary
is produced, which means RESFILE had no meaning.
Differential Revision: https://phabricator.services.mozilla.com/D86154
The resource file is always generated so being able to configure its name
is not useful. On the other hand, the way things are currently implemented,
the lack of RESFILE also makes RCFILE ignored, which we fix at the same
time.
And remove a spurious RESFILE in widget/windows/moz.build, where no binary
is produced, which means RESFILE had no meaning.
Differential Revision: https://phabricator.services.mozilla.com/D86154
Add a modular approach for the integration of `static-analysis` module in order
to be able to share components of it with other modules, like the integration of
`clangd` in `vscode` where we need to have access to the configuration of `clang-tidy`
in order to have `in-ide` `static-analysis` messages.
In this initial step we make a separate module for the clang-tidy configuration.
Differential Revision: https://phabricator.services.mozilla.com/D85979
In order to have a cross platform ide for C++ language support we've added `clangd`
extenssion and artifact part of `vscode` suite.
To generate the configuration you simply run:
`./mach ide vscode `.
Differential Revision: https://phabricator.services.mozilla.com/D85416
Contrary to python2, python3 considers the values in cl_lnotab to be signed
integers, so with python3, offsets larger than 127 would be encoded in a way
that would make them wrong, or worse, negative.
Differential Revision: https://phabricator.services.mozilla.com/D86122
These tests depend on the `mach uuid` command which was deleted with bug 1639509, and now that `mach uuid` is gone it's broken unconditionally. We could replace the reference to `uuid` with a new no-op `mach` command, but we're in the process of replacing our telemetry code with use of the `glean` API; and the new telemetry code won't have the same semantics (namely, we are unlikely to want to continue to guarantee that sub-`mach` invocations aren't covered by telemetry), so this test might as well just be deleted now.
Differential Revision: https://phabricator.services.mozilla.com/D85911
