When `./mach bootstrap` is called with the default mozconfig file,
verify the content matches the selected build target, and if not,
show a warning and ask whether to overwrite or not.
Differential Revision: https://phabricator.services.mozilla.com/D146384
The cargo-vet toolchain is auto-bootstrapped and setup for things to
work properly. We modify `mach vendor rust` to invoke `mach cargo vet`
instead of doing its own setup, but in a underhanded way to work around
bug 1772453.
Differential Revision: https://phabricator.services.mozilla.com/D148218
These were broken with Bas's region rewrite. They seem to just
pretty 'empty' now which is less useful than not having a pretty
printer at all.
Differential Revision: https://phabricator.services.mozilla.com/D148459
Upgrades to Glean v50.0.1, which comes with a rewritten core and
UniFFI-powered bindings.
Glean has some API changes, so we swap it over to that. Mostly mechanical changes.
Also upgrades to inherent v1.0 in fog.
This matches what Glean uses internally and gets rid of one duplicated crate.
Also upgrades to glean-parser==6.0.1
One crate duplication now (change in `python/mozbuild/mozbuild/vendor/vendor_rust.py` required).
Some new crates now vendored.
These are transitive dependencies of Glean dependencies, all with valid
licenses and already used in other products (mobile).
Differential Revision: https://phabricator.services.mozilla.com/D146062
The function2 library is a header-only library which provides support
for defining move-only function types, similar to the proposed
std::move_only_function in C++23, but with support for additional
customization.
This appears to be the first time we've vendored code using the boost
license, so I've added it to license.html and moz_yaml.py, and have
requested review to ensure it is OK to use code with this license.
Differential Revision: https://phabricator.services.mozilla.com/D145690
In bug 1769902, we allowed such duplicate crates when one of them is
under build/rust. This changes the logic to be more generic and allow
a crate duplicate when it depends on a crate of the same name, whether
in build/rust or elsewhere, which could happen for vendored transitional
crates (rare, but happens). This is also a more generic way to handle
our situation with build/rust and doesn't rely on parsing Cargo.toml.
Differential Revision: https://phabricator.services.mozilla.com/D147824
This is deliberately simple. Future improvements will report if the
vendoring doesn't produce the same content as what's in the tree, and
attach errors to a better location than the first line of Cargo.lock.
Differential Revision: https://phabricator.services.mozilla.com/D147466
The 'U' flag represents "universal newline". It has been deprecated
since Python3.3. Since then "universal newline" is the default when a
file is opened in text mode (not bytes). In Python3.11 using the 'U'
flag throws errors. There should be no harm in removing 'U' from 'open'
everywhere it is used, and doing allows the use of Python3.11.
For more reading see: https://docs.python.org/3.11/whatsnew/3.11.html#changes-in-the-python-api
Differential Revision: https://phabricator.services.mozilla.com/D147721
Added two additional string types to be pretty printed by the string
pretty printer.
Added the children method to smartpr, which yields 1 child. Has
the added benefit of returning (name, gdb.Value) which can be used
from an IDE with support for Python in GDB.
Differential Revision: https://phabricator.services.mozilla.com/D141404
While testing the change, I noticed that the old implementation also
considered the 'current' path as a part of all ancestors. As such, I
added an additional path component to the path that is immediately
discarded so that the behavior is consistent.
Differential Revision: https://phabricator.services.mozilla.com/D142273
In test_base.py the backslashes had to be replaced with forward slashes,
otherwise the string isn't read properly from the mozconfig during the
test. This was always 'broken', but the test just happened to pass since
the old path comparison was naive.
Differential Revision: https://phabricator.services.mozilla.com/D142272
Created GeneratedFile method implementing HTML Fragment Preprocessor.
Usage in a `moz.build` file will be as follows
```
GeneratedFile(
"example.js",
script="/python/mozbuild/mozbuild/action/html_fragment_preprocesor.py",
entry_point="generate",
inputs=["example_a.xml", "example_b.xml"],
)
```
Example XML files for testing are in `python/mozbuild/mozbuild/test/action/data/html_fragment_preprocesor/`
Differential Revision: https://phabricator.services.mozilla.com/D146255
Created GeneratedFile method implementing HTML Fragment Preprocessor.
Usage in a `moz.build` file will be as follows
```
GeneratedFile(
"example.js",
script="/python/mozbuild/mozbuild/action/html_fragment_preprocesor.py",
entry_point="generate",
inputs=["example_a.xml", "example_b.xml"],
)
```
Example XML files for testing are in `python/mozbuild/mozbuild/test/action/data/html_fragment_preprocesor/`
Differential Revision: https://phabricator.services.mozilla.com/D146255
We're going to add crates in build/rust that help reduce crate
duplication by providing older versions of crates based on newer ones,
which implies there would be at least 2 crates with the same name, one
of them being under build/rust. Allow that when vendoring.
Differential Revision: https://phabricator.services.mozilla.com/D146661
Future releases of git cinnabar may not have a "git cinnabar" command
running out of the box, and the future-proof way to download the
cinnabar helper is to use download.py.
Differential Revision: https://phabricator.services.mozilla.com/D146526
The function2 library is a header-only library which provides support
for defining move-only function types, similar to the proposed
std::move_only_function in C++23, but with support for additional
customization.
This appears to be the first time we've vendored code using the boost
license, so I've added it to license.html and moz_yaml.py, and have
requested review to ensure it is OK to use code with this license.
Differential Revision: https://phabricator.services.mozilla.com/D145690
The filesystem lastaccess check can return `None` on some platforms,
which is unexpected. Return a `DoctorCheck` indicating the check is
skipped instead.
The filesystem lastaccess check only accounts for possible values {0, 1},
when the spec returned by `fsutil` indicates 2 and 3 are also possible
values, mapped to "system managed" lastaccess. Account for these values
in the check.
Finding this bug is simple when adding type hints, since the function
with the bug will have the return type `Optional[DoctorCheck]` as
detected by the type checker, so add them as a ridealong.
Differential Revision: https://phabricator.services.mozilla.com/D145972
Since we don't encode the output, the resulting value is a `bytes`
and therefore using `"="` to split results in a `TypeError`.
Add a specified encoding to force a `str` and fix the exception.
Differential Revision: https://phabricator.services.mozilla.com/D145881
Brew's Python 3.10 causes `virtualenv==20.7.2` to produce a wonky folder
structure (`$venv/opt/homebrew/lib/python3.10/site-packages`?).
This is likely fixed with newer `virtualenv`, but the simpler workaround
here is to use `venv` instead now that Python 3 is always used.
Adds `python3-venv` to docker image so that tests and debian-based tasks
can leverage it.
Differential Revision: https://phabricator.services.mozilla.com/D144872
Brew's Python 3.10 causes `virtualenv==20.7.2` to produce a wonky folder
structure (`$venv/opt/homebrew/lib/python3.10/site-packages`?).
This is likely fixed with newer `virtualenv`, but the simpler workaround
here is to use `venv` instead now that Python 3 is always used.
Adds `python3-venv` to docker image so that tests and debian-based tasks
can leverage it.
Differential Revision: https://phabricator.services.mozilla.com/D144872
Messages used by the `about:addons` page are migrated to `aboutAddons.ftl`.
Unused messages are dropped, as is the `get_string()` helper from `test/browser/head.js`.
Differential Revision: https://phabricator.services.mozilla.com/D131793
Virtualenv-based objdir detection doesn't work for instrumented builds,
as instrumentation-specific objdirs aren't created.
Resolve this by:
* Removing redundant cwd-based topsrcdir resolution - the
`__file__`-based code path below covers this case.
* Only using virtualenv-based resolving if topsrcdir isn't resolved.
* Documenting that virtualenv-based resolution is problematic for
instrumented builds.
Differential Revision: https://phabricator.services.mozilla.com/D144030
The default download folder prefs are now counterintuitively used
whether browser.download.useDownloadDir is enabled or not, since we no
longer save downloads to the "temp" folder. But the display for these
prefs in about:preferences makes it seem as though the configured path
will not be used if the "Always ask..." option is selected. This patch
removes the radiogroup and replaces it with a checkbox, so that the
download folder path can be changed irrespective of useDownloadDir's
value (unless browser.download.dir is disabled by policy).
Differential Revision: https://phabricator.services.mozilla.com/D143555
Virtualenv-based objdir detection doesn't work for instrumented builds,
as instrumentation-specific objdirs aren't created.
Resolve this by:
* Removing redundant cwd-based topsrcdir resolution - the
`__file__`-based code path below covers this case.
* Only using virtualenv-based resolving if topsrcdir isn't resolved.
* Documenting that virtualenv-based resolution is problematic for
instrumented builds.
Differential Revision: https://phabricator.services.mozilla.com/D144030
This allows to remove the requirement on "--help" for dependable(),
which avoids dependable() implying the execution of `when` when it's
within a `with only_when`.
Differential Revision: https://phabricator.services.mozilla.com/D144404
1. add the MOZ_X11 config flag in build/gn.mozbuild and set the gn_vars
accordingly.
2. create the new gn-config/mozconfig files and delete previous ones
dom/media/webrtc/third_party_build/gn-configs/**.mozconfig with
--enable-default-toolkit=cairo-gtk3-wayland-only for the non X11 version.
New toolkit nmae is required to force disable X11 detection as
cairo-gtk3-wayland will auto-detect X11 and make generate-gn-build-files.sh
fail.
3. Add the MOZ_X11 config flag in python/mozbuild/mozbuild/gn_processor.py
Then run
dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
Differential Revision: https://phabricator.services.mozilla.com/D142904
Until clang 14, the default was -ffp-contract=fast, but some reason
didn't really enable FP contraction in practice. GCC has the same
default, but does, in fact enable FP contraction*. This leads to more
inconsistencies in FP results between builds than would normally be
expected, and has an effect on e.g. fingerprinting resistance in fdlibm
trigonometric functions, or some other places (e.g. bug 1765436).
As far as the Firefox versions Mozilla ships are concerned, FP
contraction only really affects aarch64, but this change should in
practice make no difference with the current version of clang used (13),
and would preserve our now new default behavior with clang 14.
(*) We never noticed because we neither build with GCC for arm64 nor run
tests with GCC builds.
Differential Revision: https://phabricator.services.mozilla.com/D144136
The `test_vendor.py` test needs access to `poetry` to make its
assertions. This requirement was previously satisfied by adding `poetry`
to the `python-test` site.
Unfortunately, this caused conflicts when running the `mozlint` tests,
as they use the ini "requirements" feature to install their
dependencies. Importantly, the `mozlint` dependencies aren't compatible
with the version of `poetry` we need.
Fortunately, this was a good reminder: since `poetry` is a test-specific
dependency, leverage the existing ini "requirements" feature as it
represents exactly that: the extra package dependencies of a specific
test.
Note that I had to doctor the `dataclass` line in
`vendor_requirements.txt` to include an environment marker, otherwise it
would fail to install with Python >= 3.7.
Differential Revision: https://phabricator.services.mozilla.com/D143376
1. add the MOZ_X11 config flag in build/gn.mozbuild and set the gn_vars
accordingly.
2. create the new gn-config/mozconfig files and delete previous ones
dom/media/webrtc/third_party_build/gn-configs/**.mozconfig with
--enable-default-toolkit=cairo-gtk3-wayland-only for the non X11 version.
New toolkit nmae is required to force disable X11 detection as
cairo-gtk3-wayland will auto-detect X11 and make generate-gn-build-files.sh
fail.
3. Add the MOZ_X11 config flag in python/mozbuild/mozbuild/gn_processor.py
Then run
dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh
Differential Revision: https://phabricator.services.mozilla.com/D142904
This makes some ./mach vendor improvements needed:
- Adds a replace-in-file-regex action
- Allows you to skip copying the temporary extract directory
so that a script can do it
- Removes 'commit' from the look-in-moz.yaml-for-release match
Differential Revision: https://phabricator.services.mozilla.com/D142720
This change does a few things
- Moves get_full_path from a function-local function to
a class member
- Makes yaml_file a class member so I don't need to pass
it everywhere
- Creates a new moz.yaml file directive 'generated'
- Adds that directive to libdav1d
- Detects if any files (other than generated ones) were
modified by an update; and if not - do not update the
moz.yaml. Exit with -2 so Updatebot can detect it.
Differential Revision: https://phabricator.services.mozilla.com/D141994
The `upload_geneated_sources.py` script needs `boto3`, but was:
* Running in CI with Mach having a site-packages-source of `None`
* Using the `common` site (which inherits Mach's site-packages-source)
* Installing `boto3` outside of the knowledge of the centralized
dependency system.
Make a `upload-generated-sources` site, and move `boto3` to it.
Differential Revision: https://phabricator.services.mozilla.com/D143715
* There is no `--duration` argument, but there //is// `--durations`
* `pytest` behaves better when called via its main entry points (either
`bin/pytest`, or `-m pytest`)
Differential Revision: https://phabricator.services.mozilla.com/D143609
De-duping with `set` causes order to no longer be maintained when the
items are distinct. Use the `OrderedDict` technique to de-dupe
consistently instead.
Differential Revision: https://phabricator.services.mozilla.com/D143604
I believe this is safe because from what I've been able to find
SafeLoader is FullLoader but without the bad stuff that lets you
execute arbitrary python code. We don't need (nor want) the more
comlpicated data types like 'dates' and 'numbers' so we can use
BaseLoader which does nothing except treat everything as a string.
Depends on D142125
Differential Revision: https://phabricator.services.mozilla.com/D142126
Both the dtd and properties files strings are moved to the existing storage.ftl.
A migration file was added to keep the localized strings.
Some of the strings were migrated to a declarative approach, setting `data-l10n-id` and
`data-l10n-args` attributes on elements. But in other cases, this was not easily doable.
To accomodate with the asynchronicity of `formatValue`, we translate all the strings that
don't have dynamic parameters during the initialization of the panel, and store them in
a Map for easy retrieval.
We also took this opportunity to cleanup the strings key names and have them all prefixed
with `storage-*`. Some functions where refactored so it should be easier to search the
code for a specific l10n string.
Finally, the `table.headers.indexedDB.*` strings weren't migrated as they were never read
in the UI (they are referenced in `NON_L10N_STRINGS`, which bypass localization).
Differential Revision: https://phabricator.services.mozilla.com/D143012
Verify that activating mach and command sites in various configurations
affects the `sys.path` accurately.
Also verify that the `sys.path` is correct between the "activated
process" and fresh processes spawned from created virtualenvs.
Finally, tweak existing path-management behaviour to be more consistent:
* Don't unnecessarily sort `pths` from requirement definitions - it's
more consistent (and more dependable) to keep the provided order.
* Continue removing unwanted `venv` entries from the `sys.path`, but
don't add them back (such as the `$prefix` entry)
* Always remove `venv` entries from the `sys.path` when the venv isn't
being populated. Though this doesn't affect behaviour, it makes
assertions easier and simplifies the runtime state a bit.
Differential Revision: https://phabricator.services.mozilla.com/D143201
Previously, when using the system Python packages, Mach would reuse the
values already existing in the `sys.path`. This had two benefits:
1. We didn't have to do work to calculate which paths the "system
Python" specifically referred to.
2. This allowed us to support nested Mach calls
(such as `./mach --virtualenv psutil ./mach build`).
However, it came with its own headaches, specifically around "consistent
imports" and Python subprocesses, such as in the following example
1. Mach runs "using system Python"
2. The "build" site (for example) would be activated. The current
`sys.path` is included in the virtualenv's `mach.pth` file so that
subprocesses will have access to the same packages as the primary
Mach process
3. //Something// adds something to the `sys.path`
4. The build virtualenv is redundantly re-activated (such as due to
generic setup code for some module). Though we don't physically
re-activate the virtualenv, we //do// check that it's up-to-date to
validate assumptions - **and it's not!** It's missing the
//something// that was added to the `sys.path`.
5. We can't re-create the build virtualenv because it's already active,
so:
6. An error is raised so that unexpected out-of-date-ness doesn't fly
under the radar.
-----
This patch solves this by calculating the "system Python" paths in a
deterministic way. Then, even when using the system Python, the Mach and
command sites use define themselves in a consistent way.
This means that we can't do `./mach --virtualenv <venv> ./mach ...` to
shares `venv`'s packages with the whole new Mach process. Fortunately,
this has been replaced with moving such packages into the nested Mach
command's requirements definition instead.
TL;DR: more consistent, less failures. 👍
-----
One more detail, this time around the `sys_path()` function:
Ideally, we could calculate the standard library paths *and* the
"system" paths with only one Python subprocess. Unfortunately,
that's not the case:
* If `-S` isn't provided, then it's tricky to separate the standard
library paths from the system paths
* If `-S` is provided, then doing `site.getsitepackages()` in a
virtualenv returns the *system* site packages, not the virtualenv's.
To work around this, we call the external python twice, and in
parallel. This allows resolving the information we need while avoiding
performance costs.
Differential Revision: https://phabricator.services.mozilla.com/D143200
Use `poetry` instead of `pip-compile`, then leverage `pip` to assert
that there's no conflicts between environments.
This enables simplifying `requirements.in`, since `colorama` is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.
`poetry==1.2.0a2` was chosen because it was the first version that would
include `pip`/`setuptools`/`wheel` in the `requirements.txt`, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).
Sorry about the `requirements.txt` noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
`requirements.txt` changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
`./mach vendor python`).
[1] https://github.com/python-poetry/poetry/issues/5408
Differential Revision: https://phabricator.services.mozilla.com/D142884
The PGO-specific objdir is meant for the PGO build artifacts.
The python virtualenvs just exist for the host, so there's no need to
scope them. Besides, the existing virtualenv infrastructure doesn't know
how to distinguish between "main 'build' site" and "PGO 'build' site",
which is why it raised this "out-of-date" error.
Differential Revision: https://phabricator.services.mozilla.com/D142373
Now that we are gleaning a site packages source in a process-global way
(that is *not* conditional on the site that's being managed), we've
removed an edge case from occurring: the case where Mach would use
`SitePackagesSource.NONE`, but the command site would use
`SitePackagesSource.SYSTEM`. This would've occurred if none of Mach's
dependencies were found in the system Python, but instead once the
command site was initialized, one of *its* dependencies were located.
Since that can no longer happen:
* Command sites that don't populate their VENV will *always* use the
same site packages source as Mach
* `pthfile` generation is simplified (the priority of the system paths
is no longer variable)
* We no longer need to track `site_packages_source` in metadata, since
we can use `mach_site_packages_source` instead.
While here, I moved the "`PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS`
sites shouldn't have `pypi_requirements`" check to a test, since there's
no need for it to happen at runtime.
Differential Revision: https://phabricator.services.mozilla.com/D140668
Simplify Mach's decision-making in CI for where to find native Python
packages: unless `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system` is
provided, always ignore system packages.
Removes optional dependency on system's `pip`, instead leaning on
vendored `pip` copy to see what package versions are installed.
Differential Revision: https://phabricator.services.mozilla.com/D140258
Unfortunately, the dependencies of l10nregistry-rs require tokio 1.0. This
allows this third duplicated instance. I think it makes more sense to try to
move the other uses in tree to a more modern version than to try to downgrade
the version of tokio required by l10nregistry-rs and dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D139854
The dataReportingNotification is the last place where the .properties
vendorShortName is used. As notifications already support Fluent,
porting this to use Fluent as well seems pretty strqaightforward.
Differential Revision: https://phabricator.services.mozilla.com/D141860
The Mach site is asserted to be compatible with all command sites, but
command sites are deliberately //not// asserted to be compatible with
each other - they're //supposed// to have the flexibility of being
able to be incompatible.
Accordingly, let's fail loudly if code tries to activate from one
command site to another.
Note that, due to the Mach site being a superset of common, it's safe to
activate from "common" to "<other command site>". This is needed for
cases like `./mach python --virtualenv ...`.
This required updating the `mozproxy` tests, who would deliberately
activate the `common` site so they could call a `mozproxy` entry point
script. These tests were fixed by instead invoking `mozproxy` as a
module (`-m`) of the current `python-test` site.
Differential Revision: https://phabricator.services.mozilla.com/D141659
This is needed for a few reasons:
* All mach commands can use virtualenvs, not just `build`-related
commands, so the files don't make sense to be in `build/`.
* When locking is added, more files associated with virtualenvs will be
added, and this will change will ease the related directory structure
setup.
* This removes the need for a redundant "_virtualenv_packages" keyword
as part of the manifest filenames.
Differential Revision: https://phabricator.services.mozilla.com/D140382
The intention is to remove all the files in the vendoring directory
(which may be the yaml directory or may not be) except those
specified in 'keep'.
However convert_patterns_to_paths expects a list of patterns, and
we were providing it with only one. This resulted in it iterating
over '**' as a string (one iteration for '*' and a second for '*')
This prevented it from traversing recursively. Turning it into a
list fixes it.
Depends on D141901
Differential Revision: https://phabricator.services.mozilla.com/D141902
This creates a way to override the 'tracking: tag' mechnaism
of a moz.yaml file without having to edit it. Useful to
test a library update when there is no new tag available.
Differential Revision: https://phabricator.services.mozilla.com/D141901
Update DSN for reporting exceptions to the new Sentry instance, and
update CI job to create releases on the new Sentry as well.
Differential Revision: https://phabricator.services.mozilla.com/D141122
Since bug 1717104 changed `./mach vendor python` to no longer have a
manual virtualenv activation, we can remove the test mocking that
softly handled the activation.
Differential Revision: https://phabricator.services.mozilla.com/D141674
All commands declaring a virtualenv will have them activated before the
command executes. Removes all now-redundant manual activations of
declared virtualenvs.
Commands that don't declare a virtualenv will still implicitly be
associated with the "common" virtualenv, but unlike explicit
virtualenv declarations it'll have to be activated manually, just
like it was before this patch.
To smooth the migration with existing usages, virtualenv activation
behaviour was changed slightly: if attempting to activate a new
virtualenv, but the source venv is already command venv, then raise an
exception. (In the future, we should improve testability of
virtualenv scaffolding logic so that tests can be added for this
sort of thing.) This did cause some issues with some tests, which
will be solved more cleanly with bug 1724273. In the meantime,
minimal modifications were made to failing tests to keep them green:
* `test_command_line.py` was activating the `common` virtualenv so
that it could install `mozproxy`, and use its CLI. Instead, I
modified the test to use `mozproxy` using the "module" interface
(`python -m mozproxy ...`). At that point, `MozbuildObject` was
unnecessary and usages were replaced with simpler variants.
* `test_vendor.py` needed its explicit `activate_virtualenv()` call
patched out. It still needs to use a virtualenv's Python
executable, but due to `sys.executable` now being kept up-to-date
as of bug 1717051, it could be used directly.
Differential Revision: https://phabricator.services.mozilla.com/D122892
Creates/updates virtualenvs for some mach commands, replacing
their ad-hoc usage of `install_pip_package()`, `pip install`,
and `sys.path` modifications.
Note: The `docs` virtualenv has `Sphinx==1.1.3` installed, even
though a more modern version of `Sphinx` is used when
`./mach doc` is run. This is ok for now, since `./mach doc` will
just install the newer `Sphinx` over top of the old one. Secondarily,
when we port `./mach doc` to use the centralized system, we'll
be incentivized to make the different `doc` commands use synchonized
versions of the same packages. Success!
Also, note that manual installation of `html5lib` and `requests`
isn't ported to the `wpt` site: this is because they're already
provided by the inherited Mach site.
Differential Revision: https://phabricator.services.mozilla.com/D122902
There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.
Return a `reason` string from `_is_venv_up_to_date()`, and print it in
cases where a virtualenv being out-of-date is erroneous.
Differential Revision: https://phabricator.services.mozilla.com/D140855
The Mach site management has been significantly simplified
in `mach_initialize.py`, so less of the `MachSiteManager`
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.
Differential Revision: https://phabricator.services.mozilla.com/D140854
There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.
Return a `reason` string from `_is_venv_up_to_date()`, and print it in
cases where a virtualenv being out-of-date is erroneous.
Differential Revision: https://phabricator.services.mozilla.com/D140855
The Mach site management has been significantly simplified
in `mach_initialize.py`, so less of the `MachSiteManager`
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.
Differential Revision: https://phabricator.services.mozilla.com/D140854
Previously, when trying to determine the minimal `sys.path` that just
includes the standard library (but no `pip`-installed packages), we
would resolve the `sys.path` and remove the system and user
site-packages. However, this "removal" step didn't work as-is for
`brew`'s Python, because its "system site-packages" is //different//
from its `site.getsitepackages()`.
Stepping back though, there's an easier solution: run `python` with the
`-S` flag, and no "site" paths or custom initialization logic will add
anything extra to the stdlib.
Differential Revision: https://phabricator.services.mozilla.com/D141508
Now that we're using `sysconfig` instead of `distutils`, our `purelib`
and `platlib` paths should correctly point to python environment paths
without a spurious `local/` component at their roots.
Differential Revision: https://phabricator.services.mozilla.com/D140871
Depending on the subprocesses created (such as if
`--with-ccache=sccache` is set), `./mach configure` may hang.
More details about why this is is documented in bug 1753797.
The workaround is to lean on the standard library to stream-and-format
output, rather than our existing `ProcessHandler` code.
Though this still streams both `stdout` and `stderr` in real-time, I saw
some ordering differences between the two streams locally. I don't yet
have a reason to believe that these differences are harmful or otherwise
incorrect.
Differential Revision: https://phabricator.services.mozilla.com/D141028
On newer systems, Python is intelligently populating pure-python and
platform-specific packages into //two different directories//.
Traditionally, these directories would be symlinked - or only one would
exist. However, to support these newer systems (and be better integrated
with Python conventions), we should handle both of these directories
properly.
The solution here is, when calculating the `site_packages_dir()`,
calculate all of them, and handle the multitude at all call sites.
The one use case where we want only one path (the resolving the location
for the `mach.pth` pthfile), I've opted for the `platlib`, since both
pure python dependencies //and// platform-specific dependencies are
referenced by the pthfile.
Differential Revision: https://phabricator.services.mozilla.com/D140870
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
We want `_pthfile_lines()` to consistently and accurately represent a
virtualenv's `sys.path` modifications.
However, this becomes tricky when comparing expected `pthfile_lines`
//before// activating a virtualenv versus //afterwards//, especially
since Python implicitly adds some paths (such as the path to
`site-packages`).
The current solution made this scrubbing only happen if we were
`pip install`-ing into the site. Unfortunately, it //doesn't// work for
the "use system Python for the `build` site", because:
* Pre-activation result: `site-packages` isn't added, because we aren't
using it
* Post-activation result, implicitly-added `site-packages` isn't
scrubbed, because **scrubbing only happened for
`SitePackagesSource.VENV`**
Stepping back, the solution here is:
* `pthfile_lines` only represents Mach modifications //on top// of
implicit virtualenv behaviour: since `site-packages` is always added
by default, it shouldn't be in our explicit `pthfile_lines`.
* The only time when implicit `sys.path` additions throws us off is when
`SitePackagesSource.SYSTEM`: so, move the scrubbing to happen in that
case instead.
Finally refactor the "conditional deprioritization" comment to be more
useful and more accurate: we've implemented the "nontrivial complexity"
of purging site-packages, and the other piece about nothing being
`pip install`-ed feels self-explanatory enough.
Differential Revision: https://phabricator.services.mozilla.com/D140580
So far, we've been using `virtualenv`'s `activate_this.py` script.
However, unlike earlier expectations, it adds its `sys.path` additions
to the //front//, not the back! This breaks our prioritization
requirements, such as:
* When using any package from the system environment, import *all
possible* packages from the system to avoid compatibility issues.
* Use vendored packages instead of virtualenv-installed packages
wherever possible, because it more-closely matches developer
expectations ("why is this package vendored if it's not used?")
Define an `activate_virtualenv()` function that replicates the logic
of `activate_this.py` [1], except for three differences:
* Don't modify `sys.real_prefix`, since it's a non-standard property of
`sys`.
* Only add seen-with-`venv`-module paths to the `sys.path` (`$prefix`,
`$prefix/.../$site_packages_dir`) - don't do the paths in-between.
* And, of course, append instead of prepend `sys.path` entries.
As an aside, this is one of the few remaining blockers from allowing
us to fully embrace `venv` instead of `virtualenv` - the last piece is
waiting on the fix for bug 1697833 to propagate.
[1]
https://github.com/pypa/virtualenv/blob/20.7.2/src/virtualenv/activation/python/activate_this.py
Differential Revision: https://phabricator.services.mozilla.com/D140579
