Bug 1740123: Allow invoking Mach outside of MozillaBuild r=glandium

Assuming that the `MOZILLABUILD` environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.

Note that MozillaBuild still needs to be installed, and the
`MOZILLABUILD` environment variable will have to be set.

For future reference: when I tried setting this up with Windows
Store's Python 3.9, I encountered issues when running binaries installed
via `pip`: it would fail with `abort: failed to load Python DLL
python3x.dll`.

Differential Revision: https://phabricator.services.mozilla.com/D133936

build/mach_initialize.py

@@ -8,6 +8,7 @@ import math
 import os
 import shutil
 import sys
+from pathlib import Path
 
 if sys.version_info[0] < 3:
     import __builtin__ as builtins
@@ -150,6 +151,39 @@ def _activate_python_environment(topsrcdir, get_state_dir):
     mach_environment.activate()
 
 
+def _maybe_activate_mozillabuild_environment():
+    if sys.platform != "win32":
+        return
+
+    mozillabuild = Path(os.environ.get("MOZILLABUILD", r"C:\mozilla-build"))
+    assert mozillabuild.exists(), (
+        f'MozillaBuild was not found at "{mozillabuild}".\n'
+        "If it's installed in a different location, please "
+        'set the "MOZILLABUILD" environment variable '
+        "accordingly."
+    )
+
+    for entry in os.environ.get("PATH", "").split(":"):
+        entry = Path(entry)
+        if mozillabuild in entry.parents:
+            # We're already running with MozillaBuild directories in our PATH: either
+            # that's because we're inside the MozillaBuild shell, or the active developer
+            # has manually set up the PATH entries.
+            return
+
+    use_msys2 = (mozillabuild / "msys2").exists()
+    if use_msys2:
+        mozillabuild_msys_tools_path = mozillabuild / "msys2" / "usr" / "bin"
+    else:
+        mozillabuild_msys_tools_path = mozillabuild / "msys" / "bin"
+
+    os.environ.setdefault("MOZILLABUILD", str(mozillabuild))
+    os.environ["PATH"] += (
+        f"{os.pathsep}{mozillabuild_msys_tools_path}"
+        f"{os.pathsep}{mozillabuild / 'bin'}"
+    )
+
+
 def initialize(topsrcdir):
     # This directory was deleted in bug 1666345, but there may be some ignored
     # files here. We can safely just delete it for the user so they don't have
@@ -178,6 +212,7 @@ def initialize(topsrcdir):
     _activate_python_environment(
         topsrcdir, lambda: os.path.normpath(get_state_dir(True, topsrcdir=topsrcdir))
     )
+    _maybe_activate_mozillabuild_environment()
 
     import mach.base
     import mach.main

docs/setup/windows_build.rst

@@ -153,6 +153,12 @@ start working on <https://codetribute.mozilla.org/>`_.
 See the :ref:`Firefox Contributors' Quick Reference` to learn how to test your changes,
 send patches to Mozilla, update your source code locally, and more.
 
+.. note::
+
+    If you'd like to interact with Mach from a different command line environment
+    than MozillaBuild, there's experimental support for it described
+    :ref:`over here <Using Mach on Windows Outside MozillaBuild>`.
+
 Troubleshooting
 ---------------
 

mach

@@ -53,12 +53,21 @@ def main(args):
             for guidance on how to prepare your system to build Firefox. Perhaps
             you need to update Xcode, or install Python using brew?
             """).strip())
-        elif "MOZILLABUILD" in os.environ:
+        elif "MOZILLABUILD" in os.environ and os.environ.get("TERM"):
             print(dedent("""
             Python is provided by MozillaBuild; ensure your MozillaBuild installation is
             up to date. See https://firefox-source-docs.mozilla.org/setup/windows_build.html#install-mozillabuild
             for details.
             """).strip())
+        elif sys.platform.startswith("win"):
+            print(dedent("""
+            You probably want to be interacting with Mach from within MozillaBuild, see
+            https://firefox-source-docs.mozilla.org/setup/windows_build.html for details.
+            
+            If you are deliberately using Mach from outside MozillaBuild, then see
+            https://firefox-source-docs.mozilla.org/mach/windows-usage-outside-mozillabuild.html#install-python
+            for guidance on installing native Python on your system.
+            """).strip())
         else:
             print(dedent("""
             We do not have specific instructions for your platform on how to

mach.cmd

@@ -0,0 +1,3 @@
+@ECHO OFF
+SET topsrcdir=%~dp0
+python %topsrcdir%mach %*

mach.ps1

@@ -1,11 +1,16 @@
+$mypath = $MyInvocation.MyCommand.Path
+$machpath = ($mypath -replace '\\', '/').substring(0, $mypath.length - 4)
+
+if (-not (test-path env:MACH_PS1_USE_MOZILLABUILD)) {
+  python $machpath $args
+  exit $lastexitcode
+}
+
 if (-not (test-path env:MOZILLABUILD)) {
   echo "No MOZILLABUILD environment variable found, terminating."
   exit 1
 }
 
-$mypath = $MyInvocation.MyCommand.Path
-$machpath = ($mypath -replace '\\', '/').substring(0, $mypath.length - 4)
-
 if ($machpath.contains(' ')) {
   echo @'
 The repository path contains whitespace which currently isn't supported in mach.ps1.

python/mach/docs/index.rst

@@ -85,4 +85,5 @@ best fit for you.
    logging
    settings
    telemetry
+   windows-usage-outside-mozillabuild
    faq

python/mach/docs/windows-usage-outside-mozillabuild.rst

@@ -0,0 +1,96 @@
+==========================================
+Using Mach on Windows Outside MozillaBuild
+==========================================
+
+.. note::
+
+    These docs still require that you've followed the :ref:`Building Firefox On Windows` guide.
+
+`MozillaBuild <https://wiki.mozilla.org/MozillaBuild>`__ is required to build
+Firefox on Windows, because it provides necessary unix-y tools such as ``sh`` and ``awk``.
+
+Traditionally, to interact with Mach and the Firefox Build System, Windows
+developers would have to do so from within the MozillaBuild shell. This could be
+disadvantageous for two main reasons:
+
+1. The MozillaBuild environment is unix-y and based on ``bash``, which may be unfamiliar
+   for developers used to the Windows Command Prompt or Powershell.
+2. There have been long-standing stability issues with MozillaBuild - this is due to
+   the fragile interface point between the underlying "MSYS" tools and "native Windows"
+   binaries.
+
+It is now (experimentally!) possible to invoke Mach directly from other command line
+environments, such as Powershell, Command Prompt, or even a developer-managed MSYS2
+environment. Windows Terminal should work as well, for those on the "cutting edge".
+
+.. note::
+
+    If you're using a Cygwin-based environment such as MSYS2, it'll probably be
+    best to use the Windows-native version of Python (as described below) instead of a Python
+    distribution provided by the environment's package manager. Otherwise you'll likely run into
+    compatibility issues:
+
+    * Cygin/MSYS Python will run into compatibility issues with Mach due to its unexpected Unix-y
+      conventions despite Mach assuming it's on a "Windows" platform. Additionally, there may
+      be performance issues.
+    * MinGW Python will encounter issues building native packages because they'll expect the
+      MSVC toolchain.
+
+.. warning::
+
+    This is only recommended for more advanced Windows developers: this work is experimental
+    and may run into unexpected failures!
+
+Following are steps for preparing Windows-native (Command Prompt/Powershell) usage of Mach:
+
+1. Install Python
+~~~~~~~~~~~~~~~~~
+
+Download Python from the `the official website <https://www.python.org/downloads/windows/>`__.
+
+.. note::
+
+    To avoid Mach compatibility issues with recent Python releases, it's recommended to install
+    the 2nd-most recent "major version". For example, at time of writing, the current modern Python
+    version is 3.10.1, so a safe version to install would be the most recent 3.9 release.
+
+You'll want to download the "Windows installer (64-bit)" associated with the release you've chosen.
+During installation, ensure that you check the "Add Python 3.x to PATH" option.
+
+.. note::
+
+    Due to issues with Python DLL import failures with pip-installed binaries, it's not
+    recommended to use the Windows Store release of Python.
+
+2. Modify your PATH
+~~~~~~~~~~~~~~~~~~~
+
+The Python "user site-packages directory" needs to be added to your ``PATH`` so that packages
+installed via ``pip install --user`` (such as ``hg``) can be invoked from the command-line.
+
+1. From the Start menu, go to the Control Panel entry for "Edit environment variables
+   for your account".
+2. Double-click the ``Path`` row in the top list of variables. Click "New" to add a new item to
+   the list.
+3. In a Command Prompt window, resolve the Python directory with the command
+   ``python -c "import site; import os; print(os.path.abspath(os.path.join(site.getusersitepackages(), '..', 'Scripts')))"``.
+4. Paste the output into the new item entry in the "Edit environment variable" window.
+5. Click "New" again, and add the ``bin`` folder of MozillaBuild: probably ``C:\mozilla-build\bin``.
+6. Click "OK".
+
+3. Install Version Control System
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you're using Mercurial, you'll need to install it to your Windows-native Python:
+
+.. code-block:: shell
+
+    pip3 install --user mercurial windows-curses
+
+If you're using Git with Cinnabar, follow its `setup instructions <https://github.com/glandium/git-cinnabar#setup>`__.
+
+Success!
+~~~~~~~~
+
+At this point, you should be able to invoke Mach and manage your version control system outside
+of MozillaBuild.