Bug 1661624 - Include psutil in the mach virtualenvs r=firefox-build-system-reviewers,rstewart

Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.

Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.

An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.

Differential Revision: https://phabricator.services.mozilla.com/D90914

build/common_virtualenv_packages.txt

@@ -44,9 +44,6 @@ mozilla.pth:third_party/python/pyrsistent
 mozilla.pth:third_party/python/python-hglib
 mozilla.pth:third_party/python/pluggy
 mozilla.pth:third_party/python/jsmin
-!windows:optional:setup.py:third_party/python/psutil:build_ext:--inplace
-!windows:mozilla.pth:third_party/python/psutil
-windows:mozilla.pth:third_party/python/psutil-cp27-none-win_amd64
 mozilla.pth:third_party/python/pylru
 mozilla.pth:third_party/python/pystache
 python2:mozilla.pth:third_party/python/PyYAML/lib

python/mozbuild/mozbuild/mach_commands.py

@@ -2097,6 +2097,16 @@ class CreateMachEnvironment(MachCommandBase):
 
         manager.install_pip_package("zstandard>=0.9.0,<=0.13.0")
 
+        try:
+            # `mach` can handle it perfectly fine if `psutil` is missing, so
+            # there's no reason to freak out in this case.
+            manager.install_pip_package("psutil==5.7.0")
+        except subprocess.CalledProcessError:
+            print(
+                "Could not install psutil, so telemetry will be missing some "
+                "data. Continuing."
+            )
+
         if not PY2:
             # This can fail on some platforms. See
             # https://bugzilla.mozilla.org/show_bug.cgi?id=1660120

taskcluster/ci/docker-image/kind.yml

@@ -61,6 +61,7 @@ jobs:
             - deb8-mercurial
             - deb8-python3.6
             - deb8-python3-defaults
+            - deb8-python-psutil
             - deb8-python-zstandard
             - deb8-xz-utils
     deb8-toolchain-build:

taskcluster/ci/packages/kind.yml

@@ -207,6 +207,20 @@ jobs:
                 sha256: 9727dcb3d6b655e4f2a92110f5db076a490aa50b739804be239905ecff3aacc8
             patch: gdb-jessie.diff
 
+    deb8-python-psutil:
+        description: "python-psutil rebuild for python 3.6 in Debian jessie"
+        treeherder:
+            symbol: Deb8(python-psutil)
+        run:
+            using: debian-package
+            dsc:
+                url: http://snapshot.debian.org/archive/debian/20150815T034233Z/pool/main/p/python-psutil/python-psutil_2.2.1-3.dsc
+                sha256: b47d1fc92094dfd5525cff7d0af5855f7c5335ade9de4c0e738ed490aa5bee7c
+            packages:
+                - deb8-dh-python
+                - deb8-python3.6
+                - deb8-python3-defaults
+
     deb8-python-zstandard:
         description: "python-zstandard for Debian jessie"
         treeherder:

taskcluster/docker/debian-base/Dockerfile

@@ -43,6 +43,8 @@ RUN /usr/local/sbin/setup_packages.sh $TASKCLUSTER_ROOT_URL $DOCKER_IMAGE_PACKAG
       python3-minimal \
       python-zstandard \
       python3-zstandard \
+      python-psutil \
+      python3-psutil \
       vim-tiny \
       xz-utils
 

taskcluster/docker/decision/VERSION

@@ -1 +1 @@
-2.2.1
+2.2.2

taskcluster/docker/decision/system-setup.sh

@@ -10,7 +10,9 @@ apt-get install -y --force-yes --no-install-recommends \
     python \
     sudo \
     python3 \
-    python3-yaml
+    python3-yaml \
+    python-psutil \
+    python3-psutil
 
 BUILD=/root/build
 mkdir "$BUILD"

taskcluster/docker/mingw32-build/Dockerfile

@@ -27,6 +27,7 @@ RUN apt-get update && \
       libucl1 \
       libxml2 \
       patch \
+      python3-dev \
       p7zip-full \
       scons \
       tar \

taskcluster/docker/recipes/ubuntu1804-test-system-setup.sh

@@ -19,6 +19,9 @@ pip install virtualenv==15.2.0
 pip install zstandard==0.13.0
 pip3 install zstandard==0.13.0
 
+pip install psutil==5.7.0
+pip3 install psutil==5.7.0
+
 # Cleanup
 cd /
 rm -rf /setup ~/.ccache ~/.cache ~/.npm

taskcluster/scripts/builder/build-l10n.sh

@@ -99,7 +99,10 @@ fi
 
 cd /builds/worker
 
-$GECKO_PATH/mach python $GECKO_PATH/testing/${MOZHARNESS_SCRIPT} \
+$GECKO_PATH/mach python \
+  --requirements $GECKO_PATH/taskcluster/scripts/builder/requirements.txt \
+  -- \
+  $GECKO_PATH/testing/${MOZHARNESS_SCRIPT} \
   ${config_path_cmds} \
   ${config_cmds} \
   $actions \

taskcluster/scripts/builder/build-linux.sh

@@ -114,7 +114,10 @@ fi
 
 cd /builds/worker
 
-$GECKO_PATH/mach python $GECKO_PATH/testing/${MOZHARNESS_SCRIPT} \
+$GECKO_PATH/mach python \
+  --requirements $GECKO_PATH/taskcluster/scripts/builder/requirements.txt \
+  -- \
+  $GECKO_PATH/testing/${MOZHARNESS_SCRIPT} \
   ${config_path_cmds} \
   ${config_cmds} \
   $debug_flag \

taskcluster/scripts/builder/requirements.txt

@@ -0,0 +1 @@
+psutil==5.7.0