Bug 1635514 - Delete tup CI/configure stuff r=froydnj

This includes scripts that involve `tup`, jobs that build `tup` in automation, `tup.configure`, and related infrastructure and documentation.

Differential Revision: https://phabricator.services.mozilla.com/D73921

.gitignore

@@ -158,9 +158,6 @@ tps_result.json
 # Ignore files created when running a reftest.
 lextab.py
 
-# tup database
-/.tup
-
 # Ignore Visual Studio Code workspace files.
 .vscode/
 !.vscode/extensions.json

.hgignore

@@ -180,9 +180,6 @@ _OPT\.OBJ/
 # Ignore files created when running a reftest.
 ^lextab.py$
 
-# tup database
-^\.tup
-
 # Ignore sync tps logs and reports
 tps\.log
 tps_result\.json

browser/config/mozconfigs/linux64/tup

@@ -1,16 +0,0 @@
-MOZ_AUTOMATION_BUILD_SYMBOLS=0
-MOZ_AUTOMATION_PACKAGE=0
-MOZ_AUTOMATION_UPLOAD=0
-MOZ_AUTOMATION_UPLOAD_SYMBOLS=0
-
-TOOLTOOL_DIR=${TOOLTOOL_DIR:-$topsrcdir}
-export TUP=${MOZ_FETCHES_DIR}/tup/tup
-
-. "$topsrcdir/browser/config/mozconfigs/linux64/common-opt"
-
-ac_add_options --enable-build-backends=Tup
-unset ENABLE_CLANG_PLUGIN
-# To enable the option to upload the tup database, uncomment the line below
-# ac_add_options --upload-tup-db
-
-. "$topsrcdir/build/mozconfig.common.override"

build/docs/index.rst

@@ -16,7 +16,6 @@ Important Concepts
    files-metadata
    Profile Guided Optimization <pgo>
    slow
-   tup
    environment-variables
    build-targets
    python

build/docs/slow.rst

@@ -133,16 +133,14 @@ no-op build is spent in make traversal.
 make is inefficient
 ===================
 
-Compared to modern build backends like Tup or Ninja, make is slow and
+Compared to modern build backends like Tup or Ninja, `make` is slow and
-inefficient. We can only make make so fast. At some point, we'll hit a
+inefficient. We can only make `make` so fast. At some point, we'll hit a
 performance plateau and will need to use a different tool to make builds
 faster.
 
-Please note that clobber and incremental builds are different. A clobber
+Please note that clobber and incremental builds are different. A clobber build
-build with make will likely be as fast as a clobber build with e.g. Tup.
+with `make` will likely be as fast as a clobber build with a modern build
-However, Tup should vastly outperform make when it comes to incremental
+system.
-builds. Therefore, this issue is mostly seen when performing incremental
-builds. For more information, see :ref:`tup`.
 
 C++ header dependency hell
 ==========================

build/docs/tup.rst

@@ -1,170 +0,0 @@
-.. _tup:
-
-===========
-Tup Backend
-===========
-
-The Tup build backend is an alternative to the default Make backend. The `Tup
-build system <http://gittup.org/tup/>`_ is designed for fast and correct
-incremental builds. A top-level no-op build should be under 2 seconds, and
-clobbers should rarely be required. It is currently only available for Linux
-Desktop builds -- other platforms like Windows or OSX are planned for the
-future.
-
-As part of the mozbuild architecture, the Tup backend shares a significant
-portion of frontend (developer-facing) code in the build system. When using the
-Tup backend, ``./mach build`` is still the entry point to run the build system,
-and moz.build files are still used for the build description. Familiar parts of
-the build system like configure and generating the build files (the
-``Reticulating splines...`` step) are virtually identical in both backends. The
-difference is that ``mach`` invokes Tup instead of Make under the hood to do
-the actual work of determining what needs to be rebuilt. Tup is able to perform
-this work more efficiently by loading only the parts of the DAG that are
-required for an incremental build. Additionally, Tup instruments processes to
-see what files are read from and written to in order to verify that
-dependencies are correct.
-
-For more detailed information on the rationale behind Tup, see the `Build
-System Rules and Algorithms
-<http://gittup.org/tup/build_system_rules_and_algorithms.pdf>`_ paper.
-
-Installation
-============
-
-You'll need to install the Tup executable, as well as the nightly rust/cargo
-toolchain (Note: Replace $topsrcdir with the path to your mozilla-central
-source tree)::
-
-   cd ~/.mozbuild && $topsrcdir/mach artifact toolchain --from-build linux64-tup
-   rustup install nightly
-   rustup default nightly
-
-Configuration
-=============
-
-Your mozconfig needs to describe how to find the executable if it's not in your
-PATH, and enable the Tup backend::
-
-   export TUP=~/.mozbuild/tup/tup
-   ac_add_options --enable-build-backends=Tup
-
-Configuring Parallel Jobs
--------------------------
-
-To override the default number of jobs run in parallel, set MOZ_PARALLEL_BUILD
-in your mozconfig::
-
-    mk_add_options MOZ_PARALLEL_BUILD=8
-
-What Works
-==========
-
-You should expect a Linux desktop build to generate a working Firefox binary
-from a ``./mach build``, and be able to run test suites against it (eg:
-mochitests, xpcshell, gtest). Top-level incremental builds should be fast
-enough to use them during a regular compile/edit/test cycle. If you wish to
-stop compilation partway through the build to more quickly iterate on a
-particular file, you can expect ``./mach build objdir/path/to/file.o`` to
-correctly produce all inputs required to build file.o before compiling it. For
-example, you don't have to run the build system in various subdirectories to
-get generated headers built in the right order.
-
-Currently Unsupported / Future Work
-===================================
-
-There are a number of features that you may use in the Make backend that are
-currently unsupported for the Tup backend. We plan to add support for these in
-the future according to developer demand and build team availability.
-
-* sccache - This is currently under active development to support icecream-like
-  functionality, which likely impacts the same parts that would affect Tup's
-  dependency checking mechanisms. Note that icecream itself should work with
-  Tup.
-
-* Incremental Rust compilation - see `bug 1468527 <https://bugzilla.mozilla.org/show_bug.cgi?id=1468527>`_
-
-* Watchman integration - This will allow Tup to skip the initial ``Scanning
-  filesystem...`` step, saving 1-2 seconds of startup overhead.
-
-* More platform support (Windows, OSX, etc.)
-
-* Packaging in automation - This is still quite intertwined with Makefiles
-
-* Tests in automation - Requires packaging
-
-Multiple object directories and object directories outside of the source tree
-=============================================================================
-
-Common workflows involving multiple object directories which may be outside of
-the source directory are supported by Tup, however there are some things to
-consider when using these configurations.
-
-* Using multiple object directories works as expected, however you may find
-  pulling and attempting to build will fail due to tup attempting to
-  parse a Tupfile in an object directory other than the active object
-  directory. As a workaround, activate the object directory containing the
-  failing file and run configure before proceeding.
-* If the currently active object directory is located outside of the source
-  directory, the tup backend will prompt the user to run ``tup init --no-sync``
-  in a common ancestor directory of the source directory and object directory.
-  If this ancestor contains too many files (it's the home directory, for
-  instance), this will slow down tup's initial file scan. Anecdotally multiple
-  object directories will only incur marginal scanning overhead, but care
-  should be exercised when choosing a directory layout.
-
-Tup builds in automation
-========================
-
-Tup builds run on integration branches as ``Btup`` in treeherder. There are
-some aspects of the Tup builds that are currently implemented outside of the
-make build system, and divergences may cause the ``Btup`` job to fail until
-the build is completely integrated. There are two known situations this has
-come up.
-
-* Changes to the xpidl/webidl/ipdl build - these are given special treatment
-  by the make backend that is re-created in the tup backend. An update to a
-  Makefile implementing one of these parts of the build may require a
-  corresponding change to ``python/mozbuild/mozbuild/backend/tup.py``
-* Build scripts generating new outputs - due to the implementation of Tup,
-  outputs from ``build.rs`` that run during the build must be known to Tup
-  before the build starts. The current outputs are enumerated in
-  ``python/mozbuild/mozbuild/backend/cargo_build_defs.py``. Modifying the set
-  of outputs from a build script will require a corresponding update to this
-  file.
-
-
-Partial tree builds in tup
-==========================
-
-Partial tree builds are possible in tup by invoking
-``./mach build <relative srcdir path>``, however the behavior differs from the
-make backend. A partial tree build will result in running the commands in
-Tupfiles in the specified subtree, building ``.o`` files and other outputs, but
-unlike make it will take changed dependencies into account and build everything
-necessary to update those outputs as well. Also unlike make it will not
-attempt to find downstream commands that depend on these files, i.e.
-programs and libraries from other parts of the tree will not be linked. A
-top level incremental build should be performed to run all commands that depend
-on changes to the local tree.
-
-How to Contribute
-=================
-
-At the moment we're looking for early adopters who are developing on the Linux
-desktop to try out the Tup backend, and share your experiences with the build
-team (see `Contact`_).
-
- * Are there particular issues or missing features that prevent you from using
-   the Tup backend at this time?
-
- * Do you find that top-level incremental builds are fast enough to use for
-   every build invocation?
-
- * Have you needed to perform a clobber build to fix an issue?
-
-Contact
-========
-
-If you have any issues, feel free to file a bug blocking `buildtup
-<https://bugzilla.mozilla.org/show_bug.cgi?id=827343>`_, or contact mshal or
-chmanchester in #build on IRC.

build/moz.configure/tup.configure

@@ -1,76 +0,0 @@
-# -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
-# vim: set filetype=python:
-# This Source Code Form is subject to the terms of the Mozilla Public
-# License, v. 2.0. If a copy of the MPL was not distributed with this
-# file, You can obtain one at http://mozilla.org/MPL/2.0/.
-
-# tup detection
-# ==============================================================
-
-tup = check_prog('TUP', ['tup'])
-
-@depends(tup)
-@checking('for tup version')
-@imports('re')
-def tup_version(tup):
-    tup_min_version = '0.7.8'
-    out = check_cmd_output(tup, '--version',
-                           onerror=lambda: die('Failed to get tup version'))
-    m = re.search(r'tup v?([0-9]+\.[0-9]+\.[0-9]+)', out)
-
-    if not m:
-        raise FatalCheckError('Unknown version of tup: %s' % out)
-    ver = Version(m.group(1))
-
-    if ver < tup_min_version:
-        raise FatalCheckError('To use the tup backend you must have tup version '
-                              '%s or greater in your path' % tup_min_version)
-    return ver
-
-@depends(tup)
-@checking('for tup ldpreload dependency checker')
-def tup_is_ldpreload(tup):
-    out = check_cmd_output(tup, 'server',
-                           onerror=lambda: die('Failed to get tup dependency checker'))
-    if out.rstrip() != 'ldpreload':
-        raise FatalCheckError('To use the tup backend, please use a version '
-                              'of tup compiled with the ldpreload dependency '
-                              'checker. Either compile tup locally with '
-                              'CONFIG_TUP_SERVER=ldpreload in your tup.config '
-                              'file, or use the version from the toolchain '
-                              'task via |./mach artifact toolchain '
-                              '--from-build linux64-tup|')
-    return True
-
-@depends(tup, using_sccache)
-def tup_and_sccache(tup, using_sccache):
-    if tup and using_sccache:
-        die('Cannot use sccache with tup yet. Please disable sccache or use '
-            'the make backend until it is supported.')
-
-@depends(tup, cargo)
-@imports('re')
-def check_tup_cargo_channel(tup, cargo):
-    channel = None
-    out = check_cmd_output(cargo, '--version')
-    VERSION_FORMAT = r'cargo \d\.\d+\.\d+-(\S+)'
-    m = re.match(VERSION_FORMAT, out)
-    if m:
-        channel = m.group(1)
-    if tup and channel != 'nightly':
-        die('Nightly Cargo is required when building with Tup.')
-
-@depends(tup, target, build_project)
-def tup_and_non_linux(tup, target, build_project):
-    if tup and (target.kernel != 'Linux' or build_project != 'browser'):
-        die('The tup backend can only be used to build the browser on Linux. '
-            'Use the make backend until your target is supported.')
-
-option('--upload-tup-db', help= 'Upload the tup database from an automated build.')
-
-@depends('--upload-tup-db')
-def upload_tdb(value):
-    if value:
-        return True
-
-set_config('UPLOAD_TUP_DB', upload_tdb)

moz.configure

@@ -448,21 +448,6 @@ def possible_makes(make, host):
 
 check_prog('GMAKE', possible_makes)
 
-@depends(build_backends, build_project)
-def tup_include(build_backends, build_project):
-    # We need to check the rustc version when building with tup, but
-    # rustc_info isn't available when configuring js (and build_backends isn't
-    # available from project-specific configure), so as a workaround we only
-    # include the file when we know we'll need it. This can be removed when
-    # we globally require a rustc recent enough to build with tup.
-    if build_project not in ('browser', 'mobile/android'):
-        return None
-    for backend in build_backends:
-        if 'Tup' in backend:
-            return 'build/moz.configure/tup.configure'
-
-include(tup_include)
-
 # watchman detection
 # ==============================================================
 

taskcluster/ci/build/linux.yml

@@ -1241,47 +1241,6 @@ linux64-rusttests/debug:
             - linux64-lucetc
             - wasi-sysroot
 
-linux64-tup/opt:
-    description: "Linux64 Tup"
-    index:
-        product: firefox
-        job-name: linux64-tup-opt
-    treeherder:
-        platform: linux64/opt
-        symbol: Btup
-        tier: 3
-    worker-type: b-linux
-    worker:
-        max-run-time: 3600
-        env:
-            PERFHERDER_EXTRA_OPTIONS: tup
-            RUSTC_BOOTSTRAP: '1'
-    run:
-        using: mozharness
-        actions: [get-secrets, build]
-        config:
-            - builds/releng_base_firefox.py
-            - builds/releng_base_linux_64_builds.py
-        script: "mozharness/scripts/fx_desktop_build.py"
-        secrets: true
-        custom-build-variant-cfg: tup
-        mozconfig-variant: tup
-        tooltool-downloads: public
-        need-xvfb: true
-    run-on-projects: []
-    fetches:
-        toolchain:
-            - linux64-binutils
-            - linux64-clang
-            - linux64-rust
-            - linux64-cbindgen
-            - linux64-sccache
-            - linux64-tup
-            - linux64-nasm
-            - linux64-node
-            - linux64-lucetc
-            - wasi-sysroot
-
 linux64-ccov/opt:
     description: "Linux64-CCov Opt"
     index:

taskcluster/ci/fetch/toolchains.yml

@@ -324,13 +324,6 @@ llvm-for-dsymutil:
         repo: https://github.com/llvm/llvm-project
         revision: 3b7811f6441be13c9f613f81ef93297d231b4f8e
 
-tup:
-    description: tup source code
-    fetch:
-        type: git
-        repo: https://github.com/gittup/tup
-        revision: 4371a41ba4ece660bf060b598b9eee4c2eb347d8
-
 rust-size:
     description: rust-size source code
     fetch:

taskcluster/ci/toolchain/misc.yml

@@ -100,24 +100,6 @@ linux64-mar-tools:
             - toolkit/mozapps/update/updater/bspatch/
             - tools/update-packaging/
 
-linux64-tup:
-    description: "tup toolchain build"
-    treeherder:
-        symbol: TL(tup)
-    worker:
-        max-run-time: 3600
-    run:
-        script: build-tup-linux.sh
-        toolchain-artifact: public/build/tup.tar.xz
-    run-on-projects:
-        - trunk
-        - try
-    fetches:
-        fetch:
-            - tup
-        toolchain:
-            - linux64-gcc-7
-
 linux64-upx:
     description: "UPX build"
     treeherder:

taskcluster/scripts/misc/build-tup-linux.sh

@@ -1,31 +0,0 @@
-#!/bin/bash
-set -e -v
-
-# This script is for building tup on Linux.
-
-COMPRESS_EXT=xz
-export PATH=$MOZ_FETCHES_DIR/gcc/bin:$PATH
-
-cd $MOZ_FETCHES_DIR/tup
-
-patch -p1 <<'EOF'
-diff --git a/src/tup/link.sh b/src/tup/link.sh
-index ed9a01c6..4ecc6d7d 100755
---- a/src/tup/link.sh
-+++ b/src/tup/link.sh
-@@ -4,7 +4,7 @@
- # linking, so that the version is updated whenever we change anything that
- # affects the tup binary. This used to live in the Tupfile, but to support
- # Windows local builds we need to make it an explicit shell script.
--version=`git describe`
-+version=unknown
- CC=$1
- CFLAGS=$2
- LDFLAGS=$3
-EOF
-
-(echo 'CONFIG_TUP_SERVER=ldpreload'; echo 'CONFIG_TUP_USE_SYSTEM_PCRE=n') > tup.config
-./bootstrap-ldpreload.sh
-cd ..
-tar caf tup.tar.xz tup/tup tup/tup-ldpreload.so tup/tup.1
-cp tup.tar.xz $UPLOAD_DIR

testing/mozharness/configs/builds/releng_sub_linux_configs/64_tup.py

@@ -1,24 +0,0 @@
-import os
-
-config = {
-    'stage_platform': 'linux64-tup-opt',
-    'env': {
-        'MOZBUILD_STATE_PATH': os.path.join(os.getcwd(), '.mozbuild'),
-        'DISPLAY': ':2',
-        'HG_SHARE_BASE_DIR': '/builds/hg-shared',
-        'MOZ_OBJDIR': '%(abs_obj_dir)s',
-        'MOZ_CRASHREPORTER_NO_REPORT': '1',
-        'LC_ALL': 'C',
-        'XPCOM_DEBUG_BREAK': 'stack-and-abort',
-        # 64 bit specific
-        'PATH': '/usr/local/bin:/bin:\
-/usr/bin:/usr/local/sbin:/usr/sbin:/sbin',
-        'LD_LIBRARY_PATH': '%(abs_obj_dir)s/dist/bin',
-        'TINDERBOX_OUTPUT': '1',
-
-        # sccache doesn't work yet with tup due to the way the server is
-        # spawned, and because the server does the file I/O
-        'SCCACHE_DISABLE': '1',
-    },
-    'disable_package_metrics': True, # TODO: the package needs to be created for this to work
-}

testing/mozharness/mozharness/mozilla/building/buildbase.py

@@ -314,7 +314,6 @@ class BuildOptionParser(object):
         'aarch64-debug': 'builds/releng_sub_%s_configs/%s_aarch64_debug.py',
         'android-geckoview-docs': 'builds/releng_sub_%s_configs/%s_geckoview_docs.py',
         'valgrind': 'builds/releng_sub_%s_configs/%s_valgrind.py',
-        'tup': 'builds/releng_sub_%s_configs/%s_tup.py',
     }
     build_pool_cfg_file = 'builds/build_pool_specifics.py'