mozconfig_loader is invoked with /bin/sh, which may not be bash. We could
force mozconfigs to be run with bash instead, but that might be disruptive
for existing mozconfigs, and the sole bashism used in mozconfig_loader is
rather straightforward to workaround by namespacing the variables.
Fixes a 403 error running `./mach bootstrap` with stylo enabled.
Bug 1359968 updated the tooltool manifests to point to a
publicly-available clang package. Update the macOS hash
used by mozboot to install the same package for local
use by developers who need rust-bindgen.
MozReview-Commit-ID: FtkATaLwc7n
--HG--
extra : rebase_source : 61390c729e694589b0351026aac0263440a9a3c5
The Python print() function's does not take a format string, so the extraneous %s here inadvertently prints "Downloading clang package from %s https://..." instead of "Downloading clang package from https://...".
MozReview-Commit-ID: 5Is5TK3X34e
--HG--
extra : rebase_source : 02e625fbdaab536d1a787c33edd6dfda9c1039c6
We've traditionally had per-directory 'update' scripts for
third-party media codec implementations. Instead, leverage
the new 'mach vendor' command to centralize the import and
build file generation, placing the upstream source in
third_party/aom.
Note this includes another copy of gtest and other dependencies
which we don't use, but they're required by the upstream build
process we use to generate our own build description, so I've
left them in for now.
MozReview-Commit-ID: CnWcSwvQZEh
--HG--
extra : rebase_source : 28172a41332e920c9ea4a475a6990d43ebf8185f
Add AArch64/ARM64 support to the script that produces android version
codes. Use the same scheme for AArch64 as x86, since the two
architectures don't overlap, and AArch64 should override ARM just like
x86 should override ARM.
For Stylo development, LLVM packages are required due to Stylo's
extensive use of bindgen--generating Rust bindings to Gecko's C++ code.
While people can install LLVM via their system package manager, we've
opted to download the LLVM packages used on Mozilla infrastructure for
building Gecko. Using Mozilla's packages for LLVM ensures that they
work, and also makes it easier/trivial to integrate support for other
things (e.g. Mozilla's static checkers) into `mach bootstrap`.
The way it works today, compiler warnings are logged to build output
twice. The compiler's raw output is logged. Then, if mach's compiler
warning parser detects a warning, a structured log message with info
about that warning is printed as well. Because of the order in which
callbacks fire, mach's warning message is delivered to the logger
first. So build output looks something like:
0:04.63 Warning: -Wsign-compare in /home/gps/src/firefox/security/nss/lib/dev/ckhelper.c: comparison of integers of different signs: 'CK_ULONG' (aka 'unsigned long') and 'int'
0:04.63 /home/gps/src/firefox/security/nss/lib/dev/ckhelper.c:135:45: warning: comparison of integers of different signs: 'CK_ULONG' (aka 'unsigned long') and 'int' [-Wsign-compare]
0:04.63 (obj_template[i].ulValueLen == -1)) {
0:04.63 ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~
That first line is our structured message formatted to plain text.
Subsequent lines are Clang.
The output here is redundant. But having the structured compiler warning
in the JSON logs is nice so downstream systems don't have to reinvent
the parsing wheel.
Also today, at the end of a clobber build we print a count of the
total number of compiler warnings.
Compiler warnings represent potential bugs. So we want to encourage
people to look at them. However, with build system output often spanning
several hundred lines and compiler warnings scattered in no
deterministic order because build system execution is non-deterministic,
it is easy to lose sight of compiler warnings as they go by in build
output. And, it can be difficult to find a warning in files you care
about because you don't know where in the log to look for that file.
This commit attempts to improve the visibility of compiler warnings
by printing a sorted list of warnings at the end of the build. The list
of warnings it prints are only those seen during the current build
operation. We /could/ print a list of all warnings in the persisted
database. But I think this would be annoying, particularly for
incremental or partial builds.
The structured log message previously emitted at compiler invocation
time has been removed. So now the build backend output doesn't have a
redundant "Warning: " line summarizing the warning next to the compiler
output itself. This is less confusing.
That structured log message has been reborn at the end of the build
and reformatted slightly. If there are any consumers assuming that the
log entry near this structured message is compiler output itself,
they will break. I don't think any such consumers exist. It is also
possible the structured log may not contain warning messages if a
process exit occurs. I'm fine with this: if the `mach` process dies,
we have bigger problems to worry about.
The new output looks something like this:
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:216:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:225:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/angle/src/compiler/translator/util.cpp:234:15 [-Wimplicit-fallthrough] unannotated fall-through between switch labels
18:07.97 warning: gfx/cairo/libpixman/src/pixman-bits-image.c:268:32 [-Wshift-negative-value] shifting a negative signed value is undefined
18:07.97 warning: gfx/cairo/libpixman/src/pixman-linear-gradient.c:395:6 [-Wunreachable-code] code will never be executed
18:07.97 warning: gfx/cairo/libpixman/src/pixman-x86.c:80:5 [-Wexpansion-to-defined] macro expansion producing 'defined' has undefined behavior
Note the sorting of files, including by line number and column. If we
wanted to, we could even print that line. But that's for another day
(especially since compilers themselves tend to do this already).
This change has the potential to annoy people. That's because instead
of the parsed warning summary lines being spread out over the build
(where they tend not to be noticed), they will be lumped at the end of
the build log, which people do tend to notice. In my build, I have
~363 warnings/lines. Some could perceive this as excessive and
redundant. But the recourse for this is simple: eliminate compiler
warnings. My hope is that by having the compiler warnings exposed
clearly at the end of the build log that people will notice them
and will take action to eliminate them. Sunlight is a disinfectant.
MozReview-Commit-ID: 9VHXsPhAYmr
--HG--
extra : rebase_source : 6a2da05f794a54ea54a48167a36d130dfbf34e59
Currently, the build monitor has a single compiler warnings database
that unions warnings from previous runs with warnings from the current
invocation. I want to introduce functionality that treats warnings
seen during the current invocation differently from "all warnings."
To facilitate this, this commit introduces a 2nd, non-persisted
warnings database to record warnings just for the current invocation.
MozReview-Commit-ID: FIY0GiarDmr
--HG--
extra : rebase_source : b2002e1c248ea65b2c0ee45a78b1e74d61a26f3c
Currently, the WarningsCollector (which parses warnings from line inputs)
accepts a WarningsDatabase (an object representing a collection of
warnings) and populates that instance as a new warning is parsed. In
an upcoming commit I want to introduce a 2nd WarningsDatabase.
Rather that add it to WarningsCollector, I figure it will be easier
to decouple WarningsDatabase from WarningsCollector.
This commit refactors WarningsCollector to call a callback when a
warning is parsed. This allows consumers to do anything they want
with a warning, including potentially write it to multiple databases.
MozReview-Commit-ID: 7Z9x4FMwyof
--HG--
extra : rebase_source : b4844b5c2b1926840d37e46ead38c8c358762ba8
mozharness is currently making a manual `make -k` invocation. We don't
want automation calling `make` directly. So teach `mach build` to
accept a --keep-going argument that results in `make -k`.
MozReview-Commit-ID: H3lJ4r8S4vj
--HG--
extra : rebase_source : 9feb7bcaeb855254c53c5fa9d49177c5133f2773
This patch adds a copy of vswhere.exe to build/win32, downloaded from the
current latest release (1.0.62):
https://github.com/Microsoft/vswhere/releases/download/1.0.62/vswhere.exe
It changes toolchain.configure to invoke vswhere.exe instead of reading
the registry, since that no longer works for VS2017 (but vswhere can locate
VS2015). It also removes a layer of complexity in that code by dropping
support for non-64-bit host systems, since we don't really support building
on 32-bit Windows anymore anyway.
There's a little bit of fixup in windows.configure where some LIB paths
have changed in 2017.
MozReview-Commit-ID: 5XLWjidS6W4
--HG--
extra : rebase_source : 90f79b6f4a2d8d925dd20eb0bf6ab96262c227d5
rustc generates .lib files for its libraries when compiling for Windows
(even using MinGW on Linux). But MinGW expects .a files. So we add in
rust-specific prefix and suffixes so MinGW builds can find the libs that
rustc generates. (And the RUST_LIB- variables default to the same vales
as the LIB_ variables otherwise.)
MozReview-Commit-ID: ClsA0YuJaxh
--HG--
extra : rebase_source : 7b46460c94ceb34b7a5a302ce91d3f1dca600041
Everything depending on the widget being gonk can go away, as well as
everything depending on MOZ_AUDIO_CHANNEL_MANAGER, which was only
defined on gonk builds under b2g/ (which goes away in bug 1357326).
--HG--
extra : rebase_source : 9f0aeeb7eea8417fa4e06d662d566d67ecaf2a24
There can be cases where there is simply nothing to download, especially
with the --for-job argument. So we just stop erroring out when nothing
was downloaded.
However, if the user explicitly requested a particular file(s) via the
command line and there is nothing to download, we still emit an error
code.
--HG--
extra : rebase_source : 143c15c9711bbfbbfdc110da14f3738132d53ed4
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : 81ba8bfee0ca96979cf8e30d75cdd47f06bc10ea
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : f637acff32fc8582732de932503dd696abc57877
Now that we have automated build jobs that produce toolchains, we want
to avoid the burden of uploading them to tooltool and then update the
tooltool manifests. But we don't have build jobs for all the possible
toolchains, so we allow `mach artifact toolchain` to get a mix of
tooltool and taskcluster artifacts.
For taskcluster artifacts, we can give a list of job names (conveniently
automatically normalized to begin with 'toolchain-' and end with '/opt')
for which the artifacts will be downloaded, in place of any tooltool
package with the same name (if a tooltool manifest is given).
The taskcluster artifacts that we download are the ones matching the
contents of the tree the command is run with, per the resources declared
for the corresponding toolchain build job (in
taskcluster/ci/toolchain*.yml)
So for example, a linux64 build could call the following command:
mach artifact toolchain --tooltool-manifest \
browser/config/tooltool-manifests/linux64/releng.manifest \
--from-build linux64-gcc
and get the right gcc corresponding to the build-gcc script in tree,
along with the other non-gcc files from the tooltool manifest.
Things are however planned to be even more convenient, but some commands
can already benefit from this form (even without a tooltool manifest).
See e.g. bug 1328454.
--HG--
extra : rebase_source : b12ed77bef529eb8d67476aceac0166bdfe2eeed
We're going to potentially use the same download manager for tooltool
and taskcluster artifacts, and we don't want to send the tooltool
authentication header to the taskcluster requests.
--HG--
extra : rebase_source : 79a0afdbf06cb05d7792f413ab1f6715b2a9fe2e
beb43155b7a6 changed WPT items to be 3-tuples instead of 2-tuples.
This broke test_defaults_for_path().
MozReview-Commit-ID: 7M0RcQ7bOIU
--HG--
extra : rebase_source : 28e44e5206abb7939d540ba809ec71325ead341c
At the same time, make the artifact cache directory not indexed, and
move the directory creation to the base classes that actually use it.
--HG--
extra : rebase_source : 62994499afceb5166c0041148dcc702f87166fdc
The ultimate goal is to have a generic command that pulls relevant
toolchains from either tooltool or taskcluster artifacts.
This introduces the command and makes it work to allow to wrap tooltool
in most places where it's used currently, with the ability to replace
tooltool_wrapper.sh as well.
--HG--
extra : rebase_source : f63178282f00f8698a148db484f0276c99d61f7b
Profiling revealed that mozpath.relpath() accounted for a lot of CPU
time when operating on an input of ~42,000 paths.
Due to the nature of the paths we're operating on, we don't need the
full power of mozpath.relpath() here. Instead, we can implement a
specialized version that works given already normalized paths and the
knowledge that context paths must be ancestors of the current path
being examined.
This change drops execution time of a mach command feeding ~42,000
paths to this function from ~90s to ~24s. On an input with 9131 paths,
execution time dropped from ~8.8s to ~3.7s.
MozReview-Commit-ID: EGLiJa10Zj2
--HG--
extra : rebase_source : 8e9d7a957ea3d865a6f18a3b99cca2b22f7a0385
Calling self.test_defaults_for_path() from files_info() with tens
of thousands of paths resulted in a CPU explosion in various path
normalization functions. I don't think it was so much the complexity
of the operations as much as the volume.
For an input with 9131 elements, this reduces execution time of a
mach command from ~25.7s to ~8.8s. With ~42,000 inputs, execution time
drops from <it took too long and I gave up> to ~90s.
MozReview-Commit-ID: pjQQByi2Bc
--HG--
extra : rebase_source : b8307a92e494f128ba3cf29348bc7996cfea4221
With ~42,000 entries in relpaths, this change drops execution time
of this loop from ~23s to ~0.01s.
MozReview-Commit-ID: Afm245tjWUQ
--HG--
extra : rebase_source : 86ee525b0539c80e47e4326a3f111a3fc84d0a6e
For people working on Rust code, compiling in debug mode (Cargo's "dev"
profile) is convenient: debug assertions are turned on, optimization is
turned off, and parallel compilation inside of rustc itself can be
used. These things make the build faster and the debugging experience
more pleasant.
To obtain that currently, one needs to --enable-debug at the Gecko
toplevel, which turns on debug assertions for the entire browser, which
makes things run unreasonably slowly. So it would be desirable to be
able to turn *off* debug mode for the entirety of the browser, but turn
on debug mode for the Rust code only.
Hence this added switch, --enable-rust-debug, which does what it
suggests and defaults to the value of --enable-debug. For our own
sanity and because we judge it a non-existent use case, we do not
support --enable-debug --disable-rust-debug.
Dropped support for python 2.6
Removed support for bookmarks.html
Better internal APIs
Intermediate versions had incompatibilities with old versions
of python 2.7, but 1.2.3 does not.
MozReview-Commit-ID: LfKhr8qWe28
--HG--
extra : rebase_source : 21fc39d45f22f4a6ae10523a87b0e6068f268439
This commit moves symbol dumping to the compile tier, to be run via "syms"
targets. Tracking files are used for the sake of incremental builds, because
dump_syms may genearate multiple outputs whose paths are not known ahead of
time.
Minimal changes to symbolstore.py are made here. More extensive
simplifications will be made in a future commit on the basis of symbolstore.py
handling one file at a time.
MozReview-Commit-ID: 3mOP8A6Y7iM
--HG--
extra : rebase_source : bfe97afcdfc05b9e79f01577701c83e8b00eb4e9
This creates "syms" targets that depend on the corresponding "target" for
directories containing shared libraries or programs. These targets are added
to the main compile graph in automation, and can be invoked through a special
"symbols" target. A future commit will use these targets to dump symbols for
shared libraries and programs during the compile tier.
MozReview-Commit-ID: KLuvmqsK4Zj
--HG--
extra : rebase_source : 8d76b999cb6fac8f11168ac6ebfb58774dfc2d3c
These variables are ignored in favour or reading the same values
from the passed in libdef object.
MozReview-Commit-ID: 8Xkkd68clNN
--HG--
extra : rebase_source : 767c15f1b59a6f48d09411c4f2e64eff9f8b4cc6
When downloading rust manually, it's mandatory to restart the shell,
as the script does not export the PATH environment variable.
This fix also ensures that the rust version in PATH is modern enough for a successful build.
MozReview-Commit-ID: HdLpiLPBLW7
--HG--
extra : rebase_source : dc087c36e89fd6d2957fd1e03af42229de914d5a
This adds a unit test for the expected behavior, and adds the modules
from mach_bootstrap.py that are missing in the virtualenv for the test
to run.
--HG--
extra : rebase_source : e34d0474cfb6c8c5ce9cd847b96de88906191923
In most cases, the HTTP response for the download will contain the
content-length, but if some error happens (e.g. authentication error),
there might not be one, and the download fails with a TypeError for a
division by None, instead of failing with a more friendly error message
about the HTTP error.
--HG--
extra : rebase_source : f179142e69c9ca09b05ad2ff942753fd84da6a69
This is the initial support of 'mach repackage', which can take an
existing tarball and create a DMG on either an OSX host or on a Linux
host with cross-OSX tools. Configure is needed in order to find the
tools necessary to create the DMG. On a Linux cross-compiled environment
with tooltool, this can be as simple as:
export MKFSHFS=$topsrcdir/hfsplus-tools/newfs_hfs
export DMG_TOOL=$topsrcdir/dmg/dmg
export HFS_TOOL=$topsrcdir/dmg/hfsplus
ac_add_options --disable-compile-environment
MozReview-Commit-ID: 6t2rlXpwUvu
--HG--
extra : rebase_source : 39f2f9ecac9d7e5af197f1b5dea70cd307acf488
In bug 1335309, FileFinder was made to default to not find executables,
and zip.py was made to use the default instead. Which made sense for
most uses of zip.py, except for the jsshell package.
So we add a flag to make zip.py able to strip executable (which happens
when the FileFinder is made to find them), and use that flag for the
jsshell package only.
--HG--
extra : rebase_source : 0202f9acd5e6175d3790aaef026e18c6913cf0c6
