Until rust 1.28, there was no stable way to change the allocator used by
rust code. In bug 1280578, we hooked HeadAlloc/HeapFree/HeapRealloc,
that the default rust system allocator uses. On other platforms, rust
code just ended up using malloc/free/realloc like everything else.
As of rust 1.28, though, it is now possible to use the GlobalAlloc trait
and the #[global_allocator] attribute to set an allocator. On Windows,
this can allow us to hook mozjemalloc directly, rather than using an
indirection through HeapAlloc/etc. (which require an extra call to
GetProcessHeap), so let's do this. On other platforms, this just ends up
doing the same thing as the default rust system allocator (except for
the memalign limit on 32-bits platforms).
We still need the HeapAlloc/etc. hooks for some C++ code using it, though.
Another benefit is that the HeapAlloc GlobalAlloc implementation needs
to do its own memalign, which it does by overallocating and aligning
manually. We obviously don't need to do this when we using
memalign/_aligned_malloc directly.
Differential Revision: https://phabricator.services.mozilla.com/D14820
--HG--
extra : moz-landing-system : lando
This removes the code added in bug 1458161, because the old versions of
rust that required it can't be used to build Gecko anymore. The variant
for newer versions of rust stays.
Differential Revision: https://phabricator.services.mozilla.com/D14528
--HG--
extra : moz-landing-system : lando
The current rust panic hook keeps a string for the crash reporter, and
goes on calling the default rust panic hook, which prints out a crash
stack... when RUST_BOOTSTRAP is set *and* when that works. Notably, on
both mac and Windows, it only really works for local builds, but fails
for debug builds from automation, although on automation itself, we also
do stackwalk from crash minidumps, which alleviates the problem.
Artifact debug builds are affected, though.
More importantly, C++ calls to e.g. MOZ_CRASH have a similar but
different behavior, in that they dump a stack trace on debug builds, by
default (with exceptions, see below for one). The format of those stack
traces is understood by the various fix*stack*py scripts under
tools/rb/, that are used by the various test harnesses both on
automation and locally.
Additionally, the current rust panic hook, as it calls the default rust
panic hook, ends up calling abort() on non-Windows platforms, which ends
up being verbosely redirected to mozalloc_abort per
https://dxr.mozilla.org/mozilla-central/rev/237e4c0633fda8e227b2ab3ab57e417c980a2811/memory/mozalloc/mozalloc_abort.cpp#79
which then calls MOZ_CRASH. Theoretically, /that/ would also print a
stack trace, but doesn't because currently the stack trace printing code
lives in libxul, and MOZ_CRASH only calls it when compiled from
libxul-code, which mozalloc_abort is not part of.
With this change, we make the rust panic handler call back into
MOZ_CRASH directly. This has multiple advantages:
- This is more consistent cross-platforms (Windows is not special
anymore).
- This is more consistent between C++ and rust (stack traces all look
the same, and can all be post-processed by fix*stack*py if need be)
- This is more consistent in behavior, where debug builds will show
those stack traces without caring about environment variables.
- It demangles C++ symbols in rust-initiated stack traces (for some
reason that didn't happen with the rust panic handler)
A few downsides:
- the loss of demangling for some rust symbols.
- the loss of addresses in the stacks, although they're not entirely
useful
- extra empty lines.
The first should be fixable later one. The latter two are arguably
something that should be consistent across C++ and rust, and should be
changed if necessary, independently of this patch.
Depends on D11719
Depends on D11719
Differential Revision: https://phabricator.services.mozilla.com/D11720
--HG--
extra : moz-landing-system : lando
The current rust panic hook keeps a string for the crash reporter, and
goes on calling the default rust panic hook, which prints out a crash
stack... when RUST_BOOTSTRAP is set *and* when that works. Notably, on
both mac and Windows, it only really works for local builds, but fails
for debug builds from automation, although on automation itself, we also
do stackwalk from crash minidumps, which alleviates the problem.
Artifact debug builds are affected, though.
More importantly, C++ calls to e.g. MOZ_CRASH have a similar but
different behavior, in that they dump a stack trace on debug builds, by
default (with exceptions, see below for one). The format of those stack
traces is understood by the various fix*stack*py scripts under
tools/rb/, that are used by the various test harnesses both on
automation and locally.
Additionally, the current rust panic hook, as it calls the default rust
panic hook, ends up calling abort() on non-Windows platforms, which ends
up being verbosely redirected to mozalloc_abort per
https://dxr.mozilla.org/mozilla-central/rev/237e4c0633fda8e227b2ab3ab57e417c980a2811/memory/mozalloc/mozalloc_abort.cpp#79
which then calls MOZ_CRASH. Theoretically, /that/ would also print a
stack trace, but doesn't because currently the stack trace printing code
lives in libxul, and MOZ_CRASH only calls it when compiled from
libxul-code, which mozalloc_abort is not part of.
With this change, we make the rust panic handler call back into
MOZ_CRASH directly. This has multiple advantages:
- This is more consistent cross-platforms (Windows is not special
anymore).
- This is more consistent between C++ and rust (stack traces all look
the same, and can all be post-processed by fix*stack*py if need be)
- This is more consistent in behavior, where debug builds will show
those stack traces without caring about environment variables.
- It demangles C++ symbols in rust-initiated stack traces (for some
reason that didn't happen with the rust panic handler)
A few downsides:
- the loss of demangling for some rust symbols.
- the loss of addresses in the stacks, although they're not entirely
useful
- extra empty lines.
The first should be fixable later one. The latter two are arguably
something that should be consistent across C++ and rust, and should be
changed if necessary, independently of this patch.
Depends on D11719
Differential Revision: https://phabricator.services.mozilla.com/D11720
--HG--
extra : moz-landing-system : lando
The current rust panic hook keeps a string for the crash reporter, and
goes on calling the default rust panic hook, which prints out a crash
stack... when RUST_BOOTSTRAP is set *and* when that works. Notably, on
both mac and Windows, it only really works for local builds, but fails
for debug builds from automation, although on automation itself, we also
do stackwalk from crash minidumps, which alleviates the problem.
Artifact debug builds are affected, though.
More importantly, C++ calls to e.g. MOZ_CRASH have a similar but
different behavior, in that they dump a stack trace on debug builds, by
default (with exceptions, see below for one). The format of those stack
traces is understood by the various fix*stack*py scripts under
tools/rb/, that are used by the various test harnesses both on
automation and locally.
Additionally, the current rust panic hook, as it calls the default rust
panic hook, ends up calling abort() on non-Windows platforms, which ends
up being verbosely redirected to mozalloc_abort per
https://dxr.mozilla.org/mozilla-central/rev/237e4c0633fda8e227b2ab3ab57e417c980a2811/memory/mozalloc/mozalloc_abort.cpp#79
which then calls MOZ_CRASH. Theoretically, /that/ would also print a
stack trace, but doesn't because currently the stack trace printing code
lives in libxul, and MOZ_CRASH only calls it when compiled from
libxul-code, which mozalloc_abort is not part of.
With this change, we make the rust panic handler call back into
MOZ_CRASH directly. This has multiple advantages:
- This is more consistent cross-platforms (Windows is not special
anymore).
- This is more consistent between C++ and rust (stack traces all look
the same, and can all be post-processed by fix*stack*py if need be)
- This is more consistent in behavior, where debug builds will show
those stack traces without caring about environment variables.
- It demangles C++ symbols in rust-initiated stack traces (for some
reason that didn't happen with the rust panic handler)
A few downsides:
- the loss of demangling for some rust symbols.
- the loss of addresses in the stacks, although they're not entirely
useful
- extra empty lines.
The first should be fixable later one. The latter two are arguably
something that should be consistent across C++ and rust, and should be
changed if necessary, independently of this patch.
Depends on D11719
Differential Revision: https://phabricator.services.mozilla.com/D11720
--HG--
extra : moz-landing-system : lando
Updating rkv to 0.5 enables us to un-vendor new-ordered-float, as rkv 0.4 is the last crate in the tree that depends on it.
It also enables us to un-vendor version 0.5 of uuid. We previously needed that version because multiple third-party crates depended on it, and we have limited control over third-party sub-dependencies. But rkv 0.4 was the last third-party crate that still depended on version 0.5 of uuid; rkv 0.5 depends on version 0.6 of uuid.
There would still be two internal crates that depend on version 0.5 of uuid: geckodriver and webrender_bindings. But we have more control over internal sub-dependencies, and we can update those two internal crates to depend on version 0.6 of uuid. This patch does so.
To summarize, this patch makes the following changes:
* rkv: 0.4 -> 0.5
* new-ordered-float: un-vendored
* geckodriver: uuid dependency 0.5 -> 0.6
* webrender_bindings: uuid dependency 0.5 -> 0.6
* uuid 0.5: un-vendored
* uuid 0.6: remains in tree
Differential Revision: https://phabricator.services.mozilla.com/D9160
--HG--
extra : moz-landing-system : lando
r?njn for the profiler parts
r?jrmuizel for the ELF parsing parts
Depends on D7020
Differential Revision: https://phabricator.services.mozilla.com/D7021
--HG--
extra : moz-landing-system : lando
r?njn for the profiler parts
r?jrmuizel for the ELF parsing parts
Depends on D7020
Differential Revision: https://phabricator.services.mozilla.com/D7021
--HG--
extra : moz-landing-system : lando
This introduces two new crates:
- jsrust, for standalone builds. This crate is compiled into a static library
libjsrust.a, which gets linked into the shared Spidermonkey library when it's
built, or into the static Spidermonkey library otherwise. This is just a
static library wrapping jsrust_shared below.
- jsrust_shared, for Gecko embedding. It just references other Rust
crates actively used in Spidermonkey. It is used to be embedded as part of
a new Rust dependency in Gecko (in gkrust).
--HG--
rename : js/src/wasm/cranelift/Cargo.toml => js/src/rust/Cargo.toml
extra : rebase_source : 84e440e3f669b73776653182cb7b006cc7febb10
extra : histedit_source : 3a67575ff6871b7dc3558c10a0251b73cedb090c
When I originally implemented bug 1458161, this is how it was done, but
it was suggested to use a configure-time check. This turned out to not
be great, because the rust compiler changes regularly, and we don't run
the configure tests when the version changes. When people upgraded their
rust compiler to 1.27, the code subsequently failed to build because the
features were still set for the previous version they had installed.
--HG--
extra : rebase_source : 1b5f7a02ad8495d68cd29289f7beea59b8912183
Bug 1458161 added a rust OOM handler based on an unstable API that was
removed in 1.27, replaced with something that didn't allow to get the
failed allocation size.
Latest 1.28 nightly (2018-06-13) has
https://github.com/rust-lang/rust/pull/50880,
https://github.com/rust-lang/rust/pull/51264 and
https://github.com/rust-lang/rust/pull/51241 merged, which allow to
hook the OOM handler and get the failed allocation size again.
Because this is still an unstable API, we explicitly depend on strict
versions of rustc. We also explicitly error out if automation builds
end up using a rustc version that doesn't allow us to get the allocation
size for rust OOM, because we don't want that to happen without knowing.
--HG--
extra : rebase_source : 6c097151046d088cf51f4755dd69bde97bb8bd8b
Update mp4parse-rust to 0c8e1d91464aaa63b82ebf076b63cda1df4230d1, which adds
uuid parsing support and exports the mp4parse_fallible feature from
mp4parse_capi.
Update gkrust to pass MOZ_MEMORY as a feature, and use that to conditionally
enable mp4parse_fallible/FallibleVec.
MozReview-Commit-ID: 2HDYbL2CGgJ
--HG--
extra : rebase_source : 6e8cf15241b0282406322cce29220a677edd1585
Update mp4parse-rust to 0c8e1d91464aaa63b82ebf076b63cda1df4230d1, which adds
uuid parsing support and exports the mp4parse_fallible feature from
mp4parse_capi.
Update gkrust to pass MOZ_MEMORY as a feature, and use that to conditionally
enable mp4parse_fallible/FallibleVec.
MozReview-Commit-ID: 2HDYbL2CGgJ
--HG--
extra : rebase_source : 299d9f8347d2f0ef0d66b9ea52a4ee7a31af0cd2
OOM rust crashes are currently not identified as such in crash reports
because rust libstd handles the OOMs and panics itself.
There are unstable ways to hook into this, which unfortunately are under
active changes in rust 1.27, but we're currently on 1.24 and 1.27 is not
released yet. The APIs didn't change between 1.24 and 1.26, so it's
fine-ish to use them as long as we limit their use to those versions.
As long as the Firefox versions we ship (as opposed to downstream) use
the "right" version of rust, we're good to go.
The APIs are in their phase of stabilization, so there shouldn't be too
many variants of the code to support.
--HG--
extra : rebase_source : 08a85aa102b24380b1f6764effffcc909ef3191b
This patch rewrites the rust-url-capi crate as the mozurl crate, which
provides a threadsafe MozURL object which is compatible with the
previous MozURL class.
Creating a MozURL this way performs a single allocation, which contains
only a rust-url Url object and an atomic refcnt, however it is fully
compatible with the C++ RefPtr type.
This patch also exposes methods for accessing dependent substrings of
the serialized spec, meaning that string copies can be avoided in many
situations when inspecting attributes of the MozURL.
--HG--
rename : netwerk/base/rust-url-capi/.gitignore => netwerk/base/mozurl/.gitignore
Remove the syn dependency from gkrust-shared, which was added in bug 1373878
as a workaround for this Cargo bug:
https://github.com/rust-lang/cargo/issues/3923
MozReview-Commit-ID: L34J0davEYd
--HG--
extra : rebase_source : 79e5684cba143f6215e7dcf9367f82227bca48c5
The prefs parser has two significant problems.
- It doesn't separate tokenizing from parsing.
- It is implemented as a loop around a big switch on a "current state"
variable.
As a result, it is hard to understand and modify, slower than it could be, and
in obscure cases (involving comments and whitespace) it fails to parse what
should be valid input.
This patch replaces it with a recursive descent parser (albeit one without any
recursion!) that has separate tokenization. The new parser is easier to
understand and modify, more correct, and has better error messages. It doesn't
do error recovery, but that would be much easier to add than in the old parser.
The new parser also runs about 1.9x faster than the existing parser. (As
measured by parsing greprefs.js's contents from memory 1000 times in
succession, omitting the prefs hash table construction. If the table
construction is included, it's about 1.6x faster.)
The new parser is slightly stricter than the old parser in a few ways.
- Disconcertingly, the old parser allowed arbitrary junk between prefs
(including at the start and end of the prefs file) so long as that junk
didn't include any of the following chars: '/', '#', 'u', 's', 'p'. I.e.
lines like these:
!foo@bar&pref("prefname", true);
ticky_pref("prefname", true); // missing 's' at start
User_pref("prefname", true); // should be 'u' at start
would all be treated the same as this:
pref("prefname", true);
The new parser disallows such junk because it isn't necessary and seems like
an unintentional botch by the old parser.
- The old parser allowed character 0x1a (SUB) between tokens and treated it
like '\n'.
The new parser does not allow this character. SUB was used to indicate
end-of-file (*not* end-of-line) in some old operating systems such as MS-DOS,
but this doesn't seem necessary today.
- The old parser tolerated (with a warning) invalid escape sequences within
string literals -- such as "\q" (not a valid escape) and "\x1" and "\u12"
(both of which have insufficient hex digits) -- accepting them literally.
The new parser does not tolerate invalid escape sequences because it doesn't
seem necessary and would complicate things.
- The old parser tolerated character 0x00 (NUL) within string literals; this is
dangerous because C++ code that manipulates string values with embedded NULs
will almost certainly consider those chars as end-of-string markers.
The new parser treats NUL chars as end-of-file, to avoid this danger and
because it facilitates a significant optimization (described within the
code).
- The old parser allowed integer literals to overflow, silently wrapping them.
The new parser treats integer overflow as a parse error. This seems better,
and it caught existing overflows of places.database.lastMaintenance, in
testing/profiles/prefs_general.js (bug 1424030) and
testing/talos/talos/config.py (bug 1434813).
The first of these changes meant that a couple of existing prefs with ";;" at
the end had to be changed (done in the preceding patch).
The minor increase in strictness shouldn't be a problem for default pref files
such as greprefs.js within the application (which we can modify), nor for
app-written prefs files such as prefs.js. It could affect user-written prefs
files such as user.js; the experience above suggests that integer overflow and
";;" are the most likely problems in practice. In my opinion, the risk here is
acceptable.
The new parser also does a better job of tracking line numbers because it (a)
treats "\r\n" sequences as a single end-of-line marker, and (a) pays attention
to end-of-line sequences within string literals.
Finally, the patch adds thorough tests of both valid and invalid syntax.
MozReview-Commit-ID: JD3beOQl4AJ
The prefs parser has two significant problems.
- It doesn't separate tokenizing from parsing.
- It is implemented as a loop around a big switch on a "current state"
variable.
As a result, it is hard to understand and modify, slower than it could be, and
in obscure cases (involving comments and whitespace) it fails to parse what
should be valid input.
This patch replaces it with a recursive descent parser (albeit one without any
recursion!) that has separate tokenization. The new parser is easier to
understand and modify, more correct, and has better error messages. It doesn't
do error recovery, but that would be much easier to add than in the old parser.
The new parser also runs about 1.9x faster than the existing parser. (As
measured by parsing greprefs.js's contents from memory 1000 times in
succession, omitting the prefs hash table construction. If the table
construction is included, it's about 1.6x faster.)
The new parser is slightly stricter than the old parser in a few ways.
- Disconcertingly, the old parser allowed arbitrary junk between prefs
(including at the start and end of the prefs file) so long as that junk
didn't include any of the following chars: '/', '#', 'u', 's', 'p'. I.e.
lines like these:
!foo@bar&pref("prefname", true);
ticky_pref("prefname", true); // missing 's' at start
User_pref("prefname", true); // should be 'u' at start
would all be treated the same as this:
pref("prefname", true);
The new parser disallows such junk because it isn't necessary and seems like
an unintentional botch by the old parser.
- The old parser allowed character 0x1a (SUB) between tokens and treated it
like '\n'.
The new parser does not allow this character. SUB was used to indicate
end-of-file (*not* end-of-line) in some old operating systems such as MS-DOS,
but this doesn't seem necessary today.
- The old parser tolerated (with a warning) invalid escape sequences within
string literals -- such as "\q" (not a valid escape) and "\x1" and "\u12"
(both of which have insufficient hex digits) -- accepting them literally.
The new parser does not tolerate invalid escape sequences because it doesn't
seem necessary and would complicate things.
- The old parser tolerated character 0x00 (NUL) within string literals; this is
dangerous because C++ code that manipulates string values with embedded NULs
will almost certainly consider those chars as end-of-string markers.
The new parser treats NUL chars as end-of-file, to avoid this danger and
because it facilitates a significant optimization (described within the
code).
- The old parser allowed integer literals to overflow, silently wrapping them.
The new parser treats integer overflow as a parse error. This seems better,
and it caught an existing overflow in testing/profiles/prefs_general.js, for
places.database.lastMaintenance (see bug 1424030).
The first of these changes meant that a couple of existing prefs with ";;" at
the end had to be changed (done in the preceding patch).
The minor increase in strictness shouldn't be a problem for default pref files
such as greprefs.js within the application (which we can modify), nor for
app-written prefs files such as prefs.js. It could affect user-written prefs
files such as user.js; the experience above suggests that ";;" is the most
likely problem in practice. In my opinion, the risk here is acceptable.
The new parser also does a better job of tracking line numbers because it (a)
treats "\r\n" sequences as a single end-of-line marker, and (a) pays attention
to end-of-line sequences within string literals.
Finally, the patch adds thorough tests of both valid and invalid syntax.
MozReview-Commit-ID: 8EYWH7KxGG
* * *
[mq]: win-fix
MozReview-Commit-ID: 91Bxjfghqfw
--HG--
extra : rebase_source : a8773413e5d68c33e4329df6819b6e1f82c22b85
