This patch enables SafeBrowsing in Safe Mode because features based
on SafeBrowsing are essential for Firefox(For example, Enhanced Tracking Protection).
Since Safe Browsing update is nondeterministic, we disable periodical
SafeBrowsing update to make troubleshooting easier. Manually trigger an
update via about:classifier is still enabled.
Last, SafeBrowsing tables provided by Google will be ignored in Safe
Mode to ensure the SafeBrowsing warnings are update-to-date.
Differential Revision: https://phabricator.services.mozilla.com/D62708
--HG--
extra : moz-landing-system : lando
Now mfbt/Move.h is empty except for that excellent comment about move
semantics... Should we put it somewhere else and delete the header as a
follow-up? Or just delete the header and carry on?
Differential Revision: https://phabricator.services.mozilla.com/D60297
--HG--
extra : moz-landing-system : lando
The inclusions were removed with the following very crude script and the
resulting breakage was fixed up by hand. The manual fixups did either
revert the changes done by the script, replace a generic header with a more
specific one or replace a header with a forward declaration.
find . -name "*.idl" | grep -v web-platform | grep -v third_party | while read path; do
interfaces=$(grep "^\(class\|interface\).*:.*" "$path" | cut -d' ' -f2)
if [ -n "$interfaces" ]; then
if [[ "$interfaces" == *$'\n'* ]]; then
regexp="\("
for i in $interfaces; do regexp="$regexp$i\|"; done
regexp="${regexp%%\\\|}\)"
else
regexp="$interfaces"
fi
interface=$(basename "$path")
rg -l "#include.*${interface%%.idl}.h" . | while read path2; do
hits=$(grep -v "#include.*${interface%%.idl}.h" "$path2" | grep -c "$regexp" )
if [ $hits -eq 0 ]; then
echo "Removing ${interface} from ${path2}"
grep -v "#include.*${interface%%.idl}.h" "$path2" > "$path2".tmp
mv -f "$path2".tmp "$path2"
fi
done
fi
done
Differential Revision: https://phabricator.services.mozilla.com/D55444
--HG--
extra : moz-landing-system : lando
In AsyncApplyUpdate, the Safe Browsing update thread holds a reference
to 'Classifier'. In some scenarios(See Bug 1591112), the update
thread may be the last one holding the reference; hence the update
thread releases the 'Classifier' when the task is ended.
Classifier has to be created and destroyed in the same thread
because of the constrain of LazyIdelThread, in the current
implementation, it should be released by the worker thread.
This patch transfers the ownership of the reference of 'Classifier 'from the
update thread to the worker thread before its task is finished to make
sure we release 'Classifier' in the right thread.
Differential Revision: https://phabricator.services.mozilla.com/D53156
--HG--
extra : moz-landing-system : lando
A LazyIdle thread should be created and removed by the same thread. This
patch fixes testcases that trigger the assertion.
Depends on D49874
Differential Revision: https://phabricator.services.mozilla.com/D49875
--HG--
extra : moz-landing-system : lando
Safe Browsing update thread wakes up every 30 mins to update tables from google
and 60 mins to update tables from mozilla.
Since the update thread doesn't have always to be alive, we change the
update thread to be a LazyIdle thread instead.
Differential Revision: https://phabricator.services.mozilla.com/D49874
--HG--
extra : moz-landing-system : lando
This patch replaces |ResetTables|(clear table's in-memory and on-disk
data) with |DeleteTables|(clear table's on-disk data) in GetLookupCache to avoid infinite
loop.
We can just delete on-disk data when file corruption is detected in |GetLookupCache|
without clearing the cache's internal data and refreshing current
active caches because in that scenario, the lookup cache failing to read
database has not yet added to the active caches list.
Differential Revision: https://phabricator.services.mozilla.com/D43181
--HG--
extra : moz-landing-system : lando
Before this patch, when Safe Browsing updating process discovers an error, it
quits and resets the table failing to update.
After this patch, updating process will continue to run when an error
occurs to find all the tables failing to apply an update.
Differential Revision: https://phabricator.services.mozilla.com/D42615
--HG--
extra : moz-landing-system : lando
Patch P2 & P3 refine how Safe Browsing handles Safe Browsing database
loading failure.
Safe Browsing databases are read in 3 scenarios:
1. |GetLookupCache| is called on startup. Safe Browsing reads prefix
files in this case. Metadata for updates(.sbstore, .metadata) are not
read in this scenario.
2. |TableRequest| is called before applying an update, Safe Browsing
reads update metadata to apply a partial update.
3. During an update, Safe Browsing reads both prefix files and metadata
in order to merge the update result.
For Case 1, we reset a table's database only when it returns FILE_CORRUPTED
while loading prefixes from the prefix file(.vlpset).
For Case 2, we reset a table's database when the table fails to load its
metadata file or prefix file. This is because we need to make sure both
files are complete so we can correctly perform a partial update.
Note that in this case, we don't just reset the database when "FILE_CORRUPTED"
is detected, we reset the database as long as an error occurs while loading the
database.
For Case 3, For all the tables failing to load their database during an
updating process, the databases of those tables will be reset.
Case 1 and Case 2 are done in Patch P2; Case 3 is done in Patch P3
Differential Revision: https://phabricator.services.mozilla.com/D42614
--HG--
extra : moz-landing-system : lando
This patch does the following:
1. Remove testing files from disk because they are no longer required.
2. Load completions from previous version of HashStore until an update
is applied.
3. Older version of HashStore(.sbstore) & PrefixSet(.vlpset) will be
removed during an update
Differential Revision: https://phabricator.services.mozilla.com/D36002
--HG--
extra : moz-landing-system : lando
Create test entries via update introduces performance overhead.
We can store them directly in LookupCache and do not save test entries
to disk.
Differential Revision: https://phabricator.services.mozilla.com/D34576
--HG--
extra : moz-landing-system : lando
For Safe Browsing V2, Data for lookup(LookupCache) and data for update(HashStore)
are now separated. |RegenActiveTables| doesn't need to check the chunk
number in HashStore.
Differential Revision: https://phabricator.services.mozilla.com/D34575
--HG--
extra : moz-landing-system : lando
Completions are now stored in .vlpset, we can remove it from .sbstore
Functions related to optimize reading completions from .sbstore can also
be removed because it is no longer HashStore's responsibility
Differential Revision: https://phabricator.services.mozilla.com/D34574
--HG--
extra : moz-landing-system : lando
The goal of the series of patches is to improve Safe Browsing performance by
skipping uncessary file IO.
The first two patches is to remove the dependency between LookupCache and HashStore, so HashStore is only
responsible for udpates.
Before this patch, LookupCacheV2 treats prefixes and completions
differently. It uses two data structures to maintain
prefixes:
1. mPrefixSet to store prefixes from .pset
2. mUpdateCompletions to store completions from .sbstore
After this patch
1. LookupCacheV2 & LookupCacheV4 both use variable-length
prefix set. mUpdateCompletions and mPrefixSet are removed and
mVLPrefixSet is used to store all prefixes data.
2. Move common function to base class.
Note that in this patch, conversion between 4/32 bytes prefixes and
mVLPrefixSet is not yet included, it will be handled in next patch.
This patch tries not to deal with any logic changes, only focus on refining
LookupCacheV2 & LookupCacheV4 class structure to use variable-length
prefixset for both classes.
Differential Revision: https://phabricator.services.mozilla.com/D34546
--HG--
extra : moz-landing-system : lando
We use ".pset" to find active tables, but in Bug 1353956, v4 prefix files
are renamed to ".vlpset".
This patches include both 'pset' and 'vlpset' to ScanStoreDir.
Differential Revision: https://phabricator.services.mozilla.com/D32433
--HG--
extra : moz-landing-system : lando
After calling Lookup API per table, Safe Browsing outputs too many debug
message for a single URL lookup. Refine the current output.
Differential Revision: https://phabricator.services.mozilla.com/D27066
--HG--
extra : moz-landing-system : lando
SHA256 is an expensive operation, we should avoid using them if
possible. SafeBrowsing prefix files are loaded during startup and
verify integrity with SHA256 which may affect the performance
especially on the low-end device.
This patch simply removes the SHA256 integrity check. CRC32 version
integrity check will be introduced in the other patch.
This patch also changes the behavior of recording
"Telemetry::URLCLASSIFIER_VLPS_LOAD_CORRUPT" a little bit.
It used to records only once per session(during startup, the first
time we load prefix set), now it records per update.
Differential Revision: https://phabricator.services.mozilla.com/D21461
--HG--
extra : moz-landing-system : lando
SafeBrowsing V4 protocol use SHA-256 as the checksum to check integrity
of update data and also the integrity of prefix files.
SafeBrowsing V2 HashStore use MD5 as the checksum to check integrity of
.sbstore
Since we are going to use CRC32 as the integrity check of V4 prefix files,
I think rename V4 "checksum" to SHA256 can improve readability.
Differential Revision: https://phabricator.services.mozilla.com/D21460
--HG--
extra : moz-landing-system : lando
SHA256 is an expensive operation, we should avoid using them if
possible. SafeBrowsing prefix files are loaded during startup and
verify integrity with SHA256 which may affect the performance
especially on the low-end device.
This patch simply removes the SHA256 integrity check. CRC32 version
integrity check will be introduced in the other patch.
This patch also changes the behavior of recording
"Telemetry::URLCLASSIFIER_VLPS_LOAD_CORRUPT" a little bit.
It used to records only once per session(during startup, the first
time we load prefix set), now it records per update.
Differential Revision: https://phabricator.services.mozilla.com/D21461
--HG--
extra : moz-landing-system : lando
SafeBrowsing V4 protocol use SHA-256 as the checksum to check integrity
of update data and also the integrity of prefix files.
SafeBrowsing V2 HashStore use MD5 as the checksum to check integrity of
.sbstore
Since we are going to use CRC32 as the integrity check of V4 prefix files,
I think rename V4 "checksum" to SHA256 can improve readability.
Differential Revision: https://phabricator.services.mozilla.com/D21460
--HG--
extra : moz-landing-system : lando
In Bug 1453038, |mUpdateInterrupted| is set in Classifer::Close() which is
called by PreShutdown to abort an ongoing update. That doesn't handle
all the cases.
The SafeBrowsing update involves two threads, worker thread, and update
thread. Update thread takes care of most of the update work, when it finishes
its task, it posts a task back to the worker thread to apply the updated database
and also do some cleanup stuff. Then the update is complete.
The fix in Bug 1453038 doesn't abort an update if the woker thread is doing
the job. This is because the |mUpdateInterrupted| flag is set in the
worker thread. The PreShutdown event which eventually sets the flag has to
wait until the worker thread's current task is done.
In this patch:
1. Check nsUrlClassifierDBService::ShutdownHasStarted() to abort shutdown.
This is set by main thread so both worker thread and update thread can
be interrupted now.
2. mIsClosed is now replaced by the mUpdateInterrupted. The semantics of
mUpdateInterrupted is now changed to abort update for any internal APIs
which should cause an update to abort.
3. Remove |mUpdateInterrupted| and |ShutdownHasStarted()| checks and
unify with |ShouldAbort()|
Differential Revision: https://phabricator.services.mozilla.com/D12229
--HG--
extra : moz-landing-system : lando
A directory-based file copy without checkpoint to abort may take lots
of time to finish. This cause an issue that if firefox is shutting down
and try to close an ongoing update thread, main-thread may be blocked
for a long time.
This patch adds a wrapper for copying an entire directory, within this
wrapper, we use file-based copy and add checkpoints to let update thread
has a chance to abort.
Differential Revision: https://phabricator.services.mozilla.com/D3414
--HG--
extra : moz-landing-system : lando
A directory-based file copy without checkpoint to abort may take lots
of time to finish. This cause an issue that if firefox is shutting down
and try to close an ongoing update thread, main-thread may be blocked
for a long time.
This patch adds a wrapper for copying an entire directory, within this
wrapper, we use file-based copy and add checkpoints to let update thread
has a chance to abort.
Differential Revision: https://phabricator.services.mozilla.com/D3414
--HG--
extra : moz-landing-system : lando
This makes it possible to use different lists for tracking protection
and for the features that rely on tracking annotations.
Differential Revision: https://phabricator.services.mozilla.com/D2484
--HG--
extra : moz-landing-system : lando
URL Classifier has a mUpdateInterrupted flag to abort an ongoing
update in several checkpoints, but we didn't use this while shutting down.
Set this flag in PreShutdown() to avoid url-classifier worker thread or
update thread takes too long to finish an update.
Differential Revision: https://phabricator.services.mozilla.com/D2157
--HG--
extra : moz-landing-system : lando
If Reset() is interleaved with a shutdown, there's no point in
finishing up and we may as well bail early.
MozReview-Commit-ID: Lhm6NfAEgSj
--HG--
extra : rebase_source : e74cc22a36d287a59425079a9f7c4676e7351eba
Explicitly specify the arguments to copy to avoid making a copy of
a dangling `this` pointer.
Convert nsUrlClassifierDBService::mClassifier to a RefPtr since
the update closure might need to continue to access its members
after it's been released by the main thread.
MozReview-Commit-ID: CPio3n9MmsK
--HG--
extra : rebase_source : d7a8b207336209ee37441f3429bc608140e404ae
Replace raw pointers to LookupResult with RefPtrs and eplace the
nsAutoPtr objects + raw pointers params with UniquePtrs.
Also remove unnecessarily paranoid OOM checks when creating single
LookupResult objects since those are pretty small.
MozReview-Commit-ID: G85RNnAat6H
--HG--
extra : rebase_source : a8f6a1ff1e24663d428c8d894cb624e1c67e1bd3
The existing mix of UniquePtr and raw pointers is confusing when
trying to figure out the exact lifetime of these objects.
MozReview-Commit-ID: Br4S7BXEFKs
--HG--
extra : rebase_source : ba35d5c2eeda0741eb4c5491a6caf03f20f3d0ce
This will prevent our holding on to this pointer incorrectly in the
future.
MozReview-Commit-ID: H8ueIOK1qAK
--HG--
extra : rebase_source : 937e9c702c5109b6dfc1684392a1204d8a1edc49
