forked from mirrors/gecko-dev
227 lines
10 KiB
ReStructuredText
227 lines
10 KiB
ReStructuredText
========================
|
|
How webext storage works
|
|
========================
|
|
|
|
This document describes the implementation of the the `storage.sync` part of the
|
|
`WebExtensions Storage APIs
|
|
<https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/storage>`_.
|
|
The implementation lives in the `toolkit/components/extensions/storage folder <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage>`_
|
|
|
|
Ideally you would already know about Rust and XPCOM - `see this doc for more details <../../../../writing-rust-code/index.html>`_
|
|
|
|
At a very high-level, the system looks like:
|
|
|
|
.. mermaid::
|
|
|
|
graph LR
|
|
A[Extensions API]
|
|
A --> B[Storage JS API]
|
|
B --> C{magic}
|
|
C --> D[app-services component]
|
|
|
|
Where "magic" is actually the most interesting part and the primary focus of this document.
|
|
|
|
Note: The general mechanism described below is also used for other Rust components from the
|
|
app-services team - for example, "dogear" uses a similar mechanism, and the sync engines
|
|
too (but with even more complexity) to manage the threads. Unfortunately, at time of writing,
|
|
no code is shared and it's not clear how we would, but this might change as more Rust lands.
|
|
|
|
The app-services component `lives on github <https://github.com/mozilla/application-services/blob/main/components/webext-storage>`_.
|
|
There are docs that describe `how to update/vendor this (and all) external rust code <../../../../build/buildsystem/rust.html>`_ you might be interested in.
|
|
|
|
To set the scene, let's look at the parts exposed to WebExtensions first; there are lots of
|
|
moving part there too.
|
|
|
|
WebExtension API
|
|
################
|
|
|
|
The WebExtension API is owned by the addons team. The implementation of this API is quite complex
|
|
as it involves multiple processes, but for the sake of this document, we can consider the entry-point
|
|
into the WebExtension Storage API as being `parent/ext-storage.js <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/parent/ext-storage.js>`_
|
|
|
|
This entry-point ends up using the implementation in the
|
|
`ExtensionStorageSync JS class <https://searchfox.org/mozilla-central/rev/9028b0458cc1f432870d2996b186b0938dda734a/toolkit/components/extensions/ExtensionStorageSync.jsm#84>`_.
|
|
This class/module has complexity for things like migration from the earlier Kinto-based backend,
|
|
but importantly, code to adapt a callback API into a promise based one.
|
|
|
|
Overview of the API
|
|
###################
|
|
|
|
At a high level, this API is quite simple - there are methods to "get/set/remove" extension
|
|
storage data. Note that the "external" API exposed to the addon has subtly changed the parameters
|
|
for this "internal" API, so there's an extension ID parameter and the JSON data has already been
|
|
converted to a string.
|
|
The semantics of the API are beyond this doc but are
|
|
`documented on MDN <https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync>`_.
|
|
|
|
As you will see in those docs, the API is promise-based, but the rust implementation is fully
|
|
synchronous and Rust knows nothing about Javascript promises - so this system converts
|
|
the callback-based API to a promise-based one.
|
|
|
|
xpcom as the interface to Rust
|
|
##############################
|
|
|
|
xpcom is old Mozilla technology that uses C++ "vtables" to implement "interfaces", which are
|
|
described in IDL files. While this traditionally was used to interface
|
|
C++ and Javascript, we are leveraging existing support for Rust. The interface we are
|
|
exposing is described in `mozIExtensionStorageArea.idl <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/mozIExtensionStorageArea.idl>`_
|
|
|
|
The main interface of interest in this IDL file is `mozIExtensionStorageArea`.
|
|
This interface defines the functionality - and is the first layer in the sync to async model.
|
|
For example, this interface defines the following method:
|
|
|
|
.. code-block:: rust
|
|
|
|
interface mozIExtensionStorageArea : nsISupports {
|
|
...
|
|
// Sets one or more key-value pairs specified in `json` for the
|
|
// `extensionId`...
|
|
void set(in AUTF8String extensionId,
|
|
in AUTF8String json,
|
|
in mozIExtensionStorageCallback callback);
|
|
|
|
As you will notice, the 3rd arg is another interface, `mozIExtensionStorageCallback`, also
|
|
defined in that IDL file. This is a small, generic interface defined as:
|
|
|
|
.. code-block:: cpp
|
|
|
|
interface mozIExtensionStorageCallback : nsISupports {
|
|
// Called when the operation completes. Operations that return a result,
|
|
// like `get`, will pass a `UTF8String` variant. Those that don't return
|
|
// anything, like `set` or `remove`, will pass a `null` variant.
|
|
void handleSuccess(in nsIVariant result);
|
|
|
|
// Called when the operation fails.
|
|
void handleError(in nsresult code, in AUTF8String message);
|
|
};
|
|
|
|
Note that this delivers all results and errors, so must be capable of handling
|
|
every result type, which for some APIs may be problematic - but we are very lucky with this API
|
|
that this simple XPCOM callback interface is capable of reasonably representing the return types
|
|
from every function in the `mozIExtensionStorageArea` interface.
|
|
|
|
(There's another interface, `mozIExtensionStorageListener` which is typically
|
|
also implemented by the actual callback to notify the extension about changes,
|
|
but that's beyond the scope of this doc.)
|
|
|
|
*Note the thread model here is async* - the `set` call will return immediately, and later, on
|
|
the main thread, we will call the callback param with the result of the operation.
|
|
|
|
So under the hood, what happens is something like:
|
|
|
|
.. mermaid::
|
|
|
|
sequenceDiagram
|
|
Extension->>ExtensionStorageSync: call `set` and give me a promise
|
|
ExtensionStorageSync->>xpcom: call `set`, supplying new data and a callback
|
|
ExtensionStorageSync-->>Extension: your promise
|
|
xpcom->>xpcom: thread magic in the "bridge"
|
|
xpcom-->>ExtensionStorageSync: callback!
|
|
ExtensionStorageSync-->>Extension: promise resolved
|
|
|
|
So onto the thread magic in the bridge!
|
|
|
|
webext_storage_bridge
|
|
#####################
|
|
|
|
The `webext_storage_bridge <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge>`_
|
|
is a Rust crate which, as implied by the name, is a "bridge" between this Javascript/XPCOM world to
|
|
the actual `webext-storage <https://github.com/mozilla/application-services/tree/main/components/webext-storage>`_ crate.
|
|
|
|
lib.rs
|
|
------
|
|
|
|
Is the entry-point - it defines the xpcom "factory function" -
|
|
an `extern "C"` function which is called by xpcom to create the Rust object
|
|
implementing `mozIExtensionStorageArea` using existing gecko support.
|
|
|
|
area.rs
|
|
-------
|
|
|
|
This module defines the interface itself. For example, inside that file you will find:
|
|
|
|
.. code-block:: rust
|
|
|
|
impl StorageSyncArea {
|
|
...
|
|
|
|
xpcom_method!(
|
|
set => Set(
|
|
ext_id: *const ::nsstring::nsACString,
|
|
json: *const ::nsstring::nsACString,
|
|
callback: *const mozIExtensionStorageCallback
|
|
)
|
|
);
|
|
/// Sets one or more key-value pairs.
|
|
fn set(
|
|
&self,
|
|
ext_id: &nsACString,
|
|
json: &nsACString,
|
|
callback: &mozIExtensionStorageCallback,
|
|
) -> Result<()> {
|
|
self.dispatch(
|
|
Punt::Set {
|
|
ext_id: str::from_utf8(&*ext_id)?.into(),
|
|
value: serde_json::from_str(str::from_utf8(&*json)?)?,
|
|
},
|
|
callback,
|
|
)?;
|
|
Ok(())
|
|
}
|
|
|
|
|
|
Of interest here:
|
|
|
|
* `xpcom_method` is a Rust macro, and part of the existing xpcom integration which already exists
|
|
in gecko. It declares the xpcom vtable method described in the IDL.
|
|
|
|
* The `set` function is the implementation - it does string conversions and the JSON parsing
|
|
on the main thread, then does the work via the supplied callback param, `self.dispatch` and a `Punt`.
|
|
|
|
* The `dispatch` method dispatches to another thread, leveraging existing in-tree `moz_task <https://searchfox.org/mozilla-central/source/xpcom/rust/moz_task>`_ support, shifting the `Punt` to another thread and making the callback when done.
|
|
|
|
Punt
|
|
----
|
|
|
|
`Punt` is a whimsical name somewhat related to a "bridge" - it carries things across and back.
|
|
|
|
It is a fairly simple enum in `punt.rs <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src/punt.rs>`_.
|
|
It's really just a restatement of the API we expose suitable for moving across threads. In short, the `Punt` is created on the main thread,
|
|
then sent to the background thread where the actual operation runs via a `PuntTask` and returns a `PuntResult`.
|
|
|
|
There's a few dances that go on, but the end result is that `inner_run() <https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge/src/punt.rs>`_
|
|
gets executed on the background thread - so for `Set`:
|
|
|
|
.. code-block:: rust
|
|
|
|
Punt::Set { ext_id, value } => {
|
|
PuntResult::with_change(&ext_id, self.store()?.get()?.set(&ext_id, value)?)?
|
|
}
|
|
|
|
Here, `self.store()` is a wrapper around the actual Rust implementation from app-services with
|
|
various initialization and mutex dances involved - see `store.rs`.
|
|
ie, this function is calling our Rust implementation and stashing the result in a `PuntResult`
|
|
|
|
The `PuntResult` is private to that file but is a simple struct that encapsulates both
|
|
the actual result of the function (also a set of changes to send to observers, but that's
|
|
beyond this doc).
|
|
|
|
Ultimately, the `PuntResult` ends up back on the main thread once the call is complete
|
|
and arranges to callback the JS implementation, which in turn resolves the promise created in `ExtensionStorageSync.sys.mjs`
|
|
|
|
End result:
|
|
-----------
|
|
|
|
.. mermaid::
|
|
|
|
sequenceDiagram
|
|
Extension->>ExtensionStorageSync: call `set` and give me a promise
|
|
ExtensionStorageSync->>xpcom - bridge main thread: call `set`, supplying new data and a callback
|
|
ExtensionStorageSync-->>Extension: your promise
|
|
xpcom - bridge main thread->>moz_task worker thread: Punt this
|
|
moz_task worker thread->>webext-storage: write this data to the database
|
|
webext-storage->>webext-storage: done: result/error and observers
|
|
webext-storage-->>moz_task worker thread: ...
|
|
moz_task worker thread-->>xpcom - bridge main thread: PuntResult
|
|
xpcom - bridge main thread-->>ExtensionStorageSync: callback!
|
|
ExtensionStorageSync-->>Extension: promise resolved
|