nagai/fune
3
0
Fork 0
forked from mirrors/gecko-dev
Code Pull requests Activity

Bug 1890947 - Document pipelines using JSDoc r=gregtatum a=reland CLOSED TREE

Browse source 
Differential Revision: https://phabricator.services.mozilla.com/D208469
This commit is contained in:
Tarek Ziadé 2024-04-25 06:30:38 +00:00
parent c8b8e21980
commit 4613397eba
4 changed files with 52 additions and 45 deletions
repo.diff.show_diff_stats Download patch file Download diff file Expand all files Collapse all files

1
docs/conf.py
View file

@ -72,6 +72,7 @@ js_source_path = [
"../toolkit/components/extensions",
"../toolkit/components/extensions/parent",
"../toolkit/components/featuregates",
"../toolkit/components/ml/content/ONNXPipeline.mjs",
"../toolkit/mozapps/extensions",
"../toolkit/components/prompts/src",
"../toolkit/components/pictureinpicture",

10
toolkit/components/ml/content/ONNXPipeline.mjs
View file

@ -61,12 +61,10 @@ async function echo(request, _model, _tokenizer, _processor) {
* Converts an image to text using a machine learning model.
*
* @async
* @param {T} request - The request object containing image data.
* It should contain at least one of the following fields:
* - data: Raw data for processing.
* - imageURL: A URL pointing to an image to be fetched and processed
* And the following field:
* - mimeType: The MIME type of the data or image, specifying how to interpret or process the input.
* @param {object} request - The request object containing image data.
* @param {string} [request.imageUrl] - The URL of the image to process. Either `imageUrl` or `data` must be provided, but not both.
* @param {ArrayBuffer} [request.data] - The raw image data to process. Either `data` or `imageUrl` must be provided, but not both.
* @param {string} request.mimeType - The MIME type of the image data.
* @param {object} model - The model used for inference.
* @param {object} tokenizer - The tokenizer used for decoding.
* @param {object} processor - The processor used for preparing image data.

39
toolkit/components/ml/docs/index.md
View file

@ -1,39 +0,0 @@
# Machine Learning
This component is an experimental machine learning local inference engine based on
Transformers.js and the ONNX runtime.
In the example below, an image is converted to text using the `image-to-text` task.
```js
const {PipelineOptions, EngineProcess } = ChromeUtils.importESModule("chrome://global/content/ml/EngineProcess.sys.mjs");
// First we create a pipeline options object, which contains the task name
// and any other options needed for the task
const options = new PipelineOptions({taskName: "image-to-text" });
// Next, we create an engine parent object via EngineProcess
const engineParent = await EngineProcess.getMLEngineParent();
// We then create the engine object, using the options
const engine = engineParent.getEngine(options);
// At this point we are ready to do some inference.
// We need to get the image as an array buffer and wrap it into a request object
const response = await fetch("https://huggingface.co/datasets/mishig/sample_images/resolve/main/football-match.jpg");
const buffer = await response.arrayBuffer();
const mimeType = response.headers.get('Content-Type');
const request = {
data: buffer,
mimeType: mimeType
};
// Finally, we run the engine with the request object
const res = await engine.run(request);
// The result is a string containing the text extracted from the image
console.log(res);
```

47
toolkit/components/ml/docs/index.rst Normal file
View file

@ -0,0 +1,47 @@
Machine Learning
================
This component is an experimental machine learning local inference engine based on
Transformers.js and the ONNX runtime.
In the example below, an image is converted to text using the `image-to-text` task.
.. code-block:: javascript
const {PipelineOptions, EngineProcess } = ChromeUtils.importESModule("chrome://global/content/ml/EngineProcess.sys.mjs");
// First we create a pipeline options object, which contains the task name
// and any other options needed for the task
const options = new PipelineOptions({taskName: "image-to-text" });
// Next, we create an engine parent object via EngineProcess
const engineParent = await EngineProcess.getMLEngineParent();
// We then create the engine object, using the options
const engine = engineParent.getEngine(options);
// At this point we are ready to do some inference.
// We need to get the image as an array buffer and wrap it into a request object
const response = await fetch("https://huggingface.co/datasets/mishig/sample_images/resolve/main/football-match.jpg");
const buffer = await response.arrayBuffer();
const mimeType = response.headers.get('Content-Type');
const request = {
data: buffer,
mimeType: mimeType
};
// Finally, we run the engine with the request object
const res = await engine.run(request);
// The result is a string containing the text extracted from the image
console.log(res);
Supported Inference Tasks
:::::::::::::::::::::::::
The following tasks are supported by the machine learning engine:
.. js:autofunction:: imageToText