====================================== Managing the built-in en-US dictionary ====================================== The en-US build of Firefox includes a built-in Hunspell dictionary based on the `SCOWL`_ dataset. This document describes the process to add new words to the dictionary, or update it to the current upstream version. For more information about Hunspell or the affix file format, you can check `the Ubuntu man page for hunspell `_. Requesting to add new words to the en-US dictionary =================================================== If you’d like to add new words to the dictionary, you can `file a bug`_. Try to provide information on the terms you want to add, in particular references to external sources that confirm the usage of the term. Adding new words to the en-US dictionary ======================================== This section describes the process for adding a word to the dictionary: #. Get a clone of mozilla-central (see :ref:`Firefox Contributors' Quick Reference`), if you don’t already have one, and make sure you can build it successfully. #. Get into the dictionary sources directory using this command: ``cd extensions/spellcheck/locales/en-US/hunspell/dictionary-sources`` #. There’s a special script used for editing dictionaries. The script only works if you have the environment variable ``EDITOR`` set to the executable of an editor program; if you don’t have it set, you can use ``EDITOR=vim sh edit-dictionary.sh`` to edit using ``vim`` (or you can substitute it with another editor), or you can just type ``sh edit-dictionary.sh`` if you have an ``EDITOR`` already specified. #. Add and remove words in the dictionary file, then quit the editor. #. Build Firefox and test your updated dictionary. Once you’re satisfied, use the process described in :ref:`write_a_patch` to create a patch. Note that the update script will modify 2 files, and both need to be committed: * ``en-US.dic``: the dictionary actually shipping in the build and uses ISO-8859-1 encoding. * ``utf8/en-US.dic``: a version of the same dictionary with UTF-8 encoding. This is used to work around issues with Phabricator, and it allows to display actual changes in the diff. Upgrading dictionary to a new upstream version of SCOWL ======================================================= The English dictionary available in mozilla-central is based on the `SCOWL`_ dictionary. Some scripts distributed with the SCOWL package are used to generate the files for the en-US dictionary. The working directory for this process is ``extensions/spellcheck/locales/en-US/hunspell/dictionary-sources``. #. Download the latest version of the dictionary from `SCOWL`_ homepage or `SourceForce`_ as a tarball (tag.gz) and unpack it in the working directory. Rename the resulting folder from ``scowl-YYYY.MM.DD`` to ``scowl``. #. Run the script ``sh make-new-dict.sh`` to generate a new dictionary and make sure it runs without any errors. For more details on this script, see the `make-new-dict.sh`_ section. #. Do a sanity check on the resulting dictionary file ``en_US-mozilla.dic``. For example, make sure that the size is about the same as the original dictionary (or slightly larger). #. If everything looks correct, use ``sh install-new-dict.sh`` to copy the generated file in the right position and use the process described in :ref:`write_a_patch` to create a patch. Info about the file structure ============================= mozilla-exclusions.txt ---------------------- ``mozilla-exclusions.txt`` is used to explicitly exclude some words from suggestions. The ``make-new-dict.sh`` script will add them to the dictionary file with the ``/!`` flag. Terms should be added to this file with exactly the same format used in the .dic file, including affix rules if available. mozilla-specific.txt -------------------- This file contains Mozilla-specific words that should not be submitted upstream. For example, ``Firefox`` should go in this file (see `bug 237921`_). Note that the file ``5-mozilla-specific.txt`` is generated by expanding ``mozilla-specific.txt`` and should not be edited directly. utf8 folder ----------- ``dictionary-sources/utf8`` is used to store a copy with UTF-8 encoding of the dictionary files. This is used to work around limitations in Phabricator, which treats ISO-8859-1 files as binary and won’t display a diff when updating them. Info about the included scripts =============================== make-new-dict.sh ---------------- The dictionary upgrade scripts ``make-new-dict.sh`` works by expanding (i.e. “unmunching”) the affix compression dictionaries to create wordlists and use those to generate a new dictionary. The upgrade script expects the current upstream version to be kept in the directory ``orig``. The script will create a few files in ``dictionary-sources/support_file`` in the following order: * ``0-special.txt`` contains numbers and ordinals expanded from SCOWL ``en.dic.supp``. * ``1-base.txt`` contains words expanded from ``en_US-custom.dic`` in the **previous** version of SCOWL (from the ``orig`` folder). * ``2-mozilla.txt`` contains words expanded from the current Mozilla dictionary. * ``3-upstream.txt`` contains words expanded from ``en_US-custom.dic`` in the **new** version of SCOWL (from the ``scowl/speller`` folder). * ``2-mozilla-removed.txt`` contains words that are only available in the SCOWL dictionary, i.e. removed by Mozilla. * ``2-mozilla-added.txt`` contains words that are only available in the current Mozilla dictionary, i.e. added by Mozilla. * ``4-patched.txt`` contains words from the new SCOWL dictionary (``3-upstream.txt``), with words from (``2-mozilla-removed.txt``) removed and words (``2-mozilla-added.txt``) added. * ``5-mozilla-specific.txt`` is expanded from ``mozilla-specific.txt`` using the current affix rules from the Mozilla dictionary. * ``5-mozilla-removed.txt`` and ``5-mozilla-added.txt`` contain words that are respectively removed and added by Mozilla compared to the **new** SCOWL version. These files could be used to submit upstream changes, but words included in ``5-mozilla-specific.txt`` should be removed from this list. The new dictionary is available as ``en_US-mozilla.dic`` and should be copied over using the ``install-new-dict.sh`` script. install-new-dict.sh ------------------- The script: * Creates a copy of ``orig`` as ``support_files/orig-bk`` and copies the new upstream version to ``orig``. * Copies the existing Mozilla dictionary in ``support_files/mozilla-bk``. * Converts the dictionary (.dic) generated by ``make-new-dict.sh`` from UTF-8 to ISO-8859-1 and moves it to the parent folder. * Sets the affix file (.aff) to use ``ISO8859-1`` as ``SET`` instead of the original ``UTF-8``, removes ``ICONV`` patterns (input conversion tables). .. _SCOWL: http://wordlist.aspell.net .. _file a bug: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Spelling%20checker .. _SourceForce: https://sourceforge.net/projects/wordlist/files/SCOWL/ .. _bug 237921: https://bugzilla.mozilla.org/show_bug.cgi?id=237921