c47e87c5f8
Automatic update from web-platform-tests [docs] Re-implement build process using Sphinx Prior to this commit, WPT used the Jekyll static site generator [1] to create a website from the project documentation. Jekyll has some restrictions which limit the utility of the generated website. Sphinx [2] has a number of features which will enable future improvements to the documentation - it can including content from files located outside of the designated "documentation" directory (some maintainers prefer to organize content elsewhere [3]) - it can generate documentation for Python CLIs like wptrunner [4] - the website it produces has a fully-featured client-side search interface - it can generating documentation for arbitrary Python files (the WPT documentation occasionally falls out of sync with the project's custom linter) - it detects and reports broken links and orphaned content (the WPT documentation has suffered from both problems in the past [5] [6] [7]) - as a Python tool, it presents less friction for local development than the Ruby-based Jekyll Although Sphinx parses content in the reStructured Text format by default, the existing content in WPT is formatted in Markdown. Markdown is generally preferable for this purpose because it is more familiar to the potential contributors to WPT and because the maintainers have no immediate plans to output to a format that is not a website. In recognition of this, enable the project's support for CommonMark. [1] https://jekyllrb.com/ [2] https://www.sphinx-doc.org [3] https://github.com/web-platform-tests/wpt/issues/5289 [4] https://github.com/ribozz/sphinx-argparse [5] https://github.com/web-platform-tests/wpt/issues/5299 [6] https://github.com/web-platform-tests/wpt/issues/10501 [7] https://github.com/web-platform-tests/wpt/issues/11479 -- Use rST syntax for tables -- [docs] Introduce tables of contents In addition to enhancing content discovery for site visitors, thoroughly applying Sphinx's "table of contents" feature allows the tool to report when content is not reachable from any page on the website. So-called "orphaned" content will be integrated in a subsequent commit. By using the `eval_rst` codeblock, this project can rely on advanced features available only to reStructured Text without sacrificing the familiarity of Markdown. -- [docs] Integrate orphaned content -- [docs] Rename files to preserve original URLs Previously, the project's documentation website was built using the Jekyll static site generator. That tool requires that site content be placed within directories whose name is prefixed with an underscore character (`_`). The leading underscore is removed from the generated content, so it is not present in the URLs of the resultant website. The Sphinx tool does not use this mechanism for content designation. Directories with a leading underscore are recreated in the generated site. Rename the directories by removing the leading underscore so that the content generated by Sphinx is available at the same URL as the content generated by Jekyll. Update internal links accordingly. This change set was created using the following commands: git mv {_,}writing-tests/ git mv {_,}running-tests/ git mv {_,}reviewing-tests/ git mv {_,}admin/ git mv {_,}appendix/ git ls-files . -z | \ xargs -0 sed -i 's/_\(writing-tests\|running-tests\|reviewing-tests\|admin\|appendix\)/\1/g' -- [docs] Build w/GH Actions and deploy w/GH Pages -- wp5At-commits: 02d56b7c166590601d8168dbd4e0e92532552abf, afef08f5bf1149bdf7dc4fdb1584c754d8e230a0, 238ea486f7b9adf33044dd438eb32b1151bbde4f, 22d851e14a45e0c2d044b3b595a63ba3f153f254, 970437d5648288c44152eaa4e8a73e247d343bca, 43a1f0b83446a492626ae5bb7364d080cadb2f46 wpt-pr: 16822
2.9 KiB
Manual Tests
Some testing scenarios are intrinsically difficult to automate and require a human to run the test and check the pass condition.
When to Write Manual Tests
Whenever possible it's best to write a fully automated test. For a browser vendor it's possible to run an automated test hundreds of times a day, but manual tests are likely to be run at most a handful of times a year (and quite possibly approximately never!). This makes them significantly less useful for catching regressions than automated tests.
However, there are certain scenarios in which this is not yet possible. For example:
-
Test which require observing animation (e.g., a test for CSS animation or for video playback),
-
Tests that require interaction with browser security UI (e.g., a test in which a user refuses a geolocation permissions grant),
-
Tests that require interaction with the underlying OS (e.g., tests for drag and drop from the desktop onto the browser),
-
Tests that require non-default browser configuration (e.g., images disabled), and
-
Tests that require interaction with the physical environment (e.g., tests that the vibration API causes the device to vibrate or that various sensor APIs respond in the expected way).
Requirements for a Manual Test
Manual tests are distinguished by their filename; all manual tests
have filenames of the form name-manual.ext (i.e., a -manual suffix
after the main filename but before the extension).
Manual tests must be fully self-describing. It is particularly important for these tests that it is easy to determine the result from the information provided in the page to the tester, because a tester may have hundreds of tests to get through and little understanding of the features that they are testing. As a result, minimalism is especially a virtue for manual tests.
A test should have, at a minimum step-by-step instructions for performing the test, and a clear statement of either the test result if it can be automatically determined after some setup or how to otherwise determine the outcome.
Any information other than this (e.g., quotes from the spec) should be avoided (though, as always, can be provided in HTML/CSS/JS/etc. comments).
Using testharness.js for Manual Tests
A convenient way to present the results of a test that can have the
result determined by script after some manual setup steps is to use
testharness.js to determine and present the result. In this case one
must pass {explicit_timeout: true} in a call to setup() in order
to disable the automatic timeout of the test. For example:
<!doctype html>
<title>Manual click on button triggers onclick handler</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script>
setup({explicit_timeout: true})
</script>
<p>Click on the button below. If a "PASS" result appears the test
passes, otherwise it fails</p>
<button onclick="done()">Click Here</button>