nagai/fune
3
0
Fork 0
forked from mirrors/gecko-dev
Code Pull requests Activity

Bug 1899623 - Update youtube-playback tests to use new prod URLs. r=perftest-reviewers,afinder

Browse source 
This patch updates the URLs used by the youtube-playback tests to the new prod URLs.

Differential Revision: https://phabricator.services.mozilla.com/D212056
This commit is contained in:
Greg Mierzwinski 2024-06-04 11:00:58 +00:00
parent 89c27f522a
commit ca4c620d45
4 changed files with 47 additions and 100 deletions
repo.diff.show_diff_stats Download patch file Download diff file Expand all files Collapse all files

2
testing/perfdocs/generated/raptor-metrics.rst
View file

@ -201,6 +201,6 @@ Metrics starting with VP9/H264 give the number of frames dropped, and painted.
* **Aliases**: H264, VP9 * **Aliases**: H264, VP9
* **Tests using it**: * **Tests using it**:
* **Benchmarks**: `youtube-playback <raptor.html#youtube-playback-b>`__, `youtube-playback-hfr <raptor.html#youtube-playback-hfr-b>`__ * **Benchmarks**: `youtube-playback-hfr <raptor.html#youtube-playback-hfr-b>`__

86
testing/perfdocs/generated/raptor.rst
View file

@ -8980,12 +8980,11 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
.. dropdown:: youtube-playback .. dropdown:: youtube-playback-av1-sfr
:class-container: anchor-id-youtube-playback-b :class-container: anchor-id-youtube-playback-av1-sfr-b
**Owner**: PerfTest Team **Owner**: PerfTest Team
* **alert on**: `H264.1080p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.1080p60@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.1440p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.144p15@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.2160p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.240p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.360p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.480p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.720p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `H264.720p60@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.1080p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.1080p60@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.1440p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.1440p60@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.144p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.2160p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.2160p60@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.240p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.360p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.480p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.720p30@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__, `VP9.720p60@1X_dropped_frames <raptor-metrics.html#youtube-playback-metrics>`__
* **alert threshold**: 2.0 * **alert threshold**: 2.0
* **apps**: firefox, geckoview, fenix, refbrow, chrome * **apps**: firefox, geckoview, fenix, refbrow, chrome
* **expected**: pass * **expected**: pass
@ -8999,30 +8998,7 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-av1-test&raptor=true&exclude=1,2&muted=true&command=run>`__
* **type**: benchmark
* **unit**: score
* **use live sites**: true
.. dropdown:: youtube-playback-av1-sfr
:class-container: anchor-id-youtube-playback-av1-sfr-b
**Owner**: PerfTest Team
* **alert threshold**: 2.0
* **apps**: firefox, geckoview, fenix, refbrow, chrome
* **expected**: pass
* **expose browser profiler**: true
* **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#52>`__
* **lower is better**: true
* **page cycles**: 1
* **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true
* **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-av1-test&raptor=true&exclude=1,2&muted=true&command=run>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -9089,14 +9065,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **gather cpuTime**: true * **gather cpuTime**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#56>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#28>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 20 * **page cycles**: 20
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&exclude=1,2>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&exclude=1,2>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -9291,14 +9267,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **gather cpuTime**: true * **gather cpuTime**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#62>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#34>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 20 * **page cycles**: 20
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&exclude=1,2>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&exclude=1,2>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -9493,14 +9469,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **gather cpuTime**: true * **gather cpuTime**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#68>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#40>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 20 * **page cycles**: 20
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false full-screen-api.allow-trusted-requests-only=false full-screen-api.warning.timeout=0 * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false full-screen-api.allow-trusted-requests-only=false full-screen-api.warning.timeout=0
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&fullscreen=true&exclude=1,2>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&fullscreen=true&exclude=1,2>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -9695,14 +9671,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **gather cpuTime**: true * **gather cpuTime**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#83>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#55>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 20 * **page cycles**: 20
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false full-screen-api.allow-trusted-requests-only=false full-screen-api.warning.timeout=0 * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false full-screen-api.allow-trusted-requests-only=false full-screen-api.warning.timeout=0
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&fullscreen=true&exclude=1,2>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&fullscreen=true&exclude=1,2>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -9896,14 +9872,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **expose browser profiler**: true * **expose browser profiler**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#98>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#70>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 1 * **page cycles**: 1
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -10146,14 +10122,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **expose browser profiler**: true * **expose browser profiler**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#102>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#74>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 1 * **page cycles**: 1
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&exclude=1,2&muted=true&command=run>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&exclude=1,2&muted=true&command=run>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -10406,14 +10382,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **gather cpuTime**: true * **gather cpuTime**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#110>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#82>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 20 * **page cycles**: 20
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&exclude=1,2>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&exclude=1,2>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -10608,14 +10584,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **gather cpuTime**: true * **gather cpuTime**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#116>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#88>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 20 * **page cycles**: 20
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&exclude=1,2>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&exclude=1,2>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -10810,14 +10786,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **gather cpuTime**: true * **gather cpuTime**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#122>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#94>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 20 * **page cycles**: 20
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false full-screen-api.allow-trusted-requests-only=false full-screen-api.warning.timeout=0 * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false full-screen-api.allow-trusted-requests-only=false full-screen-api.warning.timeout=0
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&fullscreen=true&exclude=1,2>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&fullscreen=true&exclude=1,2>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -11012,14 +10988,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **gather cpuTime**: true * **gather cpuTime**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#137>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#109>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 20 * **page cycles**: 20
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false full-screen-api.allow-trusted-requests-only=false full-screen-api.warning.timeout=0 * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false full-screen-api.allow-trusted-requests-only=false full-screen-api.warning.timeout=0
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&fullscreen=true&exclude=1,2>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&fullscreen=true&exclude=1,2>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -11213,14 +11189,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **expose browser profiler**: true * **expose browser profiler**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#152>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#124>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 1 * **page cycles**: 1
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -11462,14 +11438,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **expose browser profiler**: true * **expose browser profiler**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#156>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#128>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 1 * **page cycles**: 1
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false media.eme.enabled=true media.gmp-manager.updateEnabled=true media.eme.require-app-approval=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false media.eme.enabled=true media.gmp-manager.updateEnabled=true media.eme.require-app-approval=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-widevine-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -11679,14 +11655,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **expose browser profiler**: true * **expose browser profiler**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#170>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#142>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 1 * **page cycles**: 1
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false media.eme.enabled=true media.gmp-manager.updateEnabled=true media.eme.require-app-approval=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false media.eme.enabled=true media.gmp-manager.updateEnabled=true media.eme.require-app-approval=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-hfr-test&raptor=true&exclude=1,2&muted=true&command=run>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-widevine-hfr-test&raptor=true&exclude=1,2&muted=true&command=run>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true
@ -11896,14 +11872,14 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **expose browser profiler**: true * **expose browser profiler**: true
* **gecko profile interval**: 1000 * **gecko profile interval**: 1000
* **gecko profile threads**: MediaPlayback * **gecko profile threads**: MediaPlayback
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#184>`__ * **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/youtube-playback.toml#156>`__
* **lower is better**: true * **lower is better**: true
* **page cycles**: 1 * **page cycles**: 1
* **page timeout**: 2700000 * **page timeout**: 2700000
* **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false media.eme.enabled=true media.gmp-manager.updateEnabled=true media.eme.require-app-approval=false * **preferences**: media.autoplay.default=0 media.autoplay.ask-permission=false media.autoplay.blocking_policy=0 media.allowed-to-play.enabled=true media.block-autoplay-until-in-foreground=false media.eme.enabled=true media.gmp-manager.updateEnabled=true media.eme.require-app-approval=false
* **subtest lower is better**: true * **subtest lower is better**: true
* **subtest unit**: score * **subtest unit**: score
* **test url**: `<https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run>`__ * **test url**: `<https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-widevine-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run>`__
* **type**: benchmark * **type**: benchmark
* **unit**: score * **unit**: score
* **use live sites**: true * **use live sites**: true

1
testing/raptor/raptor/perfdocs/config.yml
View file

@ -181,7 +181,6 @@ suites:
wasm-misc-optimizing: "" wasm-misc-optimizing: ""
wasm-misc: "" wasm-misc: ""
webaudio: "" webaudio: ""
youtube-playback: ""
youtube-playback-av1-sfr: "" youtube-playback-av1-sfr: ""
youtube-playback-h264-1080p30: "" youtube-playback-h264-1080p30: ""
youtube-playback-h264-1080p60: "" youtube-playback-h264-1080p60: ""

58
testing/raptor/raptor/tests/benchmarks/youtube-playback.toml
View file

@ -21,49 +21,21 @@ preferences = [
"media.block-autoplay-until-in-foreground=false", "media.block-autoplay-until-in-foreground=false",
] ]
["youtube-playback"]
alert_on = [
"H264.1080p30@1X_dropped_frames",
"H264.1080p60@1X_dropped_frames",
"H264.1440p30@1X_dropped_frames",
"H264.144p15@1X_dropped_frames",
"H264.2160p30@1X_dropped_frames",
"H264.240p30@1X_dropped_frames",
"H264.360p30@1X_dropped_frames",
"H264.480p30@1X_dropped_frames",
"H264.720p30@1X_dropped_frames",
"H264.720p60@1X_dropped_frames",
"VP9.1080p30@1X_dropped_frames",
"VP9.1080p60@1X_dropped_frames",
"VP9.1440p30@1X_dropped_frames",
"VP9.1440p60@1X_dropped_frames",
"VP9.144p30@1X_dropped_frames",
"VP9.2160p30@1X_dropped_frames",
"VP9.2160p60@1X_dropped_frames",
"VP9.240p30@1X_dropped_frames",
"VP9.360p30@1X_dropped_frames",
"VP9.480p30@1X_dropped_frames",
"VP9.720p30@1X_dropped_frames",
"VP9.720p60@1X_dropped_frames",
]
apps = "firefox, geckoview, fenix, refbrow, chrome"
test_url = "http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true"
["youtube-playback-av1-sfr"] ["youtube-playback-av1-sfr"]
apps = "firefox, geckoview, fenix, refbrow, chrome" apps = "firefox, geckoview, fenix, refbrow, chrome"
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-av1-test&raptor=true&exclude=1,2&muted=true&command=run" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-av1-test&raptor=true&exclude=1,2&muted=true&command=run"
["youtube-playback-h264-1080p30"] ["youtube-playback-h264-1080p30"]
apps = "firefox" apps = "firefox"
page_cycles = 20 page_cycles = 20
gather_cpuTime = true gather_cpuTime = true
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&exclude=1,2" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&exclude=1,2"
["youtube-playback-h264-1080p60"] ["youtube-playback-h264-1080p60"]
apps = "firefox" apps = "firefox"
page_cycles = 20 page_cycles = 20
gather_cpuTime = true gather_cpuTime = true
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&exclude=1,2" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&exclude=1,2"
["youtube-playback-h264-full-1080p30"] ["youtube-playback-h264-full-1080p30"]
apps = "firefox" apps = "firefox"
@ -78,7 +50,7 @@ preferences = [
"full-screen-api.allow-trusted-requests-only=false", "full-screen-api.allow-trusted-requests-only=false",
"full-screen-api.warning.timeout=0", "full-screen-api.warning.timeout=0",
] ]
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&fullscreen=true&exclude=1,2" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-h264-test&tests=18&raptor=true&muted=true&command=run&fullscreen=true&exclude=1,2"
["youtube-playback-h264-full-1080p60"] ["youtube-playback-h264-full-1080p60"]
apps = "firefox" apps = "firefox"
@ -93,11 +65,11 @@ preferences = [
"full-screen-api.allow-trusted-requests-only=false", "full-screen-api.allow-trusted-requests-only=false",
"full-screen-api.warning.timeout=0", "full-screen-api.warning.timeout=0",
] ]
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&fullscreen=true&exclude=1,2" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=46&muted=true&command=run&fullscreen=true&exclude=1,2"
["youtube-playback-h264-sfr"] ["youtube-playback-h264-sfr"]
apps = "firefox, geckoview, fenix, refbrow, chrome" apps = "firefox, geckoview, fenix, refbrow, chrome"
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run"
["youtube-playback-hfr"] ["youtube-playback-hfr"]
alert_on = [ alert_on = [
@ -105,19 +77,19 @@ alert_on = [
"H264720p60fps@1X_dropped_frames", "H264720p60fps@1X_dropped_frames",
] ]
apps = "firefox, geckoview, fenix, refbrow, chrome" apps = "firefox, geckoview, fenix, refbrow, chrome"
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&exclude=1,2&muted=true&command=run" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&exclude=1,2&muted=true&command=run"
["youtube-playback-v9-1080p30"] ["youtube-playback-v9-1080p30"]
apps = "firefox" apps = "firefox"
page_cycles = 20 page_cycles = 20
gather_cpuTime = true gather_cpuTime = true
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&exclude=1,2" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&exclude=1,2"
["youtube-playback-v9-1080p60"] ["youtube-playback-v9-1080p60"]
apps = "firefox" apps = "firefox"
page_cycles = 20 page_cycles = 20
gather_cpuTime = true gather_cpuTime = true
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&exclude=1,2" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&exclude=1,2"
["youtube-playback-v9-full-1080p30"] ["youtube-playback-v9-full-1080p30"]
apps = "firefox" apps = "firefox"
@ -132,7 +104,7 @@ preferences = [
"full-screen-api.allow-trusted-requests-only=false", "full-screen-api.allow-trusted-requests-only=false",
"full-screen-api.warning.timeout=0", "full-screen-api.warning.timeout=0",
] ]
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&fullscreen=true&exclude=1,2" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&tests=18&muted=true&command=run&fullscreen=true&exclude=1,2"
["youtube-playback-v9-full-1080p60"] ["youtube-playback-v9-full-1080p60"]
apps = "firefox" apps = "firefox"
@ -147,11 +119,11 @@ preferences = [
"full-screen-api.allow-trusted-requests-only=false", "full-screen-api.allow-trusted-requests-only=false",
"full-screen-api.warning.timeout=0", "full-screen-api.warning.timeout=0",
] ]
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&fullscreen=true&exclude=1,2" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-hfr-test&raptor=true&tests=14&muted=true&command=run&fullscreen=true&exclude=1,2"
["youtube-playback-vp9-sfr"] ["youtube-playback-vp9-sfr"]
apps = "firefox, geckoview, fenix, refbrow, chrome" apps = "firefox, geckoview, fenix, refbrow, chrome"
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run"
["youtube-playback-widevine-h264-sfr"] ["youtube-playback-widevine-h264-sfr"]
apps = "firefox, geckoview, fenix, refbrow, chrome" apps = "firefox, geckoview, fenix, refbrow, chrome"
@ -165,7 +137,7 @@ preferences = [
"media.gmp-manager.updateEnabled=true", "media.gmp-manager.updateEnabled=true",
"media.eme.require-app-approval=false", "media.eme.require-app-approval=false",
] ]
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-widevine-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run"
["youtube-playback-widevine-hfr"] ["youtube-playback-widevine-hfr"]
apps = "firefox, geckoview, fenix, refbrow, chrome" apps = "firefox, geckoview, fenix, refbrow, chrome"
@ -179,7 +151,7 @@ preferences = [
"media.gmp-manager.updateEnabled=true", "media.gmp-manager.updateEnabled=true",
"media.eme.require-app-approval=false", "media.eme.require-app-approval=false",
] ]
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-hfr-test&raptor=true&exclude=1,2&muted=true&command=run" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-widevine-hfr-test&raptor=true&exclude=1,2&muted=true&command=run"
["youtube-playback-widevine-vp9-sfr"] ["youtube-playback-widevine-vp9-sfr"]
apps = "firefox, geckoview, fenix, refbrow, chrome" apps = "firefox, geckoview, fenix, refbrow, chrome"
@ -193,4 +165,4 @@ preferences = [
"media.gmp-manager.updateEnabled=true", "media.gmp-manager.updateEnabled=true",
"media.eme.require-app-approval=false", "media.eme.require-app-approval=false",
] ]
test_url = "https://yttest.prod.mozaws.net/2020/main.html?test_type=playbackperf-widevine-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run" test_url = "https://prod.youtube-test.prod.webservices.mozgcp.net/main.html?test_type=playbackperf-widevine-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run"