The MediaStreamRenderer handles the AudioOutput of an HTMLMediaElement. it register/unregister the AudioOutput according to the rendering status. The sink-change follows the logic of AudioOutput thus it has been moved in it.
The previous way created an error when the element had been muted and the sink had been changed before the source MediaStream was attached to the element. The error occured because it was possible, the same entry to be registered more than once in the AudioOutput's list, which resulted in the track to be unmuted when it should not.
Differential Revision: https://phabricator.services.mozilla.com/D82529
When aborting the current load, media element is undoubtedly inaudible. Therefore, we should reset its audible state in order to keep the audible state correct. That can help us show the sound indicator on the tab bar correctly.
Differential Revision: https://phabricator.services.mozilla.com/D81631
At the end of `HTMLMediaElement::SetSrcMediaStreamSink` a redirection of MozPromises, from `GenericPromise::AllPromiseType` to `GenericPromise` is taking place. However, this is not necessary since this is an internal method and the returned value of the promise is not useful. Thus the `GenericPromise::AllPromiseType` can be returned directly and avoid the extra thread dispatch.
Differential Revision: https://phabricator.services.mozilla.com/D80908
This patch would
- build the relationship between a media element and the media controller when media finishes loading, instead of building that after starting playing media
The advantage of doing this is
- the prework of being able to control media before media starts playing
Differential Revision: https://phabricator.services.mozilla.com/D79234
This patch would
- move the mechanism of creating a stop timer from media element to media controller
The advantage of doing this is
- to remove redundant timers in the content process, because we only need to maintain one global timer for a whole tab
Differential Revision: https://phabricator.services.mozilla.com/D79233
Before P1, GetCurrentThreadSerialEventTarget would have always returned the same data as NS_GetCurrentThread, making the comment incorrect Now it will properly return the running TaskQueue if any.
This change of name more clearly exposes what they are doing, as we aren't always dealing with threads directly; but a nsISerialEventTarget
Differential Revision: https://phabricator.services.mozilla.com/D80354
The SetSinkId method of HTMLMediaElement has been enhanced to handle the sink change when the element is sourced with a MediaObject through the srcObject attribute. It makes use in the corresponding method of AudioStreamTrack taking into account the AudioOutput and Volume that has to be removed and re-added to the right track.
Differential Revision: https://phabricator.services.mozilla.com/D77809
This patch will
- remove `MediaControlKeysEvent` and use `MediaControlKey` to replace it
- rename names for all `MediaControlKey` related methods, functions, classes and descriptions
The advantage of doing so are
- remove the duplicated type so that we only need to maintain `MediaControlKey`
Differential Revision: https://phabricator.services.mozilla.com/D78140
This patch will
- make `ContentMediaAgent` inherit from `IMediaInfoUpdater`
- move `MediaPlaybackState` and `MediaAudibleState` to `MediaPlaybackStatus.h`
The advantage of doing so are
- to force all methods which are related with updating information from content process to chrome process to be manged by `IMediaInfoUpdater`. It can help us only put the descriptive comment for methods on one place. (on `IMediaInfoUpdater`)
Differential Revision: https://phabricator.services.mozilla.com/D77196
Feeding a media element with the captured stream from the same media element does not make sense. Currently the spec does not mention anything about it. I'll clarify the case in the spec. In the meantime, when a cycle is detected, the setting of the `srcObject` is ignored and a warning is produced in the console.
Differential Revision: https://phabricator.services.mozilla.com/D76821
This patch will do :
- update the media status when media changes its owner browsing context
The advantage of doing so :
- make the media status in `ContextMediaInfo` correcly
More details :
`ContextMediaInfo` stores the media status of each browsing context, but actually the media doesn't always need to stay in one browsing context. We can move it to other browsing contexts (iframe) by appending it to other browsing context's document body.
For example, in [1], we move the video from the main frame to another iframe.
Therefore, when we move the media to a new browsing context, we should also update its media status (controlledMedia/playing/audio number) for its previous owner browsing context.
[1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/html/semantics/embedded-content/media-elements/playing-the-media-resource/pause-move-to-other-document.html
Differential Revision: https://phabricator.services.mozilla.com/D75477
This patch will do :
- remove audible check from the logic of registering controller
- include audio channel affect on the media element's audible state
The advantage of doing so :
- it can help to reduce the intermittent failure during testing by earlier hooking media elements in the content process to the media controller in the chrome process
More details :
In D72497, we have added the audible check to postpone the activation of the media controller, which would ensure that we only control media after it become audible. Therefore, we can remove the previous implementation which we use to achieve that in media element.
When having that audible check in media element, it would postpone the timing of adding media element to `ContentMediaController` that causes some intermitent failures when I was writing test for bug1633565. When removing those checks, we can ensure that the media element would have always been added into `ContentMediaController` after calling `video.play()`. If the element haven't been added into `ContentMediaController`, then it would miss to handle the media key events when test generates a fake media key event, which causes an intermitent failure.
Differential Revision: https://phabricator.services.mozilla.com/D73335
This patch will do :
- update the media status when media changes its owner browsing context
The advantage of doing so :
- make the media status in `ContextMediaInfo` correcly
More details :
`ContextMediaInfo` stores the media status of each browsing context, but actually the media doesn't always need to stay in one browsing context. We can move it to other browsing contexts (iframe) by appending it to other browsing context's document body.
For example, in [1], we move the video from the main frame to another iframe.
Therefore, when we move the media to a new browsing context, we should also update its media status (controlledMedia/playing/audio number) for its previous owner browsing context.
[1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/html/semantics/embedded-content/media-elements/playing-the-media-resource/pause-move-to-other-document.html
Differential Revision: https://phabricator.services.mozilla.com/D75477
This patch will do :
- remove audible check from the logic of registering controller
- include audio channel affect on the media element's audible state
The advantage of doing so :
- it can help to reduce the intermittent failure during testing by earlier hooking media elements in the content process to the media controller in the chrome process
More details :
In D72497, we have added the audible check to postpone the activation of the media controller, which would ensure that we only control media after it become audible. Therefore, we can remove the previous implementation which we use to achieve that in media element.
When having that audible check in media element, it would postpone the timing of adding media element to `ContentMediaController` that causes some intermitent failures when I was writing test for bug1633565. When removing those checks, we can ensure that the media element would have always been added into `ContentMediaController` after calling `video.play()`. If the element haven't been added into `ContentMediaController`, then it would miss to handle the media key events when test generates a fake media key event, which causes an intermitent failure.
Differential Revision: https://phabricator.services.mozilla.com/D73335
This patch will do :
- remove audible check from the logic of registering controller
- include audio channel affect on the media element's audible state
The advantage of doing so :
- it can help to reduce the intermittent failure during testing by earlier hooking media elements in the content process to the media controller in the chrome process
More details :
In D72497, we have added the audible check to postpone the activation of the media controller, which would ensure that we only control media after it become audible. Therefore, we can remove the previous implementation which we use to achieve that in media element.
When having that audible check in media element, it would postpone the timing of adding media element to `ContentMediaController` that causes some intermitent failures when I was writing test for bug1633565. When removing those checks, we can ensure that the media element would have always been added into `ContentMediaController` after calling `video.play()`. If the element haven't been added into `ContentMediaController`, then it would miss to handle the media key events when test generates a fake media key event, which causes an intermitent failure.
Differential Revision: https://phabricator.services.mozilla.com/D73335
The cast in InitWithNode is wrong. AsElement() asserts instead of
checking the flag, so we always pass an element (and if we didn't we'd
have type confusion problems). I audited the callers and we're fine.
Anyhow, always require an element, and add two convenience constructors
for C++ code.
Differential Revision: https://phabricator.services.mozilla.com/D73636
The cast in InitWithNode is wrong. AsElement() asserts instead of
checking the flag, so we always pass an element (and if we didn't we'd
have type confusion problems). I audited the callers and we're fine.
Anyhow, always require an element, and add two convenience constructors
for C++ code.
Differential Revision: https://phabricator.services.mozilla.com/D73636
This is in preparation for running the tail dispatcher off
nsIThreadObserver callbacks, which only work during regular
event processing.
Differential Revision: https://phabricator.services.mozilla.com/D72264
This patch will do :
- rename `ControlledMediaState` to `MediaPlaybackState`
- rename the related functions
The advantage of doing so :
- more consistent with `MediaAudibleState`
Differential Revision: https://phabricator.services.mozilla.com/D72060
This patch will do :
- replace `boolean` with enum class `MediaAudibleState`
The advantage of doing so :
- It's easier to understand what actually meaning of the parameter we set
Differential Revision: https://phabricator.services.mozilla.com/D72058
This patch will do :
- use current broswing context as a parameter when propagate state change to the chrome process.
The advantage of doing so :
- the chrome process can know which browsing context the state change actually comes from.
---
More details about this change :
Currently, when we propagate any controlled media related states, we would find the top level browsing context first, then pass it through IPC in order to get the correct media controller in the chrome process. However, we have implemented [1] which can find the correct media controller even if we are not passing the top level browsing context.
In addition, in bug1627999, we would like to know which browsing context those states come from. Therefore, we should replace the top browsing context with the current browsing context where controlled media exists.
[1] https://searchfox.org/mozilla-central/rev/41c3ea3ee8eab9ce7b82932257cb80b703cbba67/docshell/base/CanonicalBrowsingContext.cpp#511-515
Differential Revision: https://phabricator.services.mozilla.com/D72056
This patch will do :
- remove the inheritance relationship for `ContentControlKeyEventReceiver` and `ContentMediaAgent` and manually implement the methods we need
- `MediaControlEventListener` will inherit from `ContentControlKeyEventReceiver` directly
The advantage of doing so :
- increase the flexibilty of modification of `ContentMediaAgent` and those modification won't affect other classes inherited from `MediaControlKeysEventListener`
---
More details about this change :
As we would like to extend the class `ContentMediaAgent` and allow the `ContentMediaController` to call its extended method, but if we want to do so in current implementation, the extended method would also affect other classes inherited from `MediaControlKeysEventListener` and that is something we don't want to see.
Considering that, I decided to decouple the inheritance relationship and manually create the function I need (which will be implemented in the next patch)
Differential Revision: https://phabricator.services.mozilla.com/D72054
After applying this patch,
- notify `inaudible` state only if we already notified `audible` state before
- update `inaudible` state when audible media is paused
Therefore, in the media controller side, it would always see a pair of `audible/inaudible` notifications like what we do for the pair of `ControlledMediaState::ePlayed/ePaused` notifications. That can help us implement a way of maintaining a detailed counting about how many media is audible in bug1627999.
Differential Revision: https://phabricator.services.mozilla.com/D71429
There is actually possible to start the listener already while running `SuspendOrResumeElement()`, so we should remove the assertion and use a check instead.
Eg. JS can call `play()`, which would start the listener, before we run this method. This situation can be found when browsing Youtube on the GeckoView.
Differential Revision: https://phabricator.services.mozilla.com/D69673
When the docShell's `SuspendMediaWhenInactive` flag is true, no media should be allowed to start playing. Therefore, we add a check in `Play()`, `CanActivateAutoplay()` to prevent media from playing. In addition, we should also prevent the audio channel agant from starting.
Differential Revision: https://phabricator.services.mozilla.com/D69672
If docShell's `SuspendMediaWhenInactive` is true, then we should suspend or resume the media element according to the docshell active state when the docshell changes it active state.
Differential Revision: https://phabricator.services.mozilla.com/D69671
AutoEnter was an attempt around a race between AbstractThread and MessageLoopAbstractThreadWrap that would cause AbstractThread::GetCurrent() to return an incorrect value. MessageLoopAbstractThreadWrapper is no more and as such AutoEnter is no longer required.
Differential Revision: https://phabricator.services.mozilla.com/D71279
There is actually possible to start the listener already while running `SuspendOrResumeElement()`, so we should remove the assertion and use a check instead.
Eg. JS can call `play()`, which would start the listener, before we run this method. This situation can be found when browsing Youtube on the GeckoView.
Differential Revision: https://phabricator.services.mozilla.com/D69673
--HG--
extra : moz-landing-system : lando
When the docShell's `SuspendMediaWhenInactive` flag is true, no media should be allowed to start playing. Therefore, we add a check in `Play()`, `CanActivateAutoplay()` to prevent media from playing. In addition, we should also prevent the audio channel agant from starting.
Differential Revision: https://phabricator.services.mozilla.com/D69672
--HG--
extra : moz-landing-system : lando
If docShell's `SuspendMediaWhenInactive` is true, then we should suspend or resume the media element according to the docshell active state when the docshell changes it active state.
Differential Revision: https://phabricator.services.mozilla.com/D69671
--HG--
extra : moz-landing-system : lando
Video element can start the Picture-in-Picture mode **BEFORE** or **AFTER** we start the listener for the media cotrol, so we have to ensure we always propagate this information to the chrome process via `ContentMediaAgent`.
Differential Revision: https://phabricator.services.mozilla.com/D67712
--HG--
extra : moz-landing-system : lando
The media element being used in the PIP mode would always display on the screen even if the the tab it belongs to is in the background, so users would have high chance to interact with it again. Therefore, we don't want to start a stop-timer to cancel media control for that element.
Differential Revision: https://phabricator.services.mozilla.com/D67381
--HG--
extra : moz-landing-system : lando
In order to be able to change the audio output device, the device id is exposed to the constructor of MediaStreamGraph. Any component that gets/creates an MTG will be able to provide the device id in order to determine the desired output device. If the provided device id is null the default device will be used.
Differential Revision: https://phabricator.services.mozilla.com/D64133
--HG--
extra : moz-landing-system : lando
Intercepting media control keys would also come up with the virtual control interface in most of platforms, and for the notification sound we don't want either to show the interface or control them.
Currently we use 3s as a threshold to filter those short duration media which are possible to be a notification sound.
Differential Revision: https://phabricator.services.mozilla.com/D64478
--HG--
extra : moz-landing-system : lando
This patch is used to prevent intercepting media keys from other applications which should really receive media keys. For example, if user finishes watching a video but doesn't close that tab, and switch to other tasks. Then if they have other applications which are playing music, when they press media keys, the application which should receive media keys should be those background music app, not Firefox.
Therefore, if the media has been paused for a while, then that might indicate that user won't need to control this media anymore. Therefore, we create a timer, which default expire time is 60s, to stop listening media control key events for those kinds of media. But if the media starts again during this period, we would destroy timer and continue to control that media.
Differential Revision: https://phabricator.services.mozilla.com/D63918
--HG--
extra : moz-landing-system : lando
It's useless to update the audible state before starting listening to the media control keys event, so we make it consistent with other functions which could only be used after starting the listener.
As the audible change could happen before we start the listener, we have to manually update audible state to the controller after we start the listener.
Differential Revision: https://phabricator.services.mozilla.com/D63943
--HG--
extra : moz-landing-system : lando
When media is not audible yet, or it's inaudible at all (maybe it's used as a background video or GIF-like image), or the media is in the tab which has been muted totally, we don't want to use media control to control those media, so we should postpone starting listening to the media control keys for them event until they become audible.
Once they become audible, we would keep control them all the time even if they become inaudible again.
Differential Revision: https://phabricator.services.mozilla.com/D63915
--HG--
extra : moz-landing-system : lando
Those suspend and resume related logic were used for implemeting media control and audio focus on Fennec. After removing those things, the only purpose of `AudioChannelAgentCallback` would be muting and unmuting media element from audio channel (eg. clicking sound indicator to mute/unmute tab)
Differential Revision: https://phabricator.services.mozilla.com/D65264
--HG--
extra : moz-landing-system : lando
This attribute is used to expose the internal suspend state of media element to see if we set `SUSPENDED_PAUSE` and `SUSPENDED_PAUSE_DISPOSABLE` correctly. As we don't use those suspend state anymore and have removed the related tests, no need to keep this attribute anymore.
Differential Revision: https://phabricator.services.mozilla.com/D65263
--HG--
extra : moz-landing-system : lando
`media-playback-resumed` was a notification we used on Fennec to indicate media playback state changes and we should update the UI button on the virtual control interface. As it's not being used by anyone since we deprecate Fennec, we can remove all related code.
Differential Revision: https://phabricator.services.mozilla.com/D65258
--HG--
extra : moz-landing-system : lando
At the time CC unlink happens, we would no longer need to listen to the media control key events. Therefore, we should remember to stop and clear the listener to avoid its any method ran after CC.
Differential Revision: https://phabricator.services.mozilla.com/D64787
--HG--
extra : moz-landing-system : lando
This covers most cycle collected objects which support weak references, but
not the ones which inherit from a cycle collected class and don't do any cycle
collection on their own.
Differential Revision: https://phabricator.services.mozilla.com/D63962
--HG--
extra : moz-landing-system : lando
The check here was used to prevent calling play or pause during playing or pausing.
However, according to the spec [1], even if media session is playing, we would still want to trigger its play action handler when play action happens. So we should forward all media control key events to the content, not to block any of them.
[1] https://w3c.github.io/mediasession/#handle-media-session-action
Differential Revision: https://phabricator.services.mozilla.com/D60918
--HG--
extra : moz-landing-system : lando
When media element is inactive, asking MDSM to seek is actually useless, because all the data we decode won't be showed to user. In addition, we have to store extra pending events for `seeking` and `seeked`, which might result in memory overflow if the inactive page is calling `seek()` all the time.
Therfore, we should delay all seeking tasks while media is inactive, and perform the latest seeking task when media becomes active.
Differential Revision: https://phabricator.services.mozilla.com/D58918
--HG--
extra : moz-landing-system : lando
Two parameters in `SuspendOrResumeElement()` are acutally the same, they are both related with `IsActive()`, so using one parameter is enough.
In addition, we only suspend or resume element by checking element's active state, so rename `mPausedForInactiveDocumentOrChannel` to `mSuspendedForInactiveDocument` which fits the actual situation more.
Differential Revision: https://phabricator.services.mozilla.com/D58749
--HG--
extra : moz-landing-system : lando
Modify IPC method to support notify `ControlledMediaState` to the media controller, and remove the previous implementation.
Differential Revision: https://phabricator.services.mozilla.com/D57572
--HG--
extra : moz-landing-system : lando
We create `MediaControlEventListener` to register itself to the `MediaControlAgent` that is an event source, so that we can receive the media control event from the evnet source and operate media according to different types of event.
`MediaControlEventListener` is also used to notify controlled media state to the media controller. When media first starts playing, or leaves bfcache and has created listener before, we would notify `eStarted`. Notify `eStopped` when media destroys, or enter bfcache and has created listener before. When media's playing state changes, we would notifty `ePlayed` or `ePaused` depeding on media's `mPaused`.
Differential Revision: https://phabricator.services.mozilla.com/D57571
--HG--
extra : moz-landing-system : lando
This patch is used to change the blessed state for media element under `click-to-play` blocking autoplay policy, which only allows play invocation called from user input handler, eg. `onclick`.
When media aborts current load, we should regard it as a new media. One example is on Youtube they always reuse same media element but replace its url for different video. Therefore, we should reset media's blessed state so that the media with new url can be blocked again.
Differential Revision: https://phabricator.services.mozilla.com/D57192
--HG--
extra : moz-landing-system : lando
The inclusions were removed with the following very crude script and the
resulting breakage was fixed up by hand. The manual fixups did either
revert the changes done by the script, replace a generic header with a more
specific one or replace a header with a forward declaration.
find . -name "*.idl" | grep -v web-platform | grep -v third_party | while read path; do
interfaces=$(grep "^\(class\|interface\).*:.*" "$path" | cut -d' ' -f2)
if [ -n "$interfaces" ]; then
if [[ "$interfaces" == *$'\n'* ]]; then
regexp="\("
for i in $interfaces; do regexp="$regexp$i\|"; done
regexp="${regexp%%\\\|}\)"
else
regexp="$interfaces"
fi
interface=$(basename "$path")
rg -l "#include.*${interface%%.idl}.h" . | while read path2; do
hits=$(grep -v "#include.*${interface%%.idl}.h" "$path2" | grep -c "$regexp" )
if [ $hits -eq 0 ]; then
echo "Removing ${interface} from ${path2}"
grep -v "#include.*${interface%%.idl}.h" "$path2" > "$path2".tmp
mv -f "$path2".tmp "$path2"
fi
done
fi
done
Differential Revision: https://phabricator.services.mozilla.com/D55442
--HG--
extra : moz-landing-system : lando
The inclusions were removed with the following very crude script and the
resulting breakage was fixed up by hand. The manual fixups did either
revert the changes done by the script, replace a generic header with a more
specific one or replace a header with a forward declaration.
find . -name "*.idl" | grep -v web-platform | grep -v third_party | while read path; do
interfaces=$(grep "^\(class\|interface\).*:.*" "$path" | cut -d' ' -f2)
if [ -n "$interfaces" ]; then
if [[ "$interfaces" == *$'\n'* ]]; then
regexp="\("
for i in $interfaces; do regexp="$regexp$i\|"; done
regexp="${regexp%%\\\|}\)"
else
regexp="$interfaces"
fi
interface=$(basename "$path")
rg -l "#include.*${interface%%.idl}.h" . | while read path2; do
hits=$(grep -v "#include.*${interface%%.idl}.h" "$path2" | grep -c "$regexp" )
if [ $hits -eq 0 ]; then
echo "Removing ${interface} from ${path2}"
grep -v "#include.*${interface%%.idl}.h" "$path2" > "$path2".tmp
mv -f "$path2".tmp "$path2"
fi
done
fi
done
Differential Revision: https://phabricator.services.mozilla.com/D55442
--HG--
extra : moz-landing-system : lando
For media element, as the spec requires that we should know the whether it can be allowed to play when we call `media.play()` [1], which means that we should get the result before the play invocation.
Therefore, we request for the play permission when we create media element.
Differential Revision: https://phabricator.services.mozilla.com/D52438
--HG--
extra : moz-landing-system : lando
Move helper functions out from `AutoplayPolicy`, makes us clearly know which one could really be used to determine the blocking autoplay result. It also give other classes an ability to use those helper functions if they have a need.
Differential Revision: https://phabricator.services.mozilla.com/D52432
--HG--
extra : moz-landing-system : lando
For media element, as the spec requires that we should know the whether it can be allowed to play when we call `media.play()` [1], which means that we should get the result before the play invocation.
Therefore, we request for the play permission when we create media element.
Differential Revision: https://phabricator.services.mozilla.com/D52438
--HG--
extra : moz-landing-system : lando
Move helper functions out from `AutoplayPolicy`, makes us clearly know which one could really be used to determine the blocking autoplay result. It also give other classes an ability to use those helper functions if they have a need.
Differential Revision: https://phabricator.services.mozilla.com/D52432
--HG--
extra : moz-landing-system : lando
Unlink of mSrcStream used to rely on EndSrcMediaStreamPlayback to unhook
everything. That method does more than necessary however, and if anything in it
creates a strong reference to the media element, we risk a leak.
This patch takes what's necessary to unhook from EndSrcMediaStreamPlayback and
runs it explicitly from Unlink, to avoid anything unnecessary being run as well.
Differential Revision: https://phabricator.services.mozilla.com/D51906
--HG--
extra : moz-landing-system : lando
Unlink of mSrcStream used to rely on EndSrcMediaStreamPlayback to unhook
everything. That method does more than necessary however, and if anything in it
creates a strong reference to the media element, we risk a leak.
This patch takes what's necessary to unhook from EndSrcMediaStreamPlayback and
runs it explicitly from Unlink, to avoid anything unnecessary being run as well.
Differential Revision: https://phabricator.services.mozilla.com/D51906
--HG--
extra : moz-landing-system : lando
It is no longer needed with captured tracks' lifetimes spanning across seek
operations.
Differential Revision: https://phabricator.services.mozilla.com/D52051
--HG--
extra : moz-landing-system : lando
This reworks how media element captureStream works by removing the differences
between MediaStream and MediaDecoder capture. MediaDecoder capture will be
refactored so that ownership of MediaStreamTracks lies with the media element
instead of the OutputStreamManager. The internal MediaDecoder parts happen in a
later patch.
The new API for capturing a MediaDecoder involves a boolean on/off toggle, the
output tracks the decoder pipes data to, and the principal that data is tagged
with. If capturing is on but there are no output tracks, playback will not
happen, to ensure that no data gets accidentally skipped in the output tracks
while captured.
This also changes the logic for setting up MediaElementTrackSources in
HTMLMediaElement so it's triggered by the WatchManager and thus run in tail
dispatched runnables.
Differential Revision: https://phabricator.services.mozilla.com/D52040
--HG--
extra : moz-landing-system : lando
It is no longer needed with captured tracks' lifetimes spanning across seek
operations.
Differential Revision: https://phabricator.services.mozilla.com/D52051
--HG--
extra : moz-landing-system : lando
This reworks how media element captureStream works by removing the differences
between MediaStream and MediaDecoder capture. MediaDecoder capture will be
refactored so that ownership of MediaStreamTracks lies with the media element
instead of the OutputStreamManager. The internal MediaDecoder parts happen in a
later patch.
The new API for capturing a MediaDecoder involves a boolean on/off toggle, the
output tracks the decoder pipes data to, and the principal that data is tagged
with. If capturing is on but there are no output tracks, playback will not
happen, to ensure that no data gets accidentally skipped in the output tracks
while captured.
This also changes the logic for setting up MediaElementTrackSources in
HTMLMediaElement so it's triggered by the WatchManager and thus run in tail
dispatched runnables.
Differential Revision: https://phabricator.services.mozilla.com/D52040
--HG--
extra : moz-landing-system : lando
This is especially important for media elements playing MediaStreams, since the
streams can be manipulated to end playback of the elements synchronously by
either stopping or removing their tracks.
Differential Revision: https://phabricator.services.mozilla.com/D49385
--HG--
extra : moz-landing-system : lando
It started progressing as soon as we set up the rendering of the tracks in the
stream, which is too early according to the HTMLMediaElement spec.
Now it starts progressing when we're potentially playing. The difference being
that we now wait for mReadyState to go beyond HAVE_CURRENT_DATA before hooking
up the time progression mechanism.
Differential Revision: https://phabricator.services.mozilla.com/D49354
--HG--
extra : moz-landing-system : lando
This makes all inputs to IsPotentiallyPlayin() Watchable when we're playing a
MediaStream.
Differential Revision: https://phabricator.services.mozilla.com/D49353
--HG--
extra : moz-landing-system : lando
Everytime when we start a controller, we should also update its audible state correctly, in order to request audio focus correctly.
Differential Revision: https://phabricator.services.mozilla.com/D50151
--HG--
extra : moz-landing-system : lando
It can be unset by NotifyShutdown, to release the VideoFrameContainer in time.
This is unexpected for all paths assuming it will be unset by
EndSrcMediaStreamPlayback().
Depends on D49573
Differential Revision: https://phabricator.services.mozilla.com/D49574
--HG--
extra : moz-landing-system : lando
This change mainly removes the `mTracks` member from MediaStream and moves all
associated members up a level, so that a MediaStream in practice represents a
single track.
Classes will be renamed in a future patch to reflect this.
Other changes include:
The new `mEnded` member of MediaStream changes meaning to only become true when
all data in the stream has been processed. It stems from
StreamTracks::Track::mEnded which used to become true as soon as the last bit of
data had been added to a track, and there could still be data in the track that
would get processed in future iterations. We are moving towards not having any
future data in tracks, which is why this change is ok to make -- keeping the old
behavior will soon not make sense.
TrackUnionStream is changed to no longer take a list of streams as input and
forward the union of their tracks to itself. Instead it's limited to having one
track as input at a time.
The autofinishing functionality that TrackUnionStream had before has been
transformed into an autoending functionality to allow it to defer ending until
its been told that it's ok to end through the control API. This lets a single
TrackUnionStream span the lifetime of multiple inputs, which will be useful for
making DecodedStream spec compliant with HTMLMediaElement::CaptureStream(), and
for implementing the currently discussed MediaRecorder::ReplaceTrack(), to name
a few potential use cases.
AudioNodeStreams used to only have a track (and thus an AudioSegment) if the
EXTERNAL_OUTPUT flag was enabled on them. With all MediaStreams now representing
a track, AudioNodeStreams inherently have an AudioSegment as a member. It is
however only used with data if the EXTERNAL_OUTPUT flag is enabled.
Differential Revision: https://phabricator.services.mozilla.com/D45821
--HG--
extra : moz-landing-system : lando
This functionality is not used since we moved to only having a single track per
MediaStream (bug 1493613).
Differential Revision: https://phabricator.services.mozilla.com/D47686
--HG--
extra : moz-landing-system : lando
This change mainly removes the `mTracks` member from MediaStream and moves all
associated members up a level, so that a MediaStream in practice represents a
single track.
Classes will be renamed in a future patch to reflect this.
Other changes include:
The new `mEnded` member of MediaStream changes meaning to only become true when
all data in the stream has been processed. It stems from
StreamTracks::Track::mEnded which used to become true as soon as the last bit of
data had been added to a track, and there could still be data in the track that
would get processed in future iterations. We are moving towards not having any
future data in tracks, which is why this change is ok to make -- keeping the old
behavior will soon not make sense.
TrackUnionStream is changed to no longer take a list of streams as input and
forward the union of their tracks to itself. Instead it's limited to having one
track as input at a time.
The autofinishing functionality that TrackUnionStream had before has been
transformed into an autoending functionality to allow it to defer ending until
its been told that it's ok to end through the control API. This lets a single
TrackUnionStream span the lifetime of multiple inputs, which will be useful for
making DecodedStream spec compliant with HTMLMediaElement::CaptureStream(), and
for implementing the currently discussed MediaRecorder::ReplaceTrack(), to name
a few potential use cases.
AudioNodeStreams used to only have a track (and thus an AudioSegment) if the
EXTERNAL_OUTPUT flag was enabled on them. With all MediaStreams now representing
a track, AudioNodeStreams inherently have an AudioSegment as a member. It is
however only used with data if the EXTERNAL_OUTPUT flag is enabled.
Differential Revision: https://phabricator.services.mozilla.com/D45821
--HG--
extra : moz-landing-system : lando
This functionality is not used since we moved to only having a single track per
MediaStream (bug 1493613).
Differential Revision: https://phabricator.services.mozilla.com/D47686
--HG--
extra : moz-landing-system : lando
The request is used to track if the promise is resolved or not, so we should call its `Complete()` method once promise is resolved, in order to keep the state of the request correct.
Differential Revision: https://phabricator.services.mozilla.com/D47331
--HG--
extra : moz-landing-system : lando
In bug1583978, we are going to remove `AudibleState::eMaybeAudible`. But before that, as `AudibleState::eMaybeAudible` is only used for delaying media playback, so we can remove its usage from media element and move it to the dedicated class to reduce confusion about how and when we should use `AudibleState::eMaybeAudible` or `AudibleState::eNotAudible`.
Differential Revision: https://phabricator.services.mozilla.com/D47153
--HG--
extra : moz-landing-system : lando
Now `AudioChannelWrapper` would only start `AudioChannelAgent` when media starts and stop the agent when media stops. It won't handle any case with `SUSPENDED_BLOCK`.
It's clearer to know when we would start or stop the agent in AudioChannelWrapper.
In addition, in bug1577890, we will remove more unnecessary logic which is used for Fennec only in order to make `AudioChannelWrapper` cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D44747
--HG--
extra : moz-landing-system : lando
Separate the logic of delaying media playback from `HTMLMediaElement`'s `mAudioChannelWrapper` to a new class, because delaying media playback is a special usage of using `AudioChannelAgent`.
It would make the code cleaner and separate the normal usage and the special usage, to reduce confusion.
We usually start `AudioChannelAgent` when media playback starts and stop it when media playback stops, however, in this case, we would start the agent even if media hasn't be started yet, because we would like to use the agent as a connection with `AudioChannelService` in order to receive the resume command when we're able to play delayed playback.
Differential Revision: https://phabricator.services.mozilla.com/D44746
--HG--
extra : moz-landing-system : lando
As we have another usage to know whether media is audible or not, we remove this function from AudioChannelWrapper and make it as media element's function.
In addition, it also makes sense to have this function for media element, because the result is decided by media element's own attributes.
Differential Revision: https://phabricator.services.mozilla.com/D44744
--HG--
extra : moz-landing-system : lando
There are some checking conditions can be reordered or simplified in order to remove some unnecessary check.
Differential Revision: https://phabricator.services.mozilla.com/D47027
--HG--
extra : moz-landing-system : lando
After applying patch4, now we would pull the change from `AudioChannelService` everytime after starting the agent, so we don't need to rely on letting `NotifyStartedPlaying()` to modify the config and use it to update our state.
Differential Revision: https://phabricator.services.mozilla.com/D45753
--HG--
extra : moz-landing-system : lando
As we start audio capturing explicitly, we should also take responsibility to stop audio capturing when we don't need it.
We were hiding too many details on `AudioChannelAgent` before, which allow us hard to know who and where we handle audio capturing.
Differential Revision: https://phabricator.services.mozilla.com/D45752
--HG--
extra : moz-landing-system : lando
Instead of calling those callback functions seperately, we could provide a function to pull those changes at once after starting the agent.
In addition, `WindowXXXChanged` are callback functions of `nsIAudioChannelAgentCallback`, so they should only be called by `AudioChannelAgent`, to indicate receiving the change from `AudioChannelService`. We should not call them directly.
Differential Revision: https://phabricator.services.mozilla.com/D45751
--HG--
extra : moz-landing-system : lando
We have already had the same checking in `AudioCaptureStreamChange()`, so we can remove the checking in `AudioCaptureStreamChangeIfNeeded()`.
Differential Revision: https://phabricator.services.mozilla.com/D45749
--HG--
extra : moz-landing-system : lando
After applying patch4, now we would pull the change from `AudioChannelService` everytime after starting the agent, so we don't need to rely on letting `NotifyStartedPlaying()` to modify the config and use it to update our state.
Differential Revision: https://phabricator.services.mozilla.com/D45753
--HG--
extra : moz-landing-system : lando
As we start audio capturing explicitly, we should also take responsibility to stop audio capturing when we don't need it.
We were hiding too many details on `AudioChannelAgent` before, which allow us hard to know who and where we handle audio capturing.
Differential Revision: https://phabricator.services.mozilla.com/D45752
--HG--
extra : moz-landing-system : lando
Instead of calling those callback functions seperately, we could provide a function to pull those changes at once after starting the agent.
In addition, `WindowXXXChanged` are callback functions of `nsIAudioChannelAgentCallback`, so they should only be called by `AudioChannelAgent`, to indicate receiving the change from `AudioChannelService`. We should not call them directly.
Differential Revision: https://phabricator.services.mozilla.com/D45751
--HG--
extra : moz-landing-system : lando
We have already had the same checking in `AudioCaptureStreamChange()`, so we can remove the checking in `AudioCaptureStreamChangeIfNeeded()`.
Differential Revision: https://phabricator.services.mozilla.com/D45749
--HG--
extra : moz-landing-system : lando
People keep adding useless null-checks and it was not clear what the consensus
was from bug 1441165, but this should be unobjectionable I guess.
Differential Revision: https://phabricator.services.mozilla.com/D46781
--HG--
extra : moz-landing-system : lando
Tasks dispatched from RunInStableState() can be cancelled if there's a load in
between, so this is not sound.
See nsSyncSection::IsCancelled().
This is the only patch in this bug that is worth uplifting IMO.
Differential Revision: https://phabricator.services.mozilla.com/D46773
--HG--
extra : moz-landing-system : lando
