This blog post takes a look back at the various Multimedia-related tasks the Igalia Multimedia team was involved in during the first half of 2019.
GStreamer Editing Services
Thibault added support for the OpenTimelineIO open format for editorial timeline information. Having many editorial timeline information formats supported by OpenTimelineIO reduces vendor lock-in in the tools used by video artists in post-production studios. For instance a movie project edited in Final Cut Pro can now be easily reimported in the Pitivi free and open-source video editor. This accomplishment was made possible by implementing an OpenTimelineIO GES adapter and formatter, both upstreamed respectively in OpenTimelineIO and GES by Thibault.
Another important feature for non-linear video editors is nested timeline support. It allows teams to decouple big editing projects in smaller chunks that can later on be assembled for the final product. Another use-case is about pre-filling the timeline with boilerplate scenes, so that an initial version of the movie can be assembled before all teams involved in the project have provided the final content. To support this, Thibault implemented a GES demuxer which transparently enables playback support for GES files (through file://path/to/file.xges URIs) in any GStreamer-based media player.
As if this wasn’t impressive enough yet, Thibault greatly improved the GES unit-tests, fixing a lot of memory leaks, race conditions and generally improving the reliability of the test suite. This is very important because the Gitlab continuous integration now executes the tests harness for every submitted merge request.
For more information about this, the curious readers can dive in Thibault’s blog post. Thibault was invited to talk about these on-going efforts at SIGGRAPH during the OpenTimelineIO BOF.
Finally, Thibault is mentoring Swayamjeet Swain as part of the GSoC program, the project is about adding nested timeline support in Pitivi.
GStreamer VA-API
Víctor performed a good number of code reviews for GStreamer-VAAPI contributors, he has also started investigating GStreamer-VAAPI bugs specific to the AMDGPU Gallium driver, in order to improve the support of AMD hardware in multimedia applications.
As part of the on-going GStreamer community efforts to improve continuous integration (CI) in the project, we purchased Intel and AMD powered devices aimed to run validation tests. Running CI on real end-user hardware will help ensure regressions remain under control.
Servo and GStreamer-rs
As part of our on-going involvement in the Servo Mozilla project, Víctor has enabled zero-copy video rendering support in the GStreamer-based Servo-media crate, along with bug fixes in Glutin and finally in Servo itself.
A prior requirement for this major milestone was to enable gst-gl in the GStreamer Rust bindings and after around 20 patches were merged in the repository, we are pleased to announce Linux and Android platforms are supported. Windows and macOS platforms will also be supported, soon.
The following screencast shows how hardware-accelerated video rendering performs on Víctor’s laptop:
Víctor also delivered a talk at the MadRust meetup, about GStreamer-rs, the slides are available on his website.
WebKit’s Media Source Extensions
As part of our on-going collaboration with various device manufacturers, Alicia and Enrique validated the Youtube TV MSE 2019 test suite in WPEWebKit-based products.
Alicia developed a new GstValidate plugin to improve GStreamer pipelines testing, called validateflow. Make sure to read her blog post about it!.
In her quest to further improve MSE support, especially seek support, Alicia rewrote the GStreamer source element we use in WebKit for MSE playback. The code review is on-going in Bugzilla and on track for inclusion in WPEWebKit and WebKitGTK 2.26. This new design of the MSE source element requires the playbin3 GStreamer element and at least GStreamer 1.16. Another feature we plan to work on in the near future is multi-track support; stay tuned!
WebKit WebRTC
We announced LibWebRTC support for WPEWebKit and WebKitGTK one year ago. Since then, Thibault has been implementing new features and fixing bugs in the backend. Bridging between the WebAudio and WebRTC backend was implemented, allowing tight integration of WebAudio and WebRTC web apps. More WebKit WebRTC layout tests were unskipped in the buildbots, allowing better tracking of regressions.
Thibault also fixed various performance issues on Raspberry Pi platforms during apprtc video-calls. As part of this effort he upstreamed a GstUVCH264DeviceProvider in GStreamer, allowing applications to use already-encoded H264 streams from webcams that provide it, thus removing the need for applications to encode raw video streams.
Additionally, Thibault upstreamed a device provider for ALSA in GStreamer, allowing applications to probe for Microphones and speakers supported through the ALSA kernel driver.
Finally, Thibault tweaked the GStreamer encoders used by the WebKit LibWebRTC backend, in order to match the behavior of the Apple implementation and also fixing a few performance and rendering issues on the way.
WebKit GStreamer Multimedia maintenance
The whole team is always keeping an eye on WebKit’s Bugzilla, watching out for multimedia-related bugs reported by the community members. Charlie recently fixed a few annoying volume bugs along with an issue related with youtube.
I rewrote the WebKitWebSrc GStreamer source element we use in WebKit to download HTTP(S) media resources and feed the data to the playback pipeline. This new element is now based on the GStreamer pushsrc base class, instead of appsrc. A few seek-related issues were fixed on the way but unfortunately some regressions also slipped in; those should all be fixed by now and shipped in WPEWebKit/WebKitGTK 2.26.
An initial version of the MediaCapabilities backend was also upstreamed in WebKit. It allows web-apps (such as Youtube TV) to probe the user-agent for media decoding and encoding capabilities. The GStreamer backend relies on the GStreamer plugin registry to provide accurate information about the supported codecs and containers. H264 AVC1 profile and level information are also probed.
WPEQt
Well, this isn’t directly related with multimedia, but I finally announced the initial release of WPEQt. Any feedback is welcome, QtWebKit has phased out and if anyone out there relies on it for web-apps embedded in native QML apps, now is the time to try WPEQt!