After a year or two of hiatus I attended the GStreamer conference which happened in beautiful Edinburgh. It was great to meet the friends from the community again and learn about what’s going on in the multimedia world. The quality of the talks was great, the videos are published online as usual in Ubicast. I delivered a talk about the Multimedia support in WPEWebKit, you can watch it there and the slides are also available.
One of the many interesting presentations was about GStreamer for cloud-based live video. Usually anything with the word cloud would tend to draw my attention away but for some reason I attended this presentation, and didn’t regret it! The last demo presented by the BBC folks was about overlaying Web content on native video streams. It’s an interesting use-case for live TV broadcasting for instance. A web page provides dynamic notifications popping up and down, the web page is rendered with a transparent background and blended over the live video stream. The BBC folks implemented a GStreamer source element relying on CEF for their Brave project.
So here you wonder, why am I talking about Chromium Embedded Framework (CEF)? Isn’t this post about WPEWebKit? After seeing the demo from the Brave developers I immediately thought WPE could be a great fit for this HTML overlay use-case too! So a few weeks after the conference I finally had the time to start working on the WPE GStreamer plugin. My colleague Žan Doberšek, WPE’s founder hacker, provided a nice solution for the initial rendering issues of the prototype, many thanks to him!
Here’s a first example, a basic web-browser with gst-play:
$ gst-play-1.0 --videosink gtkglsink wpe://https://gnome.org
A GTK window opens up and the GNOME homepage should load. You can click on links too! To overlay a web page on top of a video you can use a pipeline like this one:
$ gst-launch-1.0 glvideomixer name=m sink_1::zorder=0 sink_0::height=818 sink_0::width=1920 ! gtkglsink \ wpesrc location="file:///home/phil/Downloads/plunk/index.html" draw-background=0 ! m. \ uridecodebin uri="http://192.168.1.44/Sintel.2010.1080p.mkv" name=d d. ! queue ! glupload \ ! glcolorconvert ! m.
which can be represented with this simplified graph:
The advantage of this approach is that many heavy-lifting tasks happen in the GPU. WPE loads the page using its WPENetworkProcess external process, parses everything (DOM, CSS, JS, …) and renders it as a EGLImage, shared with the UIProcess (the GStreamer application, gst-launch in this case). In most situations decodebin will use an hardware decoder. The decoded video frames are uploaded to the GPU and composited with the EGLImages representing the web-page, in a single OpenGL scene, using the glvideomixer element.
The initial version of the GstWPE plugin is now part of the gst-plugins-bad staging area, where most new plugins are uploaded for further improvements later on. Speaking of improvements, the following tasks have been identified:
- The wpesrc draw-background property is not yet operational due to missing WPEWebKit API for background-color configuration support. I expect to complete this task very soon, interested people can follow this bugzilla ticket
- Audio support, WPEWebKit currently provides only EGLImages to application side. The audio session is rendered directly to GStreamer’s autoaudiosink in WebKit, so there’s currently no audio sharing support in wpesrc.
- DMABuf support as an alternative to EGLImages. WPEWebKit internally leverages linux-dmabuf support already but doesn’t expose the file descriptors and plane informations.
- Better navigation events support. GStreamer’s navigation events API was initially designed for DVD menus navigation uses-cases mostly, the exposed input events informations are not a perfect match for WPEWebKit which expects hardware-level informations from keyboard, mouse and touch devices.
There are more ways and use-cases related with WPE, I expect to unveil another WPE embedding project very soon. Watch this space! As usual many thanks to my Igalia colleagues for sponsoring this work. We are always happy to hear what others are doing with WPE and to help improving it, don’t hesitate to get in touch!