Base-Art - Philippe Normand

From WebKit/GStreamer to rust-av, a journey on our stack’s layers

2024-04-16T22:15:00+02:00

In this post I’ll try to document the journey starting from a WebKit issue and ending up improving third-party projects that WebKitGTK and WPEWebKit depend on.

I’ve been working on WebKit’s GStreamer backends for a while. Usually some new feature needed on WebKit side would trigger work on GStreamer. That’s quite common and healthy actually, by improving GStreamer (bug fixes or implementing new features) we make the whole stack stronger (hopefully). It’s not hard to imagine other web-engines, such as Servo for instance, leveraging fixes made in GStreamer in the context of WebKit use-cases.

Sometimes though we have to go deeper and this is what this post is about!

Since version 2.44, WebKitGTK and WPEWebKit ship with a WebCodecs backend. That backend leverages the wide range of GStreamer audio and video decoders/encoders to give low-level access to encoded (or decoded) audio/video frames to Web developers. I delivered a lightning talk at gst-conf 2023 about this topic.

There are still some issues to fix regarding performance and some W3C web platform tests are still failing. The AV1 decoding tests were flagged early on while I was working on WebCodecs, I didn’t have time back then to investigate the failures further, but a couple weeks ago I went back to those specific issues.

The WebKit layout tests harness is executed by various post-commit bots, on various platforms. The WebKitGTK and WPEWebKit bots run on Linux. The WebCodec tests for AV1 currently make use of the GStreamer av1enc and dav1ddec elements. We currently don’t run the tests using the modern and hardware-accelerated vaav1enc and vaav1dec elements because the bots don’t have compatible GPUs.

The decoding tests were failing, this one for instance (the ?av1 variant). In that test both encoding and decoding are tested, but decoding was failing, for a couple reasons. Rabbit hole starts here. After debugging this for a while, it was clear that the colorspace information was lost between the encoded chunks and the decoded frames. The decoded video frames didn’t have the expected colorimetry values.

The VideoDecoderGStreamer class basically takes encoded chunks and notifies decoded VideoFrameGStreamer objects to the upper layers (JS) in WebCore. A video frame is basically a GstSample (Buffer and Caps) and we have code in place to interpret the colorimetry parameters exposed in the sample caps and translate those to the various WebCore equivalents. So far so good, but the caps set on the dav1ddec elements didn’t have those informations! I thought the dav1ddec element could be fixed, “shouldn’t be that hard” and I knew that code because I wrote it in 2018 :)

So let’s fix the GStreamer dav1ddec element. It’s a video decoder written in Rust, relying on the dav1d-rs bindings of the popular C libdav1d library. The dav1ddec element basically feeds encoded chunks of data to dav1d using the dav1d-rs bindings. In return, the bindings provide the decoded frames using a Dav1dPicture Rust structure and the dav1ddec GStreamer element basically makes buffers and caps out of this decoded picture. The dav1d-rs bindings are quite minimal, we implemented API on a per-need basis so far, so it wasn’t very surprising that… colorimetry information for decoded pictures was not exposed! Rabbit hole goes one level deeper.

So let’s add colorimetry API in dav1d-rs. When working on (Rust) bindings of a C library, if you need to expose additional API the answer is quite often in the C headers of the library. Every Dav1dPicture has a Dav1dSequenceHeader, in which we can see a few interesting fields:

typedef struct Dav1dSequenceHeader {
...
    enum Dav1dColorPrimaries pri; ///< color primaries (av1)
    enum Dav1dTransferCharacteristics trc; ///< transfer characteristics (av1)
    enum Dav1dMatrixCoefficients mtrx; ///< matrix coefficients (av1)
    enum Dav1dChromaSamplePosition chr; ///< chroma sample position (av1)
    ...
    uint8_t color_range;
    ...
...
} Dav1dSequenceHeader;

After sharing a naive branch with rust-av co-maintainers Luca Barbato and Sebastian Dröge, I came up with a couple pull-requests that eventually were shipped in version 0.10.3 of dav1d-rs. I won’t deny matching primaries, transfer, matrix and chroma-site enum values to rust-av‘s Pixel enum was a bit challenging :P Anyway, with dav1d-rs fixed up, rabbit hole level goes up one level :)

Now with the needed dav1d-rs API, the GStreamer dav1ddec element could be fixed. Again, matching the various enum values to their GStreamer equivalent was an interesting exercise. The merge request was merged, but to this date it’s not shipped in a stable gst-plugins-rs release yet. There’s one more complication here, ABI broke between dav1d 1.2 and 1.4 versions. The dav1d-rs 0.10.3 release expects the latter. I’m not sure how we will cope with that in terms of gst-plugins-rs release versioning…

Anyway, WebKit’s runtime environment can be adapted to ship dav1d 1.4 and development version of the dav1ddec element, which is what was done in this pull request. The rabbit is getting out of his hole.

The WebCodec AV1 tests were finally fixed in WebKit, by this pull request. Beyond colorimetry handling a few more fixes were needed, but luckily those didn’t require any fixes outside of WebKit.

Wrapping up, if you’re still reading this post, I thank you for your patience. Working on inter-connected projects can look a bit daunting at times, but eventually the whole ecosystem benefits from cross-project collaborations like this one. Thanks to Luca and Sebastian for the help and reviews in dav1d-rs and the dav1ddec element. Thanks to my fellow Igalia colleagues for the WebKit reviews.

GNOME Web Canary is back

2023-07-08T10:00:00+02:00

This is a short PSA post announcing the return of the GNOME Web Canary builds. Read on for the crunchy details.

A couple years ago I was blogging about the GNOME Web Canary flavor. In summary this special build of GNOME Web provides a preview of the upcoming version of the underlying WebKitGTK engine, it is potentially unstable, but allows for testing features that have not shipped in a stable release yet.

Unfortunately, Canary broke right after GNOME Web switched to GTK4, because back then the WebKit CI was missing build bots and infrastructure for hosting WebKitGTK4 build artefacts. Recently, thanks to the efforts of my Igalia colleagues, Pablo Abelenda, Lauro Moura, Diego Pino and Carlos López the WebKit CI provides WebKitGTK4 build artefacts, hosted on a server kindly provided by Igalia.

The installation instructions are already mentioned in the introductory post but I’ll just remind them again here:

flatpak --user remote-add --if-not-exists webkit https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
flatpak --user install https://nightly.gnome.org/repo/appstream/org.gnome.Epiphany.Canary.flatpakref

Update:

If you installed the older version of Canary, pre-GTK4, you might see an error related with an expired GPG key. This is due to how I update the WebKit runtime, and I’ll try to avoid it in future updates. For the time being, you can remove the flatpak remote and re-add it:

flatpak --user remote-delete webkit
flatpak --user remote-add webkit https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo

That’s all folks, happy hacking and happy testing.

WebRTC in WebKitGTK and WPE, status updates, part I

2023-02-16T21:30:00+01:00

Some time ago we at Igalia embarked on the journey to ship a GStreamer-powered WebRTC backend. This is a long journey, it is not over, but we made some progress. This post is the first of a series providing some insights of the challenges we are facing and the plans for the next release cycle(s).

Most web-engines nowadays bundle a version of LibWebRTC, it is indeed a pragmatic approach. WebRTC is a huge spec, spanning across many protocols, RFCs and codecs. LibWebRTC is in fact a multimedia framework on its own, and it’s a very big code-base. I still remember the suprised face of Emilio at the 2022 WebEngines conference when I told him we had unusual plans regarding WebRTC support in GStreamer WebKit ports. There are several reasons for this plan, explained in the WPE FAQ. We worked on a LibWebRTC backend for the WebKit GStreamer ports, my colleague Thibault Saunier blogged about it but unfortunately this backend has remained disabled by default and not shipped in the tarballs, for the reasons explained in the WPE FAQ.

The GStreamer project nowadays provides a library and a plugin allowing applications to interact with third-party WebRTC actors. This is in my opinion a paradigm shift, because it enables new ways of interoperability between the so-called Web and traditional native applications. Since the GstWebRTC announcement back in late 2017 I’ve been experimenting with the idea of shipping an alternative to LibWebRTC in WebKitGTK and WPE. The initial GstWebRTC WebKit backend was merged upstream on March 18, 2022.

As you might already know, before any audio/video call, your browser might ask permission to access your webcam/microphone and even during the call you can now share your screen. From the WebKitGTK/WPE perspective the procedure is the same of course. Let’s dive in.

WebCam/Microphone capture

Back in 2018 for the LibWebRTC backend, Thibault added support for GStreamer-powered media capture to WebKit, meaning that capture devices such as microphones and webcams would be accessible from WebKit applications using the getUserMedia spec. Under the hood, a GStreamer source element is created, using the GstDevice API. This implementation is now re-used for the GstWebRTC backend, it works fine, still has room for improvements but that’s a topic for a follow-up post.

A MediaStream can be rendered in a <audio> or <video> element, through a custom GStreamer source element that we also provide in WebKit, this is all internally wired up so that the following JS code will trigger the WebView in natively capturing and rendering a WebCam device using GStreamer:

<html>
  <head>
    <script>
    navigator.mediaDevices.getUserMedia({video: true, audio: false }).then((mediaStream) => {
        const video = document.querySelector('video');
        video.srcObject = mediaStream;
        video.onloadedmetadata = () => {
            video.play();
        };
    });
    </script>
  </head>
  <body>
    <video/>
  </body>
</html>

When this WebPage is rendered and after the user has granted access to capture devices, the GStreamer backends will create not one, but two pipelines.

flowchart LR
    pipewiresrc-->videoscale-->videoconvert-->videorate-->valve-->appsink

flowchart LR
    subgraph mediastreamsrc
    appsrc-->srcghost[src]
    end
    subgraph playbin3
       subgraph decodebin3
       end
       subgraph webkitglsink
       end
       decodebin3-->webkitglsink
    end
    srcghost-->decodebin3

The first pipeline routes video frames from the capture device using pipewiresrc to an appsink. From the appsink our capturer leveraging the Observer design pattern notifies its observers. In this case there is only one observer which is a GStreamer source element internal to WebKit called mediastreamsrc. The playback pipeline shown above is heavily simplified, in reality more elements are involved, but what matters most is that thanks to the flexibility of Gstreamer, we can leverage the existing MediaPlayer backend that we at Igalia have been maintaining for more than 10 years, to render MediaStreams. All we needed was a custom source element, the rest of our MediaPlayer didn’t need much changes to support this use-case.

One notable change we did since the initial implementation though is that for us a MediaStream can be either raw, encoded or even encapsulated in a RTP payload. So depending on which component is going to render the MediaStream, we have enough flexibility to allow zero-copy, in most scenarios. In the example above, typically the stream will be raw from source to renderer. However, some webcams can provide encoded streams. WPE and WebKitGTK will be able to internally leverage these and in some cases allow for direct streaming from hardware device to outgoing PeerConnection without third-party encoding.

Desktop capture

There is another JS API, allowing to capture from your screen or a window, called getDisplayMedia, and yes, we also support it! Thanks to these recent years ground-breaking progress of the Linux Desktop such as PipeWire and xdg-desktop-portal we can now stream your favorite desktop environment over WebRTC. Under the hood when the WebView is granted access to the desktop capture through the portal, our backend creates a pipewiresrc GStreamer element, configured to source from the file descriptor provided by the portal, and we have a healthy raw video stream.

Here’s a demo:

WebAudio capture

What more, yes you can also create a MediaStream from a WebAudio node. On the backend side, the GStreamerMediaStreamAudioSource fills GstBuffers from the audio bus channels and notifies third-parties internally observing the MediaStream, such as outgoing media sources, or simply an <audio> element that was configured to source from the given MediaStream. I have no demo for this, you’ll have to take my word.

Canvas capture

But wait there is more. Did I hear canvas? Yes we can feed your favorite <canvas> to a MediaStream. The JS API is called captureStream, its code is actually cross-platform but defers to the HTMLCanvasElement::toVideoFrame() method which has a GStreamer implementation. The code is not the most optimal yet though due to shortcomings of our current graphics pipeline implementation. Here is a demo of Canvas to WebRTC running in the WebKitGTK MiniBrowser:

Wrap-up

So we’ve got MediaStream support covered. This is only one part of the puzzle though. We are facing challenges now on the PeerConnection implementation. MediaStreams are cool but it’s even better when you can share them with your friends on the fancy A/V conferencing websites, but we’re not entirely ready for this yet in WebKitGTK and WPE. For this reason, WebRTC is not yet enabled by default in the upcoming WebKitGTK and WPE 2.40 releases. We’re just not there yet. In the next part of these series I’ll tackle the PeerConnection backend on which we’re working hard on these days, both in WebKit and in GStreamer.

Happy hacking and as always, all my gratitude goes to my fellow Igalia comrades for allowing me to keep working on these domains and to Metrological for funding some of this work. Is your organization or company interested in leveraging modern WebRTC APIs from WebKitGTK and/or WPE? If so please get in touch with us in order to help us speed-up the implementation work.

Introducing the GNOME Web Canary flavor

2021-08-02T14:00:00+02:00

Today I am happy to unveil GNOME Web Canary which aims to provide bleeding edge, most likely very unstable builds of Epiphany, depending on daily builds of the WebKitGTK development version. Read on to know more about this.

Until recently the GNOME Web browser was available for end-users in two flavors. The primary, stable release provides the vanilla experience of the upstream Web browser. It is shipped as part of the GNOME release cycle and in distros. The second flavor, called Tech Preview, is oriented towards early testers of GNOME Web. It is available as a Flatpak, included in the GNOME nightly repo. The builds represent the current state of the GNOME Web master branch, the WebKitGTK version it links to is the one provided by the GNOME nightly runtime.

Tech Preview is great for users testing the latest development of GNOME Web, but what if you want to test features that are not yet shipped in any WebKitGTK version? Or what if you are GNOME Web developer and you want to implement new features on Web that depend on API that was not released yet in WebKitGTK?

Historically, the answer was simply “you can build WebKitGTK yourself“. However, this requires some knowledge and a good build machine (or a lot of patience). Even as WebKit developer builds have become easier to produce thanks to the Flatpak SDK we provide, you would still need to somehow make Epiphany detect your local build of WebKit. Other browsers offer nightly or “Canary” builds which don’t have such requirements. This is exactly what Epiphany Canary aims to do! Without building WebKit yourself!

A brief interlude about the term: Canary typically refers to highly unstable builds of a project, they are named after Sentinel species. Canary birds were taken into mines to warn coal miners of carbon monoxide presence. For instance Chrome has been providing Canary builds of its browser for a long time. These builds are useful because they allow early testing, by end-users. Hence potentially early detection of bugs that might not have been detected by the usual automated test harness that buildbots and CI systems run.

To similar ends, a new build profile and icon were added in Epiphany, along with a new Flatpak manifest. Everything is now nicely integrated in the Epiphany project CI. WebKit builds are already done for every upstream commit using the WebKit Buildbot. As those builds are made with the WebKit Flatpak SDK, they can be reused elsewhere (x86_64 is the only arch supported for now) as long as the WebKit Flatpak platform runtime is being used as well. Build artifacts are saved, compressed, and uploaded to a web server kindly hosted and provided by Igalia. The GNOME Web CI now has a new job, called canary, that generates a build manifest that installs WebKitGTK build artifacts in the build sandbox, that can be detected during the Epiphany Flatpak build. The resulting Flatpak bundle can be downloaded and locally installed. The runtime environment is the one provided by the WebKit SDK though, so not exactly the same as the one provided by GNOME Nightly.

Back to the two main use-cases, and who would want to use this:

You are a GNOME Web developer looking for CI coverage of some shiny new WebKitGTK API you want to use from GNOME Web. Every new merge request on the GNOME Web Gitlab repo now produces installable Canary bundles, that can be used to test the code changes being submitted for review. This bundle is not automatically updated though, it’s good only for one-off testing.
You are an early tester of GNOME Web, looking for bleeding edge version of both GNOME Web and WebKitGTK. You can install Canary using the provided Flatpakref. Every commit on the GNOME Web master branch produces an update of Canary, that users can get through the usual flatpak update or through their flatpak-enabled app-store.

Update:

Due to an issue in the Flatpakref file, the WebKit SDK flatpak remote is not automatically added during the installation of GNOME Web Canary. So it needs to be manually added before attempting to install the flatpakref:

$ flatpak --user remote-add --if-not-exists webkit https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
$ flatpak --user install https://nightly.gnome.org/repo/appstream/org.gnome.Epiphany.Canary.flatpakref

As you can see in the screenshot below, the GNOME Web branding is clearly modified compared to the other flavors of the application. The updated logo, kindly provided by Tobias Bernard, has some yellow tones and the Tech Preview stripes. Also the careful reader will notice the reported WebKitGTK version in the screenshot is a development build of SVN revision r280382. Users are strongly advised to add this information to bug reports.

As WebKit developers we are always interested in getting users’ feedback. I hope this new flavor of GNOME Web will be useful for both GNOME and WebKitGTK communities. Many thanks to Igalia for sponsoring WebKitGTK build artifacts hosting and some of the work time I spent on this side project. Also thanks to Michael Catanzaro, Alexander Mikhaylenko and Jordan Petridis for the reviews in Gitlab.

Catching up on WebKit GStreamer WebAudio backends maintenance

2020-11-29T13:45:00+01:00

Over the past few months the WebKit development team has been working on modernizing support for the WebAudio specification. This post highlights some of the changes that were recently merged, focusing on the GStreamer ports.

My fellow WebKit colleague, Chris Dumez, has been very active lately, updating the WebAudio implementation for the mac ports in order to comply with the latest changes of the specification. His contributions have been documented in the Safari Technology Preview release notes for version 113, version 114, version 115 and version 116. This is great for the WebKit project! Since the initial implementation landed around 2011, there wasn’t much activity and over the years our implementation started lagging behind other web engines in terms of features and spec compliance. So, many thanks Chris, I think you’re making a lot of WebAudio web developers very happy these days :)

The flip side of the coin is that some of these changes broke the GStreamer backends, as Chris is focusing mostly on the Apple ports, a few bugs slipped in, noticed by the CI test bots and dutifully gardened by our bots sheriffs. Those backends were upstreamed in 2012 and since then I didn’t devote much time to their maintenance, aside from casual bug-fixing.

One of the WebAudio features recently supported by WebKit is the Audio Worklet interface which allows applications to perform audio processing in a dedicated thread, thus relieving some pressure off the main thread and ensuring a glitch-free WebAudio rendering. I added support for this feature in r268579. Folks eager to test this can try the GTK nightly MiniBrowser with the demos:

$ wget https://trac.webkit.org/export/270226/webkit/trunk/Tools/Scripts/webkit-flatpak-run-nightly
$ chmod +x webkit-flatpak-run-nightly
$ python3 webkit-flatpak-run-nightly --gtk MiniBrowser https://googlechromelabs.github.io/web-audio-samples/audio-worklet/

For many years our AudioFileReader implementation was limited to mono and stereo audio layouts. This limitation was lifted off in r269104 allowing for processing of up to 5.1 surround audio files in the AudioBufferSourceNode.

Our AudioDestination, used for audio playback, was only able to render stereo. It is now able to probe the GStreamer platform audio sink for the maximum number of channels it can handle, since r268727. Support for AudioContext getOutputTimestamp was hooked up in the GStreamer backend in r266109.

The WebAudio spec has a MediaStreamAudioDestinationNode for MediaStreams, allowing to feed audio samples coming from the WebAudio pipeline to outgoing WebRTC streams. Since r269827 the GStreamer ports now support this feature as well! Similarly, incoming WebRTC streams or capture devices can stream their audio samples to a WebAudio pipeline, this has been supported for a couple years already, contributed by my colleague Thibault Saunier.

Our GStreamer FFTFrame implementation was broken for a few weeks, while Chris was landing various improvements for the platform-agnostic and mac-specific implementations. I finally fixed it in r267471.

This is only the tip of the iceberg. A few more patches were merged, including some security-related bug-fixes. As the Web Platform keeps growing, supporting more and more multimedia-related use-cases, we, at the Igalia Multimedia team, are committed to maintain our position as GStreamer experts in the WebKit community.

Web-augmented graphics overlay broadcasting with WPE and GStreamer

2020-07-02T15:00:00+02:00

Graphics overlays are everywhere nowadays in the live video broadcasting industry. In this post I introduce a new demo relying on GStreamer and WPEWebKit to deliver low-latency web-augmented video broadcasts.

Readers of this blog might remember a few posts about WPEWebKit and a GStreamer element we at Igalia worked on. In december 2018 I introduced GstWPE and a few months later blogged about a proof-of-concept application I wrote for it. So, learning from this first iteration, I wrote another demo!

The first demo was already quite cool, but had a few down-sides:

It works only on desktop (running in a Wayland compositor). The Wayland compositor dependency can be a burden in some cases. Ideally we could imaginge GstWPE applications running “in the cloud”, on machines without GPU, bare metal.
While it was cool to stream to Twitch, Youtube and the like, these platforms currently can ingest only RTMP streams. That means the latency introduced can be quite significant, depending on the network conditions of course, but even in ideal conditions the latency was between one and 2 seconds. This is not great, in the world we live in.

To address the first point, WPE founding engineer, Žan Doberšek enabled software rasterizing support in WPE and its FDO backend. This is great because it allows WPE to run on machines without GPU (like continuous integration builders, test bots) but also “in the cloud” where machines with GPU are less affordable than bare metal! Following up, I enabled this feature in GstWPE. The source element caps template now has video/x-raw, in addition to video/x-raw(memory:GLMemory). To force swrast, you need to set the LIBGL_ALWAYS_SOFTWARE=true environment variable. The downside of swrast is that you need a good CPU. Of course it depends on the video resolution and framerate you want to target.

On the latency front, I decided to switch from RTMP to WebRTC! This W3C spec isn’t only about video chat! With WebRTC, sub-second live one-to-many broadcasting can be achieved, without much efforts, given you have a good SFU. For this demo I chose Janus, because its APIs are well documented, and it’s a cool project! I’m not sure it would scale very well in large deployments, but for my modest use-case, it fits very well.

Janus has a plugin called video-room which allows multiple participants to chat. But then imagine a participant only publishing its video stream and multiple “clients” connecting to that room, without sharing any video or audio stream, one-to-many broadcasting. As it turns out, GStreamer applications can already connect to this video-room plugin using GstWebRTC! A demo was developed by tobiasfriden and saket424 in Python, it recently moved to the gst-examples repository. As I kind of prefer to use Rust nowadays (whenever I can anyway) I ported this demo to Rust, it was upstreamed in gst-examples as well. This specific demo streams the video test pattern to a Janus instance.

Adapting this Janus demo was then quite trivial. By relying on a similar video mixer approach I used for the first GstWPE demo, I had a GstWPE-powered WebView streaming to Janus.

The next step was the actual graphics overlays infrastructure. In the first GstWPE demo I had a basic GTK UI allowing to edit the overlays on-the-fly. This can’t be used for this new demo, because I wanted to use it headless. After doing some research I found a really nice NodeJS app on Github, it was developed by Luke Moscrop, who’s actually one of the main developers of the Brave BBC project. The Roses CasparCG Graphics was developed in the context of the Lancaster University Students’ Union TV Station, this app starts a web-server on port 3000 with two main entry points:

An admin web-UI (in /admin/ allowing to create and manage overlays, like sports score boards, info banners, and so on.
The target overlay page (in the root location of the server), which is a web-page without predetermined background, displaying the overlays with HTML, CSS and JS. This web-page is meant to be fed to CasparCG (or GstWPE :))

After making a few tweaks in this NodeJS app, I can now:

Start the NodeJS app, load the admin UI in a browser and enable some overlays
Start my native Rust GStreamer/WPE application, which:
- connects to the overlay web-server
- mixes a live video source (webcam for instances) with the WPE-powered overlay
- encodes the video stream to H.264, VP8 or VP9
- sends the encoded RTP stream using WebRTC to a Janus server
Let “consumer” clients connect to Janus with their browser, in order to see the resulting live broadcast.

(If the video doesn’t display, here is the Youtube link.)

This is pretty cool and fun, as my colleague Brian Kardell mentions in the video. Working on this new version gave me more ideas for the next one. And very recently the audio rendering protocol was merged in WPEBackend-FDO! That means even more use-cases are now unlocked for GstWPE.

This demo’s source code is hosted on Github. Feel free to open issues there, I am always interested in getting feedback, good or bad!

GstWPE is maintained upstream in GStreamer and relies heavily on WPEWebKit and its FDO backend. Don’t hesitate to contact us if you have specific requirements or issues with these projects :)

Setting up Debian containers on Fedora Silverblue

2020-06-13T13:50:00+02:00

After almost 20 years using Debian, I am trying something different, Fedora Silverblue. However for work I still need to use Debian/Ubuntu from time to time. In this post I am explaining the steps to setup Debian containers on Silverblue.

By default Silverblue comes with Toolbox which perfectly integrates the OS. It’s a shell script which actually relies on Podman, an alternative to Docker. And it works really well with Fedora images! However I ran into various issues when I wanted to setup Debian containers pulled from docker.io:

$ toolbox create -c sid --image docker.io/debian:sid
Image required to create toolbox container.
Download docker.io/debian:sid (500MB)? [y/N]: y
Created container: sid
Enter with: toolbox enter --container sid
$ toolbox enter -c sid
toolbox: failed to start container sid

This should work, but doesn’t, because Toolbox expects specific Image requirements. After some digging into alternate Toolbox implementations and quite a bit of experimentation, I found a Toolbox pull-request adding support for Debian containers. Unfortunately this pull-request wasn’t merged yet, perhaps because the Toolbox developers are busy in the rewrite of Toolbox in Go. Scrolling down the comments, Martin Pitt provides some details and links to the approach he’s taken to achieve the same goal. Trying to follow his instructions, I finally managed to have working Debian containers in Silverblue. Many thanks to him! Here’s the run-down

Download the build-debian-toolbox script:

$ wget https://piware.de/gitweb/?p=bin.git;a=blob_plain;f=build-debian-toolbox;hb=HEAD
$ mkdir -p ~/bin
$ mv build-debian-toolbox ~/bin
$ chmod +x ~/bin/build-debian-toolbox

Modify it a little. Here I had to:
- change the shell shebang to /bin/bash
- Update toolbox call sites to ~/bin/toolbox
Make sure ~/bin/ is first in your shell $PATH
Download the patched toolbox script from this pull-request and:
- Move it to ~/bin/
- Again, switch to the /bin/bash/ shebang in that script

Then run build-debian-toolbox. By default it will create a Debian Sid container, unless you provide override as command-line arguments, so for instance to create an Ubuntu bionic container you can:

$ build-debian-toolbox bionic ubuntu
...
$ toolbox run -c bionic cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

That’s it! Hopefully the Toolbox Go rewrite will support this out of the box without requiring third-party tweaks. Thanks again to Martin for his work on this topic.

Web Engines Hackfest 2014

2020-06-13T12:05:12+02:00

Last week I attended the Web Engines Hackfest. The event was sponsored by Igalia (also hosting the event), Adobe and Collabora.

As usual I spent most of the time working on the WebKitGTK+ GStreamer backend and Sebastian Dröge kindly joined and helped out quite a bit, make sure to read his post about the event!

We first worked on the WebAudio GStreamer backend, Sebastian cleaned up various parts of the code, including the playback pipeline and the source element we use to bridge the WebCore AudioBus with the playback pipeline. On my side I finished the AudioSourceProvider patch that was abandoned for a few months (years) in Bugzilla. It’s an interesting feature to have so that web apps can use the WebAudio API with raw audio coming from Media elements.

I also hacked on GstGL support for video rendering. It’s quite interesting to be able to share the GL context of WebKit with GStreamer! The patch is not ready yet for landing but thanks to the reviews from Sebastian, Mathew Waters and Julien Isorce I’ll improve it and hopefully commit it soon in WebKit ToT.

Sebastian also worked on Media Source Extensions support. We had a very basic, non-working, backend that required… a rewrite, basically :) I hope we will have this reworked backend soon in trunk. Sebastian already has it working on Youtube!

The event was interesting in general, with discussions about rendering engines, rendering and JavaScript.

Moving to Pelican

2020-06-13T12:05:12+02:00

Time for a change! Almost 10 years ago I was starting to hack on a Blog engine with two friends, it was called Alinea and it powered this website for a long time. Back then hacking on your own Blog engine was the pre-requirement to host your blog :) But nowadays people just use Wordpress or similar platforms, if they still have a blog at all. Alinea fell into oblivion as I didn’t have time and motivation to maintain it.

Moving to Pelican was quite easy, since I’ve been writing content in ReST on the previous blog I only had to pull data from the database and hacked pelican_import.py a bit :)

Now that this website looks almost modern I’ll hopefully start to blog again, expect at least news about the WebKit work I still enjoy doing at Igalia.

GStreamer’s playbin3 overview for application developers

2020-06-13T12:05:12+02:00

Multimedia applications based on GStreamer usually handle playback with the playbin element. I recently added support for playbin3 in WebKit. This post aims to document the changes needed on application side to support this new generation flavour of playbin.

So, first of, why is it named playbin3 anyway? The GStreamer 0.10.x series had a playbin element but a first rewrite (playbin2) made it obsolete in the GStreamer 1.x series. So playbin2 was renamed to playbin. That’s why a second rewrite is nicknamed playbin3, I suppose :)

Why should you care about playbin3? Playbin3 (and the elements it’s using internally: parsebin, decodebin3, uridecodebin3 among others) is the result of a deep re-design of playbin2 (along with decodebin2 and uridecodebin) to better support:

gapless playback
audio cross-fading support (not yet implemented)
adaptive streaming
reduced CPU, memory and I/O resource usage
faster stream switching and full control over the stream selection process

This work was carried on mostly by Edward Hervey, he presented his work in detail at 3 GStreamer conferences. If you want to learn more about this and the internals of playbin3 make sure to watch his awesome presentations at the 2015 gst-conf, 2016 gst-conf and 2017 gst-conf.

Playbin3 was added in GStreamer 1.10. It is still considered experimental but in my experience it works already very well. Just keep in mind you should use at least the latest GStreamer 1.12 (or even the upcoming 1.14) release before reporting any issue in Bugzilla. Playbin3 is not a drop-in replacement for playbin, both elements share only a sub-set of GObject properties and signals. However, if you don’t want to modify your application source code just yet, it’s very easy to try playbin3 anyway:

$ USE_PLAYBIN3=1 my-playbin-based-app

Setting the USE_PLAYBIN environment variable enables a code path inside the GStreamer playback plugin which swaps the playbin element for the playbin3 element. This trick provides a glance to the playbin3 element for the most lazy people :) The problem is that depending on your use of playbin, you might get runtime warnings, here’s an example with the Totem player:

$ USE_PLAYBIN3=1 totem ~/Videos/Agent327.mp4
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'video-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'

(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'audio-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'

(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'text-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'

(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'video-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'

(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'audio-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'

(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'text-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
sys:1: Warning: g_object_get_is_valid_property: object class 'GstPlayBin3' has no property named 'n-audio'
sys:1: Warning: g_object_get_is_valid_property: object class 'GstPlayBin3' has no property named 'n-text'
sys:1: Warning: ../../../../gobject/gsignal.c:3492: signal name 'get-video-pad' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'

As mentioned previously, playbin and playbin3 don’t share the same set of GObject properties and signals, so some changes in your application are required in order to use playbin3.

If your application is based on the GstPlayer library then you should set the GST_PLAYER_USE_PLAYBIN3 environment variable. GstPlayer already handles both playbin and playbin3, so no changes needed in your application if you use GstPlayer!

Ok, so what if your application relies directly on playbin? Some changes are needed! If you previously used playbin stream selection properties and signals, you will now need to handle the GstStream and GstStreamCollection APIs. Playbin3 will emit a stream collection message on the bus, this is very nice because the collection includes information (metadata!) about the streams (or tracks) the media asset contains. In playbin this was handled with a bunch of signals (audio-tags-changed, audio-changed, etc), properties (n-audio, n-video, etc) and action signals (get-audio-tags, get-audio-pad, etc). The new GstStream API provides a centralized and non-playbin-specific access point for all these informations. To select streams with playbin3 you now need to send a select_streams event so that the demuxer can know exactly which streams should be exposed to downstream elements. That means potentially improved performance! Once playbin3 completed the stream selection it will emit a streams selected message, the application should handle this message and potentially update its internal state about the selected streams. This is also the best moment to update your UI regarding the selected streams (like audio track language, video track dimensions, etc).

Another small difference between playbin and playbin3 is about the source element setup. In playbin there is a source read-only GObject property and a source-setup GObject signal. In playbin3 only the latter is available, so your application should rely on source-setup instead of the notify::source GObject signal.

The gst-play-1.0 playback utility program already supports playbin3 so it provides a good source of inspiration if you consider porting your application to playbin3. As mentioned at the beginning of this post, WebKit also now supports playbin3, however it needs to be enabled at build time using the CMake -DUSE_GSTREAMER_PLAYBIN3=ON option. This feature is not part of the WebKitGTK+ 2.20 series but should be shipped in 2.22. As a final note I wanted to acknowledge my favorite worker-owned coop Igalia for allowing me to work on this WebKit feature and also our friends over at Centricular for all the quality work on playbin3.

The GNOME-Shell Gajim extension maintenance

2020-06-13T12:05:12+02:00

Back in January 2011 I wrote a GNOME-Shell extension allowing Gajim users to carry on with their chats using the Empathy infrastructure and UI present in the Shell. For some time the extension was also part of the official gnome-shell-extensions module and then I had to move it to Github as a standalone extension. Sadly I stopped using Gajim a few years ago and my interest in maintaining this extension has decreased quite a lot.

I don’t know if this extension is actively used by anyone beyond the few bugs reported in Github, so this is a call for help. If anyone still uses this extension and wants it supported in future versions of GNOME-Shell, please send me a mail so I can transfer ownership of the Github repository and see what I can do for the extensions.gnome.org page as well.

(Huh, also. Hi blogosphere again! My last post was in 2014 it seems :))

WebKitGTK and WPE now supporting videos in the img tag

2020-06-09T18:00:00+02:00

Using videos in the <img> HTML tag can lead to more responsive web-page loads in most cases. Colin Bendell blogged about this topic, make sure to read his post on the cloudinary website. As it turns out, this feature has been supported for more than 2 years in Safari, but only recently the WebKitGTK and WPEWebKit ports caught up. Read on for the crunchy details or skip to the end of the post if you want to try this new feature.

As WebKitGTK and WPEWebKit already heavily use GStreamer for their multimedia backends, it was natural for us to also use GStreamer to provide the video ImageDecoder implementation.

The preliminary step is to hook our new decoder into the MIMETypeRegistry and into the ImageDecoder.cpp platform-agnostic module. This is where the main decoder branches out to platform-specific backends. Then we need to add a new class implementing WebKit’s ImageDecoder virtual interface.

First you need to implement supportsMediaType(). For this method we already had all the code in place. WebKit scans the GStreamer plugin registry and depending on the plugins available on the target platform, a mime-type cache is built, by the RegistryScanner. Our new image decoder just needs to hook into this component (exposed as singleton) so that we can be sure that the decoder will be used only for media types supported by GStreamer.

The second most important places of the decoder are its constructor and the setData() method. This is the place where the decoder receives encoded data, as a SharedBuffer, and performs the decoding. Because this method is synchronously called from a secondary thread, we run the GStreamer pipeline there, until the video has been decoded entirely. Our pipeline relies on the GStreamer decodebin element and WebKit’s internal video sink. For the time being hardware-accelerated decoding is not supported. We should soon be able to fix this issue though. Once all samples have been received by the sink, the decoder notifies its caller using a callback. The caller then knows it can request decoded frames.

Last, the decoder needs to provide decoded frames! This is implemented using the createFrameImageAtIndex() method. Our decoder implementation keeps an internal Map of the decoded samples. We sub-classed the MediaSampleGStreamer to provide an image() method, which returns the Cairo surface representing the decoded frame. Again, here we don’t support GL textures yet. Some more infrastructure work is likely going to be needed in that area of our WebKit ports.

Our implementation of the video ImageDecoder lives in ImageDecoderGStreamer which will be shipped in WebKitGTK and WPEWebKit 2.30, around September/October. But what if you want to try this already? Well, building WebKit can be a tedious task. We’ve been hard a work at Igalia to make this a bit easier using a new Flatpak SDK. So, you can either try this feature in Epiphany Tech Preview or (surprise!) with our new tooling allowing to download and run nightly binaries from the upstream WebKit build bots:

$ wget https://raw.githubusercontent.com/WebKit/webkit/master/Tools/Scripts/webkit-flatpak-run-nightly
$ chmod +x webkit-flatpak-run-nightly
$ python3 webkit-flatpak-run-nightly MiniBrowser https://colinbendell.github.io/webperf/img-mp4/

This script locally installs our new Flatpak-based developer SDK in ~/.cache/wk-nightly and then downloads a zip archive of the build artefacts from servers recently brought up by my colleague Carlos Alberto Lopez Perez, many thanks to him :). The downloaded zip file is unpacked in /tmp and kept around in case you want to run this again without re-downloading the build archive. Flatpak is then used to run the binaries inside a sandbox! This is a nice way to run the bleeding edge of the web-engine, without having to build it or install any distro package.

Implementing new features in WebKit is one of the many expertize domains we are involved in at Igalia. Our multimedia team is always on the lookout to help folks in their projects involving either GStreamer or WebKit or both! Don’t hesitate to reach out.

Introducing the WebKit Flatpak SDK

2020-06-08T18:50:00+02:00

Working on a web-engine often requires a complex build infrastructure. This post documents our transition from JHBuild to Flatpak for the WebKitGTK and WPEWebKit development builds.

For the last 10 years, WebKitGTK has been relying on a custom JHBuild moduleset to handle its dependencies and (try to) ensure a reproducible test environment for the build bots. When WPEWebKit was upstreamed several years ago, a similar approach was used. We ended up with two slightly different modulesets to maintain. The biggest problem with that is that we still depend on the host OS for some dependencies. Another set of scripts was then written to install those, depending on which distro the host is running… This is a bit unfortunate. There are more issues with JHBuild, when a moduleset is updated, the bots wipe all the resulting builds and start a new build from scratch! Every WebKitGTK and WPE developer is strongly advised to use this setup, so everybody ends up building the dependencies.

In 2018, my colleague Thibault Saunier worked on a new approach, based on Flatpak. WebKit could be built as a Flatpak app relying on the GNOME SDK. This experiment was quite interesting but didn’t work for several reasons:

Developers were not really aware of this new approach
The GNOME SDK OSTree commits we were pinning we being removed from the upstream repo periodically, triggering issues on our side
Developers still had to build all the WebKit “app” dependencies

In late 2019 I started having a look at this, with a different workflow. I started to experiment with a custom SDK, based on the Freedesktop SDK. The goal was to distribute this to WebKit developers, who wouldn’t need to worry as much about build dependencies anymore and just focus on building the damn web-engine.

I first learned about flatpak-builder, built a SDK with that, and started playing with it for WebKit builds. I was almost happy with that, but soon realized flatpak-builder is cool for apps packaging, but for big SDKs, it doesn’t really work out. Flatpak-builder builds a single recipe at a time and if I want to update one, everything below in the manifest is rebuilt. As the journey goes on, I found Buildstream, which is actually used by the FDO and GNOME folks nowadays. After converting my flatpak-builder manifest to Buildstream I finally achieved happiness. Buildstream is bit like Yocto, Buildroot and all the similar NIH sysroot builders. It was one more thing to learn, but worth it.

The SDK build definitions are hosted in WebKit’s repository. The resulting Flatpak images are hosted on Igalia’s server and locally installed in a custom Flatpak UserDir so as to not interfere with the rest of the host OS Flatpak apps. The usual mix of python, perl, ruby WebKit scripts rely on the SDK to perform their job of building WebKit, running the various test suites (API tests, layout tests, JS tests, etc). All you need to do is:

clone the WebKit git repo
run Tools/Scripts/update-webkit-flatpak
run Tools/Scripts/build-webkit —gtk
run build artefacts, like the MiniBrowser, Tools/Scripts/run-minibrowser —gtk

Under the hood, the SDK will be installed in WebKitBuild/UserFlatpak and transparently used by the various build scripts. The sandbox is started using a flatpak run call which bind-mounts the WebKit checkout and build directory. This is great! We finally have a unified workflow, not depending on specific host distros. We can easily update toolchains, package handy debug tools like rr, LSP tooling such as ccls, etc and let the lazy developers actually focus on development, rather than tooling infrastructure.

Another nice tool we now support, is sccache. By deploying a “cloud” of builders in our Igalia servers we can now achieve improved build times. The SDK generates a custom sccache config based on the toolchains it includes (currently GCC 9.3.0 and clang 8). You can optionally provide an authentication token and you’re set. Access to the Igalia build cloud is restricted to Igalians, but folks who have their own sccache infra can easily set it up for WebKit. In my home office where I used to wait more than 20 minutes for a build to complete, I can now have a build done in around 12 minutes. Once we have Redis-powered cloud storage we might reach even better results. This is great because the buildbots should be able to keep the Redis cache warm enough. Also developers who can rely on this won’t need powerful build machines anymore, a standard workstation or laptop should be sufficient because the heavy C++ compilation jobs happen in the “cloud”.

If you don’t like sccache, we still support IceCC of course. The main difference is that IceCC works better on local networks.

Hacking on WebKit often involves hacking on its dependencies, such as GTK, GStreamer, libsoup and so on. With JHBuild it was fairly easy to vendor patches in the moduleset. With Buildstream the process is a bit more complicated, but actually not too bad! As we depend on the FDO SDK we can easily “patch” the junction file and also adding new dependencies from scratch is quite easy. Many thanks to the Buildstream developers for the hard work invested in the project!

As a conclusion, all our WebKitGTK and WPEWebKit bots are now using the SDK. JHBuild remains available for now, but on opt-in basis. The goal is to gently migrate most of our developers to this new setup and eventually JHBuild will be phased out. We still have some tasks pending, but we achieved very good progress on improving the WebKit developer workflow. A few more things are now easier to achieve with this new setup. Stay tuned for more! Thanks for reading!

HTML overlays with GstWPE, the demo

2019-12-08T15:00:00+01:00

Once again this year I attended the GStreamer conference and just before that, Embedded Linux conference Europe which took place in Lyon (France). Both events were a good opportunity to demo one of the use-cases I have in mind for GstWPE, HTML overlays!

As we, at Igalia, usually have a booth at ELC, I thought a GstWPE demo would be nice to have so we can show it there. The demo is a rather simple GTK application presenting a live preview of the webcam video capture with an HTML overlay blended in. The HTML and CSS can be modified using the embedded text editor and the overlay will be updated accordingly. The final video stream can even be streamed over RTMP to the main streaming platforms (Twitch, Youtube, Mixer)! Here is a screenshot:

The code of the demo is available on Igalia’s GitHub. Interested people should be able to try it as well, the app is packaged as a Flatpak. See the instructions in the GitHub repo for more details.

Having this demo running on our booth greatly helped us to explain GstWPE and how it can be used in real-life GStreamer applications. Combining the flexibility of the Multimedia framework with the wide (wild) features of the Web Platform will for sure increase the synergy and foster collaboration!

As a reminder, GstWPE is available in GStreamer since the 1.16 version. On the roadmap for the mid-term I plan to add Audio support, thus allowing even better integration between WPEWebKit and GStreamer. Imagine injecting PCM audio coming from WPEWebKit into an audio mixer in your GStreamer-based application! Stay tuned.

Review of the Igalia Multimedia team Activities (2019/H1)

2019-08-05T15:30:00+02:00

This blog post takes a look back at the various Multimedia-related tasks the Igalia Multimedia team was involved in during the first half of 2019.

GStreamer Editing Services

Thibault added support for the OpenTimelineIO open format for editorial timeline information. Having many editorial timeline information formats supported by OpenTimelineIO reduces vendor lock-in in the tools used by video artists in post-production studios. For instance a movie project edited in Final Cut Pro can now be easily reimported in the Pitivi free and open-source video editor. This accomplishment was made possible by implementing an OpenTimelineIO GES adapter and formatter, both upstreamed respectively in OpenTimelineIO and GES by Thibault.

Another important feature for non-linear video editors is nested timeline support. It allows teams to decouple big editing projects in smaller chunks that can later on be assembled for the final product. Another use-case is about pre-filling the timeline with boilerplate scenes, so that an initial version of the movie can be assembled before all teams involved in the project have provided the final content. To support this, Thibault implemented a GES demuxer which transparently enables playback support for GES files (through file://path/to/file.xges URIs) in any GStreamer-based media player.

As if this wasn’t impressive enough yet, Thibault greatly improved the GES unit-tests, fixing a lot of memory leaks, race conditions and generally improving the reliability of the test suite. This is very important because the Gitlab continuous integration now executes the tests harness for every submitted merge request.

For more information about this, the curious readers can dive in Thibault’s blog post. Thibault was invited to talk about these on-going efforts at SIGGRAPH during the OpenTimelineIO BOF.

Finally, Thibault is mentoring Swayamjeet Swain as part of the GSoC program, the project is about adding nested timeline support in Pitivi.

GStreamer VA-API

Víctor performed a good number of code reviews for GStreamer-VAAPI contributors, he has also started investigating GStreamer-VAAPI bugs specific to the AMDGPU Gallium driver, in order to improve the support of AMD hardware in multimedia applications.

As part of the on-going GStreamer community efforts to improve continuous integration (CI) in the project, we purchased Intel and AMD powered devices aimed to run validation tests. Running CI on real end-user hardware will help ensure regressions remain under control.

Servo and GStreamer-rs

As part of our on-going involvement in the Servo Mozilla project, Víctor has enabled zero-copy video rendering support in the GStreamer-based Servo-media crate, along with bug fixes in Glutin and finally in Servo itself.

A prior requirement for this major milestone was to enable gst-gl in the GStreamer Rust bindings and after around 20 patches were merged in the repository, we are pleased to announce Linux and Android platforms are supported. Windows and macOS platforms will also be supported, soon.

The following screencast shows how hardware-accelerated video rendering performs on Víctor’s laptop:

Víctor also delivered a talk at the MadRust meetup, about GStreamer-rs, the slides are available on his website.

WebKit’s Media Source Extensions

As part of our on-going collaboration with various device manufacturers, Alicia and Enrique validated the Youtube TV MSE 2019 test suite in WPEWebKit-based products.

Alicia developed a new GstValidate plugin to improve GStreamer pipelines testing, called validateflow. Make sure to read her blog post about it!.

In her quest to further improve MSE support, especially seek support, Alicia rewrote the GStreamer source element we use in WebKit for MSE playback. The code review is on-going in Bugzilla and on track for inclusion in WPEWebKit and WebKitGTK 2.26. This new design of the MSE source element requires the playbin3 GStreamer element and at least GStreamer 1.16. Another feature we plan to work on in the near future is multi-track support; stay tuned!

WebKit WebRTC

We announced LibWebRTC support for WPEWebKit and WebKitGTK one year ago. Since then, Thibault has been implementing new features and fixing bugs in the backend. Bridging between the WebAudio and WebRTC backend was implemented, allowing tight integration of WebAudio and WebRTC web apps. More WebKit WebRTC layout tests were unskipped in the buildbots, allowing better tracking of regressions.

Thibault also fixed various performance issues on Raspberry Pi platforms during apprtc video-calls. As part of this effort he upstreamed a GstUVCH264DeviceProvider in GStreamer, allowing applications to use already-encoded H264 streams from webcams that provide it, thus removing the need for applications to encode raw video streams.

Additionally, Thibault upstreamed a device provider for ALSA in GStreamer, allowing applications to probe for Microphones and speakers supported through the ALSA kernel driver.

Finally, Thibault tweaked the GStreamer encoders used by the WebKit LibWebRTC backend, in order to match the behavior of the Apple implementation and also fixing a few performance and rendering issues on the way.

WebKit GStreamer Multimedia maintenance

The whole team is always keeping an eye on WebKit’s Bugzilla, watching out for multimedia-related bugs reported by the community members. Charlie recently fixed a few annoying volume bugs along with an issue related with youtube.

I rewrote the WebKitWebSrc GStreamer source element we use in WebKit to download HTTP(S) media resources and feed the data to the playback pipeline. This new element is now based on the GStreamer pushsrc base class, instead of appsrc. A few seek-related issues were fixed on the way but unfortunately some regressions also slipped in; those should all be fixed by now and shipped in WPEWebKit/WebKitGTK 2.26.

An initial version of the MediaCapabilities backend was also upstreamed in WebKit. It allows web-apps (such as Youtube TV) to probe the user-agent for media decoding and encoding capabilities. The GStreamer backend relies on the GStreamer plugin registry to provide accurate information about the supported codecs and containers. H264 AVC1 profile and level information are also probed.

WPEQt

Well, this isn’t directly related with multimedia, but I finally announced the initial release of WPEQt. Any feedback is welcome, QtWebKit has phased out and if anyone out there relies on it for web-apps embedded in native QML apps, now is the time to try WPEQt!