In this post I’ll try to document the journey starting from a WebKit issue and ending up improving third-party projects that WebKitGTK and WPEWebKit depend on.
I’ve been working on WebKit’s GStreamer backends for a while. Usually some new feature needed on WebKit side would trigger work on GStreamer. That’s quite common and healthy actually, by improving GStreamer (bug fixes or implementing new features) we make the whole stack stronger (hopefully). It’s not hard to imagine other web-engines, such as Servo for instance, leveraging fixes made in GStreamer in the context of WebKit use-cases.
Sometimes though we have to go deeper and this is what this post is about!
Since version 2.44, WebKitGTK and WPEWebKit ship with a WebCodecs backend. That backend leverages the wide range of GStreamer audio and video decoders/encoders to give low-level access to encoded (or decoded) audio/video frames to Web developers. I delivered a lightning talk at gst-conf 2023 about this topic.
There are still some issues to fix regarding performance and some W3C web platform tests are still failing. The AV1 decoding tests were flagged early on while I was working on WebCodecs, I didn’t have time back then to investigate the failures further, but a couple weeks ago I went back to those specific issues.
The WebKit layout tests harness is executed by various post-commit bots, on
various platforms. The WebKitGTK and WPEWebKit bots run on Linux. The WebCodec
tests for AV1 currently make use of the GStreamer av1enc and
dav1ddec elements. We
currently don’t run the tests using the modern and hardware-accelerated
vaav1enc and vaav1dec elements because the bots don’t have compatible GPUs.
The decoding tests were failing, this
one
for instance (the ?av1 variant). In that test both encoding and decoding are
tested, but decoding was failing, for a couple reasons. Rabbit hole starts here.
After debugging this for a while, it was clear that the colorspace information
was lost between the encoded chunks and the decoded frames. The decoded video
frames didn’t have the expected colorimetry values.
The
VideoDecoderGStreamer
class basically takes encoded chunks and notifies decoded
VideoFrameGStreamer
objects to the upper layers (JS) in WebCore. A video frame is basically a
GstSample (Buffer and Caps) and we have code in place to interpret the
colorimetry parameters exposed in the sample caps and translate those to the
various WebCore equivalents. So far so good, but the caps set on the dav1ddec
elements didn’t have those informations! I thought the dav1ddec element could
be fixed, “shouldn’t be that hard” and I knew that code because I wrote it in
2018 :)
So let’s fix the GStreamer dav1ddec element. It’s a video decoder written in
Rust, relying on the dav1d-rs bindings of
the popular C libdav1d library. The dav1ddec element basically feeds encoded
chunks of data to dav1d using the dav1d-rs bindings. In return, the bindings
provide the decoded frames using a Dav1dPicture Rust structure and the
dav1ddec GStreamer element basically makes buffers and caps out of this
decoded picture. The dav1d-rs bindings are quite minimal, we implemented API on
a per-need basis so far, so it wasn’t very surprising that… colorimetry
information for decoded pictures was not exposed! Rabbit hole goes one level deeper.
So let’s add colorimetry API in dav1d-rs. When working on (Rust) bindings of a
C library, if you need to expose additional API the answer is quite often in the
C headers of the library. Every Dav1dPicture has a Dav1dSequenceHeader, in
which we can see a few interesting fields:
typedef struct Dav1dSequenceHeader {
...
enum Dav1dColorPrimaries pri; ///< color primaries (av1)
enum Dav1dTransferCharacteristics trc; ///< transfer characteristics (av1)
enum Dav1dMatrixCoefficients mtrx; ///< matrix coefficients (av1)
enum Dav1dChromaSamplePosition chr; ///< chroma sample position (av1)
...
uint8_t color_range;
...
...
} Dav1dSequenceHeader;
After sharing a naive branch with rust-av co-maintainers Luca
Barbato and Sebastian
Dröge, I came up with a
couple
pull-requests that eventually
were shipped in version 0.10.3 of dav1d-rs. I won’t deny matching primaries,
transfer, matrix and chroma-site enum values to rust-av‘s Pixel enum was a bit
challenging :P Anyway, with dav1d-rs fixed up, rabbit hole level goes up one
level :)
Now with the needed dav1d-rs API, the GStreamer dav1ddec element could be
fixed. Again, matching the various enum values to their GStreamer equivalent was
an interesting exercise. The merge
request
was merged, but to this date it’s not shipped in a stable gst-plugins-rs release
yet. There’s one more complication here, ABI broke between dav1d 1.2 and 1.4
versions. The dav1d-rs 0.10.3 release expects the latter. I’m not sure how we
will cope with that in terms of gst-plugins-rs release versioning…
Anyway, WebKit’s runtime environment can be adapted to ship dav1d 1.4 and
development version of the dav1ddec element, which is what was done in this
pull request. The rabbit is
getting out of his hole.
The WebCodec AV1 tests were finally fixed in WebKit, by this pull request. Beyond colorimetry handling a few more fixes were needed, but luckily those didn’t require any fixes outside of WebKit.
Wrapping up, if you’re still reading this post, I thank you for your patience.
Working on inter-connected projects can look a bit daunting at times, but
eventually the whole ecosystem benefits from cross-project collaborations like
this one. Thanks to Luca and Sebastian for the help and reviews in dav1d-rs
and the dav1ddec element. Thanks to my fellow Igalia
colleagues for the WebKit reviews.