Base-Art / Articles

GStreamer hackfest in Milan

April 01, 2013 10:29 AM

I spent Easter in Milan, attending the second official GStreamer hackfest. It has been a productive event and, as always, a pleasure to hang out with some members of the GStreamer core dev team fame :)

I focused on a few GStreamer bugs that were blocking some other WebKit bugs related to the media player and the WebAudio backend. I also had in mind to port the osxaudio sink to 1.0, a task started one week before the hackfest, but didn't have time to come back to it.

The first task I worked on was about cleaning up some code in uridecodebin related to URI protocols and on-disk buffering. In WebKit we use a custom HTTP(S) source element used by playbin when loading remote media files. The issue is that that element is also exposed to applications using WebKit itself so we made it handle a custom protocol, webkit+http://. But that exposed a bug in urideocdebin where some protocols, including http, are hardcoded in a white list used to enable on-disk buffering. Unfortunately the aforementioned bug is not solved yet even though some good progress was made!

The second task I worked on was first reported by Wim, the WebKit media player wasn't pausing when receiving buffering messages. The fix was quite simple. After that I finally added some basic codec installer support in WebKit. The GStreamer codec installer API is quite simple :)

Lastly I started working again on the WebAudio createMediaSourceElement support for the GStreamer/WebKit backend. The patch has been sleeping in Bugzilla since last Christmas. That feature is interesting for WebAudio applications that need to use audio data coming from media elements to eg, do stuff like equalizers. The current patch in bugzilla gathers the audio buffers in lists and copy them to the AudioBus when needed. Those memcpys could probably be avoided by pre-allocating the memory for the AudioBus and telling GStreamer to write directly there. So, with the help of my friend Alessandro Decina (who was doing some similar stuff on his Firefox branch) I started working on a simple GstAllocator that would be configured in the pipeline by intercepting allocation queries toward the appsinks used to gather the audio buffers. Well, this exposed some bugs on the GStreamer side! The deinterleave element always uses the default allocator when pushing data to its source pads. So I started a patch that performs allocation queries downstream and use the allocator returned, if any. Unfortunately my custom allocator doesn't work yet but I'll cary on work on this, the GstAllocator API is quite an interesting thing to play with :)

All in all it was a great and productive event. Christian wrote a nice report about the hackfest too. I also wanted to thank Collabora and Fluendo for sponsoring dinners. A special thank you as well for Igalia which covered my travel expenses and attendance to the hackfest.

WebKitGTK+ Hackfest, 2012 edition

December 15, 2012 02:12 PM

There has been some media coverage of the hackfest already so I'll just focus on one area, Multimedia!

I think this fourth edition of the event was the first with so many hackers focusing on Multimedia:

Xabier and Zan worked on the media controls and finally landed the GStreamer 1.0 build support in our JHBuild moduleset so the buildbots now run the tests with a modern GStreamer setup
Jonathon focused on the Mediastream/WebRTC backend
Victor has been working on audio pitch preservation
yours truly dedicated most of his time on porting the WebAudio backend to GStreamer 1.0 and hunting two small bugs in the deinterleave and interleave elements

We also had a small session about the current status of Multimedia in WebKitGTK+ with a summary of what's been done recently, the work in progress and plans for the future. It's interesting because there is always a new W3C specification to implement and bugs to fix :)

A warm thank you to all the sponsors and organizers that made the event possible, it was really nice to meet new (and old) friends, hack on interesting tasks and keep pushing for a vibrant project such as WebKitGTK+ :)

WebAudio support in WebKitGTK+: the demo

October 19, 2012 08:30 AM

In a previous post I was explaining how the WebAudio GStreamer backend is implemented in WebKit. What was missing was a nice screencast demonstrating the feature in Epiphany. Thankfully the valiant Mario Sanchez took up the challenge and the result is quite nice:

If the youtube embed code doesn't show up you can still see the video there.

Since I wrote the previous post the Qt and EFL WebKit ports adopted the WebAudio GStreamer backend as well. It is still experimental, we've been fixing some annoying bugs recently, the fixes will be shipped in the upcoming WebKitGTK+ 1.10.2 version.

I'm really happy to have worked on this feature as part of my daily work at Igalia. There's more to come, our WebKit team keeps expanding :)

Implementing WebAudio in WebKit with GStreamer

May 28, 2012 11:47 PM

WebAudio is a JavaScript API for processing and synthesizing audio in web applications. It has been implemented first in Google Chrome and later on in Safari. The intention behind this blog post is to explain how other WebKit ports can support the WebAudio APIs using GStreamer. The WebKit GTK+ port is already up to speed and the EFL and Qt ports are within reach as well.

The Chromium WebAudio demos don't work yet in Epiphany because WebKitGTK+ doesn't yet ship with WebAudio support enabled by default and the WebView used by Epiphany doesn't yet enable the enable-web-audio WebKitWebSetting. However, for the brave people willing to build WebKitGTK+, here are the steps to follow:

$ Tools/Scripts/build-webkit --gtk --web-audio
$ Tools/Scripts/run-launcher --gtk --enable-web-audio=TRUE http://chromium.googlecode.com/svn/trunk/samples/audio/shiny-drum-machine.html

I especially like the drum machine demo, pretty impressing stuff! So, back to the topic. Fortunately the WebAudio AudioContext was designed to allow multiple platform backends in WebCore, I'm going to talk about the GStreamer backend. It was first prototyped by Zan Dobersek and he kindly passed me on the task which I continued and upstreamed in WebKit trunk.

The two biggest features of the backend are:

1. Decoding audio data from a media file or from a memory pointer and pulling it into the WebAudio machinery of WebKit where it's going to be processed by the WebAudio pipeline.

2. Getting back audio data as WAV from the WebAudio pipeline and play it back :)

For the first step we use decodebin2 with data coming from either filesrc or giostreamsrc. The decoded data is split in mono channels using the deinterleave element, passed to appsinks and converted to AudioChannels which internally store the data in FloatArrays. Every AudioChannel is stored in an AudioBus instance which we pass onto WebCore for further internal processing. The implementation can be found in the AudioFileReaderGStreamer.cpp file.

Internally the WebAudio stack also relies on our GStreamer FFTFrame implementation which uses the gst-fft library to perform Fast Fourier Transforms.

Then let's talk about playback! Once the WebAudio stack has finished processing data and if the web developer created an AudioDestinationNode as part of his WebAudio pipeline, the user needs to hear something playing :) As the mighty reader might guess the magic is done in an AudioDestinationGStreamer.cpp file! I decided to implement most of the logic about reading data from the AudioBus as a GStreamer source element, called WebKitWebAudioSourceGStreamer. This element takes every AudioChannel of the AudioBus, convert the FloatArray data to GStreamer buffers, interleave the channels and encode the whole as WAV data using the wavenc element. In the AudioDestination GStreamer pipeline we then use this shiny source element, parse the WAV data and pass it on the audio sink!

What's next in our roadmap?

I'm not sure this source element was the correct approach in the end, I think that at some point I'll just refactor it into the AudioDestination implementation.
Port to GStreamer 0.11 APIs! This work is on-going already for the MediaPlayer.
There are still some bugs to fix, before I really feel confident about enabling this feature by default. We should also add support for reading data from the HTML5 Media elements.
The WebKit webaudio layout tests are almost passing on the build bots, this work is on-going as well.
Enable WebAudio by default in WebKitGTK's build and Epiphany, targetting GNOME 3.6.

I would like to thank Igalia for allowing me to dedicate work time on this project :)

Gajim in your GNOME3 Shell

April 08, 2011 07:57 AM

The IM integration in the GNOME3 is interesting, but only for the people using Empathy :) Because my personal preference goes to another client called Gajim I wanted it to fit more in the Shell. To reach this goal I worked on 2 small projects partly during my Igalia hackfest hours and partly during my free time.

The first bit is a Gajim plugin integrating with the GNOME Session-Manager and synchronizing your status in GNOME with your Gajim status.

The second part is a Shell extension hooking to Gajim via D-Bus to monitor chat notifications and allow chatting in the Shell without directly using the Gajim window. This behaves exactly as the Empathy integration.

There's still some work to do, like inhibiting notifications coming from Gajim because currently when a incoming chat appears the user gets a notification from the builtin Shell IM and a notification from Gajim itself. A workaround is to disable some notifications in the Gajim preferences.

As I needed some new D-Bus API in Gajim the Shell extension requires Gajim-nightly >= 20110326.

Congrats to the GNOME Team for this awesome GNOME3 release and happy chating :)

WebKitGTK+ Hackfest!

August 20, 2010 10:58 AM

This was a nice week in A CoruÃ±a, Spain. The WebKitGTK+ Hackfest went well I think :) The Igalia office is great for such an event, the organization/infrastructure was well done, the hotel near the office was awesome, what else? We managed to get things done code-wise! Even though the list is big some items were proudly erased from the TODO like:

Dan started a rewrite of some parts of libsoup with gio and worked on the soup URI loader
Xan and Gustavo worked on form authentication saving in WebKitGTK+/Epiphany
Xan did some good progress on DOM bindings
User-Agent support in Epiphany
Page cache control support in WebKitGTK+ was merged by Alex
Alex did some work on a11y support for WebKitGTK+
Bedhad and Evan worked on Harfbuzz integration (instead of Pango)
Reinout built a list of Epiphany regressions to fix
Evan and Benjamin worked on improving the build process (size and time)
Martin and Cody worked on ARGB windows support for WebKitGTK+
Benjamin went on war against crashers
and I worked on merging HTML5 video controls and the media tests
probably a lot more I didn't catch during all the discussions ;)

Still some work to do though, like for HTML5 audio/video support:

on-disk buffering
fullscreen view
closed captions (this seems very specific to SMIL)
pitch control (when scrubbing)
enhanced controls UI
and still some media tests to fix :)

So thank you to Igalia, Collabora and the GNOME foundation for sponsoring this event!

WebKitGTK+ and HTML5 fullscreen video

August 20, 2010 10:47 AM

HTML5 video is really nice and all but one annoying thing is the lack of fullscreen video specification, it is currently up to the User-Agent to allow fullscreen video display.

WebKit allows a fullscreen button in the media controls and Safari can, on user-demand only, switch a video to fullscreen and display nice controls over the video. There's also a webkit-specific DOM API that allow custom media controls to also make use of that feature, as long as it is initiated by a user gesture. Vimeo's HTML5 player uses that feature for instance.

Thanks to the efforts made by Igalia to improve the WebKit GTK+ port and its GStreamer media-player, I have been able to implement support for this fullscreen video display feature. So any application using WebKitGTK+ can now make use of it :)

The way it is done with this first implementation is that a new fullscreen gtk+ window is created and our GStreamer media-player inserts an autovideosink in the pipeline and overlays the video in the window. Some simple controls are supported, the UI is actually similar to Totem's. One improvement we could do in the future would be to allow application to override or customize that simple controls UI.

The nice thing about this is that we of course use the GstXOverlay interface and that it's implemented by the linux, windows and mac os video sinks :) So when other WebKit ports start to use our GStreamer player implementation it will be fairly easy to port the feature and allow more platforms to support fullscreen video display :)

Benjamin also suggested that we could reuse the cairo surface of our WebKit videosink and paint it in a fullscreen GdkWindow. But this should be done only when his Cairo/Pixman patches to enable hardware acceleration land.

So there is room for improvement but I believe this is a first nice step for fullscreen HTML5 video consumption from Epiphany and other WebKitGTK+-based browsers :) Thanks a lot to Gustavo, Sebastian DrÃ¶ge and Martin for the code-reviews.

Bay Area, California!

April 22, 2010 04:13 AM

Writting this post from San Francisco where I am spending some volcanic, work and little sightseeing time :)

Work/Hacking

I initially came here for the first WebKit Contributor Meeting with my Igalia coworkers Alex, Juanjo and Xan. We used the time here to also meet a customer we have in Bay Area and some of us attended ELC and Linux Collab Summit too. I was meant to attend the last day of ELC but we were a bit late asking for it so it was dropped from the long queue of people wanting to attend as well :/

People attending ELC and/or Linux Collab got a free Google Nexus. I was kinda sad to not get one because of that registration issue and Juanjo kindly gave me the one he got. So I will try it for a while, write a little N900 vs NexusOne blog post and probably give the phone back to some Igalian ;). Thanks Juanjo!

The WebKit meeting was interesting. Although I didn't make much talking I attended the session about moving to git, the one about how to deal with experimental features, the WebKit2 one and finally the one about the Media Elements. The last one was interesting for me especially because I work on the MediaPlayer stuff in WebKitGTK+ ;)

This last session motivated me again to work on video fullscreen for WebKitGTK+. I have now a working patch and hope to share some FullscreenController code with the windows port (and mac maybe). The way I did it is the following:

create a bin which will serve as video-sink for playbin2
link a ghost pad of the bin to a tee
in one branch of tee I added a queue and our internal webkit videosink
when the user requests fullscreen I create a new branch with a ffmpegcolorspace, queue and autovideosink
overlay is handed of to a window created by the fullscreen controller
when user exits from fullscreen I remove the branch from the tee.

I can now enter fullscreen by clicking on the fullscreen button of the media controls and leave fullscreen by pressing escape of f. So the next step is to draw some media controls over the video and handle the actions. Not sure what to do with the browser window when fullscreen is activated... hide? pause? Suggestions welcome :) Safari for instance animates the video box so it nicely takes all the screen estate, not sure I can do that with GTK...

Lately I've also been working on astracting some things away from the MediaPlayer backend so it can eventually be reused by other ports. I even started experimenting on mac! More to come on this. So what I needed to fix/abstract:

I moved the cairo calls from the paint method to new ImageGstreamer abstractions
The libsoup calls in the webkitsrc element have been ifdef'd. The message pausing/resuming needs to be abstracted away.
If the underlying application doesn't run on a gmainloop the gstreamer messages need to be processed with a synchronous handler.
ports implementing fullscreen need adaptations as well (use overlay when possible, see above).

I'm not yet really proud of all these patches so this seems to appear as teasing-only, sorry for that ;) I will blog about this again when the patches land in trunk.

Tourism

I was supposed to leave SF on the 15th of april but a volcano with a name a french guy like me can't pronounce decided otherwise! Alex, Juanjo and Xan managed to fly back because Madrid airport wasn't closed. So Frank and I went on a little road trip towards Muir Woods and Reyes point. This is an amazing landscape, we even saw a reserve of elephant seals :) In SF we also went to the Golden Gate park and the Japanese park inside, very nice. All the week was the occasion of eating japanese food, burritos, cable-car(ing) and walking endless miles on the hills of SF. Last saturday was Mike's birthday party and I was supposed to leave on sunday. The flight was cancelled for the second time. Mike kindly accepted to let me sleep in the guest room and I worked from a cafe in Oak street.

This time in SF was also a good occation to see other people stranded like Christian, Robert, Philippe (all 3 from Collabora), Marc (fellow fluendo coworker) and some locals like Brandon Lewis, David Shleef.

My next attempt (hopefully the last!) to reach Spain is scheduled for this friday, the Ashing seems somehow to let planes fly (or is it the airlines lobbying over politics?) so I have good hopes this time.

Dude this is the longest post I ever wrote. Thanks for reading (or skipping :)

Mirabeau on Maemo

February 11, 2010 03:20 PM

At FOSDEM Frank and I showed the work we did on Mirabeau, a screencast was made few days before FOSDEM and we showed it but I wanted to add some comments and re-arrange some parts of it, so here it is now, re-arranged with PiTiVi :)

Please install a HTML5 compliant browser. Meanwhile you can download the video from http://base-art.net/static/mirabeau2.ogg

(Video here in ogg/theora/vorbis if your browser fails to play it).

This is still work in progress, we have some issues with Telepathy MUC Tubes, I promised Sjoerd from Telepathy fame to create some new bugs in bugzilla. Also the UI itself still needs work, especially the MediaRenderer UI. I will also at some point add a chatroom window.

Oh and we also won a N900 at the XMPP developer contest thanks to this application! Thanks a lot to the XSF and Nokia :)

Guess what

January 29, 2010 11:59 AM

Arriving friday morning and leaving monday evening! I will do a quick presentation about WebKitGTK+ and a talk with my friend Frank about Mirabeau and personal media networks in the XMPP room.

We will be a whole Igalia gang hanging out there at Fosdem, looking forward socializing around beers :) Oh and don't miss Joaquim and Victor's talks!