Devlog

Using GStreamer's Rust plugins (gst-plugins-rs) on Embedded Linux just became easier, thanks to the new bitbake recipe that is now a part of OpenEmbedded-Core (OE-Core).

Yocto project and OpenEmbedded

The Yocto project has been the de facto provider of build infrastructure for creating custom Linux images, especially for embedded systems, regardless of the hardware architecture.

The OpenEmbedded build framework is the build system used by the Yocto project. It uses Bitbake to execute various build related tasks such as fetching and compiling the code, installing and packaging of the binaries etc. Bitbake executes these tasks according to the instructions provided in the form of a recipe (.bb) and other metadata files (e.g., .bbappend, .inc, .bbclass).

OE-Core contains metadata that comprises foundational recipes, classes, and associated files that are meant to be commonly-used by many different OpenEmbedded-derived systems, including the Yocto Project.

A recipe for Gstreamer Rust plugins

bitbake recipes for GStreamer have already existed in the OE-Core layer for a long time. These recipes build plugins and libraries belonging to various submodules of GStreamer.

We created a new recipe for the GStreamer Rust plugins, which was merged into OE-Core recently.

This recipe was inspired by a pre-existing patch submitted by Bartosz Golaszewski 4 years ago, but significant changes were needed. Some of them are listed below.

• Use dependencies only from crates.io

  For reproducibility, cargo.bbclass uses the --frozen flag to ensure that cargo doesn't update Cargo.lock and doesn't fetch anything from the network at build time. The necessary fetching is done by bitbake in an earlier step do_fetch, which downloads source code and other files based on the path(s) listed in SRC_URI. Refer “file download support” for more information.

  To add all the plugin dependencies to SRC_URI, we execute update_crates, which is a one-time task that auto-generates a <recipe>-update-crates.inc file. The update_crates function captures the URLs and checksums of the packages present in Cargo.lock, but only those that use the crates.io registry. It ignores all other (for e.g. git-based source) packages.

  By default, some of the dependencies of gst-plugins-rs, such as gstreamer-rs and gtk-rs, point to Git repositories in the manifest (Cargo.toml) and lock (Cargo.lock) files. As a result, they are not added to SRC_URI automatically and are not downloaded during the do_fetch stage unless we add them manually. However, as pointed out in a review comment, all the dependencies should be in the auto-generated gstreamer1.0-plguins-rs-update-crates.inc file and should not be manually specified in SRC_URI.

  To address this problem, starting with gst-plugins-rs release 0.15.3, we changed the root manifest file (Cargo.toml) for workspace dependencies like gstreamer-rs, gtk-rs etc from git repositories to the corresponding crates.io package names.

  It's also worth noting that the plugins gst-plugin-ffv1 and gst-plugin-flavors have been skipped in the recipe because they are not part of gst-plugins-rs releases, and hence do not have crates.io packages.

• Use cargo-c.bbclass instead of cargo.bbclass

  Switching to cargo-c helped to generate and ship C-ABI compatible libraries and pkg-config files. It also required a minor fix in cargo_c.bbclass to always specify the path to the directory where the library files should be installed, because the default value <prefix>/lib is not the correct lib directory for some targets.

• Skip recipe for 32-bit x86 platforms without SSE

  The ring crate, which is pulled as a dependency by various plugins, fails on 32-bit x86 platform builds that do not have SSE support. These plugins include reqwest, rswebrtc, quinn etc. It is not trivial to fix this without moving away from the ring dependency, so taking into account the small number of affected users, we decided to skip the recipe for these platforms.

Usage

By default, this recipe ships only the plugins marked essential and important, namely audiofx, closedcaption, file, fallbackswitch, tracers, threadshare, rtp, inter, isobmff, hlssink3, mpegtslive, reqwest, rtsp, webrtc, webrtc-signalling, videofx, webp.

If you want to install some or all of these in your image/rootfs, you can append those package names to IMAGE_INSTALL, f.ex.,

• IMAGE_INSTALL:append = " gstreamer1.0-plugins-rs-meta" for all plugins in the default-members list, or
• IMAGE_INSTALL:append = " gstreamer1.0-plugins-rs-audiofx gstreamer1.0-plugins-rs-webrtc gstreamer1.0-plugins-rs-reqwest" for specific plugins

You can also override the PACKAGECONFIG variable to add or remove the plugins from the default list.

Acknowledgements

Finally, a big thanks to the maintainers and reviewers from the OpenEmbedded community as well as my peers at Centricular for their valuable feedback on the patches.

Availability

This recipe is available in the master branch at moment and is expected to be part of the upcoming Yocto project release (Blacksail 6.1). This recipe will not be added to the upstream stable branches (Wrynose/Scarthgap) because Yocto policy dictates that features cannot be backported to a stable release.

If you are keen to use this recipe in the released Yocto branches, it should be straightforward on Wrynose (Yocto release 6.0). Cherry-pick the cargo-c and gstreamer1.0-plugins-rs patches, and use the License expression as per older syntax (expand the below block for the exact diff).

diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-rs_0.15.3.bb b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-rs_0.15.3.bb
index 1800dfc543..af488e1587 100644
--- a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-rs_0.15.3.bb
+++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-rs_0.15.3.bb
@@ -1,7 +1,7 @@
SUMMARY = "GStreamer Rust Plugins"
HOMEPAGE = "https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs"

-LICENSE = "Apache-2.0 OR MPL-2.0"
+LICENSE = "Apache-2.0 | MPL-2.0"

SRC_URI += "\
git://gitlab.freedesktop.org/gstreamer/gst-plugins-rs;protocol=https;tag=${PV};nobranch=1;name=default \

However, to make it work on Scarthgap (Yocto release 5.0), in addition to the above patches, further work is required - such as bumping up the rust and cargo family recipes to 1.92 (minimum supported version for gst-plugins-rs) or later, and fixing other dependents of the rust/cargo packages.

If you have questions, comments, or if you need any help using this recipe in your project, please feel free to get in touch.

RTSP stands for Real-Time Streaming Protocol, standardized by RFC 7826. It's commonly used to stream audio and video from IP cameras.

In GStreamer, RTSP handling was traditionally done by rtspsrc, a widely used and mature element, but it still had several architectural limitations that were hard to address within the existing codebase.

rtspsrc ties the server state to client/pipeline state. This resulted in issues with

• Multicast playback
• PAUSE being sent on error, immediately followed by tear down
• Flushing seeks involved state changes causing glitches and possible deadlock

As a result, the element has been rewritten from scratch in Rust as rtspsrc2. See this GStreamer Conference Talk and README for more details.

Until now, some features were missing, so if your application relied on them, rtspsrc2 couldn't replace the original.

Many of these have now been implemented:

Authentication

Basic and Digest authentication are supported. Digest covers MD5, SHA-256 and SHA-512-256, so rtspsrc2 can connect to servers that require a username/password and can perform the challenge response handshake automatically.

TLS/TCP support

rtspsrc2 can connect over TLS (the rtsps:// scheme) and supports client certificate authentication for servers that require it. Like rtspsrc, it also has tls-validation-flags property to ignore specific certificate errors like expired certificates.

HTTP tunnelling

Some networks block RTSP but allow HTTP. rtspsrc2 can tunnel RTSP over HTTP (rtsph://) and exposes an extra-http-request-headers property for custom headers (useful for proxies or authentication).

Keep alive

RTSP keep-alive is required because RTSP session state is entirely independent of the underlying transport. During active playback, RTCP serves as the primary proof of liveness. However, during paused states when media and RTCP traffic stop, explicit RTSP keep-alive requests are needed to prevent session timeout.

rtspsrc2 sends periodic keep-alive to prevent session timeout. This is enabled by default via do-rtsp-keep-alive. Disable it if you need compatibility with older servers.

Stream selection

This is one major feature that rtspsrc doesn't implement. rtspsrc2 is streams-aware.

When a source contains multiple streams (for example audio and video), rtspsrc2 exposes them as GstStream objects so applications can choose which streams to set up before playback using the StreamCollection API. For uses of this API by other elements, also see adaptive demuxers.

Consider an example scenario where the RTSP stream has audio and video. An application can choose to ignore audio and select only video via the stream collection API. rtspsrc2 then sends a SETUP request only for video and not audio. An example showing how this can be done in Rust can be seen here.

Unlinked pads

Linking of exposed source pads is now optional, for example, if you only link the video pad and leave the audio pad unlinked, rtspsrc2 handles that without error. Internally, it uses a flow combiner like the original rtspsrc, so selective linking works as expected.

GET/SET_PARAMETER

GET_PARAMETER and SET_PARAMETER requests are supported using signals similar to rtspsrc. These requests are used to retrieve or set the value of a parameter or a set of parameters for a presentation or stream specified by the URI.

Secure Real-time Transport Protocol (SRTP)

SRTP adds encryption and authentication to RTP streams. The RTSP source internally instantiates an RTP session manager element that handles the messages to and from the server, jitter removal, packet reordering along with providing a clock for the pipeline.

While rtspsrc uses rtpbin, rtspsrc2 can use rtpbin or the newer rtpbin2. rtpbin2 unlike rtpbin splits out the sender and receiver RTP session management in two separate elements rtpsend and rtprecv. See this GStreamer Conference Talk for understanding the motivations behind the new RTP elements. By setting the environment variable USE_RTP2=1, rtspsrc2 can use rtpsend and rtprecv.

Secure RTP (SRTP) as of this writing, is only supported when using rtspsrc2 with rtpbin, including MIKEY key exchange embedded in SDP.

With these additions, rtspsrc2 is much closer to being a drop-in replacement for rtspsrc. If you haven't tried it yet, now is a good time.

A GStreamer pipeline to test rtspsrc2 given a RTSP server address.

gst-launch-1.0 -e -vvv rtspsrc2 location=rtsp://some.server/url ! queue ! decodebin ! queue ! videoconvert ! autovideosink

The following key features still need to be implemented before rtspsrc2 achieves complete feature parity with rtspsrc.

• Clock sync support, such as RFC 7273 (work is on-going to bring this to rtprecv)
• Pause/seeking support with VOD
• ONVIF back-channel support

A few days ago I wrote a new GStreamer element around llama.cpp. This element takes a text stream as input, passes it together with a configurable system prompt through a locally running LLM and then forwards the LLM's output. It can also keep a history of past inputs and outputs for more consistent outputs. Check the documentation of the element for everything that can be configured on it.

This can be used for all kinds of purposes involving text. The two most obvious ones are probably translation into different languages and rephrasing.

When compiling the plugin, make sure to select the correct backend. By default only the CPU backend is compiled in but depending on your GPU the Vulkan, ROCm or CUDA backends are going to provide much better performance. For example to compile with the Vulkan backend, use

$ cargo build --release --features vulkan

Translations

Together with the whisper speech-to-text element this would for example allow you to watch a movie with English audio, transcribe that text in real-time, then have the text translated from English to German and render German subtitles on top of the video.

An example pipeline for this would be the following:

$ gst-launch-1.0 filesrc location=movie.mp4 ! decodebin3 name=dbin \
    dbin. ! audio/x-raw ! tee name=audio-tee
    audio-tee. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! audioconvert ! audioresample ! \
        whispertranscriber model-path=whisper-ggml-large-v3.bin model-preset=large-v3 chunk-duration=4000 ! \
        textaccumulate latency=0 ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! \
        llamacpp-texttransform model-path=Hunyuan-MT-7B.Q4_K_M.gguf system-prompt="Translate the following segments into German, without additional explanation." history-size=5 ! \
        textwrap columns=72 ! overlay.text_sink \
    dbin. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! videoconvert ! \
        textoverlay name=overlay ! videoconvert ! autovideosink
    audio-tee. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! audioconvert ! autoaudiosink

Running this on the famous speech at the end of The Great Dictator produces output such as the following:

Note that the plain transcribed text is rendered at the top and the corresponding translation is at the bottom.

This example also makes use of the textaccumulate element to collect full sentences or subclauses, and the textwrap element to wrap long lines so they fit on top of the video frames.

Collecting full sentences or subclauses before the text transformation can help the model with getting more context and providing better output in exchange for additional latency in live pipelines and additional buffering requirements in non-live pipelines.

Rephrasing

The same approach can also be used for rephrasing text. The following example is from the same movie and the system prompt now contained instructions to rephrase the English speech into "duck English".

While this example is not too useful apart from the comedic effect, a useful application of this would be for example rephrasing in simpler and shorter English, or removing usage of jargon.

Models

Generally, any model that is supported by llama.cpp and that supports text input and output can be used for this element. It makes little sense to use a huge model but models with 1-10B are generally more than enough to give useful results while also not requiring too many resources.

The model that should be used can be selected via the model-path property. This expects a local GGUF file, which can be downloaded e.g. from Hugging Face. Models that are known to work well are

• Gemma 4 E4B (quantizations) or Gemma 4 E2B (quantizations),
• Qwen 3.5 9B (quantizations) or Qwen 3.5 4B (quantizations),
• Ministral 3 14B Instruct (quantizations) or Ministral 3 8B Instruct (quantizations) or Ministral 3 3B Instruct (quantizations),
• Hunyuan MT 7B (quantizations). This model is specifically trained for translations and expects a system prompt in a specific format that is described on the Hugging Face page.

The examples above were all created with Hunyuan MT 7B or Qwen 3.5 9B at Q4_K_M (4 bit) quantization, and on my system via Vulkan on an AMD Radeon RX 9070 XT consumed no considerable processing resources for real-time processing and video memory corresponding to the size of the weights plus a small KV cache.

Censoring and safeguards

This all works well up to a certain point. Unfortunately most open weight LLMs come with safeguards built directly into the model weights instead of leaving it to the deployment layer. Depending on your input this might trigger:

Translating subtitles of an R-rated movie, or a documentary about certain historical events, can easily trigger these safeguards. This is apparently also a problem for more serious usage like for translation and summarization of court documents at the Swiss Federal Supreme Court. To get around that, it is possible to use uncensored / abliterated models that can be created with the help of e.g. Heretic or downloaded directly from Hugging Face (e.g. this Qwen 3.5 9B variant). Another option to get around such censorship are prompt injection patterns such as Prompt Injection as Role Confusion but they're less reliable.

Future work

The element is in a decent state and is usable as-is, but there are a couple of future extensions I would like to work on:

• Optimizing memory usage if the same model is used multiple times in the same process. The weights can be made to be in memory only once but right now every instance has a copy.

• Optimizing KV cache utilization. Once the history of past inputs/outputs runs full, currently the whole KV cache is discarded. With this PR now merged it should be possible to improve this significantly such that it is not necessary to re-process the whole prompt on each new input.

• Support for other llama.cpp elements that can work with different modalities, e.g. image or audio input and possibly combined with text input. This could, for example, be used to automatically generate scene descriptions from a low framerate video stream that is triggered on scene changes.

If any of these extensions would be useful to you, or if you have any questions, or are interested in other use-cases, please feel free to get in touch.

GStreamer's VideoToolbox-based decoder for macOS, iOS, and tvOS (vtdec) has recently gained support for decoding HEVC video with an alpha channel.

No special setup is required. When vtdec detects an HEVC+Alpha stream, it automatically negotiates an output format with alpha where possible. Existing pipelines should work without changes - if a non-alpha output format is selected, the alpha channel is replaced with a black background.

You can try it out yourself! Download this HEVC+Alpha sample file and run the following pipeline:

gst-launch-1.0 videotestsrc ! video/x-raw,width=1280,height=720 ! compositor name=comp ! autovideosink \
               filesrc location="phone-hevc-alpha.mp4" ! parsebin ! vtdec ! comp.

The sample video shows a phone rendered with a transparent background. In this pipeline, that background is replaced with the default videotestsrc SMPTE color bars. It should look something like this:

This functionality opens up simpler compositing workflows. You can place HEVC+Alpha assets like this one on top of other content for overlays or effects without needing extra processing like chroma keying.

It's worth noting that another codec for videos with an alpha channel - VP8/VP9+Alpha - is also well supported in GStreamer via our libvpx, VA-API, V4L2 and D3D12 decoders.

If you need to encode your own HEVC+Alpha assets on macOS, you can already do so using vtenc_h265a, which we added to GStreamer a while ago.

HEVC with Alpha decoding via vtdec will be available in the upcoming GStreamer 1.29.2 development snapshot - feel free to give it a go and let us know if you encounter any problems.

TL;DR

There is a new webrtcbin2 rust plugin containing split webrtcsend and webrtcrecv elements for handling a WebRTC session. The highlights of webrtcbin2 are that it uses less threads per session by using rtpsend and rtprecv (also rust), implementing DTLS handling internally, using librice (ICE in rust), and handling signalling all within an async runtime. All of these components also share threads with other instances of webrtcsend and webrtcrecv allowing for an even further reduction in the amount of resources significantly improving scalability.

The landscape

When I originally wrote the webrtcbin GStreamer element almost 10 years ago, I did not completely envision the number of users that would come to use this code in some way shape or form. From HTTP based standards such as WHIP, and WHEP and the myriad of projects that use WebRTC in some way for ingest or egress. WebRTC is still one of the best ways to transport live video into a web browser for display. WebRTC's loose compatibility with the SIP ecosystem is also a driving force behind WebRTC's continued use.

Now, webrtcbin has definitely proved itself in situations that require a small number of sessions. Using webrtcbin for a mixing server (MCU) or even SFU with hundreds or even thousands of streams in a single application is still a tall ask. The biggest reason for this is the number of threads that are created for every WebRTC session.

Threads

1. RTCP thread - rtpbin (used by webrtcbin) creates a thread per session essentially for handling timeouts required by RTCP.
2. rtpjitterbuffer creates a thread per incoming stream in order to be able to handle timeouts and deal with late or missing RTP packets.
3. dtlsenc - A thread whose sole purpose is for being able to handle DTLS timeouts.
4. webrtcbin and signalling - A dedicated thread for handling signalling related changes such as SDP generation, applying remote SDPs, handling ICE candidates, etc.
5. webrtcbin and ICE - ICE uses libnice on a dedicated ICE network thread per WebRTC session.
6. Media streaming threads - One streaming thread for sending and receiving media data.

When an application requires many WebRTC sessions, the memory requirements and context switching overhead of having 5 extra threads per WebRTC session can limit the number of sessions that can be concurrently executed.

Pipeline loops

Another concern I had is that for the server mixing/forwarding use case, pipeline loops were almost a necessity due to the basic requirement that participants in a WebRTC call wanting to be able to see and listen to each other. The obvious answer to this problem is to split the pipeline and use some wormhole elements such as appsrc/appsink, intersink/intersrc, proxysrc/proxysink, etc.

What if? - webrtcbin2

With the benefit of hindsight, we can definitely improve on this situation and reduce the number of threads that is required by each additional WebRTC session. Let us go through the list from above.

Pipeline loops

In order to solve the problem of loops in the pipeline, I took a leaf out of the design we made for rtpbin2 and created separate webrtcsend and webrtcrecv elements that interact with a shared WebRTC session object by having the same id. This allows data to flow essentially in one direction without requiring any kind of loop in the pipeline graph.

For some background on why rtpbin2 was created, you can have a look at a previous post I wrote.

Threads

1. RTCP thread - Amortised over multiple instances inside rtpbin2 using a tokio scheduler.
2. Jitter buffer per stream - rtprecv (part of rtpbin2) uses the same tokio scheduler for RTCP handling as it does for handling timeouts and packets through the jitterbuffer introducing no extra threads.
3. dtlsenc is no longer - DTLS is performed (using OpenSSL) directly just before/after ICE processing.
4. webrtcsend/webrtcrecv and signalling - Signalling occurs on a tokio runtime shared across all instances of webrtcsend/webrtcrecv.
5. webrtcsend/webrtcrecv and ICE - Uses librice on the same tokio runtime as webrtcsend/webrtcrecv.
6. Media streaming thread - Same as webrtcbin. Can be amortised by using the threadshare elements.

If we count the number of threads saved, we can see that for every WebRTC session, at least 5 threads are no longer needed in the new design. At 100 sessions, that is roughly a 500 thread saving in both memory and contention.

Features of webrtcsend/webrtcrecv

While webrtcsend and webrtcrecv are functional and can successfully communicate with a web browser such as Chrome or Firefox, there are still some missing pieces. Some of the supported features include:

• Audio and/or Video streaming. Data channels are not currently supported.
• BUNDLE is supported and required for multiple media.
• rtcp-mux is required.

A non exhaustive list of not yet supported features include:

• Retransmissions and Forward Error Correction (rtpbin2 does not support this yet).
• Data channels
• Renegotiation
• Statistics
• TURN servers (librice supports it but not yet implemented in webrtcbin2)

All of these missing features are solveable with further implementation effort.

Example

A send and receive example using webrtcbin2 is available the upstream repository and can be used with this example web page. Just make sure that data channels are not enabled as they are currently not supported.

Closing

This work will be part of the upcoming GStreamer 1.29.2 development snapshot or can be built from the main branch of gst-plugins-rs.

Writing a mature WebRTC implementation is an endeavour that requires a fair bit of implementation effort to complete. If you would like to help make a secure, mature WebRTC implementation for GStreamer please get in touch.

New udpsrc2 element

Over the past few years, I have worked on a new GStreamer UDP source element. This is finally merged now and will be part of both the GStreamer 1.30.0 release and the gst-plugins-rs 0.16.0 release.

The old element uses GIO for networking, which is quite inefficient by design. The new implementation uses about 50% less CPU on my machine compared to the old element for a 3 Gbit/s stream.

As can be seen from the docs of the new element, it preserves the API of the old element. As such it should generally be possible to use it as a drop-in replacement.

In addition to performance improvements, the new element also includes various other improvements:

• Support for faster packet receiving via Generic Receive Offload (GRO) on Linux, and for using recvmmsg() on platforms where it is available to significantly improve receive performance.

• Complete support for multicast source filtering, including negative filters, and support for platforms that do not have APIs for the IGMPv3 SSM mechanism.

• Always obtaining kernel-side packet receive times if available, which was opt-in in the old element due to GIO performance issues with socket control messages.

• New preserve-packetization property that allows outputting multiple packets in the same buffer, which improves performance for formats like MPEG-TS where the UDP packetization is not necessary.

Give it a try with your pipelines and workloads and share your feedback or any issues you encounter.

In the future, io_uring support on Linux could be added for even better receive performance.

SMPTE ST2110 capture

While udpsrc2 is an improvement in general, its primary motivation is better SMPTE ST2110 support in GStreamer. The old element could not handle the packet rates typically used for such streams very well.

ST2110 defines a UDP/RTP-based set of standards for transmitting raw or very-high bitrate audio / video / ancillary data over Ethernet. It is intended as a replacement for SDI.

Related to this, we recently also merged some other improvements:

• A new, improved raw video RTP depayloader that supports ST2110-20.

• A new, improved raw PCM audio RTP depayloader that supports ST2110-30.

• An ST291 ancillary data RTP depayloader that supports ST2110-40.

• Various improvements to the rtprecv element, especially for performance and handling of high-packet rate streams.

For all the new depayloaders there are also new, improved implementations of the corresponding payloaders available.

Together, these improvements enable reliable ST2110 stream capture in GStreamer.

An example pipeline putting it all together would look as follows:

$ gst-launch-1.0 \
    \ # Video capture pipeline part
    udpsrc2 address=239.255.64.20 port=16388 multicast-iface=enp15s0 buffer-size=20000000 caps='application/x-rtp, media=video, payload=96, clock-rate=90000, encoding-name=RAW, sampling=YCbCr-4:2:2, depth=10, width=1920, height=1080, exactframerate=60, colorimetry=BT709, pm=2110GPM, ssn=ST2110-20:2017, tp=2110TPN, a-sendonly="", a-ts-refclk="ptp=IEEE1588-2008:7C-2E-0D-FF-FE-1C-81-14:127", a-mediaclk="direct=0", ssrc-327995485-cname=E055FF0F3D6E4B349F7B786D8B6C837B' ! \
      rtprecv latency=0 ! queue max-size-bytes=0 max-size-buffers=0 max-size-time=500000000 ! rtpvrawdepay2 ! \
    \
    \ # Ancillary data capture pipeline part
    udpsrc2 address=239.255.64.20 port=16386 multicast-iface=enp15s0 buffer-size=20000000 caps='application/x-rtp, media=video, payload=98, clock-rate=90000, encoding-name=SMPTE291, vpid_code=138, a-sendonly="", a-ts-refclk="ptp=IEEE1588-2008:7C-2E-0D-FF-FE-1C-81-14:127", a-mediaclk="direct=0", ssrc-2672978631-cname=E055FF0F3D6E4B349F7B786D8B6C837B' ! \
      rtprecv latency=0 ! queue max-size-bytes=0 max-size-buffers=0 max-size-time=500000000 ! rtpsmpte291depay ! combiner.st2038 \
    \
    \ # Combination of video and ancillary data streams and output
    st2038combiner name=combiner start-time-selection=first ! videoconvert ! queue max-size-bytes=0 max-size-time=0 max-size-buffers=3 ! autovideosink \
    \
    \ # Audio capture and output pipeline part
    udpsrc2 address=239.255.64.20 port=16384 multicast-iface=enp15s0 buffer-size=20000000 caps='application/x-rtp, media=audio, payload=(int)97, clock-rate=48000, encoding-name=(string)L24, encoding-params=64, a-sendonly="", a-ptime=0.125, a-ts-refclk="ptp\=IEEE1588-2008:7C-2E-0D-FF-FE-1C-81-14:127", a-mediaclk="direct\=0", ssrc-603238248-cname=(string)E055FF0F3D6E4B349F7B786D8B6C837B' \
      rtprecv latency=0 ! queue max-size-bytes=0 max-size-buffers=0 max-size-time=500000000 ! rtpL24depay2 ! audioconvert ! autoaudiosink

This pipeline receives a 1080p60 4:2:2 YUV 10-bit video stream, ST291 ancillary data, and a 24-bit 48kHz 64-channel PCM audio stream. The video and ancillary data are combined to a single stream, and then both the combined video-ancillary stream and the audio are output.

rtprecv is used here for translating packet capture timestamps and RTP header timestamps to consistent GStreamer timestamps.

Ancillary data

The pipeline above captures all three streams and merges the ancillary data stream with the video. The ancillary data itself is not processed further.

One way to process the ancillary data further is to extract ST12 timecodes from it and overlay them over the video.

For this, insert the following elements before the video sink:

 ... ! timecodestamper source=ancillary-meta ancillary-meta-locations='8:2000,570:2000' \
     ! videoconvert ! timeoverlay time-mode=time-code \
     ! autovideosink

Here timecodes from ancillary data at positions (8,2000) and (570,2000) would be extracted and converted to GstVideoTimeCodeMeta on the video buffers.

We recently added support for extracting ST12 timecodes from ancillary meta as well.

The positions depend on the video signal standard in use and can be found in the ST12 specifications.

We've been hard at work doing numerous small and large improvements to GStreamer for people who want to target Apple platforms: macOS, iOS, and tvOS.

iOS ARM64 Simulator Support via an XCFramework

With the GStreamer 1.28.0 release, the project now releases an XCFramework for iOS. As expected, this XCFramework supports iOS arm64, iOS Simulator x86_64, and iOS Simulator arm64. The legacy iOS framework that lipo-ed iOS arm64 and iOS Simulator x86_64 is now deprecated, and will be removed in a future release.

You can download the XCFramework from the official download page.

Thanks to Amy for helping me with this!

tvOS Support

As of version 1.28.1, GStreamer officially supports tvOS, and binaries for it are shipped as part of the iOS XCFramework. This means that the GStreamer 1.28.1 iOS XCFramework contains: ios-arm64, ios-arm64_x86_64-simulator, tvos-arm64, tvos-arm64_x86_64-simulator.

Most of the relevant Apple-specific plugins are supported:

• osxaudio: Audio source/sink, using CoreAudio
• atdec: Audio decoder, using AudioToolbox
• atenc: Audio encoder, using AudioToolbox
• vtdec: Video decoder, using VideoToolbox
• vtenc: Video encoder, using VideoToolbox
• glimagesink: Deprecated EAGL video sink
• vulkansink: Metal-based video sink, using MoltenVK
• vulkancolorconvert: Metal-accelerated video conversion, using MoltenVK
• vulkanoverlaycompositor: Metal-accelerated video overlay compositor, using MoltenVK
• ... more Metal/Vulkan elements

Two elements that use AVCaptureDevice had to be disabled because they need more work to support tvOS:

• avfvideosrc: Video capture source, using AVFoundation
• avfdeviceprovider: Video capture device provider, using AVFoundation

Thanks to Remote Studio for sponsoring this work!

Improved support for using Rust plugins on Apple platforms

Linking more than one Rust plugin into your app had been broken on macOS and iOS for some time. The fix for that requires prelinking, which Amy has written about previously, but it couldn't be enabled on macOS due to some LLVM/LLD issues. We had to wait for the fixes to percolate down to a Rust toolchain release. That finally happened in Rust 1.93, but by that time a new problem had cropped up: Xcode 26.

Due to some toolchain changes in Xcode 26, linking Rust plugins was failing on macOS and also on iOS with the legacy framework. After weighing all the options, the best solution was to add -no_compact_unwind to the linker flags on macOS, and direct people to use the XCFramework when using Rust plugins on iOS.

This is now added automatically if you use pkg-config (using CMake or Meson, for example), but if you're using a plain Xcode project, you need to add -no_compact_unwind manually to linker flags in Xcode.

This fix will be available in the upcoming 1.28.3 release.

Many more macOS, iOS, tvOS improvements

Contributors have been hard at work with small and large improvements to the Apple-specific elements in GStreamer. Ranging from AV1 and VP9 decoding support in vtdec to better debug info, bugfixes, memory leak fixes, crash fixes, and much more. The patches are too many to list or even link!

GStreamer has shipped binaries for all the major platforms for many years now: Windows, Android, macOS, iOS. Linux packages are, of course, handled by all the various distros.

However, if you wanted to use the Python bindings on macOS or Windows, you had to jump through hoops. Till now. GStreamer 1.28.0 ships Python wheels supporting Python 3.9, 3.10, 3.11, 3.12, 3.13, 3.14 on macOS (GIL) and Windows (GIL and free-threading). All you need to do is to run:

python3 -m pip install gstreamer-bundle==1.28.0

And that's it! You will have a complete GStreamer install, with all the plugins you expect on macOS and Windows, and all utilities including gst-launch-1.0 gst-inspect-1.0 gst-device-monitor-1.0 ges-launch-1.0 and so on.

The gstreamer-bundle package is a complete distribution, so it will pull in all the plugins, libraries, cmd-line tools, etc. If you want to depend on a more minimal GStreamer installation or you want to avoid pulling in GPL or known-patent-encumbered ("restricted") plugins, you can use the gstreamer-meta package. That puts plugins behind "extras" like gpl cli restricted gtk4 etc.

Many thanks to Pollen Robotics for sponsoring this work. The Reachy Mini companion robot by Pollen Robotics/Hugging Face uses GStreamer via the Python bindings and is the first production user of these wheels!

We're very excited to see more people make use of these wheels.

Read on for technical details on how all this was accomplished.

Step 1: Ship Python bindings via introspection on macOS and Windows

After many years, Python bindings support was re-introduced in GStreamer 1.26 and was shipped with the installers on macOS and Windows. This required significant work:

• Re-introduce gobject-introspection support
• Re-introduce Python bindings support and ship it on macOS + Windows
• Load typelibs relocatably

Thanks to Amy for doing the bulk of the work here, and to everyone else who contributed towards this over the years: Andoni, Nacho, Thibault, Tjitte, and more that I'm sure I've missed.

Step 2: Build wheels for all supported Python versions

When shipping Python bindings for C libraries, it is necessary to also ship the accompanying libraries and plugins, lest ABI mismatches and incompatibilities arise. That's why the wheels we ship constitute a complete GStreamer distribution, including all plugin dependencies such as GTK4. This means you also have Python bindings for GTK4 available on macOS and Windows.

This wasn't easy to accomplish, especially because PyGObject doesn't use the limited Python C API. That means we can't just build for Python 3.9 and call it a day. We need separate wheels for each Python version × target.

The count goes something like this:

• We split the gstreamer libraries, plugins, and dependencies across 11 wheels
• We support 16 Python versions: 3.9 3.10 3.11 3.12 3.13 3.13t 3.14 3.14t
• And 3 platforms: macOS universal, Windows MSVC x86_64, Windows MSVC x86

That's 11 × 16 × 3 = 528 wheels. That is absolutely untenable!

So we have to do some chicanery to trim that down:

1. Put everything that links to or loads Python in one wheel called gstreamer_python, so that everything else is agnostic to the Python version being used
2. Override py_limited_api to be cp39 for all agnostic wheels and mark them as not containing ext modules
3. Rebuild the recipes responsible for generating libraries or plugins that go into gstreamer_python with each Python version we need to support
4. On macOS, override plat_name to be macosx_10_13_universal2 for all agnostic wheels even if the Python version we're using doesn't support macOS 10.13, so that they can be reused across all Python versions

That brings us down to 92 wheels. Still quite a lot, but now it's a manageable number!

The long-term solution is to port PyGObject over to the Limited Python C API—which is quite a big undertaking—but should allow us to skip most of this for Python >=3.12.

Thanks to Amy once again for doing most of the work to make this possible, and to Pollen Robotics for sponsoring us to do it. Here are the relevant merge requests:

• Support for Python wheel packaging
• Build separate wheels for each Python version
• and there were a few more fixes

Step 3: Linux support

You may have noticed that there was no mention of wheels targeting Linux. That's a much harder problem to solve than shipping on macOS or Windows, so we had to punt it for a later release, likely one of the 1.28.x stable releases.

We're planning to target manylinux_2_28 and support Python 3.9+, but there are still unknowns that could throw a spanner in our plans. For instance:

• GStreamer often utilizes subtle characteristics of the Linux graphics stack for good performance, which may break by targeting such an old base.
• The difference in library versions shipped with the wheels vs on the system may cause subtle or catastrophic breakage in apps that also load system libraries.

We're hoping that we can overcome all this and ship something that allows users on any Linux distro to get a functional GStreamer just by doing pip install gstreamer-bundle.

In the meantime, please continue to use the distro-provided GStreamer packages and Python bindings, and if they're missing plugins or are too old, please contact your distro maintainer(s).