Devlog

At the GStreamer project, we produce SDKs for lots of platforms: Linux, Android, macOS, iOS, and Windows. However, as we port more and more plugins to Rust 🦀, we are finding ourselves backed into a corner.

Rust static libraries are simply too big.

To give you an example, the AWS folks changed their SDK back in March to switch their cryptographic toolkit over to their aws-lc-rs crate ^[1]. However, that causes a 2-10x increase in code size (bug reports here and here), which gets duplicated on every plugin that makes use of their ecosystem!

What are Rust staticlibs made of?

To summarise, each Rust plugin packs a copy of its dependencies, plus a copy of the Rust standard library. This is not a problem on shared libraries and executables by their very nature, but on static libraries it causes several issues:

• Rust leaks unexported symbols from native staticlibs
• On some platform, linking against multiple Rust staticlibs is impossible

First approach: Single-Object Prelinking

I won't bore you with the details as I've written another blog post on the subject; the gist is that you can unpack the library, and then ask the linker to perform "partial linking" or "relocatable linking" (Linux term) or "Single-Object Prelinking" (the Apple term, which I'll use throughout the post) over the object files. Setting which symbols you want to be visible for downstream consumers lets dead-code elimination take place at the plugin level, ensuring your libraries are now back to a reasonable size.

Why is it not enough?

Single-Object Prelinking has two drawbacks:

• Unoptimized code: the linker won't be able to deduplicate functions between melded objects, as they've been hidden by the prelinking process.
• Windows: there are no officially supported tools (read: Visual Studio, LLVM, GCC) to perform this at the compiler level. It is possible to do this with binutils, but the PE-COFF format doesn't allow to change the visibility of unexported functions.

Melt all the object files with the power of dragons' fire!

As said earlier, no tools on Windows support prelinking officially yet, but there's another thing we can do: library deduplication.

Thanks to Rust's comprehensive crate ecosystem, I wrote a new CLI tool which I called dragonfire. Given a complete Rust workspace or list of static libraries, dragonfire:

1. reads all the static libraries in one pass
2. deduplicates the object files inside them based on their size and naming (Rust has its own, unique naming convention for object files -- pretty useful!)
3. copies the duplicate objects into a new static library (usually called gstrsworkspace as its primary use is for the GStreamer ecosystem)
4. removes the duplicates from the rest of the libraries
5. updates the symbol table in each of the libraries with the bundled LLVM tools

Thanks to the ar crate, the unpacking and writing only happens at stage 3, ensuring no wasteful I/O slowdowns takes place. The llvm-tools-preview component in turn takes care of locating and calling up llvm-ar for updating the workspace's symbol tables.

A special mention is deserved to the object files' naming convention. Assume a Rust staticlib named libfoo, its object files will be named as:

• crate_name-hash1.crate_name.hash2-cgu.nnn.rcgu.o
• On Windows only: foo.crate_name-hash1.crate_name.hash2-cgu.nnn.rcgu.o
• On non-Windows platforms: same as above, but replacing foo with libfoo-hash

In all cases, crate_name means a dependency present somewhere in the workspace tree, and nnn is a number that will be bigger than zero whenever -C codegen-units was set to higher than 1.

For dragonfire purposes, dropping the library prefix is enough to be able to deduplicate object files; however, on Windows we can also find import library stubs, which LLVM can generate on its own by the use of the #[raw-dylib] annotation ^[2]. Import stubs can have any extension, e.g. .dll, .exe and .sys (the latter two coming from private Win32 APIs). These stubs cannot be deduplicated as they are generated individually per imported function, so dragonfire must preserve them where they are.

Drawbacks of object file deduplication

Again there are several disadvantages of this approach. On Apple platforms, deduplicating libraries triggers a strange linker error, which I've not seen before:

ld: multiple errors: compact unwind must have at least 1 fixup in '<framework>/GStreamer[arm64][1021](libgstrsworkspace_a-3f2b47962471807d-lse_ldset4_acq.o)'; r_symbolnum=-19 out of range in '<framework>/GStreamer[arm64][1022](libgstrsworkspace_a-compiler_builtins-350c23344d78cfbc.compiler_builtins.5e126dca1f5284a9-cgu.162.rcgu.o)'

This also led me to find that Rust libraries were packing bitcode, which is forbidden by Apple. (This was thankfully already fixed before shipping time, but we've not yet updated our Rust minimum version to take advantage of it.)

Another drawback is that Rust's implementation of LTO causes dead-code elimination at the crate level, as opposed to the workspace level. This makes object file deduplication impossible, as each copy is different.

For the Windows platform, there is an extra drawback which affects specifically object files produced by LLVM: the COMDAT sections are set to IMAGE_COMDAT_SELECT_NODUPLICATES. This means that the linker will outright reject functions with multiple definitions, rather than realise they're all duplicates and discarding all but one of the copies. MSVC in particular performs symbol resolution before dead-code elimination. This means that linking will fail because of unresolved symbols before dead code elimination kicks in; to use deduplicated libraries, one must set the linker flags /OPT:REF /FORCE:UNRESOLVED to ensure the dead code can be successfully eliminated.

Results

With library deduplication, we can make static libraries up to 44x smaller when building under MSVC ^[3] (you can expand the tables below for the full comparison):

• gstaws.lib: from 173M to 71M (~2.5x)
• gstrswebrtc.lib: from 193M to 66M (~2.9x)
• gstwebrtchttp.lib: from 66M to 1,5M (~ 44x)

Table: before and after melding under MSVC

file	no prelinking	melded
gstaws.lib	173M	71M
gstcdg.lib	36M	572K
gstclaxon.lib	32M	568K
gstdav1d.lib	34M	936K
gstelevenlabs.lib	59M	1008K
gstfallbackswitch.lib	37M	2,3M
gstffv1.lib	34M	744K
gstfmp4.lib	39M	3,2M
gstgif.lib	34M	1,1M
gstgopbuffer.lib	30M	456K
gsthlsmultivariantsink.lib	46M	1,6M
gsthlssink3.lib	41M	1,2M
gsthsv.lib	34M	796K
gstjson.lib	31M	704K
gstlewton.lib	33M	1,2M
gstlivesync.lib	33M	728K
gstmp4.lib	38M	2,2M
gstmpegtslive.lib	31M	704K
gstndi.lib	38M	2,8M
gstoriginalbuffer.lib	34M	376K
gstquinn.lib	75M	23M
gstraptorq.lib	33M	2,4M
gstrav1e.lib	46M	11M
gstregex.lib	38M	404K
gstreqwest.lib	58M	1,4M
gstrsanalytics.lib	35M	1000K
gstrsaudiofx.lib	54M	22M
gstrsclosedcaption.lib	52M	8,4M
gstrsinter.lib	35M	604K
gstrsonvif.lib	46M	2,0M
gstrspng.lib	35M	1,2M
gstrsrtp.lib	59M	11M
gstrsrtsp.lib	57M	4,4M
gstrstracers.lib	40M	2,4M
gstrsvideofx.lib	48M	11M
gstrswebrtc.lib	193M	66M
gstrsworkspace.lib	N/A	137M
gststreamgrouper.lib	30M	376K
gsttextahead.lib	30M	332K
gsttextwrap.lib	32M	2,1M
gstthreadshare.lib	52M	12M
gsttogglerecord.lib	35M	808K
gsturiplaylistbin.lib	31M	648K
gstvvdec.lib	34M	564K
gstwebrtchttp.lib	66M	1,5M

The results from the melding above can be compared with the file sizes obtained using LTO on Windows ^[4] (remember it doesn't actually fix linking against plugins):

• gstaws.lib: from 71M (LTO) to 67M (melded) (-5.6%)
• gstrswebrtc.lib: from 105M to 66M (-37.1%)
• gstwebrtchttp.lib: from 28M to 1,5M (-94.6%)

Table: before and after LTO under MSVC (no melding involved)

file (codegen-units=1 in all cases)	no prelinking	lto=thin	opt-level=s + lto=thin	debug=1 + opt-level=s	debug=1 + lto=thin + opt-level=s
old/gstaws.lib	199M	199M	171M	78M	67M
old/gstcdg.lib	11M	11M	11M	7,5M	7,5M
old/gstclaxon.lib	11M	11M	11M	7,7M	7,7M
old/gstdav1d.lib	12M	12M	12M	7,9M	7,8M
old/gstelevenlabs.lib	52M	52M	49M	24M	22M
old/gstfallbackswitch.lib	18M	18M	17M	11M	11M
old/gstffv1.lib	11M	11M	11M	7,6M	7,6M
old/gstfmp4.lib	20M	20M	19M	12M	11M
old/gstgif.lib	12M	12M	12M	7,9M	7,9M
old/gstgopbuffer.lib	9,7M	9,7M	9,7M	7,5M	7,4M
old/gsthlsmultivariantsink.lib	16M	16M	16M	9,6M	9,4M
old/gsthlssink3.lib	14M	14M	14M	8,9M	8,8M
old/gsthsv.lib	11M	11M	11M	7,8M	7,7M
old/gstjson.lib	12M	12M	12M	8,4M	8,2M
old/gstlewton.lib	12M	12M	12M	8,1M	8,1M
old/gstlivesync.lib	12M	12M	12M	8,3M	8,2M
old/gstmp4.lib	17M	17M	17M	9,9M	9,7M
old/gstmpegtslive.lib	12M	12M	12M	8,0M	7,9M
old/gstndi.lib	21M	21M	20M	12M	11M
old/gstoriginalbuffer.lib	9,6M	9,6M	9,7M	7,4M	7,3M
old/gstquinn.lib	94M	94M	86M	39M	35M
old/gstraptorq.lib	18M	18M	17M	9,8M	9,4M
old/gstrav1e.lib	39M	39M	37M	19M	18M
old/gstregex.lib	26M	26M	25M	14M	14M
old/gstreqwest.lib	53M	53M	49M	24M	22M
old/gstrsanalytics.lib	15M	15M	14M	9,2M	8,9M
old/gstrsaudiofx.lib	57M	57M	56M	23M	22M
old/gstrsclosedcaption.lib	40M	40M	36M	20M	18M
old/gstrsinter.lib	14M	14M	13M	8,5M	8,4M
old/gstrsonvif.lib	21M	21M	20M	11M	11M
old/gstrspng.lib	13M	13M	13M	8,2M	8,2M
old/gstrsrtp.lib	47M	47M	44M	22M	20M
old/gstrsrtsp.lib	35M	35M	33M	16M	15M
old/gstrstracers.lib	28M	28M	27M	16M	15M
old/gstrsvideofx.lib	16M	16M	35M	9,2M	15M
old/gstrswebrtc.lib	329M	329M	284M	124M	105M
old/gststreamgrouper.lib	9,6M	9,6M	9,7M	7,2M	7,2M
old/gsttextahead.lib	9,6M	9,6M	9,5M	7,4M	7,3M
old/gsttextwrap.lib	13M	13M	13M	8,4M	8,4M
old/gstthreadshare.lib	49M	49M	45M	23M	20M
old/gsttogglerecord.lib	13M	13M	13M	8,5M	8,4M
old/gsturiplaylistbin.lib	11M	11M	11M	7,9M	7,9M
old/gstvvdec.lib	11M	11M	11M	7,5M	7,5M
old/gstwebrtchttp.lib	69M	69M	63M	30M	28M

Conclusion

This article presents several longstanding pain points in Rust, namely staticlib binary sizes, symbol leaking, and incompatibilities between Rust and MSVC. I demonstrate the tool dragonfire that aims to address and work around, where possible, these issues, along with remaining issues to be addressed.

As explained earlier, dragonfire treated libraries are live on all platforms except Apple's, if you use the development packages from mainline; it's on track hopefully for the 1.28 release of GStreamer. There's already a merge request pending to enable it for Apple platforms, we're only waiting to update the Rust mininum version.

If you want to have a look, dragonfire's source code is available at Freedesktop's GitLab instance. Please note that at the moment I have no plans to submit this to crates.io.

Feel free to contact me with any feedback, and thanks for reading!

---

1. See its default-https-client feature at lib.rs, you will find it throughout the AWS SDK ecosystem. ↩︎

2. https://doc.rust-lang.org/reference/items/external-blocks.html#dylib-versus-raw-dylib ↩︎

3. In all cases the -C flags are debug=1 + codegen-units=1 + opt-level=s; see this comment for the complete results across all platforms. ↩︎

4. Source: https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1895 ↩︎

Over the past few years, we've been slowly working on improving the platform-specific plugins for Windows, macOS, iOS, and Android, and making them work as well as the equivalent plugins on Linux. In this episode, we will look at audio device switching in the source and sink elements on macOS and Windows.

On Linux, if you're using the PulseAudio elements (both with the PulseAudio daemon and PipeWire), you get perfect device switching: quick, seamless, easy, and reliable. Simply set the device property whenever you want and you're off to the races. If the device gets unplugged, the pipeline will continue, and you will get notified of the unplug via the GST_MESSAGE_DEVICE_REMOVED bus message from GstDeviceMonitor so you can change the device.

As of a few weeks ago, the Windows Audio plugin wasapi2 implements the same behaviour. All you have to do is set the device property to whatever device you want (fetched using the GstDeviceMonitor API), at any time.

A merge request is open for adding the same feature to the macOS audio plugin, and is expected to be merged soon.

For graceful error handling, such as accidental device unplug or other unexpected errors, there's a new continue-on-error property. Setting that will cause the source to output silence after unplug, whereas the sink will simply discard the buffers. An element warning will be emitted to notify the app (alongside the GST_MESSAGE_DEVICE_REMOVED bus message if there was a hardware unplug), and the app can switch the device by setting the device property.

Thanks to Seungha and Piotr for working on this!

HIP (formerly known as Heterogeneous-computing Interface for Portability) is AMD’s GPU programming API that enables portable, CUDA-like development across both AMD and NVIDIA platforms.

• On AMD GPUs, HIP runs natively via the ROCm stack.
• On NVIDIA GPUs, HIP operates as a thin translation layer over the CUDA runtime and driver APIs.

This allows developers to maintain a single codebase that can target multiple GPU vendors with minimal effort.

Where HIP Is Used

HIP has seen adoption in AMD-focused GPU computing workflows, particularly in environments that require CUDA-like programmability. Examples include:

• PyTorch ROCm backend for deep learning workloads
• Select scientific applications like LAMMPS and GROMACS have experimented with HIP backends for AMD GPU support
• GPU-accelerated media processing on systems that leverage AMD hardware

While HIP adoption has been more limited compared to CUDA, its reach continues to expand as support for AMD GPUs grows across a broader range of use cases.

The Challenge: Compile-Time Platform Lock-in

Despite its cross-vendor goal, HIP still has a fundamental constraint at the build level. As of HIP 6.3, HIP requires developers to statically define their target platform at compile time via macros like:

#define __HIP_PLATFORM_AMD__    // for AMD ROCm
#define __HIP_PLATFORM_NVIDIA__ // for CUDA backend

This leads to two key limitations:

• You must compile separate binaries for AMD and NVIDIA
• A single binary cannot support both platforms simultaneously
• HIP does not support runtime backend switching natively

GstHip’s Solution

To overcome this limitation, GstHip uses runtime backend dispatch through:

• dlopen() on Linux
• LoadLibrary() on Windows

Instead of statically linking against a single HIP backend, GstHip loads both the ROCm HIP runtime and the CUDA driver/runtime API at runtime. This makes it possible to:

• Detect available GPUs dynamically
• Choose the appropriate backend per device
• Even support simultaneous use of AMD and NVIDIA GPUs in the same process

Unified Wrapper API

GstHip provides a clean wrapper layer that abstracts backend-specific APIs via a consistent naming scheme:

hipError_t HipFooBar(GstHipVendor vendor, ...);

The Hip prefix (capital H) clearly distinguishes the wrapper from native hipFooBar(...) functions. The GstHipVendor enum indicates which backend to target:

• GST_HIP_VENDOR_AMD
• GST_HIP_VENDOR_NVIDIA

Internally, each HipFooBar(...) function dispatches to the correct backend by calling either:

• hipFooBar(...) for AMD ROCm
• cudaFooBar(...) for NVIDIA CUDA

These symbols are dynamically resolved via dlopen() / LoadLibrary(), enabling runtime backend selection without GPU vendor-specific builds.

Memory Interop

All memory interop in GstHip is handled through the hipupload and hipdownload elements. While zero-copy is not supported due to backend-specific resource management and ownership ambiguity, GstHip provides optimized memory transfers between systems:

• System Memory ↔ HIP Memory: Utilizes HIP pinned memory to achieve fast upload/download operations between host and device memory
• GstGL ↔ GstHip: Uses HIP resource interop APIs to perform GPU-to-GPU memory copies between OpenGL and HIP memory
• GstCUDA ↔ GstHip (on NVIDIA platforms): Since both sides use CUDA memory, direct GPU-to-GPU memory copies are performed using CUDA APIs.

GPU-Accelerated Filter Elements

GstHip includes GPU-accelerated filters optimized for real-time media processing:

• hipconvertscale/hipconvert/hipscale: Image format conversion and image scaling
• hipcompositor: composing multiple video streams into single video stream

These filters use the same unified dispatch system and are compatible with both AMD and NVIDIA platforms.

Application Integration Support

As of Merge Request!9340, GstHip exposes public APIs that allow applications to access HIP resources managed by GStreamer. This also enables applications to implement custom GstHIP-based plugins using the same underlying infrastructure without duplicating resource management.

Summary of GstHip Advantages

• Single plugin/library binary supports both AMD and NVIDIA GPUs
• Compatible with Linux and Windows
• Supports multi-GPU systems, including hybrid AMD + NVIDIA configurations
• Seamless memory interop with System Memory, GstGL, and GstCUDA
• Provides high-performance GPU filters for video processing
• Maintains a clean API layer via HipFooBar(...) wrappers, enabling backend-neutral development

For a project recently it was necessary to collect video frames of multiple streams during a specific interval, and in the future also audio, to pass it through an inference framework for extracting additional metadata from the media and attaching it to the frames.

While GStreamer has gained quite a bit of infrastructure in the past years for machine learning use-cases in the analytics library, there was nothing for this specific use-case yet.

As part of solving this, I proposed as design for a generic interface that allows combining and batching multiple streams into a single one by using empty buffers with a GstMeta that contains the buffers of the original streams, and caps that include the caps of the original streams and allow format negotiation in the pipeline to work as usual.

While this covers my specific use case of combining multiple streams, it should be generic enough to also handle other cases that came up during the discussions.

In addition I wrote two new elements, analyticscombiner and analyticssplitter, that make use of this new API for combining and batching multiple streams in a generic, media-agnostic way over specific time intervals, and later splitting it out again into the original streams. The combiner can be configured to collect all media in the time interval, or only the first or last.

Conceptually the combiner element is similar to NVIDIA's DeepStream nvstreammux element, and in the future it should be possible to write a translation layer between the GStreamer analytics library and DeepStream.

The basic idea for the usage of these elements is to have a pipeline like

-- stream 1 --\                                                                  / -- stream 1 with metadata --
               -- analyticscombiner -- inference elements -- analyticssplitter --
-- stream 2 --/                                                                  \ -- stream 2 with metadata --
   ........                                                                           ......................
-- stream N -/                                                                     \- stream N with metadata --

The inference elements would only add additional metadata to each of the buffers, which can then be made use of further downstream in the pipeline for operations like overlays or blurring specific areas of the frames.

In the future there are likely going to be more batching elements for specific stream types, operating on multiple or a single stream, or making use of completely different batching strategies.

Special thanks also to Olivier and Daniel who provided very useful feedback during the review of the two merge requests.

With GStreamer 1.26, a new D3D12 backend GstD3D12 public library was introduced in gst-plugins-bad.

Now, with the new gstreamer-d3d12 rust crate, Rust can finally access the Windows-native GPU feature written in GStreamer in a safe and idiomatic way.

What You Get with GStreamer D3D12 Support in Rust

• Pass D3D12 textures created by your Rust application directly into GStreamer pipelines without data copying
• Likewise, GStreamer-generated GPU resources (such as frames decoded by D3D12 decoders) can be accessed directly in your Rust app
• GstD3D12 base GStreamer element can be written in Rust

Beyond Pipelines: General D3D12 Utility Layer

GstD3D12 is not limited to multimedia pipelines. It also acts as a convenient D3D12 runtime utility, providing:

• GPU resource pooling such as command allocator and descriptor heap, to reduce overhead and improve reuse
• Abstractions for creating and recycling GPU textures with consistent lifetime tracking
• Command queue and fence management helpers, greatly simplifying GPU/CPU sync
• A foundation for building custom GPU workflows in Rust, with or without the full GStreamer pipeline

As part of the GStreamer Hackfest in Nice, France I had some time to go through some outstanding GStreamer issues. One such issue that has been on my mind was this GStreamer OpenGL Wayland issue.

Now, the issue is that OpenGL is an old API and did not have some of the platform extensions it does today. As a result, most windowing system APIs allow creating an output surface (or a window) but never showing it. This also works just fine when you are creating an OpenGL context but not actually rendering anything to the screen and this approach is what is used by all of the other major OpenGL platforms (Windows, macOS, X11, etc) supported by GStreamer.

When wayland initially arrived, this was not the case. A wayland surface could be the back buffer (an OpenGL term for rendering to a surface) but could not be hidden. This is very different from how other windowing APIs worked at the time. As a result, the initial implementation using Wayland within GStreamer OpenGL used some heuristics for determining when a wayland surface would be created and used that basically boiled down to, if there is no shared OpenGL context, then create a window.

This heuristic obviously breaks in multiple different ways, the two most obvious being:

1. gltestsrc ! gldownload ! some-non-gl-sink - there should be no surface used here.
2. gltestsrc ! glimagesink gltestsrc ! glimagesink - there should be two output surfaces used here.

The good news is that issue is now fixed by adding some API that glimagesink can use to notify that it would like an output surface. This has been implemented in this merge request and will be part of GStreamer 1.28.

JPEG XS is a visually lossless, low-latency, intra-only video codec for video production workflows, standardised in ISO/IEC 21122.

A few months ago we added support for JPEG XS encoding and decoding in GStreamer, alongside MPEG-TS container support.

This initially covered progressive scan only though.

Unfortunately interlaced scan, which harks back to the days when TVs had cathode ray tube displays, is still quite common, especially in the broadcasting industry, so it was only a matter of time until support for that would be needed as well.

Long story short, GStreamer can now (with this pending Merge Request) also encode and decode interlaced video into/from JPEG XS.

When putting JPEG XS into MPEG-TS, the individual fields are actually coded separately, so there are two JPEG XS code streams per frame. Inside GStreamer pipelines interlaced raw video can be carried in multiple ways, but the most common one is an "interleaved" image, where the two fields are interleaved row by row, and this is also what capture cards such as AJA or Decklink Blackmagic produce in GStreamer.

When encoding interlaced video in this representation, we need to go twice over each frame and feed every second row of pixels to the underlying SVT JPEG XS encoder which itself is not aware of the interlaced nature of the video content. We do this by specifying double the usual stride as rowstride. This works fine, but unearthed some minor issues with the size checks on the codec side, for which we filed a pull request.

Please give it a spin, and let us know if you have any questions or are interested in additional container mappings such as MP4 or MXF, or RTP payloaders / depayloaders.

Some time ago, Edward and I wrote a new element that allows clocking a GStreamer pipeline from an MPEG-TS stream, for example received via SRT.

This new element, mpegtslivesrc, wraps around any existing live source element, e.g. udpsrc or srtsrc, and provides a GStreamer clock that approximates the sender's clock. By making use of this clock as pipeline clock, it is possible to run the whole pipeline at the same speed as the sender is producing the stream and without having to implement any kind of clock drift mechanism like skewing or resampling. Without this it is necessary currently to adjust the timestamps of media coming out of GStreamer's tsdemux element, which is problematic if accurate timestamps are necessary or the stream should be stored to a file, e.g. a 25fps stream wouldn't have exactly 40ms inter-frame timestamp differences anymore.

The clock is approximated by making use of the in-stream MPEG-TS PCR, which basically gives the sender's clock time at specific points inside the stream, and correlating that together with the local receive times via a linear regression to calculate the relative rate between the sender's clock and the local system clock.

Usage of the element is as simple as

$ gst-launch-1.0 mpegtslivesrc source='srtsrc location=srt://1.2.3.4:5678?latency=150&mode=caller' ! tsdemux skew-corrections=false ! ...
$ gst-launch-1.0 mpegtslivesrc source='udpsrc address=1.2.3.4 port=5678' ! tsdemux skew-corrections=false ! ...

Addition 2025-06-28: If you're using an older (< 1.28) version of GStreamer, you'll have to use the ignore-pcr=true property on tsdemux instead. skew-corrections=false was only added recently and allows for more reliable handling of MPEG-TS timestamp discontinuities.

A similar approach for clocking is implemented in the AJA source element and the NDI source element when the clocked timestamp mode is configured.