Devlog

In my previous post about GstHip, I introduced GStreamer's HIP backend for GPU processing on AMD and NVIDIA hardware. Since then, we have been extending the library to make it easier to integrate with inference workflows and to use in applications with custom GPU processing.

The improvements cover memory sharing, general-purpose GPU allocations, kernel execution, asynchronous resource management, and Rust bindings.

More Flexible Memory Sharing and Uploads

Moving data between a media pipeline and a GPU can introduce unnecessary copies. GstHip now supports importing DMA-BUF memory with a linear layout on AMD GPUs, allowing compatible buffers from other components to be accessed without first copying their contents through system memory. For example, compatible linear DMA-BUF output from GStreamer's VA video decoders can be imported as HIP device memory without copying the video frames. Imported resources are cached for reuse, and hipupload falls back to a CPU copy if importing is not possible.

Once imported, the frames can be processed with hipconvertscale for color conversion and scaling, or with custom HIP kernels for application-specific processing.

The hipupload element now allows upstream sources, such as a V4L2 camera source, to write directly into host-pinned memory optimized for GPU transfers. This can provide faster uploads than copying from regular system memory to HIP device memory.

From Video Frames to Inference Buffers

GstHip's memory APIs were originally designed around video frames. Inference workflows also need allocations for tensors and intermediate results, which do not necessarily have a video layout.

We have extended the allocation APIs to support arbitrary-sized linear buffers and separated video layout information from the public memory structure. This makes the same memory management infrastructure useful for general GPU computations as well as video processing. Support for floating-point and planar RGB formats also helps bridge video processing and model inputs.

The ONNX plugin now integrates HIP memory support, enabling zero-copy input sharing with a compatible GPU execution provider. Video preprocessing and inference can therefore keep data on the GPU, avoiding a round trip through system memory. Depending on the input layout and model requirements, GPU-side packing or normalization may still be needed.

Easier Integration of Custom GPU Kernels

C/C++ applications could already run custom GPU kernels using native HIP APIs. GstHip now provides a higher-level abstraction through GstHipKernelModule, covering runtime compiler loading, source compilation, compiled module loading, entry point lookup, and kernel launches on a GstHipStream.

This layer works with GStreamer's HIP memory and synchronization APIs and can be combined with the asynchronous resource cleanup described below.

The accompanying GstHipKernelArgs API handles kernel arguments, including GStreamer memory objects and scalar values. Together, these APIs reduce the backend-specific setup needed in applications and plugins. They also make kernel execution easier to expose through language bindings, allowing Rust applications to use GStreamer memory and typed scalar arguments without manually constructing the native kernel argument array.

The existing video converter now uses this layer too, and GstHipConverter itself has moved into the public library so applications can reuse its format conversion and scaling functionality.

Keeping Resource Cleanup off the Streaming Thread

Submitting GPU work asynchronously is only part of keeping a media pipeline moving. Resources must remain alive until the GPU has finished using them, and waiting for completion just to release those resources can stall the streaming thread.

GstHip now provides event completion notifications backed by internal worker threads. Applications can associate resources with a completion event and have them released in the background once the GPU work finishes. GstHipFenceData provides a convenient container for collecting those resources.

This moves completion waits and deferred resource cleanup away from the thread processing media, helping preserve its throughput. Dependencies between GPU streams can also be handled with GPU-side event waits, reducing the need for CPU synchronization between processing stages.

HIP Support from Rust

The new gstreamer-hip crate brings GstHip to Rust, including memory management, streams, events, kernel execution, and video conversion.

This allows Rust applications and GStreamer plugins to use the same GPU infrastructure without implementing their own backend dispatch and resource management. The bindings combine generated APIs with manual wrappers for operations that need additional ownership and lifetime handling.

These utilities can also be used outside a complete media pipeline: applications can use GstHip to manage GPU buffers and run their own processing kernels.

The following excerpt illustrates kernel compilation and launch, with initialization and memory setup omitted. The kernel entry point is named transform:

let module = gst_hip::HipKernelModule::from_source(
    &device,
    /* Write your kernel code here */,
    &[],
)
.unwrap();

// ... allocate and prepare input/output HIP memory ...

// Build the kernel arguments and launch the kernel.
let mut args = gst_hip::HipKernelArgs::new();
let launch_result = unsafe {
    args.add_memory(&input, 0).unwrap();
    args.add_memory(&output, 0).unwrap();
    args.add(COUNT);
    module.launch(
        "transform",
        Some(&stream),
        [COUNT.div_ceil(BLOCK_SIZE), 1, 1],
        [BLOCK_SIZE, 1, 1],
        &args,
    )
};
assert!(stream.synchronize());
launch_result.unwrap();

See the full Rust example for kernel compilation, memory allocation, upload, and reading back the results.

Give It a Try

These improvements are now available in the GStreamer main branch and will be included in the upcoming GStreamer 1.30 stable release series. If you're building GPU processing workflows with GStreamer, give it a try and let us know how it works for your use case.

If you have questions or need help with your project, please feel free to get in touch.

HIP (formerly known as Heterogeneous-computing Interface for Portability) is AMD’s GPU programming API that enables portable, CUDA-like development across both AMD and NVIDIA platforms.

• On AMD GPUs, HIP runs natively via the ROCm stack.
• On NVIDIA GPUs, HIP operates as a thin translation layer over the CUDA runtime and driver APIs.

This allows developers to maintain a single codebase that can target multiple GPU vendors with minimal effort.

Where HIP Is Used

HIP has seen adoption in AMD-focused GPU computing workflows, particularly in environments that require CUDA-like programmability. Examples include:

• PyTorch ROCm backend for deep learning workloads
• Select scientific applications like LAMMPS and GROMACS have experimented with HIP backends for AMD GPU support
• GPU-accelerated media processing on systems that leverage AMD hardware

While HIP adoption has been more limited compared to CUDA, its reach continues to expand as support for AMD GPUs grows across a broader range of use cases.

The Challenge: Compile-Time Platform Lock-in

Despite its cross-vendor goal, HIP still has a fundamental constraint at the build level. As of HIP 6.3, HIP requires developers to statically define their target platform at compile time via macros like:

#define __HIP_PLATFORM_AMD__    // for AMD ROCm
#define __HIP_PLATFORM_NVIDIA__ // for CUDA backend

This leads to two key limitations:

• You must compile separate binaries for AMD and NVIDIA
• A single binary cannot support both platforms simultaneously
• HIP does not support runtime backend switching natively

GstHip’s Solution

To overcome this limitation, GstHip uses runtime backend dispatch through:

• dlopen() on Linux
• LoadLibrary() on Windows

Instead of statically linking against a single HIP backend, GstHip loads both the ROCm HIP runtime and the CUDA driver/runtime API at runtime. This makes it possible to:

• Detect available GPUs dynamically
• Choose the appropriate backend per device
• Even support simultaneous use of AMD and NVIDIA GPUs in the same process

Unified Wrapper API

GstHip provides a clean wrapper layer that abstracts backend-specific APIs via a consistent naming scheme:

hipError_t HipFooBar(GstHipVendor vendor, ...);

The Hip prefix (capital H) clearly distinguishes the wrapper from native hipFooBar(...) functions. The GstHipVendor enum indicates which backend to target:

• GST_HIP_VENDOR_AMD
• GST_HIP_VENDOR_NVIDIA

Internally, each HipFooBar(...) function dispatches to the correct backend by calling either:

• hipFooBar(...) for AMD ROCm
• cudaFooBar(...) for NVIDIA CUDA

These symbols are dynamically resolved via dlopen() / LoadLibrary(), enabling runtime backend selection without GPU vendor-specific builds.

Memory Interop

All memory interop in GstHip is handled through the hipupload and hipdownload elements. While zero-copy is not supported due to backend-specific resource management and ownership ambiguity, GstHip provides optimized memory transfers between systems:

• System Memory ↔ HIP Memory: Utilizes HIP pinned memory to achieve fast upload/download operations between host and device memory
• GstGL ↔ GstHip: Uses HIP resource interop APIs to perform GPU-to-GPU memory copies between OpenGL and HIP memory
• GstCUDA ↔ GstHip (on NVIDIA platforms): Since both sides use CUDA memory, direct GPU-to-GPU memory copies are performed using CUDA APIs.

GPU-Accelerated Filter Elements

GstHip includes GPU-accelerated filters optimized for real-time media processing:

• hipconvertscale/hipconvert/hipscale: Image format conversion and image scaling
• hipcompositor: composing multiple video streams into single video stream

These filters use the same unified dispatch system and are compatible with both AMD and NVIDIA platforms.

Application Integration Support

As of Merge Request!9340, GstHip exposes public APIs that allow applications to access HIP resources managed by GStreamer. This also enables applications to implement custom GstHIP-based plugins using the same underlying infrastructure without duplicating resource management.

Summary of GstHip Advantages

• Single plugin/library binary supports both AMD and NVIDIA GPUs
• Compatible with Linux and Windows
• Supports multi-GPU systems, including hybrid AMD + NVIDIA configurations
• Seamless memory interop with System Memory, GstGL, and GstCUDA
• Provides high-performance GPU filters for video processing
• Maintains a clean API layer via HipFooBar(...) wrappers, enabling backend-neutral development