Devlog

A few days ago I wrote a new GStreamer element around llama.cpp. This element takes a text stream as input, passes it together with a configurable system prompt through a locally running LLM and then forwards the LLM's output. It can also keep a history of past inputs and outputs for more consistent outputs. Check the documentation of the element for everything that can be configured on it.

This can be used for all kinds of purposes involving text. The two most obvious ones are probably translation into different languages and rephrasing.

When compiling the plugin, make sure to select the correct backend. By default only the CPU backend is compiled in but depending on your GPU the Vulkan, ROCm or CUDA backends are going to provide much better performance. For example to compile with the Vulkan backend, use

$ cargo build --release --features vulkan

Translations

Together with the whisper speech-to-text element this would for example allow you to watch a movie with English audio, transcribe that text in real-time, then have the text translated from English to German and render German subtitles on top of the video.

An example pipeline for this would be the following:

$ gst-launch-1.0 filesrc location=movie.mp4 ! decodebin3 name=dbin \
    dbin. ! audio/x-raw ! tee name=audio-tee
    audio-tee. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! audioconvert ! audioresample ! \
        whispertranscriber model-path=whisper-ggml-large-v3.bin model-preset=large-v3 chunk-duration=4000 ! \
        textaccumulate latency=0 ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! \
        llamacpp-texttransform model-path=Hunyuan-MT-7B.Q4_K_M.gguf system-prompt="Translate the following segments into German, without additional explanation." history-size=5 ! \
        textwrap columns=72 ! overlay.text_sink \
    dbin. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! videoconvert ! \
        textoverlay name=overlay ! videoconvert ! autovideosink
    audio-tee. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! audioconvert ! autoaudiosink

Running this on the famous speech at the end of The Great Dictator produces output such as the following:

Note that the plain transcribed text is rendered at the top and the corresponding translation is at the bottom.

This example also makes use of the textaccumulate element to collect full sentences or subclauses, and the textwrap element to wrap long lines so they fit on top of the video frames.

Collecting full sentences or subclauses before the text transformation can help the model with getting more context and providing better output in exchange for additional latency in live pipelines and additional buffering requirements in non-live pipelines.

Rephrasing

The same approach can also be used for rephrasing text. The following example is from the same movie and the system prompt now contained instructions to rephrase the English speech into "duck English".

While this example is not too useful apart from the comedic effect, a useful application of this would be for example rephrasing in simpler and shorter English, or removing usage of jargon.

Models

Generally, any model that is supported by llama.cpp and that supports text input and output can be used for this element. It makes little sense to use a huge model but models with 1-10B are generally more than enough to give useful results while also not requiring too many resources.

The model that should be used can be selected via the model-path property. This expects a local GGUF file, which can be downloaded e.g. from Hugging Face. Models that are known to work well are

• Gemma 4 E4B (quantizations) or Gemma 4 E2B (quantizations),
• Qwen 3.5 9B (quantizations) or Qwen 3.5 4B (quantizations),
• Ministral 3 14B Instruct (quantizations) or Ministral 3 8B Instruct (quantizations) or Ministral 3 3B Instruct (quantizations),
• Hunyuan MT 7B (quantizations). This model is specifically trained for translations and expects a system prompt in a specific format that is described on the Hugging Face page.

The examples above were all created with Hunyuan MT 7B or Qwen 3.5 9B at Q4_K_M (4 bit) quantization, and on my system via Vulkan on an AMD Radeon RX 9070 XT consumed no considerable processing resources for real-time processing and video memory corresponding to the size of the weights plus a small KV cache.

Censoring and safeguards

This all works well up to a certain point. Unfortunately most open weight LLMs come with safeguards built directly into the model weights instead of leaving it to the deployment layer. Depending on your input this might trigger:

Translating subtitles of an R-rated movie, or a documentary about certain historical events, can easily trigger these safeguards. This is apparently also a problem for more serious usage like for translation and summarization of court documents at the Swiss Federal Supreme Court. To get around that, it is possible to use uncensored / abliterated models that can be created with the help of e.g. Heretic or downloaded directly from Hugging Face (e.g. this Qwen 3.5 9B variant). Another option to get around such censorship are prompt injection patterns such as Prompt Injection as Role Confusion but they're less reliable.

Future work

The element is in a decent state and is usable as-is, but there are a couple of future extensions I would like to work on:

• Optimizing memory usage if the same model is used multiple times in the same process. The weights can be made to be in memory only once but right now every instance has a copy.

• Optimizing KV cache utilization. Once the history of past inputs/outputs runs full, currently the whole KV cache is discarded. With this PR now merged it should be possible to improve this significantly such that it is not necessary to re-process the whole prompt on each new input.

• Support for other llama.cpp elements that can work with different modalities, e.g. image or audio input and possibly combined with text input. This could, for example, be used to automatically generate scene descriptions from a low framerate video stream that is triggered on scene changes.

If any of these extensions would be useful to you, or if you have any questions, or are interested in other use-cases, please feel free to get in touch.

Currently most code using the GStreamer Analytics library library is written in C or Python. To check how well the API works from Rust, and to have an excuse to play with the Rust burn deep-learning framework, I've implemented an object detection inference element based on the YOLOX model and a corresponding tensor decoder that allows usage with other elements based on the GstAnalytics API. I started this work at the last GStreamer hackfest, but this has now finally been merged and will be part of the GStreamer 1.28.0 release.

burn is a deep-learning framework in Rust that is approximately on the same level of abstraction as PyTorch. It features lots of computation backends (CPU-based, Vulkan, CUDA, ROCm, Metal, libtorch, ...), has loaders (or better: code generation) for e.g. ONNX or PyTorch models, and compiles and optimizes the model for a specific backend. It also comes with a repository containing various example models and links to other community models.

The first element is burn-yoloxinference. It takes raw RGB video frames and passes them through burn; as of the time of this writing either through a CPU-based or a Vulkan-based computation backend. The output then is the very same video frames with the raw object detection results attached as a GstTensorMeta. This is essentially a 85x8400 float matrix, which contains 8400 rows of candidate object detection boxes (4 floats) together with confidence values for the classes (80 floats for the pre-trained models on the COCO classes) and one confidence value for the overall box. The element itself is mostly boilerplate, caps negotiation code and glue code between GStreamer and burn.

The second element is yoloxtensordec. This takes the output of the first element and decodes the GstTensorMeta into a GstAnalyticsRelationMeta, which describes the detected objects with their bounding boxes in an abstract way. As part of this it also implements a non-maximum suppression (NMS) filter using intersection over unions (IoU) of bounding boxes to reduce the 8400 candidate boxes to a much lower number of actual likely object detections. The GstAnalyticsRelationMeta can then be used e.g. by the generic objectdetectionoverlay to render rectangles on top of the video, or the ioutracker elements to track objects over a sequence of frames. Again, this element is mostly boilerplate and caps negotiation code, plus around 100 SLOC of algorithm. In comparison the C YOLOv9 tensor decoder element is about 3x as much code, mostly thanks to the overhead of C memory book-keeping, lack of useful data structures and lack of abstraction language tools.

The reason why the tensor decoder is a separate element is mostly to have one such element per model and to have it implemented independently of the actual implementation and runtime of the model. The same tensor decoder should, for example, also work fine on the output of the onnxinference element with a YOLOX model. From GStreamer 1.28 onwards it will also be possible to autoplug suitable tensor decoders via the tensordecodebin element.

That the tensor decoders are independent of the actual implementation of the model also has the advantage that it can be implemented in a different language, preferably in a safer and less verbose language than C.

For using both elements together and using objectdetectionoverlay to render rectangles around the object detections, the following pipeline can be used:

gst-launch-1.0 souphttpsrc location=https://raw.githubusercontent.com/tracel-ai/models/f4444a90955c1c6fda90597aac95039a393beb5a/squeezenet-burn/samples/cat.jpg \
    ! jpegdec ! videoconvertscale ! "video/x-raw,width=640,height=640" \
    ! burn-yoloxinference model-type=large backend-type=vulkan ! yoloxtensordec label-file=COCO_classes.txt \
    ! videoconvertscale ! objectdetectionoverlay \
    ! videoconvertscale ! imagefreeze ! autovideosink -v

The output should look similar to this:

I also did a lightning talk about this at the GStreamer conference this year.

Audio source separation describes the process of splitting an already mixed audio stream into its individual, logical sources. For example, splitting a song into separate streams for its individual instruments and vocals. This can be used for example for karaoke, music practice, or isolating the speaker from background noise for easier understanding by humans or improving results of speech-to-text processing.

Starting with GStreamer 1.28.0 an element for this purpose will be included. It is based on the Python/pytorch implementation of demucs and comes with various pre-trained models with different performance and accuracy characteristics, as well as which different sets of sources they can separate. CPU-based processing is generally multiple times real-time on modern CPUs (around 8x on mine) but GPU-based processing via pytorch is also possible.

The element itself is part of the GStreamer Rust plugins and can either run demucs locally in-process using an embedded Python interpreter via pyo3, or via a small Python service over WebSockets that can run either locally or remotely (e.g. for thin clients). The used model, and chunk size and overlap between chunks can be configured. Chunk size and overlap provide control over the introduced latency (lower values give lower latency) and quality (higher values give better quality).

The separate sources are provided on individual source pads of the element and it effectively behaves like a demuxer. A pipeline for karaoke would for example look as follows:

gst-launch-1.0 uridecodebin uri=file:///path/to/music/file ! audioconvert ! tee name=t ! \
  queue max-size-time=0 max-size-bytes=0 max-size-buffers=2 ! demucs name=demucs model-name=htdemucs \
  demucs.src_vocals ! queue ! audioamplify amplification=-1 ! mixer.sink_0 \
  t. ! queue max-size-time=9000000000 max-size-bytes=0 max-size-buffers=0 ! mixer.sink_1 \
  audiomixer name=mixer ! audioconvert ! autoaudiosink

This takes an URI to a music file, passes that through the demucs element for extracting the vocals, then takes the original input via a tee and subtracts the vocals from it by first inverting all samples of the vocals stream with the audioamplify element and then mixing it with the original input with an audiomixer.

I also did a lightning talk about this at the GStreamer conference this year.

For a project recently it was necessary to collect video frames of multiple streams during a specific interval, and in the future also audio, to pass it through an inference framework for extracting additional metadata from the media and attaching it to the frames.

While GStreamer has gained quite a bit of infrastructure in the past years for machine learning use-cases in the analytics library, there was nothing for this specific use-case yet.

As part of solving this, I proposed as design for a generic interface that allows combining and batching multiple streams into a single one by using empty buffers with a GstMeta that contains the buffers of the original streams, and caps that include the caps of the original streams and allow format negotiation in the pipeline to work as usual.

While this covers my specific use case of combining multiple streams, it should be generic enough to also handle other cases that came up during the discussions.

In addition I wrote two new elements, analyticscombiner and analyticssplitter, that make use of this new API for combining and batching multiple streams in a generic, media-agnostic way over specific time intervals, and later splitting it out again into the original streams. The combiner can be configured to collect all media in the time interval, or only the first or last.

Conceptually the combiner element is similar to NVIDIA's DeepStream nvstreammux element, and in the future it should be possible to write a translation layer between the GStreamer analytics library and DeepStream.

The basic idea for the usage of these elements is to have a pipeline like

-- stream 1 --\                                                                  / -- stream 1 with metadata --
               -- analyticscombiner -- inference elements -- analyticssplitter --
-- stream 2 --/                                                                  \ -- stream 2 with metadata --
   ........                                                                           ......................
-- stream N -/                                                                     \- stream N with metadata --

The inference elements would only add additional metadata to each of the buffers, which can then be made use of further downstream in the pipeline for operations like overlays or blurring specific areas of the frames.

In the future there are likely going to be more batching elements for specific stream types, operating on multiple or a single stream, or making use of completely different batching strategies.

Special thanks also to Olivier and Daniel who provided very useful feedback during the review of the two merge requests.