Devlog

A few days ago I wrote a new GStreamer element around llama.cpp. This element takes a text stream as input, passes it together with a configurable system prompt through a locally running LLM and then forwards the LLM's output. It can also keep a history of past inputs and outputs for more consistent outputs. Check the documentation of the element for everything that can be configured on it.

This can be used for all kinds of purposes involving text. The two most obvious ones are probably translation into different languages and rephrasing.

When compiling the plugin, make sure to select the correct backend. By default only the CPU backend is compiled in but depending on your GPU the Vulkan, ROCM or CUDA backends are going to provide much better performance. For example to compile with the Vulkan backend, use

$ cargo build --release --features vulkan

Translations

Together with the whisper speech-to-text element this would for example allow you to watch a movie with English audio, transcribe that text in real-time, then have the text translated from English to German and render German subtitles on top of the video.

An example pipeline for this would be the following:

$ gst-launch-1.0 filesrc location=movie.mp4 ! decodebin3 name=dbin \
    dbin. ! audio/x-raw ! tee name=audio-tee
    audio-tee. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! audioconvert ! audioresample ! \
        whispertranscriber model-path=whisper-ggml-large-v3.bin model-preset=large-v3 chunk-duration=4000 ! \
        textaccumulate latency=0 ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! \
        llamacpp-texttransform model-path=Hunyuan-MT-7B.Q4_K_M.gguf system-prompt="Translate the following segments into German, without additional explanation." history-size=5 ! \
        textwrap columns=72 ! overlay.text_sink \
    dbin. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! videoconvert ! \
        textoverlay name=overlay ! videoconvert ! autovideosink
    audio-tee. ! queue max-size-time=10000000000 max-size-buffers=0 max-size-bytes=0 ! audioconvert ! autoaudiosink

Running this on the famous speech at the end of The Great Dictator produces output such as the following:

Note that the plain transcribed text is rendered at the top and the corresponding translation is at the bottom.

This example also makes use of the textaccumulate element to collect full sentences or subclauses, and the textwrap element to wrap long lines so they fit on top of the video frames.

Collecting full sentences or subclauses before the text transformation can help the model with getting more context and providing better output in exchange for additional latency in live pipelines and additional buffering requirements in non-live pipelines.

Rephrasing

The same approach can also be used for rephrasing text. The following example is from the same movie and the system prompt now contained instructions to rephrase the English speech into "duck English".

While this example is not too useful apart from the comedic effect, a useful application of this would be for example rephrasing in simpler and shorter English, or removing usage of jargon.

Models

Generally, any model that is supported by llama.cpp and that supports text input and output can be used for this element. It makes little sense to use a huge model but models with 1-10B are generally more than enough to give useful results while also not requiring too many resources.

The model that should be used can be selected via the model-path property. This expects a local GGUF file, which can be downloaded e.g. from Hugging Face. Models that are known to work well are

• Gemma 4 E4B (quantizations) or Gemma 4 E2B (quantizations),
• Qwen 3.5 9B (quantizations) or Qwen 3.5 4B (quantizations),
• Ministral 3 14B Instruct (quantizations) or Ministral 3 8B Instruct (quantizations) or Ministral 3 3B Instruct (quantizations),
• Hunyuan MT 7B (quantizations). This model is specifically trained for translations and expects a system prompt in a specific format that is described on the Hugging Face page.

The examples above were all created with Hunyuan MT 7B or Qwen 3.5 9B at Q4_K_M (4 bit) quantization, and on my system via Vulkan on an AMD Radeon RX 9070 XT consumed no considerable processing resources for real-time processing and video memory corresponding to the size of the weights plus a small KV cache.

Censoring and safeguards

This all works well up to a certain point. Unfortunately most open weight LLMs come with safeguards built directly into the model weights instead of leaving it to the deployment layer. Depending on your input this might trigger:

Translating subtitles of an R-rated movie, or a documentary about certain historical events, can easily trigger these safeguards. This is apparently also a problem for more serious usage like for translation and summarization of court documents at the Swiss Federal Supreme Court. To get around that, it is possible to use uncensored / abliterated models that can be created with the help of e.g. Heretic or downloaded directly from Hugging Face (e.g. this Qwen 3.5 9B variant). Another option to get around such censorship are prompt injection patterns such as Prompt Injection as Role Confusion but they're less reliable.

Future work

The element is in a decent state and is usable as-is, but there are a couple of future extensions I would like to work on:

• Optimizing memory usage if the same model is used multiple times in the same process. The weights can be made to be in memory only once but right now every instance has a copy.

• Optimizing KV cache utilization. Once the history of past inputs/outputs runs full, currently the whole KV cache is discarded. With this PR now merged it should be possible to improve this significantly such that it is not necessary to re-process the whole prompt on each new input.

• Support for other llama.cpp elements that can work with different modalities, e.g. image or audio input and possibly combined with text input. This could, for example, be used to automatically generate scene descriptions from a low framerate video stream that is triggered on scene changes.

If any of these extensions would be useful to you, or if you have any questions, or are interested in other use-cases, please feel free to get in touch.