Devlog

RTSP stands for Real-Time Streaming Protocol, standardized by RFC 7826. It's commonly used to stream audio and video from IP cameras.

In GStreamer, RTSP handling was traditionally done by rtspsrc, a widely used and mature element, but it still had several architectural limitations that were hard to address within the existing codebase.

rtspsrc ties the server state to client/pipeline state. This resulted in issues with

• Multicast playback
• PAUSE being sent on error, immediately followed by tear down
• Flushing seeks involved state changes causing glitches and possible deadlock

As a result, the element has been rewritten from scratch in Rust as rtspsrc2. See this GStreamer Conference Talk and README for more details.

Until now, some features were missing, so if your application relied on them, rtspsrc2 couldn't replace the original.

Many of these have now been implemented:

Authentication

Basic and Digest authentication are supported. Digest covers MD5, SHA-256 and SHA-512-256, so rtspsrc2 can connect to servers that require a username/password and can perform the challenge response handshake automatically.

TLS/TCP support

rtspsrc2 can connect over TLS (the rtsps:// scheme) and supports client certificate authentication for servers that require it. Like rtspsrc, it also has tls-validation-flags property to ignore specific certificate errors like expired certificates.

HTTP tunnelling

Some networks block RTSP but allow HTTP. rtspsrc2 can tunnel RTSP over HTTP (rtsph://) and exposes an extra-http-request-headers property for custom headers (useful for proxies or authentication).

Keep alive

RTSP keep-alive is required because RTSP session state is entirely independent of the underlying transport. During active playback, RTCP serves as the primary proof of liveness. However, during paused states when media and RTCP traffic stop, explicit RTSP keep-alive requests are needed to prevent session timeout.

rtspsrc2 sends periodic keep-alive to prevent session timeout. This is enabled by default via do-rtsp-keep-alive. Disable it if you need compatibility with older servers.

Stream selection

This is one major feature that rtspsrc doesn't implement. rtspsrc2 is streams-aware.

When a source contains multiple streams (for example audio and video), rtspsrc2 exposes them as GstStream objects so applications can choose which streams to set up before playback using the StreamCollection API. For uses of this API by other elements, also see adaptive demuxers.

Consider an example scenario where the RTSP stream has audio and video. An application can choose to ignore audio and select only video via the stream collection API. rtspsrc2 then sends a SETUP request only for video and not audio. An example showing how this can be done in Rust can be seen here.

Unlinked pads

Linking of exposed source pads is now optional, for example, if you only link the video pad and leave the audio pad unlinked, rtspsrc2 handles that without error. Internally, it uses a flow combiner like the original rtspsrc, so selective linking works as expected.

GET/SET_PARAMETER

GET_PARAMETER and SET_PARAMETER requests are supported using signals similar to rtspsrc. These requests are used to retrieve or set the value of a parameter or a set of parameters for a presentation or stream specified by the URI.

Secure Real-time Transport Protocol (SRTP)

SRTP adds encryption and authentication to RTP streams. The RTSP source internally instantiates an RTP session manager element that handles the messages to and from the server, jitter removal, packet reordering along with providing a clock for the pipeline.

While rtspsrc uses rtpbin, rtspsrc2 can use rtpbin or the newer rtpbin2. rtpbin2 unlike rtpbin splits out the sender and receiver RTP session management in two separate elements rtpsend and rtprecv. See this GStreamer Conference Talk for understanding the motivations behind the new RTP elements. By setting the environment variable USE_RTP2=1, rtspsrc2 can use rtpsend and rtprecv.

Secure RTP (SRTP) as of this writing, is only supported when using rtspsrc2 with rtpbin, including MIKEY key exchange embedded in SDP.

With these additions, rtspsrc2 is much closer to being a drop-in replacement for rtspsrc. If you haven't tried it yet, now is a good time.

A GStreamer pipeline to test rtspsrc2 given a RTSP server address.

gst-launch-1.0 -e -vvv rtspsrc2 location=rtsp://some.server/url ! queue ! decodebin ! queue ! videoconvert ! autovideosink

The following key features still need to be implemented before rtspsrc2 achieves complete feature parity with rtspsrc.

• Clock sync support, such as RFC 7273 (work is on-going to bring this to rtprecv)
• Pause/seeking support with VOD
• ONVIF back-channel support

When using HTTP Live Streaming (HLS), a common use case is to use MPEG-TS segments or fragmented MP4 fragments. This is done so that the overall stream is available as a sequence of small HTTP-based file downloads, each being one short chunk of an overall bounded or unbounded media stream.

The playlist file (.m3u8) contains a list of these small segments or fragments. This is the standard and most common approach for HLS. For the HLS CMAF case, a multi-segment playlist would look like below.

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-TARGETDURATION:5
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MAP:URI="init00000.mp4"
#EXTINF:5,
segment00000.m4s
#EXTINF:5,
segment00001.m4s
#EXTINF:5,
segment00002.m4s

An alternative approach is to use a single media file with the EXT-X-BYTERANGE tag. This method is primarily used for on-demand (VOD) streaming where the complete media file already exists and can reduce the number of files that needs to be managed on the server. Single file with byte-ranges requires the server and client to support HTTP byte range requests and 206 Partial Content responses.

The single media file use case wasn't supported so far with either of hlssink3 or hlscmafsink. A new property single-media-file has been added, which lets users specify the use of a single media file.

hlscmafsink.set_property("single-media-file", "main.mp4");
hlssink3.set_property("single-media-file", "main.ts");

For the HLS CMAF case, this would generate a playlist like below.

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-TARGETDURATION:5
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MAP:URI="main.mp4",BYTERANGE="768@0"
#EXT-X-BYTERANGE:100292@768
#EXTINF:5,
main.mp4
#EXT-X-BYTERANGE:98990@101060
#EXTINF:5,
main.mp4
#EXT-X-BYTERANGE:99329@200050
#EXTINF:5,
main.mp4

This can be useful if one has storage requirements where the use of a single media file for HLS might be favourable.

With GStreamer 1.26, a new D3D12 backend GstD3D12 public library was introduced in gst-plugins-bad.

Now, with the new gstreamer-d3d12 rust crate, Rust can finally access the Windows-native GPU feature written in GStreamer in a safe and idiomatic way.

What You Get with GStreamer D3D12 Support in Rust

• Pass D3D12 textures created by your Rust application directly into GStreamer pipelines without data copying
• Likewise, GStreamer-generated GPU resources (such as frames decoded by D3D12 decoders) can be accessed directly in your Rust app
• GstD3D12 base GStreamer element can be written in Rust

Beyond Pipelines: General D3D12 Utility Layer

GstD3D12 is not limited to multimedia pipelines. It also acts as a convenient D3D12 runtime utility, providing:

• GPU resource pooling such as command allocator and descriptor heap, to reduce overhead and improve reuse
• Abstractions for creating and recycling GPU textures with consistent lifetime tracking
• Command queue and fence management helpers, greatly simplifying GPU/CPU sync
• A foundation for building custom GPU workflows in Rust, with or without the full GStreamer pipeline

Up until recently, when using hlscmafsink, if you wanted to move to a new playlist you had to stop the pipeline, modify the relevant properties and then go to PLAYING again.

This was problematic when working with live sources because some data was being lost between the state changes. Not anymore!

A new-playlist signal has been added, which lets you switch output to a new location on the fly, without having any gaps between the content in each playlist.

Simply change the relevant properties first and then emit the signal:

hlscmafsink.set_property("playlist-location", new_playlist_location);
hlscmafsink.set_property("init-location", new_init_location);
hlscmafsink.set_property("location", new_segment_location);
hlscmafsink.emit_by_name::<()>("new-playlist", &[]);

This can be useful if you're capturing a live source and want to switch to a different folder every couple of hours, for example.

When using hlssink3 and hlscmafsink elements, it's now possible to track new fragments being added by listening for the hls-segment-added message:

Got message #67 from element "hlscmafsink0" (element): hls-segment-added, location=(string)segment00000.m4s, running-time=(guint64)0, duration=(guint64)3000000000;
Got message #71 from element "hlscmafsink0" (element): hls-segment-added, location=(string)segment00001.m4s, running-time=(guint64)3000000000, duration=(guint64)3000000000;
Got message #74 from element "hlscmafsink0" (element): hls-segment-added, location=(string)segment00002.m4s, running-time=(guint64)6000000000, duration=(guint64)3000000000;

This is similar to how you would listen for splitmuxsink-fragment-closed when using the older hlssink2.

webrtcsink already supported instantiating a data channel for the sole purpose of carrying navigation events from the consumer to the producer, it can also now create a generic control data channel through which the consumer can send JSON requests in the form:

{
    "id": identifier used in the response message,
    "mid": optional media identifier the request applies to,
    "request": {
        "type": currently "navigationEvent" and "customUpstreamEvent" are supported,
        "type-specific-field": ...
    }
}

The producer will reply with such messages:

{
  "id": identifier of the request,
  "error": optional error message, successful if not set
}

The example frontend was also updated with a text area for sending any arbitrary request.

The use case for this work was to make it possible for a consumer to control the mix matrix used for the audio stream, with such a pipeline running on the producer side:

gst-launch-1.0 audiotestsrc ! audioconvert ! webrtcsink enable-control-data-channel=true

As audioconvert now supports setting a mix matrix through a custom upstream event, the consumer can simply input the following text in the request field of the frontend to reverse the channels of a stereo audio stream:

{
  "type": "customUpstreamEvent",
  "structureName": "GstRequestAudioMixMatrix",
  "structure": {
    "matrix": [[0.0, 1.0], [1.0, 0.0]]
  }
}

The default signaller for webrtcsink can now produce an answer when the consumer sends the offer first.

To test this with the example, you can simply follow the usual steps but also paste the following text in the text area before clicking on the producer name:

{
  "offerToReceiveAudio": 1,
  "offerToReceiveVideo": 1
}

I implemented this in order to test multiopus support with webrtcsink, as it seems to work better when munging the SDP offered by chrome.

webrtcsink can now run both the default signalling server and a web server for static content.

gst-launch-1.0 ! videotestsrc webrtcsink name=ws run-signalling-server=true run-web-server=true

This comes in very handy for testing purposes, but could also prove useful in production.