New high-performance GStreamer UDP source element and SMPTE ST2110 capture

New udpsrc2 element

Over the past few years, I have worked on a new GStreamer UDP source element. This is finally merged now and will be part of both the GStreamer 1.30.0 release and the gst-plugins-rs 0.16.0 release.

The old element uses GIO for networking, which is quite inefficient by design. The new implementation uses about 50% less CPU on my machine compared to the old element for a 3 Gbit/s stream.

As can be seen from the docs of the new element, it preserves the API of the old element. As such it should generally be possible to use it as a drop-in replacement.

In addition to performance improvements, the new element also includes various other improvements:

• Support for faster packet receiving via Generic Receive Offload (GRO) on Linux, and for using recvmmsg() on platforms where it is available to significantly improve receive performance.

• Complete support for multicast source filtering, including negative filters, and support for platforms that do not have APIs for the IGMPv3 SSM mechanism.

• Always obtaining kernel-side packet receive times if available, which was opt-in in the old element due to GIO performance issues with socket control messages.

• New preserve-packetization property that allows outputting multiple packets in the same buffer, which improves performance for formats like MPEG-TS where the UDP packetization is not necessary.

Give it a try with your pipelines and workloads and share your feedback or any issues you encounter.

In the future, io_uring support on Linux could be added for even better receive performance.

SMPTE ST2110 capture

While udpsrc2 is an improvement in general, its primary motivation is better SMPTE ST2110 support in GStreamer. The old element could not handle the packet rates typically used for such streams very well.

ST2110 defines a UDP/RTP-based set of standards for transmitting raw or very-high bitrate audio / video / ancillary data over Ethernet. It is intended as a replacement for SDI.

Related to this, we recently also merged some other improvements:

• A new, improved raw video RTP depayloader that supports ST2110-20.

• A new, improved raw PCM audio RTP depayloader that supports ST2110-30.

• An ST291 ancillary data RTP depayloader that supports ST2110-40.

• Various improvements to the rtprecv element, especially for performance and handling of high-packet rate streams.

For all the new depayloaders there are also new, improved implementations of the corresponding payloaders available.

Together, these improvements enable reliable ST2110 stream capture in GStreamer.

An example pipeline putting it all together would look as follows:

$ gst-launch-1.0 \
    \ # Video capture pipeline part
    udpsrc2 address=239.255.64.20 port=16388 multicast-iface=enp15s0 buffer-size=20000000 caps='application/x-rtp, media=video, payload=96, clock-rate=90000, encoding-name=RAW, sampling=YCbCr-4:2:2, depth=10, width=1920, height=1080, exactframerate=60, colorimetry=BT709, pm=2110GPM, ssn=ST2110-20:2017, tp=2110TPN, a-sendonly="", a-ts-refclk="ptp=IEEE1588-2008:7C-2E-0D-FF-FE-1C-81-14:127", a-mediaclk="direct=0", ssrc-327995485-cname=E055FF0F3D6E4B349F7B786D8B6C837B' ! \
      rtprecv latency=0 ! queue max-size-bytes=0 max-size-buffers=0 max-size-time=500000000 ! rtpvrawdepay2 ! \
    \
    \ # Ancillary data capture pipeline part
    udpsrc2 address=239.255.64.20 port=16386 multicast-iface=enp15s0 buffer-size=20000000 caps='application/x-rtp, media=video, payload=98, clock-rate=90000, encoding-name=SMPTE291, vpid_code=138, a-sendonly="", a-ts-refclk="ptp=IEEE1588-2008:7C-2E-0D-FF-FE-1C-81-14:127", a-mediaclk="direct=0", ssrc-2672978631-cname=E055FF0F3D6E4B349F7B786D8B6C837B' ! \
      rtprecv latency=0 ! queue max-size-bytes=0 max-size-buffers=0 max-size-time=500000000 ! rtpsmpte291depay ! combiner.st2038 \
    \
    \ # Combination of video and ancillary data streams and output
    st2038combiner name=combiner start-time-selection=first ! videoconvert ! queue max-size-bytes=0 max-size-time=0 max-size-buffers=3 ! autovideosink \
    \
    \ # Audio capture and output pipeline part
    udpsrc2 address=239.255.64.20 port=16384 multicast-iface=enp15s0 buffer-size=20000000 caps='application/x-rtp, media=audio, payload=(int)97, clock-rate=48000, encoding-name=(string)L24, encoding-params=64, a-sendonly="", a-ptime=0.125, a-ts-refclk="ptp\=IEEE1588-2008:7C-2E-0D-FF-FE-1C-81-14:127", a-mediaclk="direct\=0", ssrc-603238248-cname=(string)E055FF0F3D6E4B349F7B786D8B6C837B' \
      rtprecv latency=0 ! queue max-size-bytes=0 max-size-buffers=0 max-size-time=500000000 ! rtpL24depay2 ! audioconvert ! autoaudiosink

This pipeline receives a 1080p60 4:2:2 YUV 10-bit video stream, ST291 ancillary data, and a 24-bit 48kHz 64-channel PCM audio stream. The video and ancillary data are combined to a single stream, and then both the combined video-ancillary stream and the audio are output.

rtprecv is used here for translating packet capture timestamps and RTP header timestamps to consistent GStreamer timestamps.

Ancillary data

The pipeline above captures all three streams and merges the ancillary data stream with the video. The ancillary data itself is not processed further.

One way to process the ancillary data further is to extract ST12 timecodes from it and overlay them over the video.

For this, insert the following elements before the video sink:

 ... ! timecodestamper source=ancillary-meta ancillary-meta-locations='8:2000,570:2000' \
     ! videoconvert ! timeoverlay time-mode=time-code \
     ! autovideosink

Here timecodes from ancillary data at positions (8,2000) and (570,2000) would be extracted and converted to GstVideoTimeCodeMeta on the video buffers.

We recently added support for extracting ST12 timecodes from ancillary meta as well.

The positions depend on the video signal standard in use and can be found in the ST12 specifications.