### State of the Union <br> GStreamer Conference 07 October 2024, Montréal <br> Tim-Philipp Müller < <tim@centricular.com> >

## Welcome Nice to see everyone again in person! <!-- who am I? -->

## Before we get started ...

## Just the other day <img width="40%" data-src="Wim Taymans - 20 Years of GStreamer.png" style="border:0px"> Wim Taymans presented "20 Years of GStreamer" at the GStreamer Conference in Lyon in 2019. That was 5 years ago!

## Happy 25th Birthday GStreamer! <img width="40%" data-src="happy_21st_birthday_by_lamarogre_defgpzc-fullview.jpg" style="border:0px"> <small><small><small>image by lamarogre</small></small></small>

## Testament to<br/>our amazing community! Speaking of which ..

## Mailing List to Discourse <img data-src="GStreamer Discourse - Screenshot.png" style="border:0px">

## Discourse Stats <img data-src="GStreamer Discourse - Stats.png" style="border:0px">

## IRC to Matrix <img data-src="GStreamer Matrix - Screenshot.png" style="border:0px">

## What else have we been up to?

## Releases! - 1.20: February 2022 - 1.22: January 2023 - 1.24: March 2024 - 1.26: ???

## 1.24 Stable Series - 8 bug-fix releases since March - 800 commits, 450 backported merge requests - Every ~3 weeks

## Upcoming 1.26 - ~2000 commits, 1400 merge requests (so far) - ca. 15-20% of commits, 35-45% of MRs are Rust!

### Let's talk a little about Rust! <i>"Fast, safe and productive - pick three."</i> <img width="10%" data-src="rust-logo.svg" alt="Rust Logo" style="border:0px">

### Why Rust? - Perfect language for us technically. - Excellent C compatibility. - Fantastic community. - Superb ecosystem. - Adoption by all the major industry players.

### The Future of GStreamer is Rust - **Bindings**: Mature and battle-tested - **Plugins**: Where cool™ new stuff happens - Can be used from any language - Shipped for macOS, Windows, Android, iOS - Linux distros still catching up

### The Future of GStreamer is Rust - Don't worry! - Change will be gradual, C isn't going anywhere. - However! - Rust plugins are needed for a fully-featured GStreamer

## Back to our feature presentation..

## RTP + RTSP stack<br/> rewrite in Rust Sponsored by the Sovereign Tech Fund <img width="25%" data-src="stf-opengraph-image.png" alt="STF logo" style="border:0px"> Why? Security and Long-Term Maintainability

## RTP session handling in Rust - rtpsend + rtprecv = rtpbin2 - Low-level RTP/RTCP packet handling all in Rust too - All the basic features are there - Advanced feature still todo

## New RTSP source in Rust - rtspsrc2 - Live streaming only for now - Can use new or existing RTP stack

## New RTP depayloaders/payloaders in Rust - New base classes in Rust - Packet parsing/handling all in Rust - Please give them a spin!

## Misc RTP improvements - RFC7273 clock sync support (PTP, NTP sync) - Easy way to get original timestamps from rtpjitterbuffer - `rtppassthroughpay` element - useful for RTP passthrough in rtsp-server - regenerate RTP timestamps based on buffer ts

## RTSP client source - Optionally timestamp RTP packets with their receive times in TCP/HTTP mode - Spec compliant control url handling - Wrongly implemented in many places - Automatic fallback for non-compliant servers - Property to force non-compliant urls if needed

## What else? WebRTC!

### WebRTC: what is it about? - <i>"How do I stream to a web browser?"</i> - Low latency, works pretty much everywhere. - Different codecs - Leverage all of GStreamer: <br/>transmuxing, hw-acceleration etc.

### webrtcbin (low level) - Better negotiation - Better spec compliance

### WebRTC made easy #### webrtcsink + webrtcsrc - Batteries-included WebRTC sender + receiver: - codec negotiation - congestion control - automatic bitrate adaptation

### WebRTC made easy - cont'd #### webrtcsink + webrtcsrc - Signallers and Associated Elements - LiveKit, Amazon Kinesis, Janus VideoRoom - WHEP, WHIP - New webrtcsrc to complement webrtcsink

### WebRTC made easy #### New webrtcsink + webrtcsrc features - Support for already-encoded streams - Raw payload support (raw audio, video) - More encoders - Built-in embedded signalling / web server

### Moving on to other topics..

## Web Technologies - Media Source Extension (MSE) - Encrypted Media Extensions (EME) - Wasm

## Analytics + Machine Learning - New GstAnalyticsRelationMeta - observations and relationships - Object detection, classification and tracking

## Analytics - cont'd - ONNX elements - onnxinference - ssdobjectdetector (to be split out)

## Analytics - cont'd - Analytics overlay - visualises objects - originalbuffersave, originalbufferrestore - Restore the original buffer after analysis on a transformed version

## Playback - playbin3, uridecodebin3, parsebin - better stream selection, gapless handling, buffering - many corner case stability/reliability issues fixed - Todo: allow decoder selection tweaking - GstPlay library + gst-play-1.0 - use playbin3 by default now

## Capture + Playout Cards - **AJA** sink + source upstream now - incl. device provider - HANC/VANC ancillary data support - **Decklink** - HDR output and input (PQ + HLG) - more modes, devices - better frame scheduling

## Pipeline Splitting - For robustness + easier handling - Multiple pipelines, either: - within the same application - in different processes on same machine (IPC) - Typically 1:1 or 1:N

### Pipeline Splitting: intra-process - New inter plugin for 1:N and any kind of data - inspired by gst-interpipes - some restrictions for now

### Pipeline Splitting: inter-process - **Send video or data to another process (IPC)**: - New **unixfdsink**, **unixfdsrc** - uses memfd + dmabuf - Windows **D3D11** + **D3D12** video - **CUDA** video sink + source - **GstMeta serialisation + deserialisation** - for certain common metas - used in unixfd and CUDA elements

### Closed Captions - cea608tocea708 - tttocea708 - cea708mux and cea608mux - cea708overlay - cea708 insertion in more encoders - caption generation improvements in transcriberbin

### Speech to Text:<br/>Transcription + Translation - Translation support in transcriberbin - awstranscriber gained translation support - multiple languages at the same time - New Speechmatics transcriber - includes translation support - can be deployed on-premises, unlike aws

### Ancillary Data - GstMeta for SMPTE ST-291M HANC/VANC <br/>Ancillary Data - SMPTE ST-2038 ANC support in MPEG-TS - ID3 in MPEG-TS

## Tracing + Debugging ### New Tracers - **Pad push durations** tracer - measures the time it takes for a buffer/buffer list push to return - **Save pad data flow** into a .pcap file

## New Codec Support - **LCEVC: Low Complexity Enh'ment Video Coding** - enhancement on top of different base codec layer - decoder and encoder - **JPEG XS** - low latency, for production workflows - MPEG-TS container support implemented - Todo: RTP, MP4, MXF

### Video Formats - More video pixel formats - More software conversion fast paths - More formats supported in OpenGL, D3D11, D3D12 - Bayer 10/12/14/16-bit depth support

### Hardware-accelerated video - **DRM modifiers support** for Linux dmabufs - plus negotiation, handling in va, wayland, msdk - waylandsink **DRM Dumb allocator support** + 10bit support - Lots of work on Vulkan and Vulkan video codecs support

### video4linux - Lots of improvements all over the place - Stateful and stateless codec support - New stateless AV1 decoder - Stateless decoders tested on CI using QEMU + visl - Encoder support for keyframe requests - Stateful decoders HDR10 support

### VA-API support - va replaces gstreamer-vaapi - Based on shared codec base classes - Just works™ \o/ - Decoders have PRIMARY+1 rank - New AV1, VP9, JPEG encoders - More rate control modes (ICQ, QVBR) - Performance improvements - Support for more pixel formats

### NVIDIA + CUDA - Desktop GPU decoders + encoders rename - `nvh{h264,h265,vp8,vp9}sldec` to `nv{h264,h265,vp8,vp9}dec` - `nvcuda{h264,h265}enc` to `nv{h264,h265}enc` - AV1 encoder added - Encoder RGB formats support - CUDA stream integration support - D3D12 integration - nvCOMP plugin for lossless compression on GPU

### Audio - ASIO: build without SDK headers - always ship the plugin - DSD audio support - LC3 bluetooth codec

### MacOS - audio sink source improvements, latency etc. - gst-validate support (gst_macos_main, lldb) - vtenc hardware encoder improvements - incl. HEVC alpha encoding support - Apple AAC encoder (atenc)

### Android - Media CODEC ported to the native API - reduces amount of Java <-> native calls<br/>= better performance - AV1 decoder/encoder - Rust plugins shipped in binary packages

### Windows - New Microsoft WebView2 based web browser source - DirectWrite text rendering - D3D12 plugin, support and integrations - d3d12swapchainsink for Windows composition API based apps - PTP clock support is now also available on Windows - Many many other improvements

## UI Toolkit Integration ### GTK4 paintable sink - performance + integration improvements - dmabuf import support - support for rotations / flipping / scaling - fullscreen property - black background setting - GL support on Windows

## UI Toolkit Integration ### Qt6 elements - OpenGL src, sink, mixer, overlay - D3D11 sink for Windows

## Editing Services Library - Reverse playback support - encodebin2 support - can use a muxing sink, e.g. HLS/DASH/splitmux

### Streaming: HLS - hlscmafsink for serving fragmented MP4 in HLS - Lots of Rust fmp4 muxer improvements - incl. support for AV1, VP8, Opus, FLAC - better handling of caps changes

### Streaming - New **QUIC source and sink**<br/>(quinnquicsrc, quinnquicsink) - New **mpegtslive element** that wraps an existing live MPEG-TS source (udp, srt) - Provides a clock based on the PCR of the stream

### Miscellanea - FFmpeg 7.0 support - ORC AVX2 support - splitmuxsrc/sink: dynamic fragment addition - jpegparse has a rank and is now autoplugged - wpesrc: WPEWebkit 2.0 support

## The Future ...

### More Rust rewrites? - Demuxers - Parsers - WebRTC? (ICE wip) - RTSP server?

### Analytics, Analytics, Analytics! ### Machine Learning! Stay tuned

### Re-organise modules? Update and simplify our module story

### Continuous Integration - Run tests and integration testsuite<br/>on Windows + macOS - validate gained macOS support - Test hardware acceleration support in upstream CI - More sanitisers (ASAN, USBSAN etc)

### Nightly builds - Has been on the list for a while - More infrastructure work required - Goal: create release binaries on the CI - Signing of binaries would be nice too

### Tracing + Debugging improvements - more tracers - nice UIs - improve output side (common log/trace formats)

### Improve PipeWire integration Existing plugins not in great shape

### HLS/DASH authoring improvements - Master playlist management for HLS alternate renditions - Low-latency HLS

## That's all! ## Thank you! ## Questions?