What is JPEG XS?
JPEG XS is a visually lossless, low-latency, intra-only video codec for video production workflows, standardised in ISO/IEC 21122.
It's wavelet based, with low computational overhead and a latency measured in scanlines, and it is designed to allow easy implementation in software, GPU or FPGAs.
Multi-generation robustness means repeated decoding and encoding will not introduce unpleasant coding artefacts or noticeably degrade image quality, which makes it suitable for video production workflows.
It is often deployed in lieu of existing raw video workflows, where it allows sending multiple streams over links designed to carry a single raw video transport.
JPEG XS encoding / decoding in GStreamer
GStreamer now gained basic support for this codec.
Encoding and decoding is supported via the Open Source Intel Scalable Video Technology JPEG XS library, but third-party GStreamer plugins that provide GPU accelerated encoding and decoding exist as well.
MPEG-TS container mapping
Support was also added for carriage inside MPEG-TS which should enable a wide range of streaming applications including those based on the Video Services Forum (VSF)'s Technical Recommendation TR-07.
JPEG XS caps in GStreamer
It actually took us a few iterations to come up with GStreamer caps that we were somewhat happy with for starters.
Our starting point was what the SVT encoder/decoder output/consume, and our initial target was MPEG-TS container format support.
We checked various specifications to see how JPEG XS is mapped there and how it could work, in particular:
- ISO/IEC 21122-3 (Part 3: Transport and container formats)
- MPEG-TS JPEG XS mapping and VSF TR-07 - Transport of JPEG XS Video in MPEG-2 Transport Stream over IP
- RFC 9134: RTP Payload Format for ISO/IEC 21122 (JPEG XS)
- SMPTE ST 2124:2020 (Mapping JPEG XS Codestreams into the MXF)
- MP4 mapping
and we think the current mapping will work for all of those cases.
Basically each mapping wants some extra headers in addition to the codestream
data, for the out-of-band signalling required to make sense of the image
data. Originally we thought about putting some form of codec_data header into
the caps, but it wouldn't really have made anything easier, and would just
have duplicated 99% of the info that's in the video caps already anyway.
The current caps mapping is based on ISO/IEC 21122-3, Annex D, with additional metadata in the caps, which should hopefully work just fine for RTP, MP4, MXF and other mappings in future.
Please give it a spin, and let us know if you have any questions or are interested in additional container mappings such as MP4 or MXF, or RTP payloaders / depayloaders.