Multimedia Formats

Since GStreamer is designed to handle multimedia, an understanding of multimedia formats will help you better understand GStreamer. Video content in streaming video, TV broadcasts, optical discs, etc., contains multiple types of information in a single stream (multiplexed), such as video, audio, subtitles, etc. GStreamer is designed to handle multimedia, so you will need to understand multimedia formats to better understand GStreamer.

In general, video and audio are compressed using a standardized method, called a codec (CODEC). Compression is called encoding and decompression is called decoding.

Multiplexing is also called multiplex/mux, and demultiplexing is also called demultiplex/demux. The term container is sometimes used for multiplexing.

Software that handles them is called encoder, decoder, multiplexer, demultiplexer, etc.

The above figure shows the simplified structure of MPEG-2 Transport Stream (ISO/IEC 13818-1 or ITU-T Rec. H.222.0). The video and audio header + Elementary Stream (= Packetized Elementary Stream) in the lower part is divided and stored in a fixed-length packet (= header + payload) in the upper part. In this way

1. Additional information can be added to the header of the packet that is not present in the video or audio itself.
   Time stamps for A/V synchronization, etc.
2. Video and audio of the same scene are close together in the stream
   Easier to resume playback from the seek position
3. The scope of damage (e.g., due to transmission path failure) is reduced.
4. Various encoding schemes can be selected for video and audio

and other features.

The third feature works in favor of streaming media. For storage media such as optical disks, which are less vulnerable to corruption, another multiplexing scheme may be used, such as MPEG-2 Program Stream (ISO/IEC 13818-1), which is more suitable.

From the fourth feature, multiplexing schemes and the codecs stored in them can be freely combined, although there are some restrictions.

Gstreamer Overview

Since multimedia multiplexing schemes and codecs can be freely combined, it is thought that software for handling multimedia can also reduce design complexity if the corresponding processing can be modularized and combined. In fact, GStreamer has also chosen such a design, combining modules with different functions to build a graph, called a pipeline, as shown in the figure below, to achieve the processing you want to accomplish.

The above figure is a typical example of reading and playing a multimedia file on disk, and the role of each module is

Input can be streaming (http, mms, rtsp) as well as file. In addition to the already introduced MPEG-2 TS/PS, GStreamer can also handle multiplexing formats such asf, avi, 3gp/mp4/mov, flv, mkv/webm, mxf, and ogg. GStreamer can also use FFmpeg, so it supports a variety of audio and video codecs.

GStreamer has been ported to many platforms and is available for Android, iOS, Linux, macOS, Windows for example.

Try it out for yourself

GStreamer comes with a utility that runs on the command line. Here we will actually try to play an ogg/vorbis audio file. The environment is assumed to be Ubuntu 22.04.

$ sudo apt install libgstreamer1.0-0 gstreamer1.0-tools
$ sudo apt install gstreamer1.0-plugins-base gstreamer1.0-plugins-good

Use the above commands to install the required applications and libraries. To try in other environments, install the corresponding packages for your environment.

$ gst-launch-1.0 filesrc location=song.ogg ! decodebin ! audioconvert ! audioresample ! autoaudiosink

To play song.ogg, provide the gst-launch-1.0 command line application with the plug-in and parameter information as shown above. When executed, song.ogg will start playing, play to the end of the file, and then the command line application will exit.

※All names, company names, product names, etc. mentioned herein are trademarks or registered trademarks of their respective companies.

• Read Previous Article
• Return To List
• Read Next Article(JP)