What is? Gstreamer？

Gstreamer It's a support Windows,Linux,Android, iOS Cross platform multimedia framework , Applications can be piped （Pipeline） The way , Concatenate the steps of multimedia processing , Achieve the desired effect . Each step passes through the element （Element） be based on GObject Object system through plug-ins （plugins） The way to achieve , Facilitate the expansion of various functions .

The figure below is based on Gstreamer Simple layering of application of framework ：

Media Applications

The top layer is application , such as gstreamer Some of our own tools （gst-launch,gst-inspect etc. ）, And based on gstreamer Wrap the library （gst-player,gst-rtsp-server,gst-editing-services etc. ) Applications implemented according to different scenarios .

Core Framework

The middle floor is Core Framework, Mainly provide ：

• Interface required for upper application
• Plugin Framework
• Pipline Framework
• The data is in each Element Transmission and processing mechanism between
• Multiple media streams （Streaming） The synchronization between the （ For example, audio and video synchronization ）
• All kinds of other tools needed

Plugins

The bottom layer is all kinds of plug-ins , Realize specific data processing and audio and video output , Applications don't need to pay attention to the details of plug-ins , Will be Core Framework Layer is responsible for the loading and management of plug-ins . The main classification is ：

• Protocols： Deal with all kinds of agreements ,file,http,rtsp etc. .
• Sources： Responsible for data source processing ,alsa,v4l2,tcp/udp etc. .
• Formats： Responsible for handling media containers ,avi,mp4,ogg etc. .
• Codecs： Be responsible for the coding and decoding of media ,mp3,vorbis etc. .
• Filters： In charge of media streaming ,converters,mixers,effects etc. .
• Sinks： Responsible for media stream output to specified devices or destinations ,alsa,xvideo,tcp/udp etc. .

Gstreamer The framework is based on the maturity of each module and the open source protocol used , take core And plugins In different source packages ：

• gstreamer: contain core framework And core elements.
• gst-plugins-base: gstreamer Necessary plug-ins for application .
• gst-plugins-good: High quality adoption LGPL Authorized plug-ins .
• gst-plugins-ugly: High-quality , But using the GPL And other library plug-ins for authorization , For example, use GPL Of x264,x265.
• gst-plugins-bad: Plugins with quality to be improved , When it's mature, it can move to good Plug in list .
• gst-libav: Yes libav encapsulation , To enable it to be in gstreamer Use... In the framework .

Gstreamer Basic concepts

Learning more about Gstreamer front , We need to master some gstreamer The basic concept of .

Element

Element yes Gstreamer One of the most important object types in . One element Implement a function （ Read the file , decode , Output, etc ）, The program needs to create multiple element, And connect them in order , Make a whole pipeline.

Pad

Pad It's a element The input of / Output interface , It is divided into src pad（ The production data ） and sink pad（ Consumption data ） Two kinds of .
Two element Must pass pad To connect ,pad Have the present element The ability to handle data types （capabilities）, Will be connected by comparing src pad and sink pad The capabilities supported in , To select the most appropriate data type for transmission , If element I won't support it , The program will exit directly . stay element adopt pad After successful connection , The data will come from the last element Of src pad Pass on to the next element Of sink pad And then deal with it .
When element When supporting multiple data processing capabilities , We can go through Cap To specify the data type .
for example , The following order passes Cap Specifies the width and height of the video ,videotestsrc The corresponding data will be generated according to the specified width and height ：

gst-launch-1.0 videotestsrc ! "video/x-raw,width=1280,height=720" ! autovideosink

Bin and Pipeline

Bin It's a container , Used to manage multiple element, change bin When ,bin Will automatically modify the included element The state of , It will also forward the received messages . without bin, We need to operate in turn what we use element. adopt bin Reduce the complexity of application .
Pipeline Inherited from bin, Provide a... For the program bus Used to transmit messages , And for all children element To synchronize . When will pipeline Is set to PLAYING when ,pipeline In a / Multiple new threads pass through element Processing data .

Let's get familiar with the concept mentioned above through an example of file playing ： The test file  sintel_trailer-480p.ogv

gst-launch-1.0 filesrc location=sintel_trailer-480p.ogv ! oggdemux name=demux ! queue ! vorbisdec ! autoaudiosink demux. ! queue ! theoradec ! videoconvert ! autovideosink

When playing a file through the above command , The following will be created pipeline：

You can see this pipeline from 8 individual element constitute , Every element All realize their own functions ：
filesrc Read the file ,oggdemux Parsing files , Extract separately audio,video data ,queue Cache data ,vorbisdec decode audio,autoaudiosink Automatically select audio device and output ,theoradec decode video,videoconvert transformation video data format ,autovideosink Automatically select display device and output .

Different element There are different numbers and types of pad, Only src pad Of element go by the name of source element, Only sink pad Is called sink element.

element You can have more than one of the same pad, for example oggdemux After parsing the file , Will audio,video Through different pad Output .

Gstreamer Data message interaction

stay pipeline In the process of operation , each element And the inevitable need for data message transmission between applications ,gstreamer Provides bus Systems and multiple data types （Buffers、Events、Messages,Queries） To achieve this goal ：

Bus

Bus yes gstreamer Internally used to make messages different from internally streaming Threads , Pass on to bus Threads , Again by bus The thread sends the message to the application . The application only needs to bus Register message handler , To receive pipline Each of them element The message sent , Use bus after , The application doesn't care which thread the message is sent from , Avoid the complexity of dealing with multiple threads sending messages at the same time .

Buffers

For from sources To sinks Media data transmission .

Events

be used for element Or apply to element The transmission of information between , For example, when playing seek The operation is through event Realized .

Messages

By element A message sent , adopt bus, Be processed by the application asynchronously . Usually used to convey errors, tags, state changes, buffering state, redirects Wait for news . Message processing is thread safe . Because most messages are processed asynchronously , So there will be a little delay in the application , If you want timely information , Need to be in streaming Thread capture processing .

Queries

For applications to gstreamer Total query time , current time , File size and other information .

gstreamer tools

Gstreamer Bring it with you gst-inspect-1.0 and gst-launch-1.0 And other command-line tools , We can use these tools to accomplish common processing tasks .
gst-inspect-1.0
see gstreamer Of plugin、element Information about . Direct will plugin/element As a parameter , The details will be listed . If it doesn't follow any parameters , The current system... Will be listed gstreamer All plugins that can be found .

$ gst-inspect-1.0 playbin

gst-launch-1.0
Used to create and execute a Pipline, So we usually use gst-launch First, verify the relevant functions , Then write the corresponding application .
Pass above ogg Examples of video playback , We've seen , One pipeline The multiple element Through between “!" Separate , At the same time, you can set element And Cap Properties of . for example ：
Play audio and video

gst-launch-1.0 playbin file:///home/root/test.mp4

transcoding

gst-launch-1.0 filesrc location=/videos/sintel_trailer-480p.ogv ! decodebin name=decode ! \
               videoscale ! "video/x-raw,width=320,height=240" ! x264enc ! queue ! \
               mp4mux name=mux ! filesink location=320x240.mp4 decode. ! audioconvert ! \
               avenc_aac ! queue ! mux.

Streaming

#Server
gst-launch-1.0 -v videotestsrc ! "video/x-raw,framerate=30/1" ! x264enc key-int-max=30 ! rtph264pay ! udpsink host=127.0.0.1 port=1234

#Client
gst-launch-1.0 udpsrc port=1234 ! "application/x-rtp, payload=96" ! rtph264depay ! decodebin ! autovideosink sync=false