Streaming audio and video in sync for mp4 container using Gstreamer framework
I am new to gstreamer and I want to stream a mp4 video which is having audio and video both from my Host(Ubuntu PC) to target board. I have successfully streamed only video. But streaming both(audio and video) is not happening.
Pipeline to stream only video is as below
Sender:
gst-launch-1.0 -v filesrc location = video.mp4 ! decodebin ! x264enc ! rtph264pay ! udpsink host=192.168.2.136 port=5000
receiver:
gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96, ssrc=3394826012, timestamp-offset=2215812541, seqnum-offset=46353" ! rtph264depay ! avdec_h264 ! videoconvert ! ximagesink sync=false
Hence, can see video on target.
Did streaming of only audio from mp4 file.
Pipeline to stream only video is as below
Sender:
gst-launch-1.0 -v filesrc location=video.mp4 ! qtdemux name=demuxer demuxer. ! queue ! rtpmp4apay pt=97 ! udpsink host=192.168.2.69 port=5000
Receiver:
gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=audio, clock-rate=44100, encoding-name=MP4A-LATM, cpresent=0, config=40002420, payload=97, ssrc=3386264266, timestamp-offset=2822202855, seqnum-offset=17719" ! rtpmp4adepay ! decodebin ! alsasink &
This too runs successfully.
But when I tried below pipeline to stream both audio/video in sync:
Sender:
gst-launch-1.0 -v filesrc location=sample.mp4 ! qtdemux name=demux demux. ! queue ! rtph264pay pt=96 ! udpsink host=192.168.3.101 port=5000 demux. ! queue ! rtpmp4apay pt=97 ! udpsink host=192.168.3.101 port=5001
Receiver:
gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96, ssrc=2226494377, timestamp-offset=3242004369, seqnum-offset=17021" ! rtph264depay ! decodebin ! ximagesink udpsrc port=5001 caps="application/x-rtp, media=audio, clock-rate=44100, encoding-name=MP4A-LATM, cpresent=0, config=40002420, payload=97, ssrc=1499706502, timestamp-offset=190741668, seqnum-offset=24774" ! rtpmp4adepay ! decodebin ! alsasink
But: get an output as below:
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin1/GstFaad:faad0: Could not decode stream.
fails. Please can someone suggest me a pipeline which can stream audio/video both on target device in sync.
Thanks in advance.
There is no easy way to stream in sync with gstreamer if you demux the video and audio stream on the sender side. Timestamp information is kept in the container (MP4 in this case), which is lost when demuxing.
Try to send the MP4 over to the receiver, and then demux there. A basic example may be found here: Streaming Mp4 video through RTP protocol using Gstreamer in Ubuntu
There seem to be issues with AAC in RTP as well as other RTP payloaders in gstreamer.
As mentioned in the other answer, it is anyway the best strategy to not demux and split video and audio if it is desired to have a synchronized playback on the receiver side. Additionally, a container format that is streamable can also improve overall functionality.
My proposal is therefore to remux the data streams into MPEG2-TS and transport that via RTP. This has several advantages, like good re-synchronization after packet loss and that A/V always stays sync.
Example
This works for me with gstreamer 1.2.4:
Sender
gst-launch-1.0 -v filesrc location=/foo/bar.mp4 !
qtdemux name=demux demux. ! h264parse config-interval=10 ! queue ! mux.
mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=10.0.0.1 port=5003
demux. ! aacparse ! queue ! mux.
Receiver (file)
gst-launch-1.0 -v udpsrc port=5003 caps="application/x-rtp" !
rtpmp2tdepay ! tsparse ! filesink location=/foo/bar.ts
Receiver (output)
This command is based on the commands in the question, could not test it with ALSA:
gst-launch-1.0 -v udpsrc port=5003 caps="application/x-rtp" !
rtpmp2tdepay ! tsparse ! tsdemux name=demux
demux. ! queue ! audioconvert ! alsasink
demux. ! queue ! xvimagesink