mirror of
https://github.com/opencv/opencv.git
synced 2025-01-19 06:53:50 +08:00
b1e0c4d119
Fix gstreamer backend with manual pipelines #24243 - Fix broken seeking in audio/video playback - Fix broken audio playback - Fix unreliable seeking - Estimate frame count if it is not available directly - Return -1 for frame count and fps if it is not available. - Return 0 for fps if the video has variable frame rate - Enable and fix tests ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [X] I agree to contribute to the project under Apache 2 License. - [X] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [X] The PR is proposed to the proper branch - [-] There is a reference to the original bug report and related work => Reproducible test provided - [-] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [X] The feature is well documented and sample code can be built with the project CMake 1. Download two test videos: ```bash wget https://github.com/ietf-wg-cellar/matroska-test-files/raw/master/test_files/test1.mkv wget https://test-videos.co.uk/vids/jellyfish/mkv/360/Jellyfish_360_10s_5MB.mkv ``` 2. I modified a OpenCV videoio sample to demonstrate the problem, here it is the patch: http://dpaste.com//C9MAT2K6W 3. Build the sample, on Ubuntu: ```bash g++ -g videocapture_audio_combination.cpp -I/usr/include/opencv4 `pkg-config --libs --cflags opencv4` -o videocapture_audio_combination ``` 4. Play an audio stream with seeking BEFORE the fix: ```bash $ ./videocapture_audio_combination --audio "filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink"[ERROR:0@0.009] global cap.cpp:164 open VIDEOIO(GSTREAMER): raised OpenCV exception: OpenCV(4.8.0-dev) ./modules/videoio/src/cap_gstreamer.cpp:153: error: (-215:Assertion failed) ptr in function 'get' [ WARN:0@0.009] global cap.cpp:204 open VIDEOIO(GSTREAMER): backend is generally available but can't be used to capture by name ERROR! Can't to open file: filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink ``` 5. Play a video stream with seeking BEFORE the fix: ```bash $ ./videocapture_audio_combination --audio "filesrc location=Jellyfish_360_10s_5MB.mkv ! queue ! matroskademux name=demux demux.video_0 ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1" [ WARN:0@0.034] global cap_gstreamer.cpp:1728 open OpenCV | GStreamer warning: Cannot query video position: status=1, value=22, duration=300 CAP_PROP_AUDIO_DATA_DEPTH: CV_16S CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 44100 CAP_PROP_AUDIO_TOTAL_CHANNELS: 0 CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.034] global cap_gstreamer.cpp:1898 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported 0 [ WARN:0@0.034] global cap_gstreamer.cpp:1817 getProperty OpenCV | GStreamer: CAP_PROP_POS_MSEC property result may be unrealiable: https://github.com/opencv/opencv/issues/19025 Timestamp: 0.6218 Timestamp: 33.1085 Timestamp: 67.1274 Timestamp: 100.1182 Timestamp: 133.1204 Timestamp: 167.1195 Timestamp: 200.1161 Timestamp: 233.1147 Timestamp: 267.1194 Timestamp: 300.1202 [ WARN:0@0.338] global cap_gstreamer.cpp:1949 setProperty OpenCV | GStreamer warning: GStreamer: unable to seek 0:00:00.338215907 3892572 0x5592899c7580 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop:<filesrc0> error: Internal data stream error. 0:00:00.338235884 3892572 0x5592899c7580 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop:<filesrc0> error: streaming stopped, reason not-linked (-1) 0:00:00.338264287 3892572 0x5592899c7580 WARN queue gstqueue.c:992:gst_queue_handle_sink_event:<queue0> error: Internal data stream error. 0:00:00.338270329 3892572 0x5592899c7580 WARN queue gstqueue.c:992:gst_queue_handle_sink_event:<queue0> error: streaming stopped, reason not-linked (-1) [ WARN:0@0.339] global cap_gstreamer.cpp:2784 handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module filesrc0 reported: Internal data stream error. [ WARN:0@0.339] global cap_gstreamer.cpp:1199 startPipeline OpenCV | GStreamer warning: unable to start pipeline Number of audio samples: 0 Number of video frames: 10 [ WARN:0@0.339] global cap_gstreamer.cpp:1164 isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created ``` 6. Play an audio stream with seeking AFTER the fix: ```bash $ ./videocapture_audio_combination --audio "filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink"CAP_PROP_AUDIO_DATA_DEPTH: CV_16S CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 48000 CAP_PROP_AUDIO_TOTAL_CHANNELS: 2 CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.025] global cap_gstreamer.cpp:1903 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported 0 Timestamp: 0.0000 Timestamp: 24.0000 Timestamp: 48.0000 Timestamp: 72.0000 Timestamp: 96.0000 Timestamp: 120.0000 Timestamp: 144.0000 Timestamp: 168.0000 Timestamp: 192.0000 Timestamp: 216.0000 Timestamp: 3500.0000 Timestamp: 3504.0000 Timestamp: 3528.0000 Timestamp: 3552.0000 Timestamp: 3576.0000 Timestamp: 3600.0000 Timestamp: 3624.0000 Timestamp: 3648.0000 Timestamp: 3672.0000 Timestamp: 3696.0000 Timestamp: 3720.0000 Timestamp: 3744.0000 Timestamp: 3768.0000 Timestamp: 3792.0000 Timestamp: 3816.0000 Timestamp: 3840.0000 Timestamp: 3864.0000 Timestamp: 3888.0000 Timestamp: 3912.0000 Timestamp: 3936.0000 ``` 7. Play a video stream with seeking AFTER the fix: ```bash $ ./videocapture_audio_combination --audio "filesrc location=Jellyfish_360_10s_5MB.mkv ! queue ! matroskademux name=demux demux.video_0 ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1" [ WARN:0@0.033] global cap_gstreamer.cpp:1746 open OpenCV | GStreamer warning: Cannot query video position: status=1, value=22, duration=300 CAP_PROP_AUDIO_DATA_DEPTH: CV_16S CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 44100 CAP_PROP_AUDIO_TOTAL_CHANNELS: 0 CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.034] global cap_gstreamer.cpp:1903 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported 0 Timestamp: 0.0000 Timestamp: 33.0000 Timestamp: 67.0000 Timestamp: 100.0000 Timestamp: 133.0000 Timestamp: 167.0000 Timestamp: 200.0000 Timestamp: 233.0000 Timestamp: 267.0000 Timestamp: 300.0000 0:00:00.335931693 3893501 0x55bbe76ad920 WARN matroskareadcommon matroska-read-common.c:759:gst_matroska_read_common_parse_skip:<demux:sink> Unknown CueTrackPositions subelement 0xf0 - ignoring 0:00:00.335952823 3893501 0x55bbe76ad920 WARN matroskareadcommon matroska-read-common.c:759:gst_matroska_read_common_parse_skip:<demux:sink> Unknown CueTrackPositions subelement 0xf0 - ignoring 0:00:00.335988029 3893501 0x55bbe76ad920 WARN basesrc gstbasesrc.c:1742:gst_base_src_perform_seek:<filesrc0> duplicate event found 184 Timestamp: 3467.0000 Timestamp: 3500.0000 Timestamp: 3533.0000 Timestamp: 3567.0000 Timestamp: 3600.0000 Timestamp: 3633.0000 Timestamp: 3667.0000 Timestamp: 3700.0000 Timestamp: 3733.0000 Timestamp: 3767.0000 Timestamp: 3800.0000 Timestamp: 3833.0000 Timestamp: 3867.0000 Timestamp: 3900.0000 Timestamp: 3933.0000 Timestamp: 3967.0000 Timestamp: 4000.0000 Timestamp: 4033.0000 Timestamp: 4067.0000 Timestamp: 4100.0000 ``` |
||
|---|---|---|
| .. | ||
| cap_obsensor | ||
| cap_winrt | ||
| backend_plugin_legacy.impl.hpp | ||
| backend_plugin.cpp | ||
| backend_static.cpp | ||
| backend.hpp | ||
| cap_android_camera.cpp | ||
| cap_android_mediandk.cpp | ||
| cap_aravis.cpp | ||
| cap_avfoundation_mac.mm | ||
| cap_avfoundation.mm | ||
| cap_dc1394_v2.cpp | ||
| cap_dshow.cpp | ||
| cap_dshow.hpp | ||
| cap_ffmpeg_hw.hpp | ||
| cap_ffmpeg_impl.hpp | ||
| cap_ffmpeg_legacy_api.hpp | ||
| cap_ffmpeg.cpp | ||
| cap_gphoto2.cpp | ||
| cap_gstreamer.cpp | ||
| cap_images.cpp | ||
| cap_interface.hpp | ||
| cap_ios_abstract_camera.mm | ||
| cap_ios_photo_camera.mm | ||
| cap_ios_video_camera.mm | ||
| cap_librealsense.cpp | ||
| cap_librealsense.hpp | ||
| cap_mfx_common.cpp | ||
| cap_mfx_common.hpp | ||
| cap_mfx_plugin.cpp | ||
| cap_mfx_reader.cpp | ||
| cap_mfx_reader.hpp | ||
| cap_mfx_writer.cpp | ||
| cap_mfx_writer.hpp | ||
| cap_mjpeg_decoder.cpp | ||
| cap_mjpeg_encoder.cpp | ||
| cap_msmf.cpp | ||
| cap_msmf.hpp | ||
| cap_obsensor_capture.cpp | ||
| cap_obsensor_capture.hpp | ||
| cap_openni2.cpp | ||
| cap_pvapi.cpp | ||
| cap_ueye.cpp | ||
| cap_v4l.cpp | ||
| cap_winrt_bridge.cpp | ||
| cap_winrt_bridge.hpp | ||
| cap_winrt_capture.cpp | ||
| cap_winrt_capture.hpp | ||
| cap_winrt_video.cpp | ||
| cap_winrt_video.hpp | ||
| cap_ximea.cpp | ||
| cap_xine.cpp | ||
| cap.cpp | ||
| container_avi.cpp | ||
| ffmpeg_codecs.hpp | ||
| plugin_api.hpp | ||
| plugin_capture_api.hpp | ||
| plugin_writer_api.hpp | ||
| precomp.hpp | ||
| videoio_c.cpp | ||
| videoio_registry.cpp | ||
| videoio_registry.hpp | ||
| wrl.h | ||