Fix gstreamer backend with manual pipelines #24243
- Fix broken seeking in audio/video playback
- Fix broken audio playback
- Fix unreliable seeking
- Estimate frame count if it is not available directly
- Return -1 for frame count and fps if it is not available.
- Return 0 for fps if the video has variable frame rate
- Enable and fix tests
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [X] I agree to contribute to the project under Apache 2 License.
- [X] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [X] The PR is proposed to the proper branch
- [-] There is a reference to the original bug report and related work => Reproducible test provided
- [-] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [X] The feature is well documented and sample code can be built with the project CMake
1. Download two test videos:
```bash
wget https://github.com/ietf-wg-cellar/matroska-test-files/raw/master/test_files/test1.mkv
wget https://test-videos.co.uk/vids/jellyfish/mkv/360/Jellyfish_360_10s_5MB.mkv
```
2. I modified a OpenCV videoio sample to demonstrate the problem, here it is the patch: http://dpaste.com//C9MAT2K6W
3. Build the sample, on Ubuntu:
```bash
g++ -g videocapture_audio_combination.cpp -I/usr/include/opencv4 `pkg-config --libs --cflags opencv4` -o videocapture_audio_combination
```
4. Play an audio stream with seeking BEFORE the fix:
```bash
$ ./videocapture_audio_combination --audio "filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink"[ERROR:0@0.009] global cap.cpp:164 open VIDEOIO(GSTREAMER): raised OpenCV exception:
OpenCV(4.8.0-dev) ./modules/videoio/src/cap_gstreamer.cpp:153: error: (-215:Assertion failed) ptr in function 'get'
[ WARN:0@0.009] global cap.cpp:204 open VIDEOIO(GSTREAMER): backend is generally available but can't be used to capture by name
ERROR! Can't to open file: filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink
```
5. Play a video stream with seeking BEFORE the fix:
```bash
$ ./videocapture_audio_combination --audio "filesrc location=Jellyfish_360_10s_5MB.mkv ! queue ! matroskademux name=demux demux.video_0 ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1"
[ WARN:0@0.034] global cap_gstreamer.cpp:1728 open OpenCV | GStreamer warning: Cannot query video position: status=1, value=22, duration=300
CAP_PROP_AUDIO_DATA_DEPTH: CV_16S
CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 44100
CAP_PROP_AUDIO_TOTAL_CHANNELS: 0
CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.034] global cap_gstreamer.cpp:1898 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported
0
[ WARN:0@0.034] global cap_gstreamer.cpp:1817 getProperty OpenCV | GStreamer: CAP_PROP_POS_MSEC property result may be unrealiable: https://github.com/opencv/opencv/issues/19025
Timestamp: 0.6218
Timestamp: 33.1085
Timestamp: 67.1274
Timestamp: 100.1182
Timestamp: 133.1204
Timestamp: 167.1195
Timestamp: 200.1161
Timestamp: 233.1147
Timestamp: 267.1194
Timestamp: 300.1202
[ WARN:0@0.338] global cap_gstreamer.cpp:1949 setProperty OpenCV | GStreamer warning: GStreamer: unable to seek
0:00:00.338215907 3892572 0x5592899c7580 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop:<filesrc0> error: Internal data stream error.
0:00:00.338235884 3892572 0x5592899c7580 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop:<filesrc0> error: streaming stopped, reason not-linked (-1)
0:00:00.338264287 3892572 0x5592899c7580 WARN queue gstqueue.c:992:gst_queue_handle_sink_event:<queue0> error: Internal data stream error.
0:00:00.338270329 3892572 0x5592899c7580 WARN queue gstqueue.c:992:gst_queue_handle_sink_event:<queue0> error: streaming stopped, reason not-linked (-1)
[ WARN:0@0.339] global cap_gstreamer.cpp:2784 handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module filesrc0 reported: Internal data stream error.
[ WARN:0@0.339] global cap_gstreamer.cpp:1199 startPipeline OpenCV | GStreamer warning: unable to start pipeline
Number of audio samples: 0
Number of video frames: 10
[ WARN:0@0.339] global cap_gstreamer.cpp:1164 isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
```
6. Play an audio stream with seeking AFTER the fix:
```bash
$ ./videocapture_audio_combination --audio "filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink"CAP_PROP_AUDIO_DATA_DEPTH: CV_16S
CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 48000
CAP_PROP_AUDIO_TOTAL_CHANNELS: 2
CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.025] global cap_gstreamer.cpp:1903 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported
0
Timestamp: 0.0000
Timestamp: 24.0000
Timestamp: 48.0000
Timestamp: 72.0000
Timestamp: 96.0000
Timestamp: 120.0000
Timestamp: 144.0000
Timestamp: 168.0000
Timestamp: 192.0000
Timestamp: 216.0000
Timestamp: 3500.0000
Timestamp: 3504.0000
Timestamp: 3528.0000
Timestamp: 3552.0000
Timestamp: 3576.0000
Timestamp: 3600.0000
Timestamp: 3624.0000
Timestamp: 3648.0000
Timestamp: 3672.0000
Timestamp: 3696.0000
Timestamp: 3720.0000
Timestamp: 3744.0000
Timestamp: 3768.0000
Timestamp: 3792.0000
Timestamp: 3816.0000
Timestamp: 3840.0000
Timestamp: 3864.0000
Timestamp: 3888.0000
Timestamp: 3912.0000
Timestamp: 3936.0000
```
7. Play a video stream with seeking AFTER the fix:
```bash
$ ./videocapture_audio_combination --audio "filesrc location=Jellyfish_360_10s_5MB.mkv ! queue ! matroskademux name=demux demux.video_0 ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1"
[ WARN:0@0.033] global cap_gstreamer.cpp:1746 open OpenCV | GStreamer warning: Cannot query video position: status=1, value=22, duration=300
CAP_PROP_AUDIO_DATA_DEPTH: CV_16S
CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 44100
CAP_PROP_AUDIO_TOTAL_CHANNELS: 0
CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.034] global cap_gstreamer.cpp:1903 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported
0
Timestamp: 0.0000
Timestamp: 33.0000
Timestamp: 67.0000
Timestamp: 100.0000
Timestamp: 133.0000
Timestamp: 167.0000
Timestamp: 200.0000
Timestamp: 233.0000
Timestamp: 267.0000
Timestamp: 300.0000
0:00:00.335931693 3893501 0x55bbe76ad920 WARN matroskareadcommon matroska-read-common.c:759:gst_matroska_read_common_parse_skip:<demux:sink> Unknown CueTrackPositions subelement 0xf0 - ignoring
0:00:00.335952823 3893501 0x55bbe76ad920 WARN matroskareadcommon matroska-read-common.c:759:gst_matroska_read_common_parse_skip:<demux:sink> Unknown CueTrackPositions subelement 0xf0 - ignoring
0:00:00.335988029 3893501 0x55bbe76ad920 WARN basesrc gstbasesrc.c:1742:gst_base_src_perform_seek:<filesrc0> duplicate event found 184
Timestamp: 3467.0000
Timestamp: 3500.0000
Timestamp: 3533.0000
Timestamp: 3567.0000
Timestamp: 3600.0000
Timestamp: 3633.0000
Timestamp: 3667.0000
Timestamp: 3700.0000
Timestamp: 3733.0000
Timestamp: 3767.0000
Timestamp: 3800.0000
Timestamp: 3833.0000
Timestamp: 3867.0000
Timestamp: 3900.0000
Timestamp: 3933.0000
Timestamp: 3967.0000
Timestamp: 4000.0000
Timestamp: 4033.0000
Timestamp: 4067.0000
Timestamp: 4100.0000
```
Adding HEVC/H265 FourCC support to MSMF video writer
* Adding HEVC/H265 fourcc to MSMF video writer
Adding HEVC/H265 fourcc to MSMF video writer. I have verified it with my own video input stream, and it works well on my workstation.
* Update video io testing
* Adding macro fence to get rid of compiler error
H265/HEVC encoder is only available in Windows or later. https://learn.microsoft.com/en-us/windows/win32/medfound/h-265---hevc-video-encoder
* Update test_video_io.cpp
videoio: HW decode/encode in FFMPEG backend; new properties with support in FFMPEG/GST/MSMF
* HW acceleration in FFMPEG backend
* fixes on Windows, remove D3D9
* HW acceleration in FFMPEG backend
* fixes on Windows, remove D3D9
* improve va test
* Copyright
* check LIBAVUTIL_BUILD >= AV_VERSION_INT(55, 78, 100) // FFMPEG 3.4+
* CAP_MSMF test on .mp4
* .mp4 in test
* improve va test
* Copyright
* check LIBAVUTIL_BUILD >= AV_VERSION_INT(55, 78, 100) // FFMPEG 3.4+
* CAP_MSMF test on .mp4
* .mp4 in test
* .avi for GStreamer test
* revert changes around seek()
* cv_writer_open_with_params
* params.warnUnusedParameters
* VideoCaptureParameters in GStreamer
* open_with_params
* params->getUnused
* Reduce PSNR threshold 33->32 (other tests use 30)
* require FFMPEG 4.0+; PSNR 30 as in other tests
* GStreamer AVI-demux plugin not installed in Ubuntu test environment?
* fix build on very old ffmpeg
* fix build on very old ffmpeg
* fix build issues
* fix build issues (static_cast)
* FFMPEG built on Windows without H264 encoder?
* fix for write_nothing test on VAAPI
* fix warnings
* fix cv_writer_get_prop in plugins
* use avcodec_get_hw_frames_parameters; more robust fallback to SW codecs
* internal function hw_check_device() for device check/logging
* two separate tests for HW read and write
* image size 640x480 in encode test
* WITH_VA=ON (only .h headers used in OpenCV, no linkage dependency)
* exception on VP9 SW encoder?
* rebase master; refine info message
* videoio: fix FFmpeg standalone plugin build
* videoio(ffmpeg): eliminate MSVC build warnings
* address review comments
* videoio(hw): update videocapture_acceleration.read test
- remove parallel decoding by SW code path
- check PSNR against the original generated image
* videoio: minor fixes
* videoio(test): disable unsupported MSMF cases (SW and HW)
* videoio(test): update PSNR thresholds for HW acceleration read
* videoio(test): update debug messages
* "hw_acceleration" whitelisting parameter
* little optimization in test
* D3D11VA supports decoders, doesn't support encoders
* videoio(test): adjust PSNR threshold in write_read_position tests
* videoio(ffmpeg): fix rejecting on acceleration device name mismatch
* videoio(ffmpeg): fix compilation USE_AV_HW_CODECS=0, add more debug logging
* videoio: rework VideoAccelerationType behavior
- enum is not a bitset
- default value is backend specific
- only '_NONE' and '_ANY' may fallback on software processing
- specific H/W acceleration doesn't fallback on software processing. It fails if there is no support for specified H/W acceleration.
* videoio(test): fix for current FFmpeg wrapper
Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>
Enable frame timestamp tests for MSMF
Add functional test for camera live timestamps
Remove trailing whitespace
Add timestamp test to all functional tests. Protect div by 0
Add Timestamps to MSMF Video Capture by index
- Suppressed FFMPEG + h264, h265 as it does not pass tests with CI configuration.
- Suppressed MediaFoundation backend as it always returns zero for now.
Update for MSMF-based VideoCapture and VideoWriter (#11092)
* MSMF based VideoCapture updated to handle video stream formats different from RGB24
* MSMF based VideoWriter updated to handle video frame top-bottom line ordering regardless of output format
* Fixed race condition in MSMF based VideoCapture
* Refactored MSMF based VideoCapture and VideoWriter
* Disabled frame rate estimation for MP43
* Removed test for unsupported avi container from MSMF VideoWriter tests
* Enabled MSMF-based VideoIO by default
