mirror/opencv
1
0
Fork 0
mirror of https://github.com/opencv/opencv.git synced 2024-12-05 09:49:12 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
33,486 Commits 6 Branches 126 Tags 3 GiB
9d0c8a9edb
Commit Graph

55 Commits

Author SHA1 Message Date
Wanli
62b5470b78
Merge pull request #24298 from WanliZhong:extend_perf_net_test 
Extend performance test models #24298

**Merged With https://github.com/opencv/opencv_extra/pull/1095**

This PR aims to extend the performance tests. 

- **YOLOv5** for object detection
- **YOLOv8** for object detection
- **EfficientNet** for classification

Models from OpenCV Zoo:

- **YOLOX** for object detection
- **YuNet** for face detection
- **SFace** for face recognization
- **MPPalm** for palm detection
- **MPHand** for hand landmark
- **MPPose** for pose estimation
- **ViTTrack** for object tracking
- **PPOCRv3** for text detection
- **CRNN** for text recognization
- **PPHumanSeg** for human segmentation

If other models should be added, **please leave some comments**. Thanks!



Build opencv with script:
```shell
-DBUILD_opencv_python2=OFF
-DBUILD_opencv_python3=OFF
-DBUILD_opencv_gapi=OFF
-DINSTALL_PYTHON_EXAMPLES=OFF
-DINSTALL_C_EXAMPLES=OFF
-DBUILD_DOCS=OFF
-DBUILD_EXAMPLES=OFF
-DBUILD_ZLIB=OFF
-DWITH_FFMPEG=OFF
```



Performance Test on **Apple M2 CPU**
```shell
MacOS 14.0
8 threads
```

**1 thread:**
| Name of Test | 4.5.5-1th | 4.6.0-1th | 4.7.0-1th | 4.8.0-1th | 4.8.1-1th |
|--------------|:---------:|:---------:|:---------:|:---------:|:---------:|
| CRNN         |  76.244   |  76.611   |  62.534   |  57.678   |  57.238   |
| EfficientNet |    ---    |    ---    |  109.224  |  130.753  |  109.076  |
| MPHand       |    ---    |    ---    |  19.289   |  22.727   |  27.593   |
| MPPalm       |  47.150   |  47.061   |  41.064   |  65.598   |  40.109   |
| MPPose       |    ---    |    ---    |  26.592   |  32.022   |  26.956   |
| PPHumanSeg   |  41.672   |  41.790   |  27.819   |  27.212   |  30.461   |
| PPOCRv3      |    ---    |    ---    |  140.371  |  187.922  |  170.026  |
| SFace        |  43.830   |  43.834   |  27.575   |  30.653   |  26.387   |
| ViTTrack     |    ---    |    ---    |    ---    |  14.617   |  15.028   |
| YOLOX        | 1060.507  | 1061.361  |  495.816  |  533.309  |  549.713  |
| YOLOv5       |    ---    |    ---    |    ---    |  191.350  |  193.261  |
| YOLOv8       |    ---    |    ---    |  198.893  |  218.733  |  223.142  |
| YuNet        |  27.084   |  27.095   |  26.238   |  30.512   |  34.439   |
| MobileNet_SSD_Caffe         |  44.742   |  44.565   |  33.005   |  29.421   |  29.286   |
| MobileNet_SSD_v1_TensorFlow |  49.352   |  49.274   |  35.163   |  32.134   |  31.904   |
| MobileNet_SSD_v2_TensorFlow |  83.537   |  83.379   |  56.403   |  42.947   |  42.148   |
| ResNet_50                   |  148.872  |  148.817  |  77.331   |  67.682   |  67.760   |


**n threads:**
| Name of Test | 4.5.5-nth | 4.6.0-nth | 4.7.0-nth | 4.8.0-nth | 4.8.1-nth |
|--------------|:---------:|:---------:|:---------:|:---------:|:---------:|
| CRNN         |  44.262   |  44.408   |  41.540   |  40.731   |  41.151   |
| EfficientNet |    ---    |    ---    |  28.683   |  42.676   |  38.204   |
| MPHand       |    ---    |    ---    |   6.738   |  13.126   |   8.155   |
| MPPalm       |  16.613   |  16.588   |  12.477   |  31.370   |  17.048   |
| MPPose       |    ---    |    ---    |  12.985   |  19.700   |  16.537   |
| PPHumanSeg   |  14.993   |  15.133   |  13.438   |  15.269   |  15.252   |
| PPOCRv3      |    ---    |    ---    |  63.752   |  85.469   |  76.190   |
| SFace        |  10.685   |  10.822   |   8.127   |   8.318   |   7.934   |
| ViTTrack     |    ---    |    ---    |    ---    |  10.079   |   9.579   |
| YOLOX        |  417.358  |  422.977  |  230.036  |  234.662  |  228.555  |
| YOLOv5       |    ---    |    ---    |    ---    |  74.249   |  75.480   |
| YOLOv8       |    ---    |    ---    |  63.762   |  88.770   |  70.927   |
| YuNet        |   8.589   |   8.731   |  11.269   |  16.466   |  14.513   |
| MobileNet_SSD_Caffe         |  12.575   |  12.636   |  11.529   |  12.114   |  12.236   |
| MobileNet_SSD_v1_TensorFlow |  13.922   |  14.160   |  13.078   |  12.124   |  13.298   |
| MobileNet_SSD_v2_TensorFlow |  25.096   |  24.836   |  22.823   |  20.238   |  20.319   |
| ResNet_50                   |  41.561   |  41.296   |  29.092   |  30.412   |  29.339   |


Performance Test on [Intel Core i7-12700K](https://www.intel.com/content/www/us/en/products/sku/134594/intel-core-i712700k-processor-25m-cache-up-to-5-00-ghz/specifications.html)
```shell
Ubuntu 22.04.2 LTS
8 Performance-cores (3.60 GHz, turbo up to 4.90 GHz)
4 Efficient-cores (2.70 GHz, turbo up to 3.80 GHz)
20 threads
```


**1 thread:**
| Name of Test | 4.5.5-1th | 4.6.0-1th | 4.7.0-1th | 4.8.0-1th | 4.8.1-1th |
|--------------|:---------:|:---------:|:---------:|:---------:|:---------:|
| CRNN         |  16.752   |  16.851   |  16.840   |  16.625   |  16.663   |
| EfficientNet |    ---    |    ---    |  61.107   |  76.037   |  53.890   |
| MPHand       |    ---    |    ---    |   8.906   |   9.969   |   8.403   |
| MPPalm       |  24.243   |  24.638   |  18.104   |  35.140   |  18.387   |
| MPPose       |    ---    |    ---    |  12.322   |  16.515   |  12.355   |
| PPHumanSeg   |  15.249   |  15.303   |  10.203   |  10.298   |  10.353   |
| PPOCRv3      |    ---    |    ---    |  87.788   |  144.253  |  90.648   |
| SFace        |  15.583   |  15.884   |  13.957   |  13.298   |  13.284   |
| ViTTrack     |    ---    |    ---    |    ---    |  11.760   |  11.710   |
| YOLOX        |  324.927  |  325.173  |  235.986  |  253.653  |  254.472  |
| YOLOv5       |    ---    |    ---    |    ---    |  102.163  |  102.621  |
| YOLOv8       |    ---    |    ---    |  87.013   |  103.182  |  103.146  |
| YuNet        |  12.806   |  12.645   |  10.515   |  12.647   |  12.711   |
| MobileNet_SSD_Caffe         |  23.556   |  23.768   |  24.304   |  22.569   |  22.602   |
| MobileNet_SSD_v1_TensorFlow |  26.136   |  26.276   |  26.854   |  24.828   |  24.961   |
| MobileNet_SSD_v2_TensorFlow |  43.521   |  43.614   |  46.892   |  44.044   |  44.682   |
| ResNet_50                   |  73.588   |  73.501   |  75.191   |  66.893   |  65.144   |


**n thread:**
| Name of Test | 4.5.5-nth | 4.6.0-nth | 4.7.0-nth | 4.8.0-nth | 4.8.1-nth | 
|--------------|:---------:|:---------:|:---------:|:---------:|:---------:|
| CRNN         |   8.665   |   8.827   |  10.643   |   7.703   |   7.743   | 
| EfficientNet |    ---    |    ---    |  16.591   |  12.715   |   9.022   |   
| MPHand       |    ---    |    ---    |   2.678   |   2.785   |   1.680   |           
| MPPalm       |   5.309   |   5.319   |   3.822   |  10.568   |   4.467   |       
| MPPose       |    ---    |    ---    |   3.644   |   6.088   |   4.608   |        
| PPHumanSeg   |   4.756   |   4.865   |   5.084   |   5.179   |   5.148   |        
| PPOCRv3      |    ---    |    ---    |  32.023   |  50.591   |  32.414   |      
| SFace        |   3.838   |   3.980   |   4.629   |   3.145   |   3.155   |       
| ViTTrack     |    ---    |    ---    |    ---    |  10.335   |  10.357   |   
| YOLOX        |  68.314   |  68.081   |  82.801   |  74.219   |  73.970   |      
| YOLOv5       |    ---    |    ---    |    ---    |  47.150   |  47.523   |    
| YOLOv8       |    ---    |    ---    |  32.195   |  30.359   |  30.267   |    
| YuNet        |   2.604   |   2.644   |   2.622   |   3.278   |   3.349   |    
| MobileNet_SSD_Caffe         |  13.005   |   5.935   |   8.586   |   4.629   |   4.713   |
| MobileNet_SSD_v1_TensorFlow |   7.002   |   7.129   |   9.314   |   5.271   |   5.213   |
| MobileNet_SSD_v2_TensorFlow |  11.939   |  12.111   |  22.688   |  12.038   |  12.086   |
| ResNet_50                   |  18.227   |  18.600   |  26.150   |  15.584   |  15.706   |
 2023-10-04 13:05:32 +03:00
Dmitry Kurtaev
c7ec0d599a
Merge pull request #23987 from dkurt:openvino_int8_backend 
OpenVINO backend for INT8 models #23987

### Pull Request Readiness Checklist

TODO:
- [x] DetectionOutput layer (https://github.com/opencv/opencv/pull/24069)
- [x] Less FP32 fallbacks (i.e. Sigmoid, eltwise sum)
- [x] Accuracy, performance tests (https://github.com/opencv/opencv/pull/24039)
- [x] Single layer tests (convolution)
- [x] ~~Fixes for OpenVINO 2022.1 (https://pullrequest.opencv.org/buildbot/builders/precommit_custom_linux/builds/100334)~~


Performace results for object detection model `coco_efficientdet_lite0_v1_1.0_quant_2021_09_06.tflite`:
| backend | performance (median time) |
|---|---|
| OpenCV | 77.42ms |
| OpenVINO 2023.0 | 10.90ms |

CPU: `11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz`

Serialized model per-layer stats (note that Convolution should use `*_I8` primitives if they are quantized correctly): https://gist.github.com/dkurt/7772bbf1907035441bb5454f19f0feef

---

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
 2023-09-28 16:24:43 +03:00
Dmitry Kurtaev
8ad5eb521a
Merge pull request #24120 from dkurt:actualize_dnn_links 
OCL_FP16 MatMul with large batch

* Workaround FP16 MatMul with large batch

* Fix OCL reinitialization

* Higher thresholds for INT8 quantization

* Try fix gemm_buffer_NT for half (columns)

* Fix GEMM by rows

* Add batch dimension to InnerProduct layer test

* Fix Test_ONNX_conformance.Layer_Test/test_basic_conv_with_padding

* Batch 16

* Replace all vload4

* Version suffix for MobileNetSSD_deploy Caffe model
 2023-08-16 15:46:11 +03:00
Maksim Shabunin
d35fbe6bfc dnn: updated YOLOv4-tiny model and tests 2022-12-22 15:49:21 +03:00
Zihao Mu
0a650b573b
Merge pull request #22840 from zihaomu:optimze_conv_memory_usage 
DNN: reduce the memory used in convolution layer

* reduce the memory in winograd and disabel the test when usage memory is larger than 2gb.

* remove VERY_LOG tag
 2022-12-08 12:57:13 +00:00
Alexander Alekhin
624d532000 Merge remote-tracking branch 'upstream/3.4' into merge-3.4 2020-12-17 21:05:34 +00:00
Alexander Alekhin
28aab134db dnn(test): update tests for OpenVINO 2021.2 2020-12-17 07:53:35 +00:00
Omar Alzaibaq
a316b11aaa
Merge pull request #18220 from Omar-AE:hddl-supported 
* added HDDL VPU support

* changed to return True in one line if any device connected

* dnn: use releaseHDDLPlugin()

* dnn(hddl): fix conditions
 2020-11-17 19:47:24 +00:00
Alexander Alekhin
6da05f7086 dnn(test): update tests for OpenVINO 2021.1 2020-10-08 10:22:31 +00:00
Alexander Alekhin
1c371d07b5 dnn(test): adjust tests for OpenVINO 2020.4 2020-07-15 23:47:40 +00:00
Alexander Alekhin
99c4b76a6d dnn(test): add YOLOv4-tiny tests 2020-07-06 21:36:19 +00:00
Alexander Alekhin
e58e545584 Merge pull request #17392 from alalek:dnn_test_yolov4 2020-05-28 22:52:21 +00:00
Dmitry Kurtaev
d9bada9867 dnn: EfficientDet 2020-05-28 17:23:42 +03:00
Alexander Alekhin
6b89154afd dnn(test): add YOLOv4 tests 2020-05-28 13:27:40 +00:00
Dmitry Kurtaev
d8e10f3a8d Enable MaxPooling with indices in Inference Engine 2019-12-04 19:14:55 +03:00
Lubov Batanina
7523c777c5 Merge pull request #15537 from l-bat:ngraph 
* Support nGraph

* Fix resize
 2019-12-02 16:16:06 +03:00
Dmitry Kurtaev
6193e403e7 Enable some tests for 2019R2 2019-08-07 09:07:53 +03:00
Dmitry Kurtaev
a0c3bb70a9 Modify SSD from TensorFlow graph generation script to enable MyriadX 2019-07-26 13:57:08 +03:00
Alexander Alekhin
416c693b3f dnn(test): OpenVINO 2019R2 2019-07-25 19:01:16 +03:00
Alexander Alekhin
13a782c039 test: fix usage of findDataFile() 
misused 'optional' mode
 2019-06-20 18:20:14 +03:00
Dmitry Kurtaev
9c0af1f675 Enable more deconvolution layer configurations with IE backend 2019-06-03 08:15:52 +03:00
Dmitry Kurtaev
44d21e5a79 Enable Slice layer on Inference Engine backend 2019-05-27 16:28:01 +03:00
Alexander Alekhin
cafa010389 dnn(test): skip tests 2019-04-03 17:49:05 +03:00
Lubov Batanina
7d3d6bc4e2 Merge pull request #13932 from l-bat:MyriadX_master_dldt 
* Fix precision in tests for MyriadX

* Fix ONNX tests

* Add output range in ONNX tests

* Skip tests on Myriad OpenVINO 2018R5

* Add detect MyriadX

* Add detect MyriadX on OpenVINO R5

* Skip tests on Myriad next version of OpenVINO

* dnn(ie): VPU type from environment variable

* dnn(test): validate VPU type

* dnn(test): update DLIE test skip conditions
 2019-03-29 16:42:58 +03:00
Dmitry Kurtaev
ed710eaa1c Make Inference Engine R3 as a minimal supported version 2019-02-21 09:32:26 +03:00
Liubov Batanina
183c0fcab1 Changed condition for resize and lrn layers 2019-02-14 13:11:14 +03:00
Dmitry Kurtaev
f0ddf302b2 Move Inference Engine to new API 2019-01-17 14:28:48 +03:00
Maksim Shabunin
fe459c82e5 Merge pull request #13332 from mshabunin:dnn-backends 
DNN backends registry (#13332)

* Added dnn backends registry

* dnn: process DLIE/FPGA target
 2018-12-05 18:11:45 +03:00
Dmitry Kurtaev
0d117312c9 DNN_TARGET_FPGA using Intel's Inference Engine 2018-11-19 11:41:43 +03:00
Alexander Alekhin
96c71dd3d2 dnn: reduce set of ignored warnings 2018-11-15 13:15:59 +03:00
Alexander Alekhin
c557193b8c dnn(test): use dnnBackendsAndTargets() param generator 2018-08-31 15:11:58 +03:00
Dmitry Kurtaev
8e034053af Faster-RCNN from TensorFlow on CPU with Intel's Inference Engine backend 2018-08-01 11:29:58 +03:00
Dmitry Kurtaev
2c291bc2fb Enable FastNeuralStyle and OpenFace networks with IE backend 2018-06-09 15:57:12 +03:00
Dmitry Kurtaev
40765c5f8d Enable SSD models from TensorFlow with OpenCL plugin of Intel's Inference Engine 2018-06-08 16:55:21 +03:00
David
7175f257b5 Added ResizeBilinear op for tf (#11050) 
* Added ResizeBilinear op for tf

Combined ResizeNearestNeighbor and ResizeBilinear layers into Resize (with an interpolation param).

Minor changes to tf_importer and resize layer to save some code lines

Minor changes in init.cpp

Minor changes in tf_importer.cpp

* Replaced implementation of a custom ResizeBilinear layer to all layers

* Use Mat::ptr. Replace interpolation flags
 2018-06-07 16:29:04 +03:00
Dmitry Kurtaev
f3a6ae5f00 Wrap Inference Engine init to try-catch 2018-06-07 12:55:52 +03:00
Vadim Pisarevsky
3cbd2e2764 Merge pull request #11650 from dkurt:dnn_default_backend 2018-06-06 09:30:39 +00:00
Alexander Alekhin
6816495bee dnn(test): reuse test/test_common.hpp, eliminate dead code warning 2018-06-05 12:52:53 +03:00
Dmitry Kurtaev
b781ac7346 Make Intel's Inference Engine backend is default if no preferable backend is specified. 2018-06-04 18:31:46 +03:00
Dmitry Kurtaev
f96f934426 Update Intel's Inference Engine deep learning backend (#11587) 
* Update Intel's Inference Engine deep learning backend

* Remove cpu_extension dependency

* Update Darknet accuracy tests
 2018-05-31 14:05:21 +03:00
Li Peng
1b517a45ae add fp16 accuracy and perf test 
Signed-off-by: Li Peng <peng.li@intel.com>
 2018-05-16 22:45:07 +08:00
Dmitry Kurtaev
bd77d100e1 Enable some tests for clDNN plugin from Intel's Inference Engine 2018-04-20 10:47:46 +03:00
Dmitry Kurtaev
97fec07d96 Support YOLOv3 model from Darknet 2018-04-16 18:44:12 +03:00
Dmitry Kurtaev
709cf5d038 OpenCL GPU target for Inference Engine deep learning backend 
Enable FP16 GPU target for DL Inference Engine backend.
 2018-04-09 17:21:35 +03:00
Dmitry Kurtaev
7972f47ed4 Load networks from intermediate representation of Intel's Deep learning deployment toolkit. 2018-03-26 07:24:21 +03:00
Dmitry Kurtaev
7fe97376c2 MobileNet-SSD from TensorFlow 1.3 and Inception-V2-SSD using Inference Engine backend 2018-02-09 13:45:45 +03:00
Dmitry Kurtaev
ed94136548 OpenCV face detection network using Inference Engine backend 2018-02-06 17:53:24 +03:00
Alexander Alekhin
2a1f46c42d Merge pull request #9770 from alalek:refactor_test_files 2018-02-06 09:33:58 +00:00
Dmitry Kurtaev
10e1de74d2 Intel Inference Engine deep learning backend (#10608) 
* Intel Inference Engine deep learning backend.

* OpenFace network using Inference Engine backend
 2018-02-06 11:57:35 +03:00
Alexander Alekhin
4a297a2443 ts: refactor OpenCV tests 
- removed tr1 usage (dropped in C++17)
- moved includes of vector/map/iostream/limits into ts.hpp
- require opencv_test + anonymous namespace (added compile check)
- fixed norm() usage (must be from cvtest::norm for checks) and other conflict functions
- added missing license headers
 2018-02-03 19:39:47 +00:00