mirror of
https://github.com/opencv/opencv.git
synced 2025-06-10 19:24:07 +08:00
0521a3a384
dnn: add attention layer #24476 Resolves #24609 Merge with: https://github.com/opencv/opencv_extra/pull/1128. Attention operator spec from onnxruntime: https://github.com/microsoft/onnxruntime/blob/v1.16.1/docs/ContribOperators.md#com.microsoft.Attention. TODO: - [x] benchmark (before this PR vs. with this PR vs. ORT). - [x] Layer fusion: Take care Slice with end=INT64_MAX. - [x] Layer fusion: match more potential attention (VIT) patterns. - [x] Single-head attention is supported. - [x] Test AttentionSubgraph fusion. - [x] Add acc tests for VIT_B_32 and VitTrack - [x] Add perf tests for VIT_B_32 and VitTrack ## Benchmarks Platform: Macbook Air M1. ### Attention Subgraph Input scale: [1, 197, 768]. | | mean (ms) | median (ms) | min (ms) | | ---------------------- | --------- | ----------- | -------- | | w/ Attention (this PR) | 3.75 | 3.68 | 3.22 | | w/o Attention | 9.06 | 9.01 | 8.24 | | ORT (python) | 4.32 | 2.63 | 2.50 | ### ViTs All data in millisecond (ms). | ViTs | With Attention | Without Attention | ORT | | -------- | -------------- | ----------------- | ------ | | vit_b_16 | 302.77 | 365.35 | 109.70 | | vit_b_32 | 89.92 | 116.22 | 30.36 | | vit_l_16 | 1593.32 | 1730.74 | 419.92 | | vit_l_32 | 468.11 | 577.41 | 134.12 | | VitTrack | 3.80 | 3.87 | 2.25 | ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake |
||
|---|---|---|
| .. | ||
| caffe | ||
| cuda | ||
| cuda4dnn | ||
| darknet | ||
| int8layers | ||
| layers | ||
| ocl4dnn | ||
| onnx | ||
| opencl | ||
| tensorflow | ||
| tflite | ||
| torch | ||
| vkcom | ||
| webnn | ||
| backend.cpp | ||
| backend.hpp | ||
| debug_utils.cpp | ||
| dnn_common.hpp | ||
| dnn_params.cpp | ||
| dnn_read.cpp | ||
| dnn_utils.cpp | ||
| dnn.cpp | ||
| factory.hpp | ||
| graph_simplifier.cpp | ||
| graph_simplifier.hpp | ||
| halide_scheduler.cpp | ||
| halide_scheduler.hpp | ||
| ie_ngraph.cpp | ||
| ie_ngraph.hpp | ||
| init.cpp | ||
| layer_factory.cpp | ||
| layer_internals.hpp | ||
| layer.cpp | ||
| legacy_backend.cpp | ||
| legacy_backend.hpp | ||
| math_utils.hpp | ||
| model.cpp | ||
| net_cann.cpp | ||
| net_impl_backend.cpp | ||
| net_impl_fuse.cpp | ||
| net_impl.cpp | ||
| net_impl.hpp | ||
| net_openvino.cpp | ||
| net_quantization.cpp | ||
| net.cpp | ||
| nms.cpp | ||
| nms.inl.hpp | ||
| op_cann.cpp | ||
| op_cann.hpp | ||
| op_cuda.cpp | ||
| op_cuda.hpp | ||
| op_halide.cpp | ||
| op_halide.hpp | ||
| op_inf_engine.cpp | ||
| op_inf_engine.hpp | ||
| op_timvx.cpp | ||
| op_timvx.hpp | ||
| op_vkcom.cpp | ||
| op_vkcom.hpp | ||
| op_webnn.cpp | ||
| op_webnn.hpp | ||
| plugin_api.hpp | ||
| plugin_wrapper.impl.hpp | ||
| precomp.hpp | ||
| registry.cpp | ||