mirror of
https://github.com/opencv/opencv.git
synced 2025-06-07 09:25:45 +08:00
Open Source Computer Vision Library
0521a3a384
dnn: add attention layer #24476 Resolves #24609 Merge with: https://github.com/opencv/opencv_extra/pull/1128. Attention operator spec from onnxruntime: https://github.com/microsoft/onnxruntime/blob/v1.16.1/docs/ContribOperators.md#com.microsoft.Attention. TODO: - [x] benchmark (before this PR vs. with this PR vs. ORT). - [x] Layer fusion: Take care Slice with end=INT64_MAX. - [x] Layer fusion: match more potential attention (VIT) patterns. - [x] Single-head attention is supported. - [x] Test AttentionSubgraph fusion. - [x] Add acc tests for VIT_B_32 and VitTrack - [x] Add perf tests for VIT_B_32 and VitTrack ## Benchmarks Platform: Macbook Air M1. ### Attention Subgraph Input scale: [1, 197, 768]. | | mean (ms) | median (ms) | min (ms) | | ---------------------- | --------- | ----------- | -------- | | w/ Attention (this PR) | 3.75 | 3.68 | 3.22 | | w/o Attention | 9.06 | 9.01 | 8.24 | | ORT (python) | 4.32 | 2.63 | 2.50 | ### ViTs All data in millisecond (ms). | ViTs | With Attention | Without Attention | ORT | | -------- | -------------- | ----------------- | ------ | | vit_b_16 | 302.77 | 365.35 | 109.70 | | vit_b_32 | 89.92 | 116.22 | 30.36 | | vit_l_16 | 1593.32 | 1730.74 | 419.92 | | vit_l_32 | 468.11 | 577.41 | 134.12 | | VitTrack | 3.80 | 3.87 | 2.25 | ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake |
||
|---|---|---|
| .github | ||
| 3rdparty | ||
| apps | ||
| cmake | ||
| data | ||
| doc | ||
| include | ||
| modules | ||
| platforms | ||
| samples | ||
| .editorconfig | ||
| .gitattributes | ||
| .gitignore | ||
| CMakeLists.txt | ||
| CONTRIBUTING.md | ||
| COPYRIGHT | ||
| LICENSE | ||
| README.md | ||
| SECURITY.md | ||
OpenCV: Open Source Computer Vision Library
Keep OpenCV Free
OpenCV is raising funds to keep the library free for everyone, and we need the support of the entire community to do it. Donate to OpenCV on IndieGoGo before the campaign ends on December 16 to show your support.
Resources
- Homepage: https://opencv.org
- Courses: https://opencv.org/courses
- Docs: https://docs.opencv.org/4.x/
- Q&A forum: https://forum.opencv.org
- previous forum (read only): http://answers.opencv.org
- Issue tracking: https://github.com/opencv/opencv/issues
- Additional OpenCV functionality: https://github.com/opencv/opencv_contrib
Contributing
Please read the contribution guidelines before starting work on a pull request.
Summary of the guidelines:
- One pull request per issue;
- Choose the right base branch;
- Include tests and documentation;
- Clean up "oops" commits before submitting;
- Follow the coding style guide.
Additional Resources
- Submit your OpenCV-based project for inclusion in Community Friday on opencv.org
- Subscribe to the OpenCV YouTube Channel featuring OpenCV Live, an hour-long streaming show
- Follow OpenCV on LinkedIn for daily posts showing the state-of-the-art in computer vision &AI
- Apply to be an OpenCV Volunteer to help organize events and online campaigns as well as amplify them
- Follow OpenCV on Mastodon in the Fediverse
- Follow OpenCV on Twitter
- OpenCV.ai: Computer Vision and AI development services from the OpenCV team.