G-API: Fix incorrect OpaqueKind for Kernel outputs #23843
### Pull Request Readiness Checklist
#### Overview
The PR is going to fix several problems:
1. Major: `GKernel` doesn't hold `kind` for its outputs. Since `GModelBuilder` traverse graph from outputs to inputs once it reaches any output of the operation it will use its `kind` to create `Data` meta for all operation outputs. Since it essential for `python` to know `GTypeInfo` (which is `shape` and `kind`) it will be confused.
Consider this operation:
```
@cv.gapi.op('custom.square_mean', in_types=[cv.GArray.Int], out_types=[cv.GOpaque.Float, cv.GArray.Int])
class GSquareMean:
@staticmethod
def outMeta(desc):
return cv.empty_gopaque_desc(), cv.empty_array_desc()
```
Even though `GOpaque` is `Float`, corresponding metadata might have `Int` kind because it might be taken from `cv.GArray.Int`
so it will be a problem if one of the outputs of these operation is graph output because python will cast it to the wrong type based on `Data` meta.
2. Minor: Some of the OpenVINO `IR`'s doesn't any layout information for input. It's usually true only for `IRv10` but since `OpenVINO 2.0` need this information to correctly configure resize we need to put default layout if there no such assigned in `ov::Model`.
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [ ] I agree to contribute to the project under Apache 2 License.
- [ ] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
G-API: Integration branch for ONNX & Python-related changes #23597
# Changes overview
## 1. Expose ONNX backend's Normalization and Mean-value parameters in Python
* Since Python G-API bindings rely on `Generic` infer to express Inference, the `Generic` specialization of `onnx::Params` was extended with new methods to control normalization (`/255`) and mean-value; these methods were exposed in the Python bindings
* Found some questionable parts in the existing API which I'd like to review/discuss (see comments)
UPD:
1. Thanks to @TolyaTalamanov normalization inconsistencies have been identified with `squeezenet1.0-9` ONNX model itself; tests using these model were updated to DISABLE normalization and NOT using mean/value.
2. Questionable parts were removed and tests still pass.
### Details (taken from @TolyaTalamanov's comment):
`squeezenet1.0.*onnx` - doesn't require scaling to [0,1] and mean/std because the weights of the first convolution already scaled. ONNX documentation is broken. So the correct approach to use this models is:
1. ONNX: apply preprocessing from the documentation: https://github.com/onnx/models/blob/main/vision/classification/imagenet_preprocess.py#L8-L44 but without normalization step:
```
# DON'T DO IT:
# mean_vec = np.array([0.485, 0.456, 0.406])
# stddev_vec = np.array([0.229, 0.224, 0.225])
# norm_img_data = np.zeros(img_data.shape).astype('float32')
# for i in range(img_data.shape[0]):
# norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i]
# # add batch channel
# norm_img_data = norm_img_data.reshape(1, 3, 224, 224).astype('float32')
# return norm_img_data
# INSTEAD
return img_data.reshape(1, 3, 224, 224)
```
2. G-API: Convert image from BGR to RGB and then pass to `apply` as-is with configuring parameters:
```
net = cv.gapi.onnx.params('squeezenet', model_filename)
net.cfgNormalize('data_0', False)
```
**Note**: Results might be difference because `G-API` doesn't apply central crop but just do resize to model resolution.
---
`squeezenet1.1.*onnx` - requires scaling to [0,1] and mean/std - onnx documentation is correct.
1. ONNX: apply preprocessing from the documentation: https://github.com/onnx/models/blob/main/vision/classification/imagenet_preprocess.py#L8-L44
2. G-API: Convert image from BGR to RGB and then pass to `apply` as-is with configuring parameters:
```
net = cv.gapi.onnx.params('squeezenet', model_filename)
net.cfgNormalize('data_0', True) // default
net.cfgMeanStd('data_0', [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
```
**Note**: Results might be difference because `G-API` doesn't apply central crop but just do resize to model resolution.
## 2. Expose Fluid & kernel package-related functionality in Python
* `cv::gapi::combine()`
* `cv::GKernelPackage::size()` (mainly for testing purposes)
* `cv::gapi::imgproc::fluid::kernels()`
Added a test for the above.
## 3. Fixed issues with Python stateful kernel handling
Fixed error message when `outMeta()` of custom python operation fails.
## 4. Fixed various issues in Python tests
1. `test_gapi_streaming.py` - fixed behavior of Desync test to avoid sporadic issues
2. `test_gapi_infer_onnx.py` - fixed model lookup (it was still using the ONNX Zoo layout but was NOT using the proper env var we use to point to one).
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
1) Document GFrame/MediaFrame (and also other G-API types)
- Added doxygen comments for GMat, GScalar, GArray<T>, GOpaque classes;
- Documented GFrame and its host-side counterpart MediaFrame;
- Added some more notes to the data type classes.
2) Give @brief descriptions to most of the cv::gapi::* namespaces
3) Make some symbols private
- These structures are mainly internal and shouldn't be used directly
[G-API] Introduce cv.gin/cv.descr_of for python
* Implement cv.gin/cv.descr_of
* Fix macos build
* Fix gcomputation tests
* Add test
* Add using to a void exceeded length for windows build
* Add using to a void exceeded length for windows build
* Fix comments to review
* Fix comments to review
* Update from latest master
* Avoid graph compilation to obtain in/out info
* Fix indentation
* Fix comments to review
* Avoid using default in switches
* Post output meta for giebackend
- desync() is a new (and for now, the only one) intrinsic
which splits the graph execution into asynchronous parts
when running in Streaming mode;
- desync() makes no effect when compiling in Traditional mode;
- Added tests on desync() working in various scenarios;
- Extended GStreamingExecutor to support desync(); also extended
GStreamingCompiled() with a new version of pull() returning a
vector of optional values;
- Fixed various issues with storing the type information & proper
construction callbacks for GArray<> and GOpaque;
- Introduced a new infer(Roi,GMat) overload with a sample;
- Introduced an internal API for Islands to control fusion
procedure (to fuse or not to fuse);
- Introduced handleStopStream() callback for island executables;
- Added GCompileArgs to metadata of the graph (required for other
features).
* Implement G-API python bindings
* Fix hdr_parser
* Drop initlization with brackets using regexp
* Handle bracket initilization another way
* Add test for core operations
* Declaration and definition of View constructor now in different files
* Refactor tests
* Remove combine decorator from tests
* Fix comment to review
* Fix test
* Fix comments to review
* Remove GCompilerArgs implementation from python
Co-authored-by: Pinaev <danil.pinaev@intel.com>
* G-API: Introduce a new gapi::infer2 overload + gaze estimation sample
* G-API/infer2: Introduced static type checking for infer2
- Also added extra tests on the type check routine
* G-API/infer2: Addressed self-review comments in the sample app
- Also fix build on Linux;
* G-API/infer2: Remove incorrect SetLayout(HWC) + dead code
- Also fixed comments in the backend
* G-API/infer2: Continue with self-review
- Fix warnings/compile errors in gaze estimation
- Dropped the use of RTTI (VectorRef::holds()) from the giebackend
- Replaced it with a trait-based enums for GArray<T> and std::vector<T>
- The enums and traits are temporary and need to be unified with
the S11N when it comes
* G-API/infer2: Final self-review items
- Refactored ROIList test to cover 70% for infer<> and infer2<>;
- Fixed the model data discovery routine to be compatible with new
OpenVINO;
- Hopefully fixed the final issues (warnings) with the sample.
* G-API/infer2: address review problems
- Fixed typo in comments;
- Fixed public (Doxygen) comment on GArray<GMat> input case for infer2;
- Made model lookup more flexible to allow new & old OMZ dir layouts.
* G-API/infer2: Change the model paths again
* G-API/infer2: Change the lookup path for test data
* G-API/infer2: use randu instead of imread. CI war is over
G-API Serialization routines
* Serialization backend in tests, initial version
* S11N/00: A Great Rename
- "Serialization" is too long and too error-prone to type,
so now it is renamed to "s11n" everywhere;
- Same applies to "SRLZ";
- Tests also renamed to start with 'S11N.*' (easier to run);
- Also updated copyright years in new files to 2020.
* S11N/01: Some basic interface segregation
- Moved some details (low-level functions) out of serialization.hpp;
- Introduced I::IStream and I::OStream interfaces;
- Implemented those via the existing [De]SerializationStream classes;
- Moved all operators to use interfaces instead of classes;
- Moved the htonl/ntohl handling out of operators (to the classes).
The implementation didn't change much, it is a subject to the further
refactoring
* S11N/02: Basic operator reorg, basic tests, vector support
- Reorganized operators on atomic types to follow >>/<< model
(put them closer in the code for the respective types);
- Introduce more operators for basic (scalar) types;
- Drop all vector s11n overloads -- replace with a generic
(template-based) one;
- Introduced a new test suite where low-level s11n functionality
is tested (for the basic types).
* S11N/03: Operators reorganization
- Sorted the Opaque types enum by complexity;
- Reorganized the existing operators for basic types, also ordered by
complexity;
- Organized operators in three groups (Basics, OpenCV, G-API);
- Added a generic serialization for variant<>;
- Reimplemented some of the existing operators (for OpenCV and G-API
data structures);
- Introduced new operators for cv::gimpl data types. These operators
(and so, the data structures) are not yet used in the graph
dump/reconstruction routine, it will be done as a next step.
* S11N/04: The Great Clean-up
- Drop the duplicates of GModel data structures from the
serialization, serialize the GModel data structures themselve
instead (hand-written code replaced with operators).
- Also removed usuned code for printing, etc.
* S11N/05: Internal API Clean-up
- Minimize the serialization API to just Streams and Operators;
- Refactor and fix the graph serialization (deconstruction and
reconstruction) routines, fix data addressing problems there;
- Move the serialization.[ch]pp files to the core G-API library
* S11N/06: Top-level API introduction
- !!!This is likely the most invasive commit in the series!!!
- Introduced a top-level API to serialize and deserialize a GComputation
- Extended the compiler to support both forms of a GComputation:
an expession based and a deserialized one. This has led to changes in
the cv::GComputation::Priv and in its dependent components (even the
transformation tests);
- Had to extend the kernel API (GKernel) with extra information on
operations (mainly `outMeta`) which was only available for expression
based graphs. Now the `outMeta` can be taken from kernels too (and for
the deserialized graphs it is the only way);
- Revisited the internal serialization API, had to expose previously
hidden entities (like `GSerialized`);
- Extended the serialized graph info with new details (object counter,
protocol). Added unordered_map generic serialization for that;
- Reworked the very first pipeline test to be "proper"; GREEN now, the rest
is to be reworked in the next iteration.
* S11N/07: Tests reworked
- Moved the sample pipeline tests w/serialization to
test the public API (`cv::gapi::serialize`, then
followed by `cv::gapi::deserialize<>`). All GREEN.
- As a consequence, dropped the "Serialization" test
backend as no longer necessary.
* S11N/08: Final touches
- Exposed the C++ native data types at Streams level;
- Switched the ByteMemoryIn/OutStreams to store data in `char`
internally (2x less memory for sample pipelines);
- Fixed and refactored Mat dumping to the stream;
- Renamed S11N pipeline tests to their new meaning.
* linux build fix
* fix RcDesc and int uint warnings
* more Linux build fix
* white space and virtual android error fix (attempt)
* more warnings to be fixed
* android warnings fix attempt
* one more attempt for android build fix
* android warnings one more fix
* return back override
* avoid size_t
* static deserialize
* and how do you like this, elon? anonymous namespace to fix android warning.
* static inline
* trying to fix standalone build
* mat dims fix
* fix mat r/w for standalone
Co-authored-by: Dmitry Matveev <dmitry.matveev@intel.com>
G-API: Using functors as kernel implementation
* Implement ability to create kernel impls from functors
* Clean up
* Replace make_ocv_functor to ocv_kernel
* Clean up
* Replace GCPUFunctor -> GOCVFunctor
* Move GOCVFunctor to cv::gapi::cpu namespace
* Implement override for rvalue and lvalue cases
* Fix comments to review
* Remove GAPI_EXPORT for template functions
* Fix indentation
* G-API: Addressed various documentation issues
- Fixed various typos and missing references;
- Added brief documentaion on G_TYPED_KERNEL and G_COMPOUND_KERNEL macros;
- Briefly described GComputationT<>;
- Briefly described G-API data objects (in a group section).
* G-API: Some clean-ups in doxygen, also a chapter on Render API
* G-API: Expose more graph compilation arguments in the documentation
* G-API: Address documentation review comments
* G-API-NG/API: Introduced inference API and IE-based backend
- Very quick-n-dirty implementation
- OpenCV's own DNN module is not used
- No tests so far
* G-API-NG/IE: Refined IE backend, added more tests
* G-API-NG/IE: Fixed various CI warnings & build issues + tests
- Added tests on multi-dimensional own::Mat
- Added tests on GMatDesc with dimensions
- Documentation on infer.hpp
- Fixed more warnings + added a ROI list test
- Fix descr_of clash for vector<Mat> & standalone mode
- Fix build issue with gcc-4.8x
- Addressed review comments
* G-API-NG/IE: Addressed review comments
- Pass `false` to findDataFile()
- Add deprecation warning suppression macros for IE
* Introduce GAPI_TRANSFORM initial interface
Comes along with simple tests and kernel package changes
* Fix documentation and adjust combine() function
* Fix stuff after rebasing on master
* Remove redundant functionality
* Refactoring according to review feedback provided
* Fixes according to review feedback
* Reconsider transformations return and fix a warning
* Fixes from code review
* Add a new simple test
* Cleanup, added tests on GScalar, GMatP, GArray
G-API: Kernel package design (#13851)
* Remove cv::unite_policy from API
* Add check that all id in kernel package are unique
* Refactor checker id procedure
* Remove cv::gapi::GLookupOrder from API
* Implement cv::gapi::use_only
* Fix samples
* Fix docs
* Fix comments to review
* Remove unite_policy
* Fix GKernelPackage::backends()
* Fix comments to review
* Fix all_unique
* Fix comments to review
* Fix comments to review
* Remove out of date tests
G-API: Doxygen class reference
* G-API Doxygen documentation: covered cv::GComputation
* G-API Doxygen documentation: added sections on compile arguments
* G-API Doxygen documentation: restructuring & more text
* Added new sections (organized API reference into it);
* Documented GCompiled, compile args, backends, etc.
* G-API Doxygen documentation: documented GKernelPackage and added group for meta
* G-API: First steps with tutorial
* G-API Tutorial: First iteration
* G-API port of anisotropic image segmentation tutorial;
* Currently works via OpenCV only;
* Some new kernels have been required.
* G-API Tutorial: added chapters on execution code, inspection, and profiling
* G-API Tutorial: make Fluid kernel headers public
For some reason, these headers were not moved to the public
headers subtree during the initial development. Somehow it even
worked for the existing workloads.
* G-API Tutorial: Fix a couple of issues found during the work
* Introduced Phase & Sqrt kernels, OCV & Fluid versions
* Extended GKernelPackage to allow kernel removal & policies on include()
All the above stuff needs to be tested, tests will be added later
* G-API Tutorial: added chapter on running Fluid backend
* G-API Tutorial: fix a number of issues in the text
* G-API Tutorial - some final updates
- Fixed post-merge issues after Sobel kernel renaming;
- Simplified G-API code a little bit;
- Put a conclusion note in text.
* G-API Tutorial - fix build issues in test/perf targets
Public headers were refactored but tests suites were not updated in time
* G-API Tutorial: Added tests & reference docs on new kernels
* Phase
* Sqrt
* G-API Tutorial: added link to the tutorial from the main module doc
* G-API Tutorial: Added tests on new GKernelPackage functionality
* G-API Tutorial: Extended InRange tests to cover 32F
* G-API Tutorial: Misc fixes
* Avoid building examples when gapi module is not there
* Added a volatile API disclaimer to G-API root documentation page
* G-API Tutorial: Fix perf tests build issue
This change came from master where Fluid kernels are still used
incorrectly.
* G-API Tutorial: Fixed channels support in Sqrt/Phase fluid kernels
Extended tests to cover this case
* G-API Tutorial: Fix text problems found on team review
* G-API Documentation: first submission
This PR introduces a number of new OpenCV documentation chapters for
Graph API module.
In particular, the following topics are covered:
- Introduction & background information;
- High-level design overview;
- Kernel API;
- Pipeline example.
All changes are done in Markdown files, no headers, etc modified.
Doxygen references for main API classes will be added later.
Also, a tutorial will be introduced soon (in the common Tutorials place)
* G-API Documentation - fix warnings & trailing whitespaces
* G-API Documentation: address review issues
* G-API Documentation: export code snippets to compileable files
* gapi: move documentation samples
* G-API Initial code upload
* Update G-API code base to Sep-24-2018
* The majority of OpenCV buildbot problems was addressed
* Update G-API code base to 24-Sep-18 EOD
* G-API code base update 25-Sep-2018
* Linux warnings should be resolved
* Documentation build should become green
* Number of Windows warnings should be reduced
* Update G-API code base to 25-Sep-18 EOD
* ARMv7 build issue should be resolved
* ADE is bumped to latest version and should fix Clang builds for macOS/iOS
* Remaining Windows warnings should be resolved
* New Linux32 / ARMv7 warnings should be resolved
* G-API code base update 25-Sep-2018-EOD2
* Final Windows warnings should be resolved now
* G-API code base update 26-Sep-2018
* Fixed issues with precompiled headers in module and its tests
