mirror/opencv
1
0
Fork 0
mirror of https://github.com/opencv/opencv.git synced 2024-11-23 18:50:21 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #20406 from MarkGHX:gsoc_2021_webnn

Browse Source 
[GSoC] OpenCV.js: Accelerate OpenCV.js DNN via WebNN

* Add WebNN backend for OpenCV DNN Module

Update dnn.cpp

Update dnn.cpp

Update dnn.cpp

Update dnn.cpp

Add WebNN head files into OpenCV 3rd partiy files

Create webnn.hpp

update cmake

Complete README and add OpenCVDetectWebNN.cmake file

add webnn.cpp

Modify webnn.cpp

Can successfully compile the codes for creating a MLContext

Update webnn.cpp

Update README.md

Update README.md

Update README.md

Update README.md

Update cmake files and

update README.md

Update OpenCVDetectWebNN.cmake and README.md

Update OpenCVDetectWebNN.cmake

Fix OpenCVDetectWebNN.cmake and update README.md

Add source webnn_cpp.cpp and libary libwebnn_proc.so

Update dnn.cpp

Update dnn.cpp

Update dnn.cpp

Update dnn.cpp

update dnn.cpp

update op_webnn

update op_webnn

Update op_webnn.hpp

update op_webnn.cpp & hpp

Update op_webnn.hpp

Update op_webnn

update the skeleton

Update op_webnn.cpp

Update op_webnn

Update op_webnn.cpp

Update op_webnn.cpp

Update op_webnn.hpp

update op_webnn

update op_webnn

Solved the problems of released variables.

Fixed the bugs in op_webnn.cpp

Implement op_webnn

Implement Relu by WebNN API

Update dnn.cpp for better test

Update elementwise_layers.cpp

Implement ReLU6

Update elementwise_layers.cpp

Implement SoftMax using WebNN API

Implement Reshape by WebNN API

Implement PermuteLayer by WebNN API

Implement PoolingLayer using WebNN API

Update pooling_layer.cpp

Update pooling_layer.cpp

Update pooling_layer.cpp

Update pooling_layer.cpp

Update pooling_layer.cpp

Update pooling_layer.cpp

Implement poolingLayer by WebNN API and add more detailed logs

Update dnn.cpp

Update dnn.cpp

Remove redundant codes and add more logs for poolingLayer

Add more logs in the pooling layer implementation

Fix the indent issue and resolve the compiling issue

Fix the build problems

Fix the build issue

FIx the build issue

Update dnn.cpp

Update dnn.cpp

* Fix the build issue

* Implement BatchNorm Layer by WebNN API

* Update convolution_layer.cpp

This is a temporary file for Conv2d layer implementation

* Integrate some general functions into op_webnn.cpp&hpp

* Update const_layer.cpp

* Update convolution_layer.cpp

Still have some bugs that should be fixed.

* Update conv2d layer and fc layer

still have some problems to be fixed.

* update constLayer, conv layer, fc layer

There are still some bugs to be fixed.

* Fix the build issue

* Update concat_layer.cpp

Still have some bugs to be fixed.

* Update conv2d layer, fully connected layer and const layer

* Update convolution_layer.cpp

* Add OpenCV.js DNN module WebNN Backend (both using webnn-polyfill and electron)

* Delete bib19450.aux

* Add WebNN backend for OpenCV DNN Module

Update dnn.cpp

Update dnn.cpp

Update dnn.cpp

Update dnn.cpp

Add WebNN head files into OpenCV 3rd partiy files

Create webnn.hpp

update cmake

Complete README and add OpenCVDetectWebNN.cmake file

add webnn.cpp

Modify webnn.cpp

Can successfully compile the codes for creating a MLContext

Update webnn.cpp

Update README.md

Update README.md

Update README.md

Update README.md

Update cmake files and

update README.md

Update OpenCVDetectWebNN.cmake and README.md

Update OpenCVDetectWebNN.cmake

Fix OpenCVDetectWebNN.cmake and update README.md

Add source webnn_cpp.cpp and libary libwebnn_proc.so

Update dnn.cpp

Update dnn.cpp

Update dnn.cpp

Update dnn.cpp

update dnn.cpp

update op_webnn

update op_webnn

Update op_webnn.hpp

update op_webnn.cpp & hpp

Update op_webnn.hpp

Update op_webnn

update the skeleton

Update op_webnn.cpp

Update op_webnn

Update op_webnn.cpp

Update op_webnn.cpp

Update op_webnn.hpp

update op_webnn

update op_webnn

Solved the problems of released variables.

Fixed the bugs in op_webnn.cpp

Implement op_webnn

Implement Relu by WebNN API

Update dnn.cpp for better test

Update elementwise_layers.cpp

Implement ReLU6

Update elementwise_layers.cpp

Implement SoftMax using WebNN API

Implement Reshape by WebNN API

Implement PermuteLayer by WebNN API

Implement PoolingLayer using WebNN API

Update pooling_layer.cpp

Update pooling_layer.cpp

Update pooling_layer.cpp

Update pooling_layer.cpp

Update pooling_layer.cpp

Update pooling_layer.cpp

Implement poolingLayer by WebNN API and add more detailed logs

Update dnn.cpp

Update dnn.cpp

Remove redundant codes and add more logs for poolingLayer

Add more logs in the pooling layer implementation

Fix the indent issue and resolve the compiling issue

Fix the build problems

Fix the build issue

FIx the build issue

Update dnn.cpp

Update dnn.cpp

* Fix the build issue

* Implement BatchNorm Layer by WebNN API

* Update convolution_layer.cpp

This is a temporary file for Conv2d layer implementation

* Integrate some general functions into op_webnn.cpp&hpp

* Update const_layer.cpp

* Update convolution_layer.cpp

Still have some bugs that should be fixed.

* Update conv2d layer and fc layer

still have some problems to be fixed.

* update constLayer, conv layer, fc layer

There are still some bugs to be fixed.

* Update conv2d layer, fully connected layer and const layer

* Update convolution_layer.cpp

* Add OpenCV.js DNN module WebNN Backend (both using webnn-polyfill and electron)

* Update dnn.cpp

* Fix Error in dnn.cpp

* Resolve duplication in conditions in convolution_layer.cpp

* Fixed the issues in the comments

* Fix building issue

* Update tutorial

* Fixed comments

* Address the comments

* Update CMakeLists.txt

* Offer more accurate perf test on native

* Add better perf tests for both native and web

* Modify per tests for better results

* Use more latest version of Electron

* Support latest WebNN Clamp op

* Add definition of HAVE_WEBNN macro

* Support group convolution

* Implement Scale_layer using WebNN

* Add Softmax option for native classification example

* Fix comments

* Fix comments
This commit is contained in:
Hanxi Guo 2021-11-24 05:15:31 +08:00 committed by GitHub
parent 12c1e1d149
commit 1fcf7ba5bc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
32 changed files with 2260 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
CMakeLists.txt
View File

@ -296,6 +296,9 @@ OCV_OPTION(WITH_INF_ENGINE "Include Intel Inference Engine support" OFF
OCV_OPTION(WITH_NGRAPH "Include nGraph support" WITH_INF_ENGINE
VISIBLE_IF TRUE
VERIFY TARGET ngraph::ngraph)
OCV_OPTION(WITH_WEBNN "Include WebNN support" OFF
VISIBLE_IF TRUE
VERIFY HAVE_WEBNN)
OCV_OPTION(WITH_JASPER "Include JPEG2K support (Jasper)" ON
VISIBLE_IF NOT IOS
VERIFY HAVE_JASPER)
@ -793,6 +796,11 @@ if(WITH_VULKAN)
include(cmake/OpenCVDetectVulkan.cmake)
endif()
# --- WebNN ---
if(WITH_WEBNN)
include(cmake/OpenCVDetectWebNN.cmake)
endif()
# --- Inference Engine ---
if(WITH_INF_ENGINE)
include(cmake/OpenCVDetectInferenceEngine.cmake)
@ -1624,6 +1632,15 @@ if(WITH_VULKAN OR HAVE_VULKAN)
endif()
endif()
if(WITH_WEBNN OR HAVE_WEBNN)
status("")
status(" WebNN:" HAVE_WEBNN THEN "YES" ELSE "NO")
if(HAVE_WEBNN AND NOT EMSCRIPTEN)
status(" Include path:" WEBNN_HEADER_DIRS THEN "${WEBNN_HEADER_DIRS}" ELSE "NO")
status(" Link libraries:" WEBNN_LIBRARIES THEN "${WEBNN_LIBRARIES}" ELSE "NO")
endif()
endif()
if(WITH_OPENCL OR HAVE_OPENCL)
ocv_build_features_string(opencl_features
IF HAVE_OPENCL_SVM THEN "SVM"

49
cmake/OpenCVDetectWebNN.cmake Normal file
View File

@ -0,0 +1,49 @@
if(NOT EMSCRIPTEN)
if(WITH_WEBNN)
ocv_check_environment_variables(WEBNN_HEADER_DIRS)
ocv_check_environment_variables(WEBNN_INCLUDE_DIRS)
ocv_check_environment_variables(WEBNN_LIBRARIES)
if(NOT DEFINED WEBNN_HEADER_DIRS)
set(WEBNN_HEADER_DIRS "$ENV{WEBNN_NATIVE_DIR}/gen/src/include")
endif()
if(NOT DEFINED WEBNN_INCLUDE_DIRS)
set(WEBNN_INCLUDE_DIRS "$ENV{WEBNN_NATIVE_DIR}/../../src/include")
endif()
if(NOT DEFINED WEBNN_LIBRARIES)
set(WEBNN_LIBRARIES "$ENV{WEBNN_NATIVE_DIR}/libwebnn_native.so;$ENV{WEBNN_NATIVE_DIR}/libwebnn_proc.so")
endif()
endif()
try_compile(VALID_WEBNN
"${OpenCV_BINARY_DIR}"
SOURCES "${OpenCV_SOURCE_DIR}/cmake/checks/webnn.cpp"
"$ENV{WEBNN_NATIVE_DIR}/gen/src/webnn/webnn_cpp.cpp"
CMAKE_FLAGS "-DINCLUDE_DIRECTORIES:STRING=${WEBNN_INCLUDE_DIRS}\;${WEBNN_HEADER_DIRS}"
"-DLINK_LIBRARIES:STRING=${WEBNN_LIBRARIES}"
OUTPUT_VARIABLE TRY_OUT
)
else()
try_compile(VALID_WEBNN
"${OpenCV_BINARY_DIR}"
SOURCES "${OpenCV_SOURCE_DIR}/cmake/checks/webnn.cpp"
OUTPUT_VARIABLE TRY_OUT
)
endif()
if(NOT VALID_WEBNN)
if(NOT EMSCRIPTEN)
message(WARNING "Can't use WebNN-native")
return()
else()
message(WARNING "Can't use WebNN")
return()
endif()
else()
set(HAVE_WEBNN ON)
message(STATUS "Set HAVE_WEBNN = ${HAVE_WEBNN}")
endif()
if(NOT EMSCRIPTEN)
message(AUTHOR_WARNING "Use WebNN-native")
else()
message(AUTHOR_WARNING "Use WebNN")
endif()

23
cmake/checks/webnn.cpp Normal file
View File

@ -0,0 +1,23 @@
#include <webnn/webnn_cpp.h>
#include <webnn/webnn.h>
#ifdef __EMSCRIPTEN__
#include <emscripten.h>
#include <emscripten/html5.h>
#include <emscripten/html5_webnn.h>
#else
#include <webnn/webnn_proc.h>
#include <webnn_native/WebnnNative.h>
#endif
int main(int /*argc*/, char** /*argv*/)
{
#ifdef __EMSCRIPTEN__
ml::Context ml_context = ml::Context(emscripten_webnn_create_context());
#else
WebnnProcTable backendProcs = webnn_native::GetProcs();
webnnProcSetProcs(&backendProcs);
ml::Context ml_context = ml::Context(webnn_native::CreateContext());
#endif
return 0;
}

269
doc/js_tutorials/js_assets/js_image_classification_webnn_polyfill.html Normal file
View File

@ -0,0 +1,269 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Image Classification Example</title>
<link href="js_example_style.css" rel="stylesheet" type="text/css" />
<script src="./webnn-polyfill.js"></script>
</head>
<body>
<h2>Image Classification Example</h2>
<p>
This tutorial shows you how to write an image classification example with OpenCV.js.<br>
To try the example you should click the <b>modelFile</b> button(and <b>configFile</b> button if needed) to upload inference model.
You can find the model URLs and parameters in the <a href="#appendix">model info</a> section.
Then You should change the parameters in the first code snippet according to the uploaded model.
Finally click <b>Try it</b> button to see the result. You can choose any other images.<br>
</p>
<div class="control"><button id="tryIt" disabled>Try it</button></div>
<div>
<table cellpadding="0" cellspacing="0" width="0" border="0">
<tr>
<td>
<canvas id="canvasInput" width="400" height="400"></canvas>
</td>
<td>
<table style="visibility: hidden;" id="result">
<thead>
<tr>
<th scope="col">#</th>
<th scope="col" width=300>Label</th>
<th scope="col">Probability</th>
</tr>
</thead>
<tbody>
<tr>
<th scope="row">1</th>
<td id="label0" align="center"></td>
<td id="prob0" align="center"></td>
</tr>
<tr>
<th scope="row">2</th>
<td id="label1" align="center"></td>
<td id="prob1" align="center"></td>
</tr>
<tr>
<th scope="row">3</th>
<td id="label2" align="center"></td>
<td id="prob2" align="center"></td>
</tr>
</tbody>
</table>
<p id='status' align="left"></p>
</td>
</tr>
<tr>
<td>
<div class="caption">
canvasInput <input type="file" id="fileInput" name="file" accept="image/*">
</div>
</td>
<td></td>
</tr>
<tr>
<td>
<div class="caption">
modelFile <input type="file" id="modelFile">
</div>
</td>
</tr>
<tr>
<td>
<div class="caption">
configFile <input type="file" id="configFile">
</div>
</td>
</tr>
</table>
</div>
<div>
<p class="err" id="errorMessage"></p>
</div>
<div>
<h3>Help function</h3>
<p>1.The parameters for model inference which you can modify to investigate more models.</p>
<textarea class="code" rows="13" cols="100" id="codeEditor" spellcheck="false"></textarea>
<p>2.Main loop in which will read the image from canvas and do inference once.</p>
<textarea class="code" rows="17" cols="100" id="codeEditor1" spellcheck="false"></textarea>
<p>3.Load labels from txt file and process it into an array.</p>
<textarea class="code" rows="7" cols="100" id="codeEditor2" spellcheck="false"></textarea>
<p>4.Get blob from image as input for net, and standardize it with <b>mean</b> and <b>std</b>.</p>
<textarea class="code" rows="17" cols="100" id="codeEditor3" spellcheck="false"></textarea>
<p>5.Fetch model file and save to emscripten file system once click the input button.</p>
<textarea class="code" rows="17" cols="100" id="codeEditor4" spellcheck="false"></textarea>
<p>6.The post-processing, including softmax if needed and get the top classes from the output vector.</p>
<textarea class="code" rows="35" cols="100" id="codeEditor5" spellcheck="false"></textarea>
</div>
<div id="appendix">
<h2>Model Info:</h2>
</div>
<script src="utils.js" type="text/javascript"></script>
<script src="js_dnn_example_helper.js" type="text/javascript"></script>
<script id="codeSnippet" type="text/code-snippet">
inputSize = [224,224];
mean = [104, 117, 123];
std = 1;
swapRB = false;
// record if need softmax function for post-processing
needSoftmax = false;
// url for label file, can from local or Internet
labelsUrl = "https://raw.githubusercontent.com/opencv/opencv/master/samples/data/dnn/classification_classes_ILSVRC2012.txt";
</script>
<script id="codeSnippet1" type="text/code-snippet">
main = async function() {
const labels = await loadLables(labelsUrl);
const input = getBlobFromImage(inputSize, mean, std, swapRB, 'canvasInput');
let net = cv.readNet(configPath, modelPath);
net.setPreferableBackend(6);
net.setInput(input);
let result = net.forward();
const start = performance.now();
for (i=0;i<200;i++)
{
result = net.forward();
}
const time = performance.now()-start;
const probs = softmax(result);
const classes = getTopClasses(probs, labels);
updateResult(classes, time/200);
input.delete();
net.delete();
result.delete();
}
</script>
<script id="codeSnippet5" type="text/code-snippet">
softmax = function(result) {
let arr = result.data32F;
if (needSoftmax) {
const maxNum = Math.max(...arr);
const expSum = arr.map((num) => Math.exp(num - maxNum)).reduce((a, b) => a + b);
return arr.map((value, index) => {
return Math.exp(value - maxNum) / expSum;
});
} else {
return arr;
}
}
</script>
<script type="text/javascript">
let jsonUrl = "js_image_classification_model_info.json";
drawInfoTable(jsonUrl, 'appendix');
let utils = new Utils('errorMessage');
utils.loadCode('codeSnippet', 'codeEditor');
utils.loadCode('codeSnippet1', 'codeEditor1');
let loadLablesCode = 'loadLables = ' + loadLables.toString();
document.getElementById('codeEditor2').value = loadLablesCode;
let getBlobFromImageCode = 'getBlobFromImage = ' + getBlobFromImage.toString();
document.getElementById('codeEditor3').value = getBlobFromImageCode;
let loadModelCode = 'loadModel = ' + loadModel.toString();
document.getElementById('codeEditor4').value = loadModelCode;
utils.loadCode('codeSnippet5', 'codeEditor5');
let getTopClassesCode = 'getTopClasses = ' + getTopClasses.toString();
document.getElementById('codeEditor5').value += '\n' + '\n' + getTopClassesCode;
let canvas = document.getElementById('canvasInput');
let ctx = canvas.getContext('2d');
let img = new Image();
img.crossOrigin = 'anonymous';
img.src = 'space_shuttle.jpg';
img.onload = function() {
ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
};
let tryIt = document.getElementById('tryIt');
tryIt.addEventListener('click', () => {
initStatus();
document.getElementById('status').innerHTML = 'Running function main()...';
utils.executeCode('codeEditor');
utils.executeCode('codeEditor1');
if (modelPath === "") {
document.getElementById('status').innerHTML = 'Runing failed.';
utils.printError('Please upload model file by clicking the button first.');
} else {
setTimeout(main, 1);
}
});
let fileInput = document.getElementById('fileInput');
fileInput.addEventListener('change', (e) => {
initStatus();
loadImageToCanvas(e, 'canvasInput');
});
let configPath = "";
let configFile = document.getElementById('configFile');
configFile.addEventListener('change', async (e) => {
initStatus();
configPath = await loadModel(e);
document.getElementById('status').innerHTML = `The config file '${configPath}' is created successfully.`;
});
let modelPath = "";
let modelFile = document.getElementById('modelFile');
modelFile.addEventListener('change', async (e) => {
initStatus();
modelPath = await loadModel(e);
document.getElementById('status').innerHTML = `The model file '${modelPath}' is created successfully.`;
configPath = "";
configFile.value = "";
});
utils.loadOpenCv(() => {
tryIt.removeAttribute('disabled');
});
var main = async function() {};
var softmax = function(result){};
var getTopClasses = function(mat, labels, topK = 3){};
utils.executeCode('codeEditor1');
utils.executeCode('codeEditor2');
utils.executeCode('codeEditor3');
utils.executeCode('codeEditor4');
utils.executeCode('codeEditor5');
function updateResult(classes, time) {
try{
classes.forEach((c,i) => {
let labelElement = document.getElementById('label'+i);
let probElement = document.getElementById('prob'+i);
labelElement.innerHTML = c.label;
probElement.innerHTML = c.prob + '%';
});
let result = document.getElementById('result');
result.style.visibility = 'visible';
document.getElementById('status').innerHTML = `<b>Model:</b> ${modelPath}<br>
<b>Inference time:</b> ${time.toFixed(2)} ms`;
} catch(e) {
console.log(e);
}
}
function initStatus() {
document.getElementById('status').innerHTML = '';
document.getElementById('result').style.visibility = 'hidden';
utils.clearError();
}
</script>
</body>
</html>

268
doc/js_tutorials/js_assets/webnn-electron/js_image_classification_webnn_electron.html Normal file
View File

@ -0,0 +1,268 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Image Classification Example</title>
<link href="js_example_style.css" rel="stylesheet" type="text/css" />
</head>
<body>
<h2>Image Classification Example</h2>
<p>
This tutorial shows you how to write an image classification example with OpenCV.js.<br>
To try the example you should click the <b>modelFile</b> button(and <b>configFile</b> button if needed) to upload inference model.
You can find the model URLs and parameters in the <a href="#appendix">model info</a> section.
Then You should change the parameters in the first code snippet according to the uploaded model.
Finally click <b>Try it</b> button to see the result. You can choose any other images.<br>
</p>
<div class="control"><button id="tryIt" disabled>Try it</button></div>
<div>
<table cellpadding="0" cellspacing="0" width="0" border="0">
<tr>
<td>
<canvas id="canvasInput" width="400" height="400"></canvas>
</td>
<td>
<table style="visibility: hidden;" id="result">
<thead>
<tr>
<th scope="col">#</th>
<th scope="col" width=300>Label</th>
<th scope="col">Probability</th>
</tr>
</thead>
<tbody>
<tr>
<th scope="row">1</th>
<td id="label0" align="center"></td>
<td id="prob0" align="center"></td>
</tr>
<tr>
<th scope="row">2</th>
<td id="label1" align="center"></td>
<td id="prob1" align="center"></td>
</tr>
<tr>
<th scope="row">3</th>
<td id="label2" align="center"></td>
<td id="prob2" align="center"></td>
</tr>
</tbody>
</table>
<p id='status' align="left"></p>
</td>
</tr>
<tr>
<td>
<div class="caption">
canvasInput <input type="file" id="fileInput" name="file" accept="image/*">
</div>
</td>
<td></td>
</tr>
<tr>
<td>
<div class="caption">
modelFile <input type="file" id="modelFile">
</div>
</td>
</tr>
<tr>
<td>
<div class="caption">
configFile <input type="file" id="configFile">
</div>
</td>
</tr>
</table>
</div>
<div>
<p class="err" id="errorMessage"></p>
</div>
<div>
<h3>Help function</h3>
<p>1.The parameters for model inference which you can modify to investigate more models.</p>
<textarea class="code" rows="13" cols="100" id="codeEditor" spellcheck="false"></textarea>
<p>2.Main loop in which will read the image from canvas and do inference once.</p>
<textarea class="code" rows="17" cols="100" id="codeEditor1" spellcheck="false"></textarea>
<p>3.Load labels from txt file and process it into an array.</p>
<textarea class="code" rows="7" cols="100" id="codeEditor2" spellcheck="false"></textarea>
<p>4.Get blob from image as input for net, and standardize it with <b>mean</b> and <b>std</b>.</p>
<textarea class="code" rows="17" cols="100" id="codeEditor3" spellcheck="false"></textarea>
<p>5.Fetch model file and save to emscripten file system once click the input button.</p>
<textarea class="code" rows="17" cols="100" id="codeEditor4" spellcheck="false"></textarea>
<p>6.The post-processing, including softmax if needed and get the top classes from the output vector.</p>
<textarea class="code" rows="35" cols="100" id="codeEditor5" spellcheck="false"></textarea>
</div>
<div id="appendix">
<h2>Model Info:</h2>
</div>
<script src="utils_webnn_electron.js" type="text/javascript"></script>
<script src="js_dnn_example_helper.js" type="text/javascript"></script>
<script id="codeSnippet" type="text/code-snippet">
inputSize = [224,224];
mean = [104, 117, 123];
std = 1;
swapRB = false;
// record if need softmax function for post-processing
needSoftmax = false;
// url for label file, can from local or Internet
labelsUrl = "https://raw.githubusercontent.com/opencv/opencv/master/samples/data/dnn/classification_classes_ILSVRC2012.txt";
</script>
<script id="codeSnippet1" type="text/code-snippet">
main = async function() {
const labels = await loadLables(labelsUrl);
const input = getBlobFromImage(inputSize, mean, std, swapRB, 'canvasInput');
let net = cv.readNet(configPath, modelPath);
net.setPreferableBackend(6);
net.setInput(input);
let result = net.forward();
const start = performance.now();
for (i=0;i<200;i++)
{
result = net.forward();
}
const time = performance.now()-start;
const probs = softmax(result);
const classes = getTopClasses(probs, labels);
updateResult(classes, time/200);
input.delete();
net.delete();
result.delete();
}
</script>
<script id="codeSnippet5" type="text/code-snippet">
softmax = function(result) {
let arr = result.data32F;
if (needSoftmax) {
const maxNum = Math.max(...arr);
const expSum = arr.map((num) => Math.exp(num - maxNum)).reduce((a, b) => a + b);
return arr.map((value, index) => {
return Math.exp(value - maxNum) / expSum;
});
} else {
return arr;
}
}
</script>
<script type="text/javascript">
let jsonUrl = "js_image_classification_model_info.json";
drawInfoTable(jsonUrl, 'appendix');
let utils = new Utils('errorMessage');
utils.loadCode('codeSnippet', 'codeEditor');
utils.loadCode('codeSnippet1', 'codeEditor1');
let loadLablesCode = 'loadLables = ' + loadLables.toString();
document.getElementById('codeEditor2').value = loadLablesCode;
let getBlobFromImageCode = 'getBlobFromImage = ' + getBlobFromImage.toString();
document.getElementById('codeEditor3').value = getBlobFromImageCode;
let loadModelCode = 'loadModel = ' + loadModel.toString();
document.getElementById('codeEditor4').value = loadModelCode;
utils.loadCode('codeSnippet5', 'codeEditor5');
let getTopClassesCode = 'getTopClasses = ' + getTopClasses.toString();
document.getElementById('codeEditor5').value += '\n' + '\n' + getTopClassesCode;
let canvas = document.getElementById('canvasInput');
let ctx = canvas.getContext('2d');
let img = new Image();
img.crossOrigin = 'anonymous';
img.src = 'space_shuttle.jpg';
img.onload = function() {
ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
};
let tryIt = document.getElementById('tryIt');
tryIt.addEventListener('click', () => {
initStatus();
document.getElementById('status').innerHTML = 'Running function main()...';
utils.executeCode('codeEditor');
utils.executeCode('codeEditor1');
if (modelPath === "") {
document.getElementById('status').innerHTML = 'Runing failed.';
utils.printError('Please upload model file by clicking the button first.');
} else {
setTimeout(main, 1);
}
});
let fileInput = document.getElementById('fileInput');
fileInput.addEventListener('change', (e) => {
initStatus();
loadImageToCanvas(e, 'canvasInput');
});
let configPath = "";
let configFile = document.getElementById('configFile');
configFile.addEventListener('change', async (e) => {
initStatus();
configPath = await loadModel(e);
document.getElementById('status').innerHTML = `The config file '${configPath}' is created successfully.`;
});
let modelPath = "";
let modelFile = document.getElementById('modelFile');
modelFile.addEventListener('change', async (e) => {
initStatus();
modelPath = await loadModel(e);
document.getElementById('status').innerHTML = `The model file '${modelPath}' is created successfully.`;
configPath = "";
configFile.value = "";
});
utils.loadOpenCv(() => {
tryIt.removeAttribute('disabled');
});
var main = async function() {};
var softmax = function(result){};
var getTopClasses = function(mat, labels, topK = 3){};
utils.executeCode('codeEditor1');
utils.executeCode('codeEditor2');
utils.executeCode('codeEditor3');
utils.executeCode('codeEditor4');
utils.executeCode('codeEditor5');
function updateResult(classes, time) {
try{
classes.forEach((c,i) => {
let labelElement = document.getElementById('label'+i);
let probElement = document.getElementById('prob'+i);
labelElement.innerHTML = c.label;
probElement.innerHTML = c.prob + '%';
});
let result = document.getElementById('result');
result.style.visibility = 'visible';
document.getElementById('status').innerHTML = `<b>Model:</b> ${modelPath}<br>
<b>Inference time:</b> ${time.toFixed(2)} ms`;
} catch(e) {
console.log(e);
}
}
function initStatus() {
document.getElementById('status').innerHTML = '';
document.getElementById('result').style.visibility = 'hidden';
utils.clearError();
}
</script>
</body>
</html>

56
doc/js_tutorials/js_assets/webnn-electron/main.js Normal file
View File

@ -0,0 +1,56 @@
// Modules to control application life and create native browser window
const {app, BrowserWindow} = require('electron')
const path = require('path')
// Keep a global reference of the window object, if you don't, the window will
// be closed automatically when the JavaScript object is garbage collected.
let mainWindow = {}
function createWindow() {
// Create the browser window.
mainWindow = new BrowserWindow({
width: 1220,
height: 840,
webPreferences: {
nodeIntegration: true,
contextIsolation: false,
preload: app.getAppPath()+"/node_setup.js"
}
})
// Load the index.html with 'numRunsParm' to run inference multiple times.
let url = `file://${__dirname}/js_image_classification_webnn_electron.html`
const numRunsParm = '?' + process.argv[2]
mainWindow.loadURL(url + numRunsParm)
// Emitted when the window is closed.
mainWindow.on('closed', function() {
// Dereference the window object, usually you would store windows
// in an array if your app supports multi windows, this is the time
// when you should delete the corresponding element.
mainWindow = null
})
}
// This method will be called when Electron has finished
// initialization and is ready to create browser windows.
// Some APIs can only be used after this event occurs.
app.on('ready', createWindow)
// Quit when all windows are closed.
app.on('window-all-closed', function() {
// On macOS it is common for applications and their menu bar
// to stay active until the user quits explicitly with Cmd + Q
if (process.platform !== 'darwin') app.quit()
})
app.on(
'activate',
function() {
// On macOS it's common to re-create a window in the app when the
// dock icon is clicked and there are no other windows open.
if (mainWindow === null) createWindow()
})
// In this file you can include the rest of your app's specific main process
// code. You can also put them in separate files and require them here.

12
doc/js_tutorials/js_assets/webnn-electron/node_setup.js Normal file
View File

@ -0,0 +1,12 @@
const cv = require('./opencv');
const webnn = require(process.env.WEBNN_NATIVE_DIR+'/../../node/lib/webnn');
// navigator is undefined in node.js, but defined in electron.js.
if (global.navigator === undefined) {
global.navigator = {};
}
global.navigator.ml = webnn.ml;
global.MLContext = webnn.MLContext
global.MLGraphBuilder = webnn.MLGraphBuilder
global.MLGraph = webnn.MLGraph
global.MLOperand = webnn.MLOperand
global.cv = cv;

14
doc/js_tutorials/js_assets/webnn-electron/package.json Normal file
View File

@ -0,0 +1,14 @@
{
"name": "image_classification",
"version": "0.0.1",
"description": "An Electon.js example of image_classification using webnn-native",
"main": "main.js",
"author": "WebNN-native Authors",
"license": "Apache-2.0",
"scripts": {
"start": "electron ."
},
"dependencies": {
"electron": "^15.1.2"
}
}

159
doc/js_tutorials/js_assets/webnn-electron/utils_webnn_electron.js Normal file
View File

@ -0,0 +1,159 @@
function Utils(errorOutputId) { // eslint-disable-line no-unused-vars
let self = this;
this.errorOutput = document.getElementById(errorOutputId);
const OPENCV_URL = 'opencv.js';
this.loadOpenCv = async function(onloadCallback) {
if (cv.getBuildInformation)
{
console.log(cv.getBuildInformation());
onloadCallback();
}
else
{
// WASM
if (cv instanceof Promise) {
cv = await cv;
console.log(cv.getBuildInformation());
onloadCallback();
} else {
cv['onRuntimeInitialized']=()=>{
console.log(cv.getBuildInformation());
onloadCallback();
}
}
}
};
this.createFileFromUrl = function(path, url, callback) {
let request = new XMLHttpRequest();
request.open('GET', url, true);
request.responseType = 'arraybuffer';
request.onload = function(ev) {
if (request.readyState === 4) {
if (request.status === 200) {
let data = new Uint8Array(request.response);
cv.FS_createDataFile('/', path, data, true, false, false);
callback();
} else {
self.printError('Failed to load ' + url + ' status: ' + request.status);
}
}
};
request.send();
};
this.loadImageToCanvas = function(url, cavansId) {
let canvas = document.getElementById(cavansId);
let ctx = canvas.getContext('2d');
let img = new Image();
img.crossOrigin = 'anonymous';
img.onload = function() {
canvas.width = img.width;
canvas.height = img.height;
ctx.drawImage(img, 0, 0, img.width, img.height);
};
img.src = url;
};
this.executeCode = function(textAreaId) {
try {
this.clearError();
let code = document.getElementById(textAreaId).value;
eval(code);
} catch (err) {
this.printError(err);
}
};
this.clearError = function() {
this.errorOutput.innerHTML = '';
};
this.printError = function(err) {
if (typeof err === 'undefined') {
err = '';
} else if (typeof err === 'number') {
if (!isNaN(err)) {
if (typeof cv !== 'undefined') {
err = 'Exception: ' + cv.exceptionFromPtr(err).msg;
}
}
} else if (typeof err === 'string') {
let ptr = Number(err.split(' ')[0]);
if (!isNaN(ptr)) {
if (typeof cv !== 'undefined') {
err = 'Exception: ' + cv.exceptionFromPtr(ptr).msg;
}
}
} else if (err instanceof Error) {
err = err.stack.replace(/\n/g, '<br>');
}
this.errorOutput.innerHTML = err;
};
this.loadCode = function(scriptId, textAreaId) {
let scriptNode = document.getElementById(scriptId);
let textArea = document.getElementById(textAreaId);
if (scriptNode.type !== 'text/code-snippet') {
throw Error('Unknown code snippet type');
}
textArea.value = scriptNode.text.replace(/^\n/, '');
};
this.addFileInputHandler = function(fileInputId, canvasId) {
let inputElement = document.getElementById(fileInputId);
inputElement.addEventListener('change', (e) => {
let files = e.target.files;
if (files.length > 0) {
let imgUrl = URL.createObjectURL(files[0]);
self.loadImageToCanvas(imgUrl, canvasId);
}
}, false);
};
function onVideoCanPlay() {
if (self.onCameraStartedCallback) {
self.onCameraStartedCallback(self.stream, self.video);
}
};
this.startCamera = function(resolution, callback, videoId) {
const constraints = {
'qvga': {width: {exact: 320}, height: {exact: 240}},
'vga': {width: {exact: 640}, height: {exact: 480}}};
let video = document.getElementById(videoId);
if (!video) {
video = document.createElement('video');
}
let videoConstraint = constraints[resolution];
if (!videoConstraint) {
videoConstraint = true;
}
navigator.mediaDevices.getUserMedia({video: videoConstraint, audio: false})
.then(function(stream) {
video.srcObject = stream;
video.play();
self.video = video;
self.stream = stream;
self.onCameraStartedCallback = callback;
video.addEventListener('canplay', onVideoCanPlay, false);
})
.catch(function(err) {
self.printError('Camera Error: ' + err.name + ' ' + err.message);
});
};
this.stopCamera = function() {
if (this.video) {
this.video.pause();
this.video.srcObject = null;
this.video.removeEventListener('canplay', onVideoCanPlay);
}
if (this.stream) {
this.stream.getVideoTracks()[0].stop();
}
};
};

6
doc/js_tutorials/js_setup/js_setup/js_setup.markdown
View File

@ -145,6 +145,12 @@ Building OpenCV.js from Source
python ./platforms/js/build_js.py build_js --cmake_option="-DOPENCV_EXTRA_MODULES_PATH=opencv_contrib/modules"
@endcode
-# [optional] To enable WebNN backend, append `--webnn` option.
For example:
@code{.bash}
emcmake python ./opencv/platforms/js/build_js.py build_js --webnn
@endcode
Running OpenCV.js Tests
---------------------------------------

16
modules/dnn/CMakeLists.txt
View File

@ -19,6 +19,10 @@ if(OPENCV_DNN_OPENCL AND HAVE_OPENCL)
add_definitions(-DCV_OCL4DNN=1)
endif()
if(WITH_WEBNN AND HAVE_WEBNN)
add_definitions(-DHAVE_WEBNN=1)
endif()
ocv_option(OPENCV_DNN_CUDA "Build with CUDA support"
HAVE_CUDA
AND HAVE_CUBLAS
@ -142,6 +146,16 @@ if(HAVE_TENGINE)
list(APPEND libs -Wl,--whole-archive ${TENGINE_LIBRARIES} -Wl,--no-whole-archive)
endif()
set(webnn_srcs "")
if(NOT EMSCRIPTEN)
if(HAVE_WEBNN)
list(APPEND include_dirs ${WEBNN_HEADER_DIRS})
list(APPEND include_dirs ${WEBNN_INCLUDE_DIRS})
list(APPEND libs -Wl,--whole-archive ${WEBNN_LIBRARIES} -Wl,--no-whole-archive)
list(APPEND webnn_srcs $ENV{WEBNN_NATIVE_DIR}/gen/src/webnn/webnn_cpp.cpp)
endif()
endif()
ocv_module_include_directories(${include_dirs})
if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
ocv_append_source_files_cxx_compiler_options(fw_srcs "-Wno-suggest-override") # GCC
@ -171,7 +185,7 @@ if(HAVE_NGRAPH)
list(APPEND dnn_runtime_libs ngraph::ngraph)
endif()
ocv_glob_module_sources(${sources_options} SOURCES ${fw_srcs})
ocv_glob_module_sources(${sources_options} SOURCES ${fw_srcs} ${webnn_srcs})
ocv_create_module(${libs} ${dnn_runtime_libs})
ocv_add_samples()
ocv_add_accuracy_tests(${dnn_runtime_libs})

3
modules/dnn/include/opencv2/dnn/dnn.hpp
View File

@ -74,6 +74,7 @@ CV__DNN_INLINE_NS_BEGIN
DNN_BACKEND_OPENCV,
DNN_BACKEND_VKCOM,
DNN_BACKEND_CUDA,
DNN_BACKEND_WEBNN,
#ifdef __OPENCV_BUILD
DNN_BACKEND_INFERENCE_ENGINE_NGRAPH = 1000000, // internal - use DNN_BACKEND_INFERENCE_ENGINE + setInferenceEngineBackendType()
DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019, // internal - use DNN_BACKEND_INFERENCE_ENGINE + setInferenceEngineBackendType()
@ -307,6 +308,8 @@ CV__DNN_INLINE_NS_BEGIN
virtual Ptr<BackendNode> initVkCom(const std::vector<Ptr<BackendWrapper> > &inputs);
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> > &inputs, const std::vector<Ptr<BackendNode> >& nodes);
/**
* @brief Returns a CUDA backend node
*

313
modules/dnn/src/dnn.cpp
View File

@ -45,6 +45,7 @@
#include "ie_ngraph.hpp"
#include "op_vkcom.hpp"
#include "op_cuda.hpp"
#include "op_webnn.hpp"
#ifdef HAVE_CUDA
#include "cuda4dnn/init.hpp"
@ -224,6 +225,13 @@ private:
#endif
#endif // HAVE_INF_ENGINE
#ifdef HAVE_WEBNN
if (haveWebnn())
{
backends.push_back(std::make_pair(DNN_BACKEND_WEBNN, DNN_TARGET_CPU));
}
#endif // HAVE_WEBNN
#ifdef HAVE_OPENCL
if (cv::ocl::useOpenCL())
{
@ -1114,6 +1122,14 @@ static Ptr<BackendWrapper> wrapMat(int backendId, int targetId, cv::Mat& m)
return Ptr<BackendWrapper>(new NgraphBackendWrapper(targetId, m));
#else
CV_Error(Error::StsNotImplemented, "This OpenCV version is built without support of Inference Engine + nGraph");
#endif
}
else if (backendId == DNN_BACKEND_WEBNN)
{
#ifdef HAVE_WEBNN
return Ptr<BackendWrapper>(new WebnnBackendWrapper(targetId, m));
#else
CV_Error(Error::StsNotImplemented, "This OpenCV version is built without support of WebNN");
#endif
}
else if (backendId == DNN_BACKEND_VKCOM)
@ -1259,6 +1275,12 @@ struct Net::Impl : public detail::NetImplBase
{
return wrapMat(preferableBackend, preferableTarget, host);
}
else if (preferableBackend == DNN_BACKEND_WEBNN)
{
#ifdef HAVE_WEBNN
return wrapMat(preferableBackend, preferableTarget, host);
#endif
}
else if (preferableBackend == DNN_BACKEND_VKCOM)
{
#ifdef HAVE_VULKAN
@ -1399,6 +1421,13 @@ struct Net::Impl : public detail::NetImplBase
preferableTarget == DNN_TARGET_FPGA
);
}
#endif
#ifdef HAVE_WEBNN
if (preferableBackend == DNN_BACKEND_WEBNN)
{
CV_Assert(preferableTarget == DNN_TARGET_CPU ||
preferableTarget == DNN_TARGET_OPENCL);
}
#endif
CV_Assert(preferableBackend != DNN_BACKEND_VKCOM ||
preferableTarget == DNN_TARGET_VULKAN);
@ -1622,6 +1651,14 @@ struct Net::Impl : public detail::NetImplBase
initNgraphBackend(blobsToKeep_);
#else
CV_Error(Error::StsNotImplemented, "This OpenCV version is built without support of Inference Engine + nGraph");
#endif
}
else if (preferableBackend == DNN_BACKEND_WEBNN)
{
#ifdef HAVE_WEBNN
initWebnnBackend(blobsToKeep_);
#else
CV_Error(Error::StsNotImplemented, "This OpenCV version is built without support of WebNN");
#endif
}
else if (preferableBackend == DNN_BACKEND_VKCOM)
@ -2341,6 +2378,270 @@ struct Net::Impl : public detail::NetImplBase
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
void addWebnnOutputs(LayerData &ld)
{
CV_TRACE_FUNCTION();
Ptr<WebnnNet> layerNet;
auto it = ld.backendNodes.find(preferableBackend);
if (it != ld.backendNodes.end())
{
Ptr<BackendNode> node = it->second;
if (!node.empty())
{
Ptr<WebnnBackendNode> webnnNode = node.dynamicCast<WebnnBackendNode>();
CV_Assert(!webnnNode.empty()); CV_Assert(!webnnNode->net.empty());
layerNet = webnnNode->net;
}
}
for (int i = 0; i < ld.inputBlobsId.size(); ++i)
{
LayerData &inpLd = layers[ld.inputBlobsId[i].lid];
Ptr<BackendNode> inpNode = inpLd.backendNodes[preferableBackend];
if (!inpNode.empty())
{
Ptr<WebnnBackendNode> webnnInpNode = inpNode.dynamicCast<WebnnBackendNode>();
CV_Assert(!webnnInpNode.empty()); CV_Assert(!webnnInpNode->net.empty());
if (layerNet != webnnInpNode->net)
{
webnnInpNode->net->addOutput(webnnInpNode->name);
webnnInpNode->net->setUnconnectedNodes(webnnInpNode);
}
}
}
}
void initWebnnBackend(const std::vector<LayerPin>& blobsToKeep_)
{
CV_TRACE_FUNCTION();
CV_Assert_N(preferableBackend == DNN_BACKEND_WEBNN, haveWebnn());
MapIdToLayerData::iterator it;
Ptr<WebnnNet> net;
for (it = layers.begin(); it != layers.end(); ++it)
{
LayerData &ld = it->second;
if (ld.id == 0)
{
CV_Assert((netInputLayer->outNames.empty() && ld.outputBlobsWrappers.size() == 1) ||
(netInputLayer->outNames.size() == ld.outputBlobsWrappers.size()));
for (int i = 0; i < ld.outputBlobsWrappers.size(); ++i)
{
Ptr<WebnnBackendWrapper> wrapper = ld.outputBlobsWrappers[i].dynamicCast<WebnnBackendWrapper>();
std::string outputName = netInputLayer->outNames.empty() ? ld.name : netInputLayer->outNames[i];
outputName = ld.outputBlobsWrappers.size() > 1 ? (outputName + "." + std::to_string(i)) : outputName;
wrapper->name = outputName;
}
}
else
{
for (int i = 0; i < ld.outputBlobsWrappers.size(); ++i)
{
Ptr<WebnnBackendWrapper> wrapper = ld.outputBlobsWrappers[i].dynamicCast<WebnnBackendWrapper>();
std::string outputName = ld.outputBlobsWrappers.size() > 1 ? (ld.name + "." + std::to_string(i)) : ld.name;
wrapper->name = outputName;
}
}
}
// Build WebNN networks from sets of layers that support this
// backend. Split a whole model on several WebNN networks if
// some of layers are not implemented.
for (it = layers.begin(); it != layers.end(); ++it)
{
LayerData &ld = it->second;
if (ld.id == 0 && ld.skip)
continue;
bool fused = ld.skip;
Ptr<Layer> layer = ld.layerInstance;
if (!fused && !layer->supportBackend(preferableBackend))
{
// For test use. when not using WebNN, the test case will fail
// with the following code.
CV_LOG_WARNING(NULL, "Layer " + ld.type + " name " + ld.name + " is unsupported by WebNN backend.");
addWebnnOutputs(ld);
net = Ptr<WebnnNet>();
layer->preferableTarget = DNN_TARGET_CPU;
for (int i = 0; i < ld.inputBlobsId.size(); ++i)
{
LayerData &inpLd = layers[ld.inputBlobsId[i].lid];
Ptr<BackendNode> inpNode = inpLd.backendNodes[preferableBackend];
if (!inpNode.empty()) {
Ptr<WebnnBackendNode> webnnNode = inpNode.dynamicCast<WebnnBackendNode>();
CV_Assert(!webnnNode.empty());
webnnNode->net->setUnconnectedNodes(webnnNode);
}
}
continue;
}
ld.skip = true; // Initially skip all WebNN supported layers.
// Create a new network if one of inputs from different WebNN graph.
std::vector<Ptr<BackendNode>> inputNodes;
for (int i = 0; i < ld.inputBlobsId.size(); ++i)
{
// Layer_Test_ROIPooling.Accuracy has 2 inputs inpLD = 0, 0 -> has 4 inputNodes (input, rois, input, rois)
if (inputNodes.size() == ld.inputBlobsId.size()) {
break;
}
LayerData &inpLd = layers[ld.inputBlobsId[i].lid];
Ptr<BackendNode> inpNode = inpLd.backendNodes[preferableBackend];
if (!inpNode.empty())
{
Ptr<WebnnBackendNode> webnnInpNode = inpNode.dynamicCast<WebnnBackendNode>();
CV_Assert(!webnnInpNode.empty()); CV_Assert(!webnnInpNode->net.empty());
if (webnnInpNode->net == net && !fused) {
inputNodes.push_back(inpNode);
continue;
}
}
if (net.empty()) {
net = Ptr<WebnnNet>(new WebnnNet());
}
if (!fused) {
std::vector<std::string> inputNames;
std::vector<cv::Mat> inputs;
auto curr_pos = inpLd.consumers.begin();
auto compare = [&ld] (const LayerPin& lp) { return lp.lid == ld.id; };
auto cons = curr_pos;
while ((cons = std::find_if(curr_pos, inpLd.consumers.end(), compare)) !=
inpLd.consumers.end()) {
int cons_inp = cons->oid;
Ptr<WebnnBackendWrapper> inpWrapper = inpLd.outputBlobsWrappers[cons_inp].
dynamicCast<WebnnBackendWrapper>();
CV_Assert(!inpWrapper.empty());
auto iter = std::find(inputNames.begin(), inputNames.end(),
inpWrapper->name);
if (iter == inputNames.end()) {
inputNames.push_back(inpWrapper->name);
inputs.push_back(inpLd.outputBlobs[cons_inp]);
}
curr_pos = cons + 1;
}
auto inps = net->setInputs(inputs, inputNames);
for (auto& inp : inps) {
WebnnBackendNode* node = new WebnnBackendNode(inp);
node->net = net;
inputNodes.emplace_back(Ptr<BackendNode>(node));
}
}
}
Ptr<BackendNode> node;
if (!net.empty())
{
if (fused)
{
bool inPlace = ld.inputBlobsId.size() == 1 && ld.outputBlobs.size() == 1 &&
ld.inputBlobs[0]->data == ld.outputBlobs[0].data;
CV_Assert(inPlace);
node = layers[ld.inputBlobsId[0].lid].backendNodes[preferableBackend];
ld.inputBlobsWrappers = layers[ld.inputBlobsId[0].lid].inputBlobsWrappers;
}
}
else {
net = Ptr<WebnnNet>(new WebnnNet());
}
if (!fused)
{
CV_Assert(ld.inputBlobsId.size() == inputNodes.size());
for (int i = 0; i < ld.inputBlobsId.size(); ++i)
{
int lid = ld.inputBlobsId[i].lid;
int oid = ld.inputBlobsId[i].oid;
if (oid == 0 || lid == 0)
continue;
auto webnnInpNode = inputNodes[i].dynamicCast<WebnnBackendNode>();
inputNodes[i] = Ptr<BackendNode>(new WebnnBackendNode(webnnInpNode->operand));
}
if (layer->supportBackend(preferableBackend))
{
if (ld.type == "Const") {
ml::Operand fake_operand;
Ptr<WebnnBackendNode> fake_input_node = Ptr<WebnnBackendNode>(new WebnnBackendNode(fake_operand));
fake_input_node->net = net;
inputNodes.push_back(fake_input_node);
}
node = layer->initWebnn(ld.inputBlobsWrappers, inputNodes);
for (int i = 0; i < ld.outputBlobsWrappers.size(); ++i)
{
Ptr<WebnnBackendWrapper> wrapper = ld.outputBlobsWrappers[i].dynamicCast<WebnnBackendWrapper>();
node.dynamicCast<WebnnBackendNode>()->name = wrapper->name;
}
}
else
{
continue;
}
}
else if (node.empty())
continue;
ld.backendNodes[preferableBackend] = node;
Ptr<WebnnBackendNode> webnnNode = node.dynamicCast<WebnnBackendNode>();
CV_Assert(!webnnNode.empty());
webnnNode->net = net;
if (ld.consumers.empty()) {
// TF EAST_text_detection
webnnNode->net->setUnconnectedNodes(webnnNode);
}
for (const auto& pin : blobsToKeep_)
{
if (pin.lid == ld.id)
{
webnnNode->net->addOutput(webnnNode->name);
break;
}
}
net->addBlobs(ld.inputBlobsWrappers);
net->addBlobs(ld.outputBlobsWrappers);
addWebnnOutputs(ld);
}
// Initialize all networks.
for (MapIdToLayerData::reverse_iterator it = layers.rbegin(); it != layers.rend(); ++it)
{
LayerData &ld = it->second;
auto iter = ld.backendNodes.find(preferableBackend);
if (iter == ld.backendNodes.end())
continue;
Ptr<BackendNode>& node = iter->second;
if (node.empty())
continue;
Ptr<WebnnBackendNode> webnnNode = node.dynamicCast<WebnnBackendNode>();
if (webnnNode.empty())
continue;
CV_Assert(!webnnNode->net.empty());
if (!webnnNode->net->isInitialized())
{
webnnNode->net->setUnconnectedNodes(webnnNode);
webnnNode->net->createNet((Target)preferableTarget);
ld.skip = false;
}
}
}
#endif
void initVkComBackend()
{
CV_TRACE_FUNCTION();
@ -3393,6 +3694,10 @@ struct Net::Impl : public detail::NetImplBase
else if (preferableBackend == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH)
{
forwardNgraph(ld.outputBlobsWrappers, node, isAsync);
}
else if (preferableBackend == DNN_BACKEND_WEBNN)
{
forwardWebnn(ld.outputBlobsWrappers, node, isAsync);
}
else if (preferableBackend == DNN_BACKEND_VKCOM)
{
@ -4830,6 +5135,7 @@ string Net::Impl::dump()
case DNN_BACKEND_OPENCV: backend = "OCV/"; break;
case DNN_BACKEND_VKCOM: backend = "VULKAN/"; break;
case DNN_BACKEND_CUDA: backend = "CUDA/"; break;
case DNN_BACKEND_WEBNN: backend = "WEBNN/"; break;
// don't use default:
}
out << "digraph G {\n";
@ -5421,6 +5727,13 @@ Ptr<BackendNode> Layer::initNgraph(const std::vector<Ptr<BackendWrapper> > & inp
return Ptr<BackendNode>();
}
Ptr<BackendNode> Layer::initWebnn(const std::vector<Ptr<BackendWrapper> > & inputs, const std::vector<Ptr<BackendNode> >& nodes)
{
CV_Error(Error::StsNotImplemented, "WebNN pipeline of " + type +
" layers is not defined.");
return Ptr<BackendNode>();
}
void Layer::applyHalideScheduler(Ptr<BackendNode>& node, const std::vector<Mat*> &inputs,
const std::vector<Mat> &outputs, int targetId) const
{

23
modules/dnn/src/layers/batch_norm_layer.cpp
View File

@ -15,6 +15,7 @@ Implementation of Batch Normalization layer.
#include "../op_halide.hpp"
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_webnn.hpp"
#include <opencv2/dnn/shape_utils.hpp>
@ -172,6 +173,7 @@ public:
return (backendId == DNN_BACKEND_OPENCV) ||
backendId == DNN_BACKEND_CUDA ||
(backendId == DNN_BACKEND_HALIDE && haveHalide()) ||
backendId == DNN_BACKEND_WEBNN ||
((backendId == DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019 || backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH) && haveInfEngine() && (preferableTarget == DNN_TARGET_CPU || dims == 4));
}
@ -421,6 +423,27 @@ public:
return true;
}
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand = node->operand;
auto& webnnGraphBuilder = node->net->builder;
std::vector<int32_t> weights_shape = webnn::getShape(weights_);
ml::Operand weights = webnn::BuildConstant(webnnGraphBuilder, weights_shape, weights_.data, weights_.total()*weights_.elemSize(), ml::OperandType::Float32);
std::vector<int32_t> shape(dims, 1);
shape[1] = weights_shape[1];
ml::Operand weights_reshaped = webnnGraphBuilder.Reshape(weights, shape.data(), shape.size());
ml::Operand mul_res = webnnGraphBuilder.Mul(webnnInpOperand, weights_reshaped);
std::vector<int32_t> bias_shape = webnn::getShape(bias_);
ml::Operand bias = webnn::BuildConstant(webnnGraphBuilder, bias_shape, bias_.data, bias_.total()*bias_.elemSize(), ml::OperandType::Float32);
shape[1] = bias_shape[1];
ml::Operand bias_reshaped = webnnGraphBuilder.Reshape(bias, shape.data(), shape.size());
ml::Operand add_res = webnnGraphBuilder.Add(mul_res, bias_reshaped);
return Ptr<BackendNode>(new WebnnBackendNode(add_res));
}
#endif
virtual int64 getFLOPS(const std::vector<MatShape> &inputs,
const std::vector<MatShape> &outputs) const CV_OVERRIDE
{

18
modules/dnn/src/layers/concat_layer.cpp
View File

@ -47,6 +47,7 @@
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_vkcom.hpp"
#include "../op_webnn.hpp"
#ifdef HAVE_OPENCL
#include "opencl_kernels_dnn.hpp"
@ -117,6 +118,7 @@ public:
(backendId == DNN_BACKEND_HALIDE && haveHalide() && axis == 1 && !padding) || // By channels
(backendId == DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019 && haveInfEngine() && !padding) ||
backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH ||
(backendId == DNN_BACKEND_WEBNN && !padding) ||
(backendId == DNN_BACKEND_VKCOM && haveVulkan() && !padding);
}
@ -408,6 +410,22 @@ public:
params.set("padding_value", zeropoints[1][0]);
return true;
}
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnGraphBuilder = node->net->builder;
std::vector<ml::Operand> inputsOperand;
for (int i = 0; i < nodes.size(); i++)
{
inputsOperand.push_back(nodes[i].dynamicCast<WebnnBackendNode>()->operand);
}
auto operand = webnnGraphBuilder.Concat(inputsOperand.size(), inputsOperand.data(), axis);
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
#endif
};
Ptr<ConcatLayer> ConcatLayer::create(const LayerParams& params)

12
modules/dnn/src/layers/const_layer.cpp
View File

@ -10,6 +10,7 @@
#include "../op_cuda.hpp"
#include "layers_common.hpp"
#include "../ie_ngraph.hpp"
#include "../op_webnn.hpp"
#ifdef HAVE_OPENCL
#include "opencl_kernels_dnn.hpp"
@ -36,6 +37,7 @@ public:
return backendId == DNN_BACKEND_OPENCV ||
backendId == DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019 ||
backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH ||
backendId == DNN_BACKEND_WEBNN ||
backendId == DNN_BACKEND_CUDA;
}
@ -97,6 +99,16 @@ public:
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
ml::Operand operand = nullptr;
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnGraphBuilder = node->net->builder;
operand = webnn::BuildConstant(webnnGraphBuilder, webnn::getShape(blobs[0]), blobs[0].data, blobs[0].total()*blobs[0].elemSize(), ml::OperandType::Float32);
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
#endif
#ifdef HAVE_CUDA
Ptr<BackendNode> initCUDA(

120
modules/dnn/src/layers/convolution_layer.cpp
View File

@ -47,6 +47,7 @@
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_vkcom.hpp"
#include "../op_webnn.hpp"
#include <opencv2/core/utils/configuration.private.hpp>
#include <opencv2/core/utils/logger.hpp>
@ -80,6 +81,9 @@ class BaseConvolutionLayerImpl : public ConvolutionLayer
public:
bool fusedWeights, fusedBias;
std::vector<double> weightsMultipliers;
#ifdef HAVE_WEBNN
int groups;
#endif
BaseConvolutionLayerImpl(const LayerParams &params)
{
setParamsFrom(params);
@ -87,6 +91,9 @@ public:
numOutput = params.get<int>("num_output");
int ngroups = params.get<int>("group", 1);
#ifdef HAVE_WEBNN
groups = ngroups;
#endif
CV_Assert(numOutput % ngroups == 0);
if (kernel_size.size() == 2) {
@ -347,6 +354,17 @@ public:
#ifdef HAVE_VULKAN
if (backendId == DNN_BACKEND_VKCOM)
return ksize == 2;
#endif
#ifdef HAVE_WEBNN
if (backendId == DNN_BACKEND_WEBNN)
{
if (ksize != 2)
{
CV_LOG_WARNING(NULL, "WebNN only supports Conv2d.");
return false;
}
return true;
}
#endif
return false;
}
@ -896,6 +914,108 @@ public:
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
CV_Assert_N(inputs.size() >= 1, nodes.size() >= 1);
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand = node->operand;
auto& webnnGraphBuilder = node->net->builder;
ml::Operand webnnWeights = nodes.size() > 1 ? nodes[1].dynamicCast<WebnnBackendNode>()->operand : nullptr;
if (nodes.size() > 1)
CV_Assert(webnnWeights);
const int inpCn = weightsMat.total()/(kernel_size[0]*kernel_size[1]*numOutput);
const int group = groups;
const int inpGroupCn = inpCn / group;
std::vector<int32_t> kernel_shape;
if (group != 1)
{
kernel_shape.push_back(group);
}
kernel_shape.push_back(numOutput / group);
kernel_shape.push_back(inpGroupCn);
std::copy(kernel_size.begin(), kernel_size.end(), back_inserter(kernel_shape));
if (nodes.size() == 1)
{
webnnWeights = webnn::BuildConstant(webnnGraphBuilder, webnn::getShape(blobs[0]), blobs[0].data, blobs[0].total()*blobs[0].elemSize(), ml::OperandType::Float32);
if (fusedWeights)
{
if (weightsMat.isContinuous())
{
webnnWeights = webnn::BuildConstant(webnnGraphBuilder, webnn::getShape(weightsMat), weightsMat.data, weightsMat.total()*weightsMat.elemSize(), ml::OperandType::Float32);
}
else
{
Mat newWeights;
Mat cvWeights = weightsMat.colRange(0, blobs[0].total() / numOutput);
cvWeights.copyTo(newWeights);
webnnWeights = webnn::BuildConstant(webnnGraphBuilder, webnn::getShape(newWeights), newWeights.data, newWeights.total()*newWeights.elemSize(), ml::OperandType::Float32);
}
}
}
else
{
webnnWeights = webnnGraphBuilder.Reshape(webnnWeights, kernel_shape.data(), kernel_shape.size());
}
ml::AutoPad pad_type = ml::AutoPad::Explicit;
if (!padMode.empty())
pad_type = padMode == "VALID" ? ml::AutoPad::Explicit : ml::AutoPad::SameUpper;
ml::Conv2dOptions options = {};
options.groups = group;
options.autoPad = pad_type;
std::vector<int32_t> Strides(strides.begin(), strides.end());
if (!Strides.empty())
{
options.stridesCount = Strides.size();
options.strides = Strides.data();
}
std::vector<int32_t> Padding;
if (padMode.empty())
{
Padding = {static_cast<int32_t>(pads_begin[0]),
static_cast<int32_t>(pads_end[0]),
static_cast<int32_t>(pads_begin[1]),
static_cast<int32_t>(pads_end[1])};
}
else if (padMode == "VALID")
{
Padding = {0, 0, 0, 0};
}
if (!Padding.empty())
{
options.paddingCount = Padding.size();
options.padding = Padding.data();
}
std::vector<int32_t> Dilations(dilations.begin(), dilations.end());
if (!Dilations.empty())
{
options.dilationsCount = Dilations.size();
options.dilations = Dilations.data();
}
ml::Operand operand = webnnGraphBuilder.Conv2d(webnnInpOperand, webnnWeights, &options);
// ml::Operand result = operand;
if (hasBias() || fusedBias || nodes.size() == 3)
{
ml::Operand webnnBias = nullptr;
if (nodes.size() == 3)
{
std::vector<int32_t> bias_shape = {1, numOutput, 1, 1};
webnnBias = webnnGraphBuilder.Reshape(nodes[2].dynamicCast<WebnnBackendNode>()->operand, bias_shape.data(), bias_shape.size());
}
else
{
webnnBias = webnn::BuildConstant(webnnGraphBuilder, {1, numOutput, 1, 1}, biasvec.data(), (numOutput) * sizeof(float), ml::OperandType::Float32);
}
operand = webnnGraphBuilder.Add(operand, webnnBias);
}
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
#endif // HAVE_WEBNN
class ParallelConv : public cv::ParallelLoopBody
{
public:

135
modules/dnn/src/layers/elementwise_layers.cpp
View File

@ -47,8 +47,11 @@
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_vkcom.hpp"
#include "../op_webnn.hpp"
#include <opencv2/dnn/shape_utils.hpp>
#include <iostream>
#include <limits>
#include <cfenv>
#ifdef HAVE_OPENCL
@ -59,6 +62,7 @@
#include "../cuda4dnn/primitives/activation.hpp"
using namespace cv::dnn::cuda4dnn;
#endif
#include <opencv2/core/utils/logger.hpp>
namespace cv
{
@ -186,6 +190,17 @@ public:
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand = node->operand;
auto& webnnGraphBuilder = node->net->builder;
auto operand = func.initWebnnAPI(webnnGraphBuilder, webnnInpOperand);
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
#endif
virtual Ptr<BackendNode> initVkCom(const std::vector<Ptr<BackendWrapper> >& inputs) CV_OVERRIDE
{
#ifdef HAVE_VULKAN
@ -319,6 +334,16 @@ struct ReLUFunctor : public BaseFunctor
#ifdef HAVE_DNN_NGRAPH
if (backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH)
return true;
#endif
#ifdef HAVE_WEBNN
if (backendId == DNN_BACKEND_WEBNN) {
// TODO: support PRELU
if (slope != 0)
{
CV_LOG_WARNING(NULL, "PRELU is not supported now.");
}
return slope == 0;
}
#endif
return backendId == DNN_BACKEND_OPENCV ||
backendId == DNN_BACKEND_CUDA ||
@ -441,6 +466,13 @@ struct ReLUFunctor : public BaseFunctor
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
return builder.Relu(input);
}
#endif
#ifdef HAVE_VULKAN
std::shared_ptr<vkcom::OpBase> initVkCom()
{
@ -491,6 +523,7 @@ struct ReLU6Functor : public BaseFunctor
return backendId == DNN_BACKEND_OPENCV ||
backendId == DNN_BACKEND_CUDA ||
backendId == DNN_BACKEND_HALIDE ||
backendId == DNN_BACKEND_WEBNN ||
backendId == DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019 || backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH;
}
@ -587,6 +620,18 @@ struct ReLU6Functor : public BaseFunctor
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
ml::ClampOptions clampOptions;
clampOptions.minValue = minValue;
clampOptions.maxValue = maxValue;
return builder.Clamp(input, &clampOptions);
}
#endif
#ifdef HAVE_VULKAN
std::shared_ptr<vkcom::OpBase> initVkCom()
{
@ -684,6 +729,15 @@ struct BaseDefaultFunctor : public BaseFunctor
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
#ifdef HAVE_VULKAN
std::shared_ptr<vkcom::OpBase> initVkCom()
{
@ -742,6 +796,15 @@ struct TanHFunctor : public BaseDefaultFunctor<TanHFunctor>
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
int64 getFLOPSPerElement() const { return 1; }
};
@ -845,6 +908,15 @@ struct MishFunctor : public BaseDefaultFunctor<MishFunctor>
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
int64 getFLOPSPerElement() const { return 3; }
};
@ -897,6 +969,15 @@ struct SigmoidFunctor : public BaseDefaultFunctor<SigmoidFunctor>
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
int64 getFLOPSPerElement() const { return 3; }
};
@ -1007,6 +1088,15 @@ struct AbsValFunctor : public BaseDefaultFunctor<AbsValFunctor>
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
int64 getFLOPSPerElement() const { return 1; }
};
@ -1136,6 +1226,15 @@ struct LogFunctor : public BaseDefaultFunctor<LogFunctor>
return log(x);
}
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
#ifdef HAVE_CUDA
Ptr<BackendNode> initCUDA(int target, csl::Stream stream)
{
@ -1233,6 +1332,15 @@ struct SqrtFunctor : public BaseDefaultFunctor<SqrtFunctor>
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
int64 getFLOPSPerElement() const { return 1; }
};
@ -1415,6 +1523,15 @@ struct PowerFunctor : public BaseFunctor
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
#ifdef HAVE_VULKAN
std::shared_ptr<vkcom::OpBase> initVkCom()
{
@ -1517,6 +1634,15 @@ struct ExpFunctor : public BaseDefaultFunctor<ExpFunctor>
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
int64 getFLOPSPerElement() const { return 3; }
};
@ -1649,6 +1775,15 @@ struct ChannelsPReLUFunctor : public BaseFunctor
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
ml::Operand initWebnnAPI(const ml::GraphBuilder& builder, const ml::Operand& input)
{
CV_Error(Error::StsNotImplemented, "");
ml::Operand operand;
return operand;
}
#endif
#ifdef HAVE_VULKAN
std::shared_ptr<vkcom::OpBase> initVkCom()
{

36
modules/dnn/src/layers/fully_connected_layer.cpp
View File

@ -46,6 +46,7 @@
#include "../op_halide.hpp"
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_webnn.hpp"
#include <opencv2/dnn/shape_utils.hpp>
@ -150,6 +151,7 @@ public:
return backendId == DNN_BACKEND_OPENCV ||
backendId == DNN_BACKEND_CUDA ||
(backendId == DNN_BACKEND_HALIDE && haveHalide() && axis == 1) ||
(backendId == DNN_BACKEND_WEBNN && axis == 1) ||
(((backendId == DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019 && !blobs.empty()) ||
backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH) && axis == 1);
}
@ -657,6 +659,40 @@ public:
return true;
}
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand = node->operand;
auto& webnnGraphBuilder = node->net->builder;
ml::GemmOptions gemmOptions = {};
if (bias)
{
std::vector<int32_t> biasDims = {(int32_t)blobs[1].size[1]};
ml::Operand bias = webnn::BuildConstant(webnnGraphBuilder, biasDims, blobs[1].data, blobs[1].total()*blobs[1].elemSize(), ml::OperandType::Float32);
gemmOptions.c = bias;
}
ml::Operand result = nullptr;
if (nodes.size() == 2)
{
auto& inp2 = nodes[1].dynamicCast<WebnnBackendNode>()->operand;
result = webnnGraphBuilder.Gemm(webnnInpOperand, inp2, &gemmOptions);
}
else
{
std::vector<int32_t> input_shape(2, -1);
input_shape[1] = blobs[0].size[1];
ml::Operand webnnInpOperand_reshaped = webnnGraphBuilder.Reshape(webnnInpOperand, input_shape.data(), input_shape.size());
std::vector<int32_t> weight_shape = {(int32_t)blobs[0].size[0], (int32_t)blobs[0].size[1]};
// std::cout<<"weight size: "<<weight_shape[1]<<" "<<weight_shape[0]<<std::endl;
ml::Operand inp2 = webnn::BuildConstant(webnnGraphBuilder, weight_shape, blobs[0].data, blobs[0].total()*blobs[0].elemSize(), ml::OperandType::Float32);
gemmOptions.bTranspose = true;
result = webnnGraphBuilder.Gemm(webnnInpOperand_reshaped, inp2, &gemmOptions);
}
return Ptr<BackendNode>(new WebnnBackendNode(result));
}
#endif // HAVE_WEBNN
virtual int64 getFLOPS(const std::vector<MatShape> &inputs,
const std::vector<MatShape> &outputs) const CV_OVERRIDE
{

16
modules/dnn/src/layers/permute_layer.cpp
View File

@ -46,6 +46,7 @@
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_vkcom.hpp"
#include "../op_webnn.hpp"
#include <float.h>
#include <algorithm>
@ -119,6 +120,7 @@ public:
#endif
return backendId == DNN_BACKEND_OPENCV ||
backendId == DNN_BACKEND_CUDA ||
backendId == DNN_BACKEND_WEBNN ||
((backendId == DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019 || backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH) && haveInfEngine()) ||
(backendId == DNN_BACKEND_VKCOM && haveVulkan());
}
@ -439,6 +441,20 @@ public:
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand = node->operand;
auto& webnnGraphBuilder = node->net->builder;
std::vector<int32_t> permutation(_order.begin(), _order.end());
ml::TransposeOptions options;
options.permutation = permutation.data();
options.permutationCount = permutation.size();
auto operand = webnnGraphBuilder.Transpose(webnnInpOperand, &options);
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
#endif
#ifdef HAVE_CUDA
Ptr<BackendNode> initCUDA(

86
modules/dnn/src/layers/pooling_layer.cpp
View File

@ -46,6 +46,7 @@
#include "../op_cuda.hpp"
#include "../op_halide.hpp"
#include "../op_inf_engine.hpp"
#include "../op_webnn.hpp"
#ifdef HAVE_DNN_NGRAPH
#include "../ie_ngraph.hpp"
@ -85,6 +86,7 @@ typedef int HALIDE_DIFF_T;
#include "../cuda4dnn/primitives/max_unpooling.hpp"
using namespace cv::dnn::cuda4dnn;
#endif
#include <opencv2/core/utils/logger.hpp>
namespace cv
@ -246,6 +248,51 @@ public:
(type == MAX || type == AVE);
return false;
}
else if (backendId == DNN_BACKEND_WEBNN)
{
if (kernel_size.empty() || kernel_size.size() == 2)
{
if (!haveWebnn())
{
return false;
}
else
{
if (!ceilMode)
{
CV_LOG_WARNING(NULL, "ceilMode is not supported by WebNN backend.");
return false;
}
if (computeMaxIdx)
{
CV_LOG_WARNING(NULL, "Mask is not supported by WebNN backend.");
return false;
}
if (type != MAX && type != AVE)
{
if (type == STOCHASTIC)
{
CV_LOG_WARNING(NULL, "Stochastic Pooling is not supported by WebNN backend.");
}
if (type == SUM)
{
CV_LOG_WARNING(NULL, "Sum Pooling is not supported by WebNN backend.");
}
if (type == ROI)
{
CV_LOG_WARNING(NULL, "ROI Pooling is not supported by WebNN backend.");
}
if (type == PSROI)
{
CV_LOG_WARNING(NULL, "Position-sensitive ROI Pooling is not supported by WebNN backend.");
}
CV_LOG_WARNING(NULL, "WebNN backend only supports MaxPooling and AveragePooling currently.");
return false;
}
}
return true;
}
}
return false;
}
@ -607,6 +654,45 @@ public:
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
// std::cout << "Use WebNN Pooling Layer's Implementation." << std::endl;
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand = node->operand;
auto& webnnGraphBuilder = node->net->builder;
webnn::Pool2dOptions options;
std::vector<int32_t> kernelSize(kernel_size.begin(), kernel_size.end());
std::vector<int32_t> Strides(strides.begin(), strides.end());
std::vector<int32_t> Padding;
if (padMode.empty()) {
Padding = {static_cast<int32_t>(pads_begin[0]),
static_cast<int32_t>(pads_end[0]),
static_cast<int32_t>(pads_begin[1]),
static_cast<int32_t>(pads_end[1])};
} else if (padMode == "VALID") {
Padding = {0, 0, 0, 0};
} else if (padMode == "SAME") {
options.autoPad = ml::AutoPad::SameUpper;
}
// std::cout << "padMode: " << padMode << std::endl;
options.windowDimensions = kernelSize;
options.strides = Strides;
options.padding = Padding;
if (type == MAX)
{
auto operand = webnnGraphBuilder.MaxPool2d(webnnInpOperand, options.AsPtr());
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
else if (type == AVE)
{
auto operand = webnnGraphBuilder.AveragePool2d(webnnInpOperand, options.AsPtr());
return Ptr<BackendNode>(new WebnnBackendNode(operand));
} else {
CV_Error(Error::StsNotImplemented, "Unsupported pooling type");
}
}
#endif // HAVE_WEBNN
class PoolingInvoker : public ParallelLoopBody
{

13
modules/dnn/src/layers/reshape_layer.cpp
View File

@ -45,6 +45,7 @@
#include "../op_cuda.hpp"
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_webnn.hpp"
#include <opencv2/dnn/shape_utils.hpp>
@ -203,6 +204,7 @@ public:
{
return backendId == DNN_BACKEND_OPENCV ||
backendId == DNN_BACKEND_CUDA ||
backendId == DNN_BACKEND_WEBNN ||
((backendId == DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019 || backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH) && haveInfEngine());
}
@ -330,6 +332,17 @@ public:
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand = node->operand;
auto& webnnGraphBuilder = node->net->builder;
const std::vector<int32_t> out(outShapes[0].begin(), outShapes[0].end());
auto operand = webnnGraphBuilder.Reshape(webnnInpOperand, out.data(), out.size());
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
#endif
#ifdef HAVE_CUDA
Ptr<BackendNode> initCUDA(

59
modules/dnn/src/layers/scale_layer.cpp
View File

@ -15,6 +15,7 @@ Implementation of Scale layer.
#include "../op_halide.hpp"
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_webnn.hpp"
#include <opencv2/imgproc.hpp>
#include <opencv2/dnn/shape_utils.hpp>
@ -32,6 +33,10 @@ namespace dnn
class ScaleLayerImpl CV_FINAL : public ScaleLayer
{
public:
#ifdef HAVE_WEBNN
mutable int dims;
mutable int numChannels;
#endif
ScaleLayerImpl(const LayerParams& params)
{
setParamsFrom(params);
@ -47,6 +52,15 @@ public:
std::vector<MatShape> &internals) const CV_OVERRIDE
{
outputs.assign(1, inputs[0]);
#ifdef HAVE_WEBNN
dims = inputs[0].size();
numChannels = 1;
if (inputs.size() > 1)
{
for (const size_t& dim : inputs[1])
numChannels *= dim;
}
#endif
return true;
}
@ -68,7 +82,8 @@ public:
backendId == DNN_BACKEND_CUDA ||
backendId == DNN_BACKEND_HALIDE ||
(backendId == DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019 && axis == 1 && !blobs.empty()) ||
(backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH && axis > 0);
(backendId == DNN_BACKEND_INFERENCE_ENGINE_NGRAPH && axis > 0) ||
(backendId == DNN_BACKEND_WEBNN && axis >0);
}
template<typename T>
@ -375,6 +390,48 @@ public:
}
#endif // HAVE_DNN_NGRAPH
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand0 = node->operand;
auto& webnnGraphBuilder = node->net->builder;
auto webnnInpOperand1 = nodes.size() > 1 ? nodes[1].dynamicCast<WebnnBackendNode>()->operand : nullptr;
auto webnnInpOperand2 = nodes.size() > 2 ? nodes[1].dynamicCast<WebnnBackendNode>()->operand : nullptr;
std::vector<int32_t> shape(dims, 1);
size_t channels = 1;
if (blobs.empty())
channels = numChannels;
else
channels = blobs[0].total();
int cAxis = normalize_axis(axis, shape.size());
shape[cAxis] = channels;
ml::Operand operand = webnnInpOperand0;
if (hasWeights)
{
ml::Operand webnnWeights = blobs.empty() ? webnnInpOperand1 : webnn::BuildConstant(webnnGraphBuilder, webnn::getShape(blobs[0]), blobs[0].data, blobs[0].total()*blobs[0].elemSize(), ml::OperandType::Float32);
webnnWeights = webnnGraphBuilder.Reshape(webnnWeights, shape.data(), shape.size());
operand = webnnGraphBuilder.Mul(operand, webnnWeights);
}
if (hasBias)
{
ml::Operand webnnBias;
if(!hasWeights)
webnnBias = blobs.empty() ? webnnInpOperand1 : webnn::BuildConstant(webnnGraphBuilder, webnn::getShape(blobs.back()), blobs.back().data, blobs.back().total()*blobs.back().elemSize(), ml::OperandType::Float32);
else
webnnBias = blobs.empty() ? webnnInpOperand2 : webnn::BuildConstant(webnnGraphBuilder, webnn::getShape(blobs.back()), blobs.back().data, blobs.back().total()*blobs.back().elemSize(), ml::OperandType::Float32);
webnnBias = webnnGraphBuilder.Reshape(webnnBias, shape.data(), shape.size());
operand = webnnGraphBuilder.Add(operand, webnnBias);
}
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
#endif
void getScaleShift(Mat& scale, Mat& shift) const CV_OVERRIDE
{
scale = (hasWeights && !blobs.empty()) ? blobs[0] : Mat();

24
modules/dnn/src/layers/softmax_layer.cpp
View File

@ -47,9 +47,11 @@
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
#include "../op_vkcom.hpp"
#include "../op_webnn.hpp"
#include <algorithm>
#include <stdlib.h>
#include <opencv2/core/utils/logger.hpp>
using std::max;
#ifdef HAVE_OPENCL
@ -97,6 +99,16 @@ public:
virtual bool supportBackend(int backendId) CV_OVERRIDE
{
#ifdef HAVE_WEBNN
if (backendId == DNN_BACKEND_WEBNN) {
// TODO: support logSoftMax
if (logSoftMax)
{
CV_LOG_WARNING(NULL, "logSoftMax is not supported by WebNN backend.")
}
return !logSoftMax;
}
#endif
return backendId == DNN_BACKEND_OPENCV ||
backendId == DNN_BACKEND_CUDA ||
(backendId == DNN_BACKEND_HALIDE && haveHalide() && axisRaw == 1) ||
@ -390,6 +402,18 @@ public:
return true;
}
#ifdef HAVE_WEBNN
virtual Ptr<BackendNode> initWebnn(const std::vector<Ptr<BackendWrapper> >& inputs, const std::vector<Ptr<BackendNode> >& nodes) CV_OVERRIDE
{
Ptr<WebnnBackendNode> node = nodes[0].dynamicCast<WebnnBackendNode>();
auto& webnnInpOperand = node->operand;
auto& webnnGraphBuilder = node->net->builder;
auto operand = webnnGraphBuilder.Softmax(webnnInpOperand);
return Ptr<BackendNode>(new WebnnBackendNode(operand));
}
#endif
int64 getFLOPS(const std::vector<MatShape> &inputs,
const std::vector<MatShape> &outputs) const CV_OVERRIDE
{

249
modules/dnn/src/op_webnn.cpp Normal file
View File

@ -0,0 +1,249 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
#include <fstream>
#include "op_webnn.hpp"
#include <opencv2/core/utils/configuration.private.hpp>
#include <opencv2/core/utils/logger.hpp>
#include "opencv2/core/utils/filesystem.hpp"
#include "opencv2/core/utils/filesystem.private.hpp"
#include <opencv2/dnn/shape_utils.hpp>
namespace cv { namespace dnn {
#ifdef HAVE_WEBNN
namespace webnn {
ml::Operand BuildConstant(const ml::GraphBuilder& builder,
const std::vector<int32_t>& dimensions,
const void* value,
size_t size,
ml::OperandType type) {
ml::OperandDescriptor desc;
desc.type = type;
desc.dimensions = dimensions.data();
desc.dimensionsCount = (uint32_t)dimensions.size();
ml::ArrayBufferView resource;
resource.buffer = const_cast<void*>(value);
resource.byteLength = size;
return builder.Constant(&desc, &resource);
}
}
static std::string kDefaultInpLayerName = "opencv_webnn_empty_inp_layer_name";
static std::vector<Ptr<WebnnBackendWrapper> >
webnnWrappers(const std::vector<Ptr<BackendWrapper> >& ptrs)
{
std::vector<Ptr<WebnnBackendWrapper> > wrappers(ptrs.size());
for (int i = 0; i < ptrs.size(); ++i)
{
CV_Assert(!ptrs[i].empty());
wrappers[i] = ptrs[i].dynamicCast<WebnnBackendWrapper>();
CV_Assert(!wrappers[i].empty());
}
return wrappers;
}
// WebnnNet
WebnnNet::WebnnNet()
{
hasNetOwner = false;
device_name = "CPU";
#ifdef __EMSCRIPTEN__
context = ml::Context(emscripten_webnn_create_context());
#else
WebnnProcTable backendProcs = webnn_native::GetProcs();
webnnProcSetProcs(&backendProcs);
context = ml::Context(webnn_native::CreateContext());
#endif
builder = ::ml::CreateGraphBuilder(context);
namedOperands = ::ml::CreateNamedOperands();
}
void WebnnNet::addOutput(const std::string& name)
{
requestedOutputs.push_back(name);
}
void WebnnNet::createNet(Target targetId) {
init(targetId);
}
void WebnnNet::init(Target targetId)
{
switch (targetId)
{
case DNN_TARGET_CPU:
device_name = "CPU";
break;
case DNN_TARGET_OPENCL:
device_name = "GPU";
break;
default:
CV_Error(Error::StsNotImplemented, "Unknown target");
};
graph = builder.Build(namedOperands);
CV_Assert(graph!=nullptr);
isInit = true;
}
std::vector<ml::Operand> WebnnNet::setInputs(const std::vector<cv::Mat>& inputs,
const std::vector<std::string>& names) {
CV_Assert_N(inputs.size() == names.size());
std::vector<ml::Operand> current_inp;
for (size_t i = 0; i < inputs.size(); i++)
{
auto& m = inputs[i];
std::vector<int32_t> dimensions = webnn::getShape(m);
ml::OperandDescriptor descriptor;
descriptor.dimensions = dimensions.data();
descriptor.dimensionsCount = dimensions.size();
if (m.type() == CV_32F)
{
descriptor.type = ml::OperandType::Float32;
}
else
{
CV_Error(Error::StsNotImplemented, format("Unsupported data type %s", typeToString(m.type()).c_str()));
}
ml::Operand inputOperand = builder.Input(names[i].c_str(), &descriptor);
current_inp.push_back(std::move(inputOperand));
}
inputNames = names;
return current_inp;
}
void WebnnNet::setUnconnectedNodes(Ptr<WebnnBackendNode>& node) {
outputNames.push_back(node->name);
namedOperands.Set(outputNames.back().c_str(), node->operand);
}
bool WebnnNet::isInitialized()
{
return isInit;
}
void WebnnNet::reset()
{
allBlobs.clear();
isInit = false;
}
void WebnnNet::addBlobs(const std::vector<cv::Ptr<BackendWrapper> >& ptrs)
{
auto wrappers = webnnWrappers(ptrs);
for (const auto& wrapper : wrappers)
{
std::string name = wrapper->name;
name = name.empty() ? kDefaultInpLayerName : name;
allBlobs.insert({name, wrapper});
}
}
void WebnnNet::forward(const std::vector<Ptr<BackendWrapper> >& outBlobsWrappers, bool isAsync)
{
CV_LOG_DEBUG(NULL, "WebnnNet::forward(" << (isAsync ? "async" : "sync") << ")");
ml::NamedInputs named_inputs = ::ml::CreateNamedInputs();
std::vector<ml::Input> inputs(inputNames.size());
for (int i = 0; i < inputNames.size(); ++i) {
const std::string& name = inputNames[i];
ml::Input& input = inputs[i];
auto blobIt = allBlobs.find(name);
CV_Assert(blobIt != allBlobs.end());
const Ptr<WebnnBackendWrapper> wrapper = blobIt->second;
input.resource.buffer = wrapper->host->data;
input.resource.byteLength = wrapper->size;
named_inputs.Set(name.c_str(), &input);
}
std::vector<Ptr<WebnnBackendWrapper> > outs = webnnWrappers(outBlobsWrappers);
ml::NamedOutputs named_outputs = ::ml::CreateNamedOutputs();
std::vector<ml::ArrayBufferView> outputs(outs.size());
for (int i = 0; i < outs.size(); ++i) {
const std::string& name = outs[i]->name;
ml::ArrayBufferView& output = outputs[i];
output.buffer = outs[i]->host->data;
// std::cout<<"host data size: "<<outs[i]->host->total()*outs[i]->host->elemSize()<<std::endl;
output.byteLength = outs[i]->size;
// std::cout<<"outs[i]->size: "<< outs[i]->size << std::endl;
named_outputs.Set(name.c_str(), &output);
}
ml::ComputeGraphStatus status = graph.Compute(named_inputs, named_outputs);
if (status != ::ml::ComputeGraphStatus::Success) {
CV_Error(Error::StsAssert, format("Failed to compute: %d", int(status)));
}
}
// WebnnBackendNode
WebnnBackendNode::WebnnBackendNode(ml::Operand&& _operand)
: BackendNode(DNN_BACKEND_WEBNN), operand(std::move(_operand)) {}
WebnnBackendNode::WebnnBackendNode(ml::Operand& _operand)
: BackendNode(DNN_BACKEND_WEBNN), operand(_operand) {}
// WebnnBackendWrapper
WebnnBackendWrapper::WebnnBackendWrapper(int targetId, cv::Mat& m)
: BackendWrapper(DNN_BACKEND_WEBNN, targetId)
{
size = m.total() * m.elemSize();
// buffer.reset(new char[size]);
// std::memcpy(buffer.get(), m.data, size);
// dimensions = getShape<int32_t>(m);
// descriptor.dimensions = dimensions.data();
// descriptor.dimensionsCount = dimensions.size();
if (m.type() == CV_32F)
{
descriptor.type = ml::OperandType::Float32;
}
else
{
CV_Error(Error::StsNotImplemented, format("Unsupported data type %s", typeToString(m.type()).c_str()));
}
host = &m;
}
WebnnBackendWrapper::~WebnnBackendWrapper()
{
// nothing
}
void WebnnBackendWrapper::copyToHost()
{
CV_LOG_DEBUG(NULL, "WebnnBackendWrapper::copyToHost()");
//CV_Error(Error::StsNotImplemented, "");
}
void WebnnBackendWrapper::setHostDirty()
{
CV_LOG_DEBUG(NULL, "WebnnBackendWrapper::setHostDirty()");
//CV_Error(Error::StsNotImplemented, "");
}
void forwardWebnn(const std::vector<Ptr<BackendWrapper> >& outBlobsWrappers,
Ptr<BackendNode>& node, bool isAsync)
{
CV_Assert(!node.empty());
Ptr<WebnnBackendNode> webnnNode = node.dynamicCast<WebnnBackendNode>();
CV_Assert(!webnnNode.empty());
webnnNode->net->forward(outBlobsWrappers, isAsync);
}
#else
void forwardWebnn(const std::vector<Ptr<BackendWrapper> >& outBlobsWrappers,
Ptr<BackendNode>& operand, bool isAsync)
{
CV_Assert(false && "WebNN is not enabled in this OpenCV build");
}
#endif
}
}

171
modules/dnn/src/op_webnn.hpp Normal file
View File

@ -0,0 +1,171 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
#ifndef __OPENCV_DNN_OP_WEBNN_HPP__
#define __OPENCV_DNN_OP_WEBNN_HPP__
#include "opencv2/core/cvdef.h"
#include "opencv2/core/cvstd.hpp"
#include "opencv2/dnn.hpp"
#ifdef HAVE_WEBNN
#include <webnn/webnn_cpp.h>
#include <webnn/webnn.h>
#ifdef __EMSCRIPTEN__
#include <emscripten.h>
#include <emscripten/html5.h>
#include <emscripten/html5_webnn.h>
#else
#include <webnn/webnn_proc.h>
#include <webnn_native/WebnnNative.h>
#endif
#include <unordered_map>
#include <unordered_set>
#endif // HAVE_WEBNN
namespace cv { namespace dnn {
constexpr bool haveWebnn() {
#ifdef HAVE_WEBNN
return true;
#else
return false;
#endif
}
#ifdef HAVE_WEBNN
class WebnnBackendNode;
class WebnnBackendWrapper;
namespace webnn {
inline std::vector<int32_t> getShape(const Mat& mat)
{
std::vector<int32_t> result(mat.dims);
for (int i = 0; i < mat.dims; i++)
result[i] = (int32_t)mat.size[i];
return result;
}
ml::Operand BuildConstant(const ml::GraphBuilder& builder,
const std::vector<int32_t>& dimensions,
const void* value,
size_t size,
ml::OperandType type);
struct Pool2dOptions {
public:
std::vector<int32_t> windowDimensions;
std::vector<int32_t> padding;
std::vector<int32_t> strides;
std::vector<int32_t> dilations;
ml::AutoPad autoPad = ml::AutoPad::Explicit;
ml::InputOperandLayout layout = ml::InputOperandLayout::Nchw;
const ml::Pool2dOptions* AsPtr() {
if (!windowDimensions.empty()) {
mOptions.windowDimensionsCount = windowDimensions.size();
mOptions.windowDimensions = windowDimensions.data();
}
if (!padding.empty()) {
mOptions.paddingCount = padding.size();
mOptions.padding = padding.data();
}
if (!strides.empty()) {
mOptions.stridesCount = strides.size();
mOptions.strides = strides.data();
}
if (!dilations.empty()) {
mOptions.dilationsCount = dilations.size();
mOptions.dilations = dilations.data();
}
mOptions.layout = layout;
mOptions.autoPad = autoPad;
return &mOptions;
}
private:
ml::Pool2dOptions mOptions;
};
}
class WebnnNet
{
public:
WebnnNet();
void addOutput(const std::string& name);
bool isInitialized();
void init(Target targetId);
void forward(const std::vector<Ptr<BackendWrapper> >& outBlobsWrappers, bool isAsync);
std::vector<ml::Operand> setInputs(const std::vector<cv::Mat>& inputs, const std::vector<std::string>& names);
void setUnconnectedNodes(Ptr<WebnnBackendNode>& node);
void addBlobs(const std::vector<cv::Ptr<BackendWrapper> >& ptrs);
void createNet(Target targetId);
// void setNodePtr(std::shared_ptr<ngraph::Node>* ptr);
void reset();
ml::GraphBuilder builder;
ml::Context context;
ml::Graph graph;
std::unordered_map<std::string, cv::Ptr<WebnnBackendWrapper>> allBlobs;
bool hasNetOwner;
std::string device_name;
bool isInit = false;
std::vector<std::string> requestedOutputs;
std::vector<std::string> inputNames;
std::vector<std::string> outputNames;
ml::NamedOperands namedOperands;
};
class WebnnBackendNode : public BackendNode
{
public:
WebnnBackendNode(ml::Operand&& operand);
WebnnBackendNode(ml::Operand& operand);
std::string name;
ml::Operand operand;
Ptr<WebnnNet> net;
};
class WebnnBackendWrapper : public BackendWrapper
{
public:
WebnnBackendWrapper(int targetId, Mat& m);
~WebnnBackendWrapper();
virtual void copyToHost() CV_OVERRIDE;
virtual void setHostDirty() CV_OVERRIDE;
std::string name;
Mat* host;
std::unique_ptr<char> buffer;
size_t size;
std::vector<int32_t> dimensions;
ml::OperandDescriptor descriptor;
};
#endif // HAVE_WebNN
void forwardWebnn(const std::vector<Ptr<BackendWrapper> >& outBlobsWrappers,
Ptr<BackendNode>& node, bool isAsync);
}} // namespace cv::dnn
#endif // __OPENCV_DNN_OP_WEBNN_HPP__

11
modules/dnn/src/webnn/README.md Normal file
View File

@ -0,0 +1,11 @@
## Build Instructions
### Build WebNN-native and set the environment variable
Refer to [WebNN's build instructions](https://github.com/webmachinelearning/webnn-native) to complete the build of WebNN-native.
Set environment variable `WEBNN_NATIVE_DIR` to enable native DNN_BACKEND_WEBNN build: `export WEBNN_NATIVE_DIR=${PATH_TO_WebNN}`. Please let `WEBNN_NATIVE_DIR` points the output directory of webnn-native build (e.g. webnn-native/out/Release).
### Test native DNN_BACKEND_WEBNN backend
Add -DWITH_WEBNN=ON to the cmake command to build the WebNN module such as:
`cmake -DWITH_WEBNN=ON ../opencv` (according to the @ref tutorial_linux_install)

3
modules/dnn/test/test_common.hpp
View File

@ -135,7 +135,8 @@ testing::internal::ParamGenerator< tuple<Backend, Target> > dnnBackendsAndTarget
bool withCpuOCV = true,
bool withVkCom = true,
bool withCUDA = true,
bool withNgraph = true
bool withNgraph = true,
bool withWebnn = true
);
testing::internal::ParamGenerator< tuple<Backend, Target> > dnnBackendsAndTargetsIE();

15
modules/dnn/test/test_common.impl.hpp
View File

@ -29,6 +29,7 @@ void PrintTo(const cv::dnn::Backend& v, std::ostream* os)
case DNN_BACKEND_CUDA: *os << "CUDA"; return;
case DNN_BACKEND_INFERENCE_ENGINE_NN_BUILDER_2019: *os << "DLIE"; return;
case DNN_BACKEND_INFERENCE_ENGINE_NGRAPH: *os << "NGRAPH"; return;
case DNN_BACKEND_WEBNN: *os << "WEBNN"; return;
} // don't use "default:" to emit compiler warnings
*os << "DNN_BACKEND_UNKNOWN(" << (int)v << ")";
}
@ -247,7 +248,8 @@ testing::internal::ParamGenerator< tuple<Backend, Target> > dnnBackendsAndTarget
bool withCpuOCV /*= true*/,
bool withVkCom /*= true*/,
bool withCUDA /*= true*/,
bool withNgraph /*= true*/
bool withNgraph /*= true*/,
bool withWebnn /*= false*/
)
{
#ifdef HAVE_INF_ENGINE
@ -302,6 +304,17 @@ testing::internal::ParamGenerator< tuple<Backend, Target> > dnnBackendsAndTarget
}
#endif
#ifdef HAVE_WEBNN
if (withWebnn)
{
for (auto target : getAvailableTargets(DNN_BACKEND_WEBNN)) {
targets.push_back(make_tuple(DNN_BACKEND_WEBNN, target));
}
}
#else
CV_UNUSED(withWebnn);
#endif
{
available = getAvailableTargets(DNN_BACKEND_OPENCV);
for (std::vector< Target >::const_iterator i = available.begin(); i != available.end(); ++i)

15
platforms/js/build_js.py
View File

@ -67,6 +67,8 @@ class Builder:
self.options = options
self.build_dir = check_dir(options.build_dir, create=True)
self.opencv_dir = check_dir(options.opencv_dir)
print('-----------------------------------------------------------')
print('options.opencv_dir:', options.opencv_dir)
self.emscripten_dir = check_dir(options.emscripten_dir)
def get_toolchain_file(self):
@ -84,6 +86,7 @@ class Builder:
"-DCMAKE_BUILD_TYPE=Release",
"-DCMAKE_TOOLCHAIN_FILE='%s'" % self.get_toolchain_file(),
"-DCPU_BASELINE=''",
"-DCMAKE_INSTALL_PREFIX=/usr/local",
"-DCPU_DISPATCH=''",
"-DCV_TRACE=OFF",
"-DBUILD_SHARED_LIBS=OFF",
@ -136,10 +139,10 @@ class Builder:
"-DBUILD_opencv_js=ON",
"-DBUILD_opencv_python2=OFF",
"-DBUILD_opencv_python3=OFF",
"-DBUILD_EXAMPLES=OFF",
"-DBUILD_EXAMPLES=ON",
"-DBUILD_PACKAGE=OFF",
"-DBUILD_TESTS=OFF",
"-DBUILD_PERF_TESTS=OFF"]
"-DBUILD_TESTS=ON",
"-DBUILD_PERF_TESTS=ON"]
if self.options.cmake_option:
cmd += self.options.cmake_option
if self.options.build_doc:
@ -162,6 +165,9 @@ class Builder:
else:
cmd.append("-DBUILD_WASM_INTRIN_TESTS=OFF")
if self.options.webnn:
cmd.append("-DWITH_WEBNN=ON")
flags = self.get_build_flags()
if flags:
cmd += ["-DCMAKE_C_FLAGS='%s'" % flags,
@ -184,6 +190,8 @@ class Builder:
flags += "-msimd128 "
if self.options.build_flags:
flags += self.options.build_flags
if self.options.webnn:
flags += "-s USE_WEBNN=1 "
return flags
def config(self):
@ -243,6 +251,7 @@ if __name__ == "__main__":
# Write a path to modify file like argument of this flag
parser.add_argument('--config', default=os.path.join(os.path.dirname(os.path.abspath(__file__)), 'opencv_js.config.py'),
help="Specify configuration file with own list of exported into JS functions")
parser.add_argument('--webnn', action="store_true", help="Enable WebNN Backend")
args = parser.parse_args()

2
platforms/js/opencv_js.config.py
View File

@ -129,7 +129,7 @@ video = {
'TrackerMIL_Params': [],
}
dnn = {'dnn_Net': ['setInput', 'forward'],
dnn = {'dnn_Net': ['setInput', 'forward', 'setPreferableBackend'],
'': ['readNetFromCaffe', 'readNetFromTensorflow', 'readNetFromTorch', 'readNetFromDarknet',
'readNetFromONNX', 'readNet', 'blobFromImage']}

70
samples/dnn/classification.cpp
View File

@ -1,5 +1,6 @@
#include <fstream>
#include <sstream>
#include <iostream>
#include <opencv2/dnn.hpp>
#include <opencv2/imgproc.hpp>
@ -17,6 +18,7 @@ std::string keys =
"{ std | 0.0 0.0 0.0 | Preprocess input image by dividing on a standard deviation.}"
"{ crop | false | Preprocess input image by center cropping.}"
"{ framework f | | Optional name of an origin framework of the model. Detect it automatically if it does not set. }"
"{ needSoftmax | false | Use Softmax to post-process the output of the net.}"
"{ classes | | Optional path to a text file with names of classes. }"
"{ backend | 0 | Choose one of computation backends: "
"0: automatically (by default), "
@ -24,7 +26,8 @@ std::string keys =
"2: Intel's Deep Learning Inference Engine (https://software.intel.com/openvino-toolkit), "
"3: OpenCV implementation, "
"4: VKCOM, "
"5: CUDA },"
"5: CUDA, "
"6: WebNN }"
"{ target | 0 | Choose one of target computation devices: "
"0: CPU target (by default), "
"1: OpenCL, "
@ -70,6 +73,9 @@ int main(int argc, char** argv)
String framework = parser.get<String>("framework");
int backendId = parser.get<int>("backend");
int targetId = parser.get<int>("target");
bool needSoftmax = parser.get<bool>("needSoftmax");
std::cout<<"mean: "<<mean<<std::endl;
std::cout<<"std: "<<std<<std::endl;
// Open file with classes names.
if (parser.has("classes"))
@ -141,28 +147,62 @@ int main(int argc, char** argv)
net.setInput(blob);
//! [Set input blob]
//! [Make forward pass]
Mat prob = net.forward();
//! [Make forward pass]
//! [Get a class with a highest score]
Point classIdPoint;
// double t_sum = 0.0;
// double t;
int classId;
double confidence;
minMaxLoc(prob.reshape(1, 1), 0, &confidence, 0, &classIdPoint);
int classId = classIdPoint.x;
//! [Get a class with a highest score]
cv::TickMeter timeRecorder;
timeRecorder.reset();
Mat prob = net.forward();
double t1;
timeRecorder.start();
prob = net.forward();
timeRecorder.stop();
t1 = timeRecorder.getTimeMilli();
// Put efficiency information.
std::vector<double> layersTimes;
double freq = getTickFrequency() / 1000;
double t = net.getPerfProfile(layersTimes) / freq;
std::string label = format("Inference time: %.2f ms", t);
timeRecorder.reset();
for(int i = 0; i < 200; i++) {
//! [Make forward pass]
timeRecorder.start();
prob = net.forward();
timeRecorder.stop();
//! [Get a class with a highest score]
Point classIdPoint;
minMaxLoc(prob.reshape(1, 1), 0, &confidence, 0, &classIdPoint);
classId = classIdPoint.x;
//! [Get a class with a highest score]
// Put efficiency information.
// std::vector<double> layersTimes;
// double freq = getTickFrequency() / 1000;
// t = net.getPerfProfile(layersTimes) / freq;
// t_sum += t;
}
if (needSoftmax == true)
{
float maxProb = 0.0;
float sum = 0.0;
Mat softmaxProb;
maxProb = *std::max_element(prob.begin<float>(), prob.end<float>());
cv::exp(prob-maxProb, softmaxProb);
sum = (float)cv::sum(softmaxProb)[0];
softmaxProb /= sum;
Point classIdPoint;
minMaxLoc(softmaxProb.reshape(1, 1), 0, &confidence, 0, &classIdPoint);
classId = classIdPoint.x;
}
std::string label = format("Inference time of 1 round: %.2f ms", t1);
std::string label2 = format("Average time of 200 rounds: %.2f ms", timeRecorder.getTimeMilli()/200);
putText(frame, label, Point(0, 15), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0));
putText(frame, label2, Point(0, 35), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0));
// Print predicted class.
label = format("%s: %.4f", (classes.empty() ? format("Class #%d", classId).c_str() :
classes[classId].c_str()),
confidence);
putText(frame, label, Point(0, 40), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0));
putText(frame, label, Point(0, 55), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0));
imshow(kWinName, frame);
}