mirror of
https://github.com/opencv/opencv.git
synced 2024-11-24 11:10:21 +08:00
050085c996
Diffusion Inpainting Sample #25950 This PR adds inpaiting sample that is based on [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/pdf/2112.10752) paper (reference github [repository](https://github.com/CompVis/latent-diffusion)). Steps to run the model: 1. Firstly needs ONNX graph of the Latent Diffusion Model. You can get it in two different ways. > a. Generate the using this [repo](https://github.com/Abdurrahheem/latent-diffusion/tree/ash/export2onnx) and follow instructions below ```bash git clone https://github.com/Abdurrahheem/latent-diffusion.git cd latent-diffusion conda env create -f environment.yaml conda activate ldm wget -O models/ldm/inpainting_big/last.ckpt https://heibox.uni-heidelberg.de/f/4d9ac7ea40c64582b7c9/?dl=1 python -m scripts.inpaint.py --indir data/inpainting_examples/ --outdir outputs/inpainting_results --export=True ``` > b. Download the ONNX graph (there 3 fiels) using this link: TODO make a link 2. Build opencv (preferebly with CUDA support enabled 3. Run the script ```bash cd opencv/samples/dnn python ldm_inpainting.py python ldm_inpainting.py -e=<path-to-InpaintEncoder.onnx file> -d=<path-to-InpaintDecoder.onnx file> -df=<path-to-LatenDiffusion.onnx file> -i=<path-to-image> ``` Right after the last command you will be prompted with image. You can click on left mouse bottom and starting selection a region you would like to be inpainted (deleted). Once you finish marking the region, click on left mouse botton again and press esc button on your keyboard. The inpainting proccess will start. Note: If you are running it on CPU it might take a large chank of time. Also make sure to have about 15GB of RAM to make process faster (other wise swapping will click in and everything will be slower) Current challenges: 1. Diffusion process is slow (many layers fallback to CPU with running with CUDA backend) 2. The diffusion result is does exactly mach that of the original torch pipeline ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x]The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake |
||
|---|---|---|
| .. | ||
| android | ||
| cpp | ||
| data | ||
| directx | ||
| dnn | ||
| gdb | ||
| gpu | ||
| hal | ||
| install | ||
| java | ||
| opencl | ||
| opengl | ||
| python | ||
| semihosting | ||
| swift/ios | ||
| sycl | ||
| tapi | ||
| va_intel | ||
| winrt | ||
| winrt_universal | ||
| wp8 | ||
| _winpack_build_sample.cmd | ||
| _winpack_run_python_sample.cmd | ||
| CMakeLists.example.in | ||
| CMakeLists.txt | ||
| samples_utils.cmake | ||