Commit Graph

1 Commits

Author SHA1 Message Date
Abduragim Shtanchaev
050085c996
Merge pull request #25950 from Abdurrahheem:ash/add-inpainting-sample
Diffusion Inpainting Sample #25950

This PR adds inpaiting sample that is based on [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/pdf/2112.10752) paper (reference github [repository](https://github.com/CompVis/latent-diffusion)).


Steps to run the model:

1. Firstly needs ONNX graph of the Latent Diffusion Model. You can get it in two different ways. 

> a. Generate the using this [repo](https://github.com/Abdurrahheem/latent-diffusion/tree/ash/export2onnx) and follow instructions below

```bash
git clone https://github.com/Abdurrahheem/latent-diffusion.git
cd latent-diffusion
conda env create -f environment.yaml
conda activate ldm
wget -O models/ldm/inpainting_big/last.ckpt https://heibox.uni-heidelberg.de/f/4d9ac7ea40c64582b7c9/?dl=1
python -m scripts.inpaint.py --indir data/inpainting_examples/ --outdir outputs/inpainting_results --export=True
```

> b. Download the ONNX graph (there 3 fiels) using this link: TODO make a link

2. Build opencv (preferebly with CUDA support enabled
3. Run the script 

```bash
cd opencv/samples/dnn
python ldm_inpainting.py 
python ldm_inpainting.py -e=<path-to-InpaintEncoder.onnx file> -d=<path-to-InpaintDecoder.onnx file> -df=<path-to-LatenDiffusion.onnx file> -i=<path-to-image>
```
Right after the last command you will be prompted with image. You can click on left mouse bottom and starting selection a region you would like to be inpainted (deleted). Once you finish marking the region, click on left mouse botton again and press esc button on your keyboard. The inpainting proccess will start. 

Note: If you are running it on CPU it might take a large chank of time. Also make sure to have about 15GB of RAM to make process faster (other wise swapping will click in and everything will be slower)
 
Current challenges: 

1. Diffusion process is slow (many layers fallback to CPU with running with CUDA backend) 
2. The diffusion result is does exactly mach that of the original torch pipeline

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x]The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
2024-08-21 14:48:37 +03:00