AI实战营第二期 第十节 《MMagic 代码课》——条记11

AI实战营第二期 第十节 《MMagic 代码课》

MMagic (Multimodal Advanced, Generative, and Intelligent Creation) 是一个供专业人工智能研究职员和机器学习工程师去处置惩罚、编辑和天生图像与视频的开源 AIGC 工具箱。

MMagic 允许研究职员和工程师使用开始进的预训练模型，而且可以轻松训练和开发新的定制模型。
MMagic 支持各种根本天生模型，包罗：

• 无条件天生对抗网络 (GANs)
  
• 条件天生对抗网络 (GANs)
  
• 内部学习
  
• 扩散模型
  
• 尚有许多其他天生模型即将推出！
  

MMagic 支持各种应用步调，包罗：

• 图文天生
  
• 图像翻译
  
• 3D 天生
  
• 图像超分辨率
  
• 视频超分辨率
  
• 视频插帧
  
• 图像补全
  
• 图像抠图
  
• 图像修复
  
• 图像上色
  
• 图像天生
  
• 尚有许多其他应用步调即将推出！
  

【课程链接】https://www.bilibili.com/video/BV1gM4y1n7vP/
【讲师先容】张子豪 OpenMMLab算法工程师
OpenMMLab 天生模型+底层视觉+AIGC+多模态 算法库 MMagic
MMagic主页：https://github.com/open-mmlab/mmagic
代码教程：https://github.com/TommyZihao/MMagic_Tutorials
中文文档：https://mmagic.readthedocs.io/zh_CN/latest/
【代码教程目次】
安装设置MMagic环境
优劣老照片上色
文生图-Stable Diffusion
文生图-Dreambooth
图生图-ControlNet
安装设置MMagic

安装Pytorch

1. !pip3 install install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

复制代码

安装MMCV和MMEngine环境

[2]

1. !pip3 install openmim
2. !mim install 'mmcv>=2.0.0'
3. !mim install 'mmengine'

复制代码

安装MMagic

1. !mim install 'mmagic'
2. 从源码安装MMagic
3. !rm -rf mmagic # 删除原有的 mmagic 文件夹（如有）
4. !git clone https://github.com/open-mmlab/mmagic.git # 下载 mmagic 源代码

复制代码

1. import os
2. os.chdir('mmagic')
3. !pip3 install -e .

复制代码

查抄安装乐成

1. # 检查 Pytorch
2. import torch, torchvision
3. print('Pytorch 版本', torch.__version__)
4. print('CUDA 是否可用',torch.cuda.is_available())

复制代码

[
查抄 mmcv

1. import mmcv
2. from mmcv.ops import get_compiling_cuda_version, get_compiler_version
3. print('MMCV版本', mmcv.__version__)
4. print('CUDA版本', get_compiling_cuda_version())
5. print('编译器版本', get_compiler_version())

复制代码

查抄 mmagic

1. import mmagic
2. print('MMagic版本', mmagic.__version__)
3. MMagic版本 1.0.2dev0

复制代码

安装别的工具包

1. !pip install opencv-python pillow matplotlib seaborn tqdm -i https://pypi.tuna.tsinghua.edu.cn/simple
2. !pip install clip transformers gradio 'httpx[socks]' diffusers==0.14.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
3. !mim install 'mmdet>=3.0.0'

复制代码

优劣照片上色

进入 MMagic 主目次

1. import os
2. os.chdir('mmagic')

复制代码

下载样例图片

[2]

1. !wget https://zihao-openmmlab.obs.cn-east-3.myhuaweicloud.com/20230613-MMagic/data/test_colorization.jpg -O test_colorization.jpg

复制代码

运行推测

[3]
!python demo/mmagic_inference_demo.py
–model-name inst_colorization
–img test_colorization.jpg
–result-out-dir out_colorization.png
文生图-Stable Diffusion

1. from mmagic.apis import MMagicInferencer
2. sd_inferencer = MMagicInferencer(model_name='stable_diffusion')
3. text_prompts = 'A panda is having dinner at KFC'
5. text_prompts = 'A Persian cat walking in the streets of New York'
7. sd_inferencer.infer(text=text_prompts, low_cpu_mem_usage=True,result_out_dir='output/sd_res.png')

复制代码

文生图-Dreambooth

新建文件夹data/dreambooth/imgs/

修改config/dreambooth/文件夹中的dreambooth-lora.py脚本

1. dataset = dict(
2.     type='DreamBoothDataset',
3.     data_root='./data/dreambooth',
4.     # TODO: rename to instance
5.     concept_dir='imgs',
6.     prompt='a photo of gril',
7.     pipeline=pipeline)

复制代码

然后实行下令：

1. !bash tools/dist_train.sh configs/dreambooth/dreambooth-lora.py 1

复制代码

用训练好的模型做推测

1. from mmengine import Config
3. from mmagic.registry import MODELS
4. from mmagic.utils import register_all_modules
6. register_all_modules()
7. cfg = Config.fromfile('./mmagic/configs/dreambooth/dreambooth-lora.py')
8. dreambooth_lora = MODELS.build(cfg.model)
10. state = torch.load('mmagic/work_dirs/dreambooth-lora/iter_1000.pth')['state_dict']
12. def convert_state_dict(state):
13.     state_dict_new = {}
14.     for k, v in state.items():
15.         if '.module' in k:
16.             k_new = k.replace('.module', '')
17.         else:
18.             k_new = k
19.         if 'vae' in k:
20.             if 'to_q' in k:
21.                 k_new = k.replace('to_q', 'query')
22.             elif 'to_k' in k:
23.                 k_new = k.replace('to_k', 'key')
24.             elif 'to_v' in k:
25.                 k_new = k.replace('to_v', 'value')
26.             elif 'to_out' in k:
27.                 k_new = k.replace('to_out.0', 'proj_attn')
28.         state_dict_new[k_new] = v
29.     return state_dict_new
30. dreambooth_lora.load_state_dict(convert_state_dict(state))
31. dreambooth_lora = dreambooth_lora.cuda()
32. samples = dreambooth_lora.infer('side view of gril', guidance_scale=5)
33. samples['samples'][0]

复制代码

图生图-ControlNet-Canny

导入工具包

1. import cv2
2. import numpy as np
3. import mmcv
4. from mmengine import Config
5. from PIL import Image
7. from mmagic.registry import MODELS
8. from mmagic.utils import register_all_modules
10. register_all_modules()

复制代码

载入ControlNet模型

1. cfg = Config.fromfile('configs/controlnet/controlnet-canny.py')
2. controlnet = MODELS.build(cfg.model).cuda()

复制代码

输入Canny边缘图

1. control_url = 'https://user-images.githubusercontent.com/28132635/230288866-99603172-04cb-47b3-8adb-d1aa532d1d2c.jpg'
2. control_img = mmcv.imread(control_url)
3. control = cv2.Canny(control_img, 100, 200)
4. control = control[:, :, None]
5. control = np.concatenate([control] * 3, axis=2)
6. control = Image.fromarray(control)

复制代码

咒语Prompt

1. prompt ='Room with blue walls and a yellow ceiling.'

复制代码

实行推测

1. output_dict = controlnet.infer(prompt, control=control)
2. samples = output_dict['samples']
3. for idx, sample in enumerate(samples):
4.     sample.save(f'sample_{idx}.png')
5. controls = output_dict['controls']
6. for idx, control in enumerate(controls):
7.     control.save(f'control_{idx}.png')

复制代码

图生图-ControlNet-Pose

1. import mmcv
2. from mmengine import Config
3. from PIL import Image
5. from mmagic.registry import MODELS
6. from mmagic.utils import register_all_modules
8. register_all_modules()

复制代码

载入ControlNet模型

1. cfg = Config.fromfile('configs/controlnet/controlnet-pose.py')
2. # convert ControlNet's weight from SD-v1.5 to Counterfeit-v2.5
3. cfg.model.unet.from_pretrained = 'gsdf/Counterfeit-V2.5'
4. cfg.model.vae.from_pretrained = 'gsdf/Counterfeit-V2.5'
5. cfg.model.init_cfg['type'] = 'convert_from_unet'
6. controlnet = MODELS.build(cfg.model).cuda()
7. # call init_weights manually to convert weight
8. controlnet.init_weights()

复制代码

咒语Prompt

1. prompt = 'masterpiece, best quality, sky, black hair, skirt, sailor collar, looking at viewer, short hair, building, bangs, neckerchief, long sleeves, cloudy sky, power lines, shirt, cityscape, pleated skirt, scenery, blunt bangs, city, night, black sailor collar, closed mouth'

复制代码

输入Pose图

1. control_url = 'https://user-images.githubusercontent.com/28132635/230380893-2eae68af-d610-4f7f-aa68-c2f22c2abf7e.png'
2. control_img = mmcv.imread(control_url)
3. control = Image.fromarray(control_img)
4. control.save('control.png')

复制代码

实行推测

1. output_dict = controlnet.infer(prompt, control=control, width=512, height=512, guidance_scale=7.5)
2. samples = output_dict['samples']
3. for idx, sample in enumerate(samples):
4.     sample.save(f'sample_{idx}.png')
5. controls = output_dict['controls']
6. for idx, control in enumerate(controls):
7.     control.save(f'control_{idx}.png')

复制代码

图生图-ControlNet Animation

方式一：Gradio下令行

1. !python demo/gradio_controlnet_animation.py

复制代码

点击URL，打开Gradio在线交互式网站，上传视频，实行推测
方式二：MMagic API

1. # 导入工具包
2. from mmagic.apis import MMagicInferencer
4. # Create a MMEdit instance and infer
5. editor = MMagicInferencer(model_name='controlnet_animation')
6. # 指定 prompt 咒语
7. prompt = 'a girl, black hair, T-shirt, smoking, best quality, extremely detailed'
8. negative_prompt = 'longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality'
10. # 待测视频
11. # https://user-images.githubusercontent.com/12782558/227418400-80ad9123-7f8e-4c1a-8e19-0892ebad2a4f.mp4
12. video = '../run_forrest_frames_rename_resized.mp4'
13. save_path = '../output_video.mp4'
14. # 执行预测
15. editor.infer(video=video, prompt=prompt, image_width=512, image_height=512, negative_prompt=negative_prompt, save_path=save_path)

复制代码

训练自己的ControlNet

下载数据集

1. !rm -rf fill50k.zip fill50k
2. !wget https://huggingface.co/lllyasviel/ControlNet/blob/main/training/fill50k.zip
3. !unzip fill50k.zip >> /dev/null # 解压压缩包
4. !rm -rf fill50k.zip # 删除压缩包

复制代码

训练

1. !bash tools/dist_train.sh configs/controlnet/controlnet-1xb1-demo_dataset 1

复制代码

1. from mmagic.apis import MMagicInferencerimport matplotlib.pyplot as pltsd_inferencer = MMagicInferencer(model_name='stable_diffusion')import cv2
2. import numpy as np
3. import mmcv
4. from mmengine import Config
5. from PIL import Image
7. from mmagic.registry import MODELS
8. from mmagic.utils import register_all_modules
10. register_all_modules()
11. cfg = Config.fromfile('configs/controlnet/controlnet-canny.py')
12. controlnet = MODELS.build(cfg.model).cuda()
13. control_img = mmcv.imread('11.JPG')control = cv2.Canny(control_img, 100, 200)control = control[:, :, None]control = np.concatenate([control] * 3, axis=2)control = Image.fromarray(control)plt.subplot(121)plt.imshow(control_img)plt.subplot(122)plt.imshow(control)plt.show()prompt = 'Make this room full of warmth.'output_dict = controlnet.infer(prompt, control=control)
14. samples = output_dict['samples']
15. for idx, sample in enumerate(samples):
16.     sample.save(f'sample_{idx}.png')
17. controls = output_dict['controls']
18. for idx, control in enumerate(controls):
19.     control.save(f'control_{idx}.png')
20. plt.subplot(121)plt.imshow(control_img)plt.subplot(122)sample_0 = mmcv.imread('./sample_0.png')plt.imshow(sample_0)plt.show()

复制代码

来源：https://blog.csdn.net/hhhhhhhhhhwwwwwwwwww/article/details/131251962
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！