Stable Diffusion之Scheduler模块比对天生结果

显示全部楼层 · 2023-8-16 16:32:45

项目场景：

更换Stable Diffusion的Scheduler模块并对结果进行分析

diffusers包含多个用于扩散过程的预置scheduler function，用于接收经过训练的模子的输出，扩散过程正在迭代的样本，以及返归去噪样本的时间步长。在其他扩散模子又被称为采样器。
Schedulers

Schedulers define the methodology for iteratively adding noise to an image or for updating a sample based on model outputs.

adding noise in different manners represent the algorithmic processes to train a diffusion model by adding noise to images.
for inference, the scheduler defines how to update a sample based on an output from a pretrained model.

Schedulers are often defined by a noise schedule and an update rule to solve the differential equation solution.
加载 Stable Diffusion pipeline

from huggingface_hub import login
from diffusers import DiffusionPipeline
import torch
# first we need to login with our access token
login()
# Now we can download the pipeline
pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
# Move it to GPU
pipeline.to("cuda")
# Access the scheduler
pipeline.scheduler

复制代码

StableDiffusion模子各组件默认设置
StableDiffusionPipeline {
“_class_name”: “StableDiffusionPipeline”,
“_diffusers_version”: “0.11.1”,
“feature_extractor”: [
“transformers”,
“CLIPFeatureExtractor”
],
“requires_safety_checker”: true,
“safety_checker”: [
“stable_diffusion”,
“StableDiffusionSafetyChecker”
],
“scheduler”: [
“diffusers”,
“PNDMScheduler”
],
“text_encoder”: [
“transformers”,
“CLIPTextModel”
],
“tokenizer”: [
“transformers”,
“CLIPTokenizer”
],
“unet”: [
“diffusers”,
“UNet2DConditionModel”
],
“vae”: [
“diffusers”,
“AutoencoderKL”
]
}

StableDiffusion的scheduler模块默认设置
PNDMScheduler {
“_class_name”: “PNDMScheduler”,
“_diffusers_version”: “0.11.1”,
“beta_end”: 0.012,
“beta_schedule”: “scaled_linear”,
“beta_start”: 0.00085,
“clip_sample”: false,
“num_train_timesteps”: 1000,
“prediction_type”: “epsilon”,
“set_alpha_to_one”: false,
“skip_prk_steps”: true,
“steps_offset”: 1,
“trained_betas”: null
}
实验思绪

我们可以看到StableDiffusion的调治器范例是PNDMScheduler。现在我想比力这个调治器与其他调治器的性能。起首，我们定义了一个prompt，我们将在这个prompt上测试全部差别的调治器。为包管天生图像具有高度相似性，先用random seed创建了一个天生器，它将确保我们可以天生类似的图像:

prompt = "A photograph of an astronaut riding a horse on Mars, high resolution, high definition."
generator = torch.Generator(device="cuda").manual_seed(8)
image = pipeline(prompt, generator=generator).images[0]
image

复制代码

每个调治器的属性schedulermix .compatibles 定义了全部兼容的调治器。检察Stable Diffusion管道的全部可用的、兼容的调治步伐。

pipeline.scheduler.compatibles

复制代码

[diffusers.schedulers.scheduling_euler_ancestral_discrete.EulerAncestralDiscreteScheduler,
diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler,
diffusers.schedulers.scheduling_ddim.DDIMScheduler,
diffusers.schedulers.scheduling_euler_discrete.EulerDiscreteScheduler,
diffusers.schedulers.scheduling_ddpm.DDPMScheduler,
diffusers.schedulers.scheduling_pndm.PNDMScheduler,
diffusers.schedulers.scheduling_dpmsolver_singlestep.DPMSolverSinglestepScheduler,
diffusers.schedulers.scheduling_heun_discrete.HeunDiscreteScheduler,
diffusers.schedulers.scheduling_lms_discrete.LMSDiscreteScheduler]
PNDMScheduler类的设置

pipeline.scheduler.config

复制代码

FrozenDict([(‘num_train_timesteps’, 1000),
(‘beta_start’, 0.00085),
(‘beta_end’, 0.012),
(‘beta_schedule’, ‘scaled_linear’),
(‘trained_betas’, None),
(‘skip_prk_steps’, True),
(‘set_alpha_to_one’, False),
(‘prediction_type’, ‘epsilon’),
(‘steps_offset’, 1),
(‘_class_name’, ‘PNDMScheduler’),
(‘_diffusers_version’, ‘0.11.1’),
(‘clip_sample’, False)])
利用此设置实例化一个与管道兼容的差别类的调治器。这里将调治步伐更改为DDIMScheduler。并重新运行pipeline比力天生质量

from diffusers import DDIMSchedulerpipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config
)generator = torch.Generator(device="cuda").manual_seed(8)image = pipeline(prompt, generator=generator).images[0]image

复制代码

换成接纳更少timestep的采样器LMSDiscreteScheduler

from diffusers import LMSDiscreteSchedulerpipeline.scheduler = LMSDiscreteScheduler.from_config(pipeline.scheduler.config
)generator = torch.Generator(device="cuda").manual_seed(8)image = pipeline(prompt, generator=generator).images[0]image

复制代码

DPMSolverMultistepScheduler提供了最好权衡天生速率/天生质量的Scheduler，可以运行只有20步。

from diffusers import DPMSolverMultistepSchedulerpipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config
)generator = torch.Generator(device="cuda").manual_seed(8)image = pipeline(prompt, generator=generator, num_inference_steps=20).images[0]image

复制代码

接纳差别的scheduler用于文本引导图像翻译结果：

大多数图像看起来非常相似，质量也非常相似，但DPMSolver所需天生时间最短，可在20个·inference_step天生结果。调治器的选择还要取决于详细的用例。

来源：https://blog.csdn.net/wangyunpeng33/article/details/130033116
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！