How to schedule LoRAs weight into your ComfyUI workflow to achieve a better composition control. This method allows applying different weights at different steps.

Requirements

• Updated ComfyUI: requires at least version 2890 released on December 5th, 2024
• ComfyUI Manager: Recommended to manage plugins.

Introduction

Whenever you add a lora to your workflow, the generated image composition may deviate from the original model. This is more noticeable with character loras, where instead of just adopting our character’s facial features and clothing, the model composition is poisoned by unexpected styles. A common example is when some character LoRA tends to convert all models to anime style because of the way it was trained.

The purple colored nodes here are those required for LoRA scheduling.

In this workflow, both LoRAs are gradually applied with increasing strength, and we also control when they are applied during the sampling process. This results in a smooth image composition retaining the original model features.

Workflow explained

Create Hook LoRA node

This node replaces the standard Load LoRA node. As you can see it does not require any model or clip input. Simply select your lora files, taking into account that its strength will not be increased by the scheduler node. You can apply multiple loras with the same scheduling by chaining them with the prev_hooks input.

Create Hook Keyframes Interp node

This node determines the LoRa’s weight at each sampling step. See the chart in this page to interpret its current values during the sampling process. Understanding this node is essential to use LoRA scheduling, and I tried my best to make its settings as clear as possible here.

Start applying LoRAs from the beginning with 0.000 strength and gradually increase it to 1.000 at 60% of the sampling process using 5 evenly-spaced keyframes.

• strength_start: the lora strength at the first keyframe to be created.
• strength_end: the lora strength at the last keyframe to be created.
• start_percent: when to start applying the lora keyframes, eg:
  • a value of 0 would start from the first step
  • a value of 0.5 would start at 50% of the sampling process
• end_percent: when to end the lora scheduling.
  • this value should be greater than start_percent.
  • the rest of the sampling process will continue to use the strength_end (if I understood the node’s behaviour; correct me otherwise in the comments section)
• keyframes_count: determines how many total keyframes to create, including the start and end keyframes.
• interpolation: determines how to scale the strengths in between; the start_percents of the keyframes will remain evenly spaced (linear) no matter the selection.
• print_keyframes: show the actual values assigned to the keyframes in the console.

Both Create Hook Model as LoRA and Create Hook LoRA nodes have an optional prev_hooks input – this can be used to chain multiple hooks, allowing to use multiple LoRAs and/or Model-as-LoRAs together, at whatever strengths you desire.

With our current values, the console has shown this during sampling:

Hook Keyframe - start_percent:0.0 = 0.0
Hook Keyframe - start_percent:0.15000000596046448 = 0.25
Hook Keyframe - start_percent:0.30000001192092896 = 0.5
Hook Keyframe - start_percent:0.45000001788139343 = 0.75
Hook Keyframe - start_percent:0.6000000238418579 = 1.0

Those values have been expanded into a 25-row data series using Chat GPT’s data analysis, representing the scheduled LoRAs weight at each step of the sampling process. Based on this data, Google Sheets has created the following chart showing exactly how the LoRA was applied in our sample workflow:

This chart shows a visual representation of the scheduled LoRAs weight in the sample workflow with our Create Hook Keyframes Interp node values. As you can see, instead of a a constant strength, it starts with 0.00 and it’s gradually increased to 1.0 around step 17. This ensures the LoRAs are gracefully applied as the model is forming the initial image composition. Fine tuning this curve depends on the model and LoRAs used, and the desired image composition.

Set CLIP Hooks node

When apply_to_conds is true (default), it means that the nodes that perform Text Encode will automatically attach the same hooks that were on the CLIP to the conditioning that comes out of them.

When schedule_clip is false, scheduling weights will ONLY apply to conditioning [model], not CLIP; instead, CLIP will only use the original strength_clip defined in the Create Hook LoRA/Model as LoRA nodes. When set to true, conditioning will be created for each scheduled CLIP weight; depending on how many keyframes you have, you will notice the CLIP Text Encode nodes take longer to run (and a progress bar will appear on the node).

Since those explanations from the ComfyUI blog are too technical to translate to plain English, here’s a visual comparison to help you assimilate its different settings:

The first image in that set has the best composition for our PDXL Kagome Higurashi LoRA.

Workflow comparison

Here are the results for comparing our workflow images with and without LoRA scheduling using different checkpoint models.

Perfect Pony XL V2 model comparison

Pony Diffusion V6 XL model comparison

Without scheduling, both models generated almost identical images, meaning that our LoRAs have managed to seriously interfere with each model’s styles. You can see how the scheduled LoRA workflows resulted in a more graceful image composition, retaining most of the original styles as showcased in each model’s civitai download page.