ComfyUI ControlNet Guide: Precise Control Over AI Image Generation

What is ControlNet?

ControlNet is a neural network that adds structural control to AI image generation. Instead of relying on text prompts alone, you provide a reference image — an edge map, a depth map, a skeleton pose — and ControlNet ensures the output follows that structure.

Think of it as giving the AI a blueprint. The text prompt says what to draw, and ControlNet says how to arrange it.

Why Use ControlNet?

Control Types at a Glance

ControlNet supports many control methods. Each extracts different information from a reference image:

Line and Edge Controls

Depth and Spatial Controls

Style and Other Controls

How ControlNet Works in ComfyUI

The workflow has three stages:

1. Preprocessing — A preprocessor node extracts control information from your reference image (e.g., the Canny node extracts edges, a Depth preprocessor generates a depth map)
2. Condition injection — The Apply ControlNet node combines the extracted features with your text prompt conditions
3. Sampling — The KSampler generates the image, respecting both the text description and the structural constraints

Setting Up Your First ControlNet Workflow

What You Need

File Placement

ComfyUI/
├── models/
│   ├── checkpoints/
│   │   └── dreamshaper_8.safetensors
│   └── controlnet/
│       └── control_v11p_sd15_canny.pth
└── custom_nodes/
    └── comfyui_controlnet_aux/   (installed via Manager)

Building the Workflow

1. Load Checkpoint — loads your base model
2. Load Image — loads your reference photo
3. Preprocessor node (e.g., Canny) — extracts control features from the reference
4. Load ControlNet Model — loads the matching ControlNet model file
5. Apply ControlNet — merges extracted features into your prompt conditioning
6. CLIP Text Encode (x2) — positive and negative prompts
7. KSampler — generates the image
8. VAE Decode → Save Image — decodes and saves the result

Key Parameters

Using Multiple ControlNets Together

You can chain multiple ControlNet models in a single workflow. For example, combine Depth (for spatial layout) with OpenPose (for pose) to control both scene structure and character pose simultaneously.

Connect them in series: the output of the first Apply ControlNet node feeds into the conditioning input of the second Apply ControlNet node.

Common Issues and Fixes

ControlNet has no visible effect

• Verify the ControlNet model matches your checkpoint version (SD1.5 with SD1.5, SDXL with SDXL)
• Increase the strength value
• Make sure the preprocessor output actually contains meaningful features (preview it)

Output follows structure too rigidly

• Lower strength to 0.5–0.7
• Set end_percent to 0.7 so the AI has freedom in later sampling steps

Deformed or broken anatomy

• Use OpenPose ControlNet alongside your other ControlNet to enforce correct body structure
• Add anatomy-related negative prompts: deformed, extra limbs, bad anatomy

"ControlNet model not found" error

• Check that the .pth or .safetensors file is in ComfyUI/models/controlnet/
• Restart ComfyUI after adding new model files

Preprocessor node missing

• Install the ComfyUI ControlNet Auxiliary Preprocessors plugin
• Restart ComfyUI after installation

Related Guides

Ready to try specific ControlNet types? Check out:

• Canny ControlNet — Edge-based structure control
• Depth ControlNet — Spatial depth and perspective
• OpenPose ControlNet — Human pose control

Source References

• ComfyUI ControlNet usage example
• ControlNetApply node documentation
• ComfyUI models concept