ControlNet

ControlNet is a series of settings, which gives users targeted and nuanced control of their outputs. Users simply select a reference image, select a mode, and retain important information from their image that other tools fail to keep

When Should You Use ControlNet?

The best time to use ControlNet Modes is when you are trying to emulate the structure, linework, or general architecture of your reference image. Different modes can pick up poses, edges, lines, and even depth.

ControlNet Modes

Pose Mode

Pose Mode is ideal for character creation. It works best with realistic or semi-realistic human images, as that is what it was trained on. Pose Mode is not as useful for non-character work but is incredibly powerful at detecting faces and poses. It interprets the skeletal structure or the pose of the figures within your reference image, helping you generate images that maintain the same stance or gesture.

Depth Mode

Depth Mode is a wonderful tool for differentiating the background and foreground of your reference image, as well as the various leveled elements in an image. This mode is particularly useful in landscapes or scenes where understanding the spatial arrangement is critical. It retains both the outer structure and many of the finer details, helping create a sense of depth and dimension similar to the original image.

Structure Mode

Structure Mode picks out and highlights all the fine edgework in an image, focusing mainly on what it considers the subject. This mode is ideal for replicating architectural forms, intricate designs, or any composition where maintaining the integrity of the edges is crucial. Structure Mode closely resembles the original input in terms of shape and form, but as is true with ControlNet, it will not carry over any of the original reference colors.

Segmentation Mode

Segmentation Mode identifies the areas of space taken up by subjects in an image, separating different objects and often distinguishing between foreground and background elements. This mode is particularly useful when you need to control how the model interprets different objects within a scene, allowing for targeted manipulation of specific elements within the generated image. However, it relies more on the underlying model for composition details rather than the reference image itself.

Illusion Mode

Illusion Mode is designed to seamlessly integrate patterns like graphic elements into an image. It’s ideal for blending these patterns into the background or overall design in a way that feels natural and cohesive. Whether you want to subtly embed a logo or create intricate designs with repeating patterns, Illusion Mode helps to merge these elements smoothly into the final image, making them a part of the overall composition without standing out too much.

Going further

You can combine Controlnet Mode with other Pipelines like Style or Character References. Follow the link to have some examples and explanations about combinations.