Manually Caption Your Dataset

Master the art of image captioning for LoRA training on SDXL to enhance your AI model's accuracy and consistency.

This guide is tailored for intermediate users. Default settings are optimized for automated captioning, so if manual captioning doesn’t yield the expected results, consider using the auto-generated captions when uploading your dataset.

Introduction

Image captioning in generative AI helps you define the key elements of your dataset that you want the AI to recognize and learn from while ignoring non-essential details. These captions teach the AI model to interpret various components within an image, ultimately guiding it in producing accurate and consistent outputs.

The Basics of Captioning for LoRA on SDXL

1. Understand the Caption Style

   Captions typically consist of three main elements: trigger, class, and descriptors.

   • Trigger: A unique keyword that activates a specific subject or style in the model’s outputs. The trigger should be a unique token not already recognized by the foundational model. Test its uniqueness by using it as a prompt in SDXL under the "Foundational" model to ensure it doesn’t produce any recognizable output. For instance, you can create a trigger by taking a memorable word and removing its vowels—like turning "chicken" into "chkn."

   • Class: This refers to the main subjects in an image, such as a man, woman, sword, etc.

   • Descriptors: These include specific details like actions, colors, or emotions unique to an image, or styles of art.

   For advanced training, keep a record of any words used in captions, especially trigger words. Otherwise, you can find them on your Model page, tab Details, under the Caption Words category. More info on Managing Your Model article.

2. Creating Effective Captions

Effective captions vary depending on training goals. Here are some helpful rules of thumb. Considering the image below, here are some methods for approaching captioning based on the goal of your training.

• If the goal is to train the style of the image a user might describe various aspects of the scene, including the airship, the clouds, and the houses on the ground below. In this type of caption, a unique trigger word is not indicated. A caption might read:

  an airship flying through the clouds above islands and houses

• If the goal is to train the airship as the subject of a LoRA model a user typically needs fewer descriptive words. In this case, a trigger word, also named token, is optional and can be used to indicate a particular kind of airship. It is less important to describe all the details in the scene, and more important to describe them as they relate to the airship. It is also useful to use any words you may want to be able to prompt again, such as colors, which the model may have more association with in the future due to their inclusion. An example of this type of caption could be:

  a cldlk airship in the sky with a big yellow blob and red and blue patterns on it

Effective captions should be concise yet detailed enough to convey the essential elements of the image. Balancing brevity with detail ensures the AI model learns effectively.

3. Consistency is Key

Maintain a consistent captioning approach across your dataset to help the AI model learn and apply concepts more reliably.

Best Practices

• Adapt Based on Desired Results: Tailor your captioning to the specific model you wish to train. Adjusting the learning rate of the text encoder may be necessary if captions aren't being learned effectively.

• Leverage the Foundational Model: Utilize concepts, styles, and aesthetics already associated with the foundational SDXL model in your captions and prompts. Working with the model’s strengths often yields better results.

• Clean Up Images: Before captioning, ensure your images are free of unwanted elements like stray objects or watermarks.