Dataset Card for Dalle3 1 Million+ High Quality Captions

Alt name: Human Preference Synthetic Dataset

Description:

This dataset comprises of AI-generated images sourced from various websites and individuals, primarily focusing on Dalle 3 content, along with contributions from other AI systems of sufficient quality like Stable Diffusion and Midjourney (MJ v5 and above). As users typically share their best results online, this dataset reflects a diverse and high quality compilation of human preferences and high quality creative works. Captions for the images were generated using 4-bit CogVLM with custom caption failure detection and correction. The short captions were created using Dolphin 2.6 Mistral 7b - DPO on the CogVLM captions.

This dataset is composed of over a million unique and high quality human chosen Dalle 3 images, a few tens of thousands of Midjourney v5 & v6 images, and a handful of Stable Diffusion images.

CogVLM was prompted to produce captions for the images with this prompt:

'Provide a very detailed description for this image.'

Usage Ideas:

The dataset uses the WebDataset format and can be easily accessed and used with HuggingFace's datasets library like so:

from datasets import load_dataset
base_url = "https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-high-quality-captions/resolve/main/data/data-{i:06d}.tar"
num_shards = 52  # Number of webdataset tar files
urls = [base_url.format(i=i) for i in range(num_shards)]
dataset = load_dataset("webdataset", data_files={"train": urls}, split="train", streaming=True)

# Example of iterating through the dataset
for image in dataset:
    print(image)  # single image in row with associated columns
    break

• Note that as long as streaming=True in the above example, the dataset does not have to be downloaded in full.

This dataset can be used for a variety of purposes such as research, enhancing the diversity of existing datasets, exploration, psychological research, evaluating model performance, fine-tuning existing models, and training new models. This dataset also represents a historical archive of human interactions with AI systems.

Source Code:

The code used to generate the long and short captions can be found in here. Examples of the code used to produce the hashes can also be found in this repo.

Known Limitations:

The images may contain biases from users who made them. They are also not representative of the training datasets used for training the models and they only represent an extremely small fraction of what users have made with the models.

Prefixes

CogVLM often uses common prefix substrings in output captions. This function provides a way to remove those if desired.

import re

def modify_caption(caption: str) -> str:
    """
    Removes common prefix substrings from CogVLM outputs.

    Args:
        caption (str): A string containing a cogvlm caption.

    Returns:
        str: The caption with the prefix substring removed
            or altered if it was present.
    """
    base_words = ['showcases ', 'portrays ', 'appears to be ', 'is ', 'depicts ', 'features ']
    prefix_substrings = [("The image " + s, '') for s in base_words] + [("This image " + s, '') for s in base_words]
    prefix_substrings += [("In this " + s, '') for s in ["picture, ", "depiction, ",  "piece, ", "image, ", "scene, "]]
    prefix_substrings  += [
        ('In this artwork, ', 'Artwork of '),
        ('In this illustration, ', 'Illustration of '),
        ('In this art piece, ', 'Art of ')
    ]
    pattern = '|'.join([re.escape(opening) for opening, _ in prefix_substrings])
    replacers = {opening: replacer for opening, replacer in prefix_substrings}
    
    def replace_fn(match):
        return replacers[match.group(0)]
    
    return re.sub(pattern, replace_fn, caption, count=1, flags=re.IGNORECASE).capitalize()

Content

The dataset encompasses a wide array of concepts including art styles, landscapes, themes, holidays, vehicles, objects, and beyond. The unique circumstances of Dall-E 3's unpredictable and often nonsensical censorship forced users to be more creative than they would have been normally, and thus the human driven creativity in this dataset is made even more valuable.

The following categories of content were filtered out from the dataset using extensive automated and manual filtering methods: images made with non-AI means, failed rendering attempts, content positively portraying hate & hateful groups, along with acts and content deemed to be unethical (i.e. sexually explicit images of real people). Manual review was performed for over 300,000 images based on detected image contents and download source.

Deduplication was performed using md5_pil_hash column values.

Contributors

• Ben Egan
• SilentAntagonist
• Alex Redden
• XWAVE

Data Attributes

• image: The image file. Data type: image.
• image_name: The name of the image file. Data type: string.
• md5_pil_hash: The MD5 hash of the image as generated by PIL. See example code for more details. Data type: string.
• md5_file_hash: The MD5 hash of the image file. See example code for more details. Data type: string.
• sha512_hash: The SHA512 hash of the image file. See example code for more details. Data type: string.
• short_caption: A brief description of the image created by summarizing the long_caption with Dolphin 2.6 Mistral 7b - DPO. Data type: string.
• long_caption: A detailed description of the image created using CogVLM. Data type: string.
• resolution: The image height and width of the image in the format of <width>x<height>. Data type: string.
• width: The width of the image in pixels. Data type: integer.
• height: The height of the image in pixels. Data type: integer.
• url: The URL where the image can be accessed. Note that some urls are no longer functional. Data type: string.
• source: The general source where the image was found, if available. This includes information not available from the other attributes, like Reddit subreddits. Data type: string.
• original_prompt: The original prompt used to generate the image, if available. Data type: string.
• short_caption2: If available, one or more additional short captions made with Dolphin 2.6 Mistral 7b - DPO corresponding to long_caption2. Data type: list[string].
• long_caption2: If available, one or more additional long captions generated with CogVLM. Data type: list[string].