Flux Mini 3B

I'm not the original author of the model, created by TencentARC all credit goes to them.

Important Notes:

• I have 8GB VRAM GPU and I saw times between 12 seconds and 120 seconds to generate an image between 512x512 to 1024x1024 so it depends what settings you use

• I highly recommend switching from KSAMPLER to SHARKSAMPLER as it delivers way better results for this model https://huggingface.co/TencentARC/flux-mini/tree/main

  diffusion_models version

  • this is the original format as provided by the authors

  • This is a Flux Transformers Format based model that means Load Checkpoint node will not work in ComfyUI for this version

  • The file must be placed in the diffusion_models directory for it to be found by `Load Diffusion Model` node.

  • To use it you must use the Load Diffusion Model node to load the model and then just the traditional Flux setup for the rest will work.

  checkpoints version

  • this is the format that has been converted by me to work with Load Checkpoint node

  • This is still in Flux Transformers Format but has had the keys of the structure prefixed to work with Load Checkpoint

  • This file must be placed in the checkpoints directory for it to be found by the Load Checkpoint node.

  • To use it you must use the Load Checkpoint node to load the model, it does not have the CLIP and VAE baked in so you must still source that elsewhere

  • Use DualClipLoader and Load VAE to get those into your workflow as usual

  q8 version

  • this is the format that has been converted to unet format and then quantsized to q8 gguf

  • This file must be placed in the unet directory for it to be found by the UNET Loader node.

  • To use this you must have this third party extension installed Unet Loader (GGUF)

  • This reduces the file to 3GB in size and further improvements in speed and total VRAM requirements with only a 0.1% difference between this and the original file.

  aio version (all in one)

  • this is similar to the checkpoints format if you just want to use Load Checkpoint without having to also use DualClipLoader and Load VAE seperately

  • You can use this with the default workflow just pick the file and it should work out the box

  • Baked in T5 Model: t5xxl_fp8_e4m3fn.safetensors

  • Baked in CLIP-L: ViT-L-14-TEXT-detail-improved-hiT-GmP-TE-only-HF.safetensors

  • This one should deliver the fastest results ideally, q8 might still be faster, need further testing to determine.

A 3.2B MMDiT distilled from Flux-dev for efficient text-to-image generation

Nowadays, text-to-image (T2I) models are growing stronger but larger, which limits their practical applicability, especially on consumer-level devices. To bridge this gap, we distilled the 12B Flux-dev model into a 3.2B Flux-mini model, trying to preserve its strong image generation capabilities. Specifically, we prune the original Flux-dev by reducing its depth from 19 + 38 (number of double blocks and single blocks) to 5 + 10. The pruned model is further tuned with denoising and feature alignment objectives on a curated image-text dataset.

We empirically found that different blocks have different impacts on the generation quality, thus we initialize the student model with several most important blocks. The distillation process consists of three objectives: the denoise loss, the output alignment loss as well as the feature alignment loss. The feature alignment loss is designed in a way such that the output of block-x in the student model is encouraged to match that of block-4x in the teacher model. The distillation process is performed with 512x512 Laion images recaptioned with Qwen-VL in the first stage for 90k steps, and 1024x1024 images generated by Flux using the prompts in JourneyDB with another 90k steps.

Github link: https://github.com/TencentARC/flux-toolkits