(HoJ) - High on Juice - Semi-realistic IllustriousXL

V4!

As i was working on an other checkpoint and wanted to integrate HoJ in it, i started using sd-mecha. Not your casual tool for merging, but very powerful. But what was the most interesting was a strange warning i ????????? of the keys for one of the layers of CLIP-L was containing non-finite values (so, NaN, -Inf or +Inf). Oh oh, that's not good. This means that when the prompt is parsed, some conditions may be corrupted, resulting in strange stuff in the picture.

After careful investigation, this error is in almost ALL of my checkpoints o_O

I did a quick fix:

from safetensors.torch import load_file, save_file

h = load_file("mine\\Hoj\\HoJ_IXL-V1.1.fp16.safetensors")
to_fix = [ k for k in h.keys() if not h[k].isfinite().all() ]

print(to_fix)
['conditioner.embedders.0.transformer.text_model.encoder.layers.11.mlp.fc1.weight', 'conditioner.embedders.0.transformer.text_model.encoder.layers.11.mlp.fc2.weight', 'conditioner.embedders.0.transformer.text_model.encoder.layers.11.self_attn.k_proj.weight', 'conditioner.embedders.0.transformer.text_model.encoder.layers.11.self_attn.out_proj.weight', 'conditioner.embedders.0.transformer.text_model.encoder.layers.11.self_attn.q_proj.weight', 'conditioner.embedders.0.transformer.text_model.encoder.layers.11.self_attn.v_proj.weight']

b = load_file("base\\illustriousXL_v01.safetensors")
test_base = [ k for k in b.keys() if not b[k].isfinite().all() ]

print(test_base)
[]

for k in to_fix:
  h[k] = b[k]

save_file(h, "hoj11.safetensors")

And based on this fixed version of Hoj 1.1, 2.0, 3.0 and AnBan ShinV2, i built HoJ 4.0c (for fixed CLIP).

V3

V3.0 is out! It is based on V2.0 with an effort to remove the unwanted influence from Niji. (longer version of the story below). Special thanks to @GZees!

V3 + DMD2

Since i am going to play a bit with the DMD2 LoRA, i have integrated it in HoJ V3. This version is really experimental, prefer V3.0 if you want to use a Negative Prompt as usual. Otherwise, for this version, use 8 steps, Euler A and CFG 1 for very fast generation 😁

Short Story

HoJ is a checkpoint focused on my personal tastes: high-constrasted, vividly colored, semi-realistic anime-like pictures.

Recommended usage:

• Sample/Scheduler:

  • DPM++ 2M SDE Heun & SGM Uniform (what i used for showcase)

  • Euler A & either "automatic", "SGM Uniform" or "Beta" for fun

  • DPM++ 2M & Karras works too, but it is not the best

• Steps: between 30 and 40

• CFG: 6 (this is my preference, feel free to experiment)

• VAE: Standard SDXL VAE is baked in, no need for an "extra colored VAE"

• Upscaler: 4xNMKDSuperscale 20 steps, denoising max 0.3

• Adetailer: 0.15 denoising maximum, same number of step, same VAE (or None)

• Positive prompt: Quality tags are not needed (masterpiece and so on)

• Negative prompt: IllusN3g

• Artist tags: I don't recommend using any of those, they will not respond correctly

• Clip Skip: 2 (but 1 can works fine too)

• 🚩 ???? Warning: Be sure to include model age or some measure of security in this direction or you may land with far too young characters. That's something i noticed and that i'll need to fix in a future version.

Permissions: Feel free to use this model in a merge, just make sure to mention the source 😉

Don't upload this model in other website,

i'll handle myself the upload on TA and SA.

@GZees kindly added HoJ on https://socialdiff.net also if you want to give it a try 🥰

To comply with source resources, commercial use of pictures generated is forbidden.

V3.0 update "long" story

As mentionned in this article, i am starting to feel overwhelmed by the omnipresence of Niji influence in my own stuff. I have nothing against this style, but i don't want it baked in by default. That's why i built this LoRA.

V2.0 of HoJ finally got the "undo Niji" treatment in this update.

With the authorisation of @GZees i "piggybacked" on the fantastic improvement of iLustMix to smooth some crazyness of HoJ will keeping its distinct style:

• I have extracted a LoRA from HoJ V2.0 minus iLustMix V2.0

• Applied it to iLustMix V7.0

• and merged the result with Cosine to HoJ V2.0 (all with Un-Niji-fy on top)

• Then, to get back some of the original style, i brought HoJ V1.0 in the mix in small proportion but only for the BASE/CLIP and MID blocks.

• Lastly, to add some spice, i merged at 0.1 Add Micro Details V4 in the result (from what i see on the model, merge is allowed)

This allowed me to get HoJ V3.0 without too much influence from the correcting checkpoint

The (famous) long story (V1.0)

It was build on top of my own Illustrious merge checkpoint, with some elaborate mixing of JunkJuice Psychonex (a Pony model) without losing the Illustrious goodness.

For the sake of openess, the resources i leveraged to do so are:

• Featureless 2.5D

• Coco Illustrious 5.5 (-)

• (Psychonex - Pony) as an "add difference" merge

• realistic filter [IL]

• Painted_World_IllustriousXL

• Smooth Booster

• Pino_Art_Style_by_UOC

• Hands 5.5

And then manually trained on a subset of "best pictures" for a few epoch as explained in this article.

The resulting trained model was carefully merged in the original checkpoint to fix its defaults. To do so, using merge block, i created two versions (the similarity of blocks is here):

• Original + Training, similar looking blocks only (IN/OUT pairs 02,03,05,06 + MID, at strength 1.0)

• Original + Training, different looking blocks only (IN/OUT pairs 00,01,04,07,08, at strength 1.0, except for the 01 pair at 0.25 to avoid over-saturation)

And then, using Supermerger and cosine calculation mode, these two versions were mixed for a finished checkpoint (Similar + 0.25 different, CosineA).

Thanks for reading! 💖

EDIT: the dataset used to fine-tune HoJ is available here: https://huggingface.co/datasets/n-Arno/juicy