Save base model in ControlNet training scripts when saving trained model or pushing to hub

[pcuenca]

This came up in the PR to provide ControlNet support to Apple's ml-stable-diffusion: apple/ml-stable-diffusion#153. I've also seen people training ControlNet models for Stable Diffusion v2 and v2.1, so it could be useful to save this as a metadata field for users of the trained models.

This would be similar to the base_model recently added to the textual inversion training script: huggingface/notebooks@944dc4b, but could maybe go inside the config.json of the saved model.

@williamberman @sayakpaul @yiyixuxu @patrickvonplaten do you think this could be useful? If so, I can submit a PR for both the PyTorch and the Flax training scripts.

[sayakpaul]

I think it would make sense to add that to the YAML metadata like we do in the DreamBooth training as well:

diffusers/examples/dreambooth/train_dreambooth_lora.py

Line 70 in ca783a0

base_model: {base_model}

[patrickvonplaten]

We could definitely add such a flag to the training script @pcuenca - I'd say be default we only save controlnet, but having an option to save the whole model along-side makes a lot of sense!

[pcuenca]

Thanks, @sayakpaul and @patrickvonplaten, I'll prepare a PR then! I thought we just needed the metadata info, didn't think about saving the whole model to the same repo.

Actually the metadata field already appears in some models, for example here: https://huggingface.co/lllyasviel/control_v11f1p_sd15_depth/blob/main/README.md?code=true#L3, but couldn't find it being automatically added in either the training or conversion scripts.

[atiorh]

Thank you all! We currently assume a runwayml/stable-diffusion-v1-5 base model and warn the users who attempt to export ControlNet models to Core ML using another model version. I will update the behavior once the base model attribute is made available in the model metadata.

[pcuenca]

Closing as fixed by #3229.