Interesting Stable Diffusion Open-Source Models You Wish You Knew Earlier

Interesting Stable Diffusion Open-Source Models You Wish You Knew Earlier

by

Among many text-to-image models, Stable Diffusion is one of the most welcomed and free-to-use AI art generators for its true open-source nature.

In this article, you will get to know the amazing open-source contributions that the community has built upon Stable Diffusion and their uses. All of the contributions are in the form of a Google Colab notebook for easy access.

Along with the amazing open-source models, I have attached their Twitter and GitHub links to get to know the latest updates and guides if any. 

Stable Diffusion and Its Popularity

Stable Diffusion (SD) is a text-to-image latent diffusion model that was developed by Stability AI in collaboration with a bunch of researchers at LMU Munich and Runway.

Stable Diffusion is primarily used to generate images from text descriptions known as prompts. However, it can also be used for inpainting, outpainting, and image-to-image generations guided by prompts.

Unlike other AI art generators in the market, Stable Diffusion is a pure open-source model. Anyone can see the model weights and view as well as execute its code.

Due to its openness, it exploded in popularity rapidly among ML research enthusiasts as well as ordinary people. As a result, it has seen amazing open-source contributions since its release in August 2022.

How to Use Stable Diffusion?

Stable Diffusion (SD) model can be run both online and offline. To run SD online, you need to sign in to DreamStudio. DreamStudio is an in-browser application that was developed by Stability AI. Since it has an intuitive user interface, anyone can use SD for free in DreamStudio but with a limited free credit of 200.

To run SD on the local machine, the system requirements below are essential:

  • NVIDIA GPU with at least 4 GB VRAM memory (6 GB VRAM will be faster)
  • 10 GB of local storage
  • An OS of Linux, Windows 11, 10, 8.1, 8, or Mac

If your computer meets the above-mentioned requirements, you can run SD on your local machine.

If you have exceeded the free credits in DreamStudio and didn’t have the essential system requirement, then try SD for free on Google Colab notebook. Trying SD on Colab notebook is a time-consuming process and you may get intimidated due to the coding environment.

To sum up, if you are not a coding person and want to run Stable Diffusion, then try on DreamStudio or follow this article to run it locally. Alternatively, you can also use Stable Diffusion for free in Canva, Photoshop as well as blender.

How to Use Stable Diffusion Open-Source Models Contributed by the Community?

Before knowing how to use the Stable Diffusion models developed by the community for several purposes, you must know what are those models.

Stable Diffusion models that I’m going to explain here are the models that are built upon the base Stable Diffusion model by AI enthusiasts. These models are built to fulfill certain tasks that the base SD model could not.

For example, animations generated by the base Stable Diffusion will not give more personalized results. Hence, the need for other models arises.

Interesting Stable Diffusion Open-Source Models You Wish You Knew Earlier
Generated by “Deforum Stable Diffusion Local Version”

Image source

Unlike Stable Diffusion, there is no graphical user interface (GUI) available for the majority of the open-source models that are built upon Stable Diffusion by the community members. I.e. there are no DreamStudio-like applications available for these models.

Hence, these models need to be used on the Google Colab notebook or on the command-line/ integrated development environment software (IDE) in your local computer if you have enough GPU memory. Actually, it isn’t that difficult to work on Colab notebooks. You just need to read the instructions given in the Colab notebook or on the GitHub page of the model and run every cell in the Colab notebook.

However, this will not be the case and GUI’s will be available in the future for easy interaction.

Interesting Stable Diffusion Open-Source Models You Wish You Knew Earlier

  1. Deforum Stable Diffusion
  2. Video Killed the Radio Star …Diffusion
  3. Doohickey Diffusion
  4. Stable Diffusion Interpolation
  5. Prompt Parrot 🦜
  6. Easy Diffusion
  7.  Stable-DreamFusion
  8. Stable Conceptualizer Inference
  9. Stable Diffusion DreamBooth Inference
  10. Stable Craiyon 
  11. Stable Diffusion Videos
  12. Seamless Texture Inpainting
  13. Stable Worlds
  14. Grokking Stable Diffusion
  15. Openjourney

1. Deforum Stable Diffusion 

Deforum Stable Diffusion is a model that is built upon the Stable Diffusion base model that allows users to generate 3D and 2D animations.

It has Waifu and Robo Diffusion models for animations.

The features of Deforum Stable Diffusion include weighted prompts, perspective 2D flipping, dynamic video making, custom MATH expressions, and Waifu and Robo Diffusion models.

Interesting Stable Diffusion Open-Source Models

Google Colab Notebook: https://colab.research.google.com/github/deforum/stable-diffusion/blob/main/Deforum_Stable_Diffusion.ipynb 

GitHub: https://github.com/deforum/stable-diffusion 

Discord: https://discord.com/invite/deforum 

Created by: https://twitter.com/deforum_art 

2. Video Killed the Radio Star …Diffusion

This amazing model allows users to create Stable Diffusion animation prompted by the lyrics of any YouTube video. It not only uses Stable Diffusion but also OpenAI’s Whisper, a speech-to-text model.

The features of Video Killed the Radio Star …Diffusion includes an in-notebook spreadsheet for prompt editing, overriding, and flagging images for regeneration, an optional DreamStudio API, and Google Drive connectivity.

Google Colab Notebook: https://colab.research.google.com/github/dmarx/video-killed-the-radio-star/blob/main/Video_Killed_The_Radio_Star_Defusion.ipynb

GitHub: https://github.com/dmarx/video-killed-the-radio-star 

Created by: https://twitter.com/DigThatData 

3. Doohickey Diffusion

Doohickey Diffusion is one model that has all the tools every user dreamt of for Stable Diffusion. It has:

  • CLIP guidance
  • Perceptual guidance/ Learned Perceptual Image Patch Similarity (LPIPS)
  • Perlin initial noise
  • Attention slicing for more memory-efficient sampling
  • Image inits
  • Multiple models to choose from

Google Colab Notebook: https://colab.research.google.com/github/aicrumb/doohickey/blob/main/Doohickey_Diffusion.ipynb 

GitHub: https://github.com/aicrumb/doohickey 

Created by: https://twitter.com/aicrumb 

4. Stable Diffusion Interpolation

Stable Diffusion Interpolation is a model that morphs two different images. For example, morphing between “blueberry spaghetti” and “strawberry spaghetti”.

Interesting Stable Diffusion Open-Source Models

The Stable Diffusion Interpolation V2 includes features such as multiple prompts and multiple seeds.

Google Colab Notebook: https://colab.research.google.com/drive/1EHZtFjQoRr-bns1It5mTcOVyZzZD9bBc?usp=sharing 

Created by: https://twitter.com/ygantigravity & https://twitter.com/pharmapsychotic 

5. Prompt Parrot 🦜

Prompt Parrot 🦜 is a boon for all who are struggling to create good prompts. It uses a GPT-2 model to train on a list of your prompts. Once the training was over, this model throws multiple tweaked prompts in your own style.

Prompt Parrot has made to create brilliant and hilarious prompts.

The features of Prompt Parrot 2.0 includes integrated Stable Diffusion and pre-loaded 148 signature Kyrick prompts.

Interesting Stable Diffusion Open-Source ModelsImage source

Google Colab Notebook: https://colab.research.google.com/drive/1GtyVgVCwnDfRvfsHbeU0AlG-SgQn1p8e?usp=sharing

Created by: https://twitter.com/KyrickYoung 

6. Easy Diffusion

Easy Diffusion model notebook is also created to enhance the output of the base Stable Diffusion like Doohickey Diffusion model.

Though the Easy Diffusion is a fork of NOP’s notebook, it has evolved into a whole new model with image upscaling features, such as depth of field, interrogation with CLIP, and more.

Google Colab Notebook: https://colab.research.google.com/github/WASasquatch/easydiffusion/blob/main/Stability_AI_Easy_Diffusion.ipynb 

GitHub: https://github.com/WASasquatch/easydiffusion 

Created by: WASasquatch

7. Stable-DreamFusion

Stable-DreamFusion by ashawkey is a cool model for those who love text-to-3D. It is powered by Stable Diffusion and built based on the DreamFusion paper from Google.

Google Colab Notebook: https://colab.research.google.com/drive/1MXT3yfOFvO0ooKEfiUUvTKwUkrrlCHpF?usp=sharing 

GitHub: https://github.com/ashawkey/stable-dreamfusion 

Created by: https://twitter.com/_akhaliq 

8. Stable Conceptualizer Inference

Stable Conceptualizer Inference is one of the most welcomed models that allows the generation of images with new concepts through Textual Inversion. You can use the already pre-trained concepts as well as teach your own concept.

Interesting Stable Diffusion Open-Source Models

Google Colab Notebook: https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_conceptualizer_inference.ipynb (pre-trained model) & https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb (To train your own model)

HuggingFace: https://huggingface.co/spaces/sd-concepts-library/stable-diffusion-conceptualizer (check out for pre-trained models)

Created by: https://twitter.com/huggingface 

9. Stable Diffusion DreamBooth Inference

Stable Diffusion DreamBooth Inference is a similar kind of “Stable Conceptualizer Inference” model but for Google’s DreamBooth model.

DreamBooth is known for subject-driven AI generation that highly contextualizes the subject. The Stable Diffusion DreamBooth model also inherits this property. Hence, you can even turn yourself into AI art using this model.

Interesting Stable Diffusion Open-Source Models

Google Colab Notebook: https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_dreambooth_training.ipynb 

HuggingFace: https://huggingface.co/sd-dreambooth-library (check out for pre-trained models)

Created by: https://twitter.com/huggingface

10. Stable Craiyon 

Stable Craiyon is a model that combines both the Craiyon AI and Stable Diffusion to give great results. Those who want to leverage the features of both models can try this colab notebook.

Google Colab Notebook: https://colab.research.google.com/drive/1EIS7mAu4qcLsoEzQqREuxxaAX9Vmwt7H?usp=sharing 

GitHub: https://github.com/GeeveGeorge/Stable-Craiyon

Created by: https://twitter.com/GeeveGeorge 

11. Stable Diffusion Videos

Stable Diffusion Videos model makes text-to-video possible in Stable Diffusion. The model generates videos by interpolating between prompts as well as based on audio input.

You can create AI music videos with Stable Diffusion with this model!

Google Colab Notebook: https://colab.research.google.com/github/nateraw/stable-diffusion-videos/blob/main/stable_diffusion_videos.ipynb

GitHub: https://github.com/nateraw/stable-diffusion-videos 

Created by: https://twitter.com/_nateraw 

12. Seamless Texture Inpainting

As the name suggests, Seamless Texture Inpainting model generates textures of all kinds. This model uses the CompVis SD checkpoint provided by HuggingFace. This model gave me a seam that was 100% perfect.

Google Colab Notebook: https://colab.research.google.com/drive/14gt9Z1wqQRS8jVfzJA1K9_PAYlXUAFrR?usp=sharing 

Created by: https://twitter.com/metasemantic 

In case you want to use AI-generated texture in Blender, a plug-in is available that was created by another AI enthusiast. 

13. Stable Worlds

Want to create panoramas in Stable Diffusion? This model has got you covered. Stable Worlds not just create panoramas but immersive panoramas!

Google Colab Notebook: https://colab.research.google.com/drive/1RXRrkKUnpNiPCxTJg0Imq7sIM8ltYFz2?usp=sharing 

Created by: https://twitter.com/NaxAlpha 

14. Grokking Stable Diffusion

Grokking Stable Diffusion is a model that acts as a hands-on experiment on how Stable Diffusion works! By utilizing this model, users will achieve better interaction with Stable Diffusion thus leading to the creation of cool stuff.

Grokking Stable Diffusion covers the diffusion loop, messing with text embeddings, img2img, and arbitrary guidance.

Google Colab Notebook: https://colab.research.google.com/drive/1dlgggNa5Mz8sEAGU0wFCHhGLFooW_pf1?usp=sharing

Created by: https://twitter.com/johnowhitaker 

15. Openjourney

Openjourney, formerly known as “midjourney-v4-diffusion”, is one of the hottest models that accumulated a slew of attractions just after its release. In fact, it was the #1 trending model on Hugging Face for several weeks, even higher than the Stable Diffusion model itself.

The Openjourney is a custom model trained on the style of Midjourney version 4 by PromptHero, a prompt marketplace. This means you can get Midjourney V4 styles for free in Stable Diffusion. Midjourney V4 is famous for creating ultra-realistic portraits with perfect skin and hair.

To use Openjourney, head to the PromptHero website.

 

Google Colab Notebook: –

GitHub: https://github.com/prompthero/openjourney 

Created by: https://twitter.com/prompthero

Conclusion

With these interesting Stable Diffusion open-source models, you can enjoy Stable Diffusion in many forms according to your requirement. Immerse yourself in AI art!

Also, stay away from random anonymous Colab notebooks with mysterious features that you come across.

Share this:

Leave a Comment