Among many text-to-image models, Stable Diffusion is one of the most welcomed and free-to-use AI art generators for its true open-source nature.
In this article, you will get to know the amazing open-source contributions that the community has built upon Stable Diffusion and their uses. All of the contributions are in the form of a Google Colab notebook for easy access.
Along with the amazing open-source models, I have attached their Twitter and GitHub links to get to know the latest updates and guides if any.
Stable Diffusion and Its Popularity
Stable Diffusion (SD) is a text-to-image latent diffusion model that was developed by Stability AI in collaboration with a bunch of researchers at LMU Munich and Runway.
Stable Diffusion is primarily used to generate images from text descriptions known as prompts. However, it can also be used for inpainting, outpainting, and image-to-image generations guided by prompts.
Unlike other AI art generators in the market, Stable Diffusion is a pure open-source model. Anyone can see the model weights and view as well as execute its code.
Due to its openness, it exploded in popularity rapidly among ML research enthusiasts as well as ordinary people. As a result, it has seen amazing open-source contributions since its release in August 2022.
How to Use Stable Diffusion?
Stable Diffusion (SD) model can be run both online and offline. To run SD online, you need to sign in to DreamStudio. DreamStudio is an in-browser application that was developed by Stability AI. Since it has an intuitive user interface, anyone can use SD for free in DreamStudio but with a limited free credit of 200.
To run SD on the local machine, the system requirements below are essential:
- NVIDIA GPU with at least 4 GB VRAM memory (6 GB VRAM will be faster)
- 10 GB of local storage
- An OS of Linux, Windows 11, 10, 8.1, 8, or Mac
If your computer meets the above-mentioned requirements, you can run SD on your local machine.
If you have exceeded the free credits in DreamStudio and didn’t have the essential system requirement, then try SD for free on Google Colab notebook. Trying SD on Colab notebook is a time-consuming process and you may get intimidated due to the coding environment.
To sum up, if you are not a coding person and want to run Stable Diffusion, then try on DreamStudio or follow this article to run it locally. Alternatively, you can also use Stable Diffusion for free in Canva, Photoshop as well as blender.
How to Use Stable Diffusion Open-Source Models Contributed by the Community?
Before knowing how to use the Stable Diffusion models developed by the community for several purposes, you must know what are those models.
Stable Diffusion models that I’m going to explain here are the models that are built upon the base Stable Diffusion model by AI enthusiasts. These models are built to fulfill certain tasks that the base SD model could not.
For example, animations generated by the base Stable Diffusion will not give more personalized results. Hence, the need for other models arises.
Unlike Stable Diffusion, there is no graphical user interface (GUI) available for the majority of the open-source models that are built upon Stable Diffusion by the community members. I.e. there are no DreamStudio-like applications available for these models.
Hence, these models need to be used on the Google Colab notebook or on the command-line/ integrated development environment software (IDE) in your local computer if you have enough GPU memory. Actually, it isn’t that difficult to work on Colab notebooks. You just need to read the instructions given in the Colab notebook or on the GitHub page of the model and run every cell in the Colab notebook.
However, this will not be the case and GUI’s will be available in the future for easy interaction.
Interesting Stable Diffusion Open-Source Models You Wish You Knew Earlier
- Deforum Stable Diffusion
- Video Killed the Radio Star …Diffusion
- Doohickey Diffusion
- Stable Diffusion Interpolation
- Prompt Parrot 🦜
- Easy Diffusion
- Stable Conceptualizer Inference
- Stable Diffusion DreamBooth Inference
- Stable Craiyon
- Stable Diffusion Videos
- Seamless Texture Inpainting
- Stable Worlds
- Grokking Stable Diffusion
1. Deforum Stable Diffusion
Deforum Stable Diffusion is a model that is built upon the Stable Diffusion base model that allows users to generate 3D and 2D animations.
It has Waifu and Robo Diffusion models for animations.
The features of Deforum Stable Diffusion include weighted prompts, perspective 2D flipping, dynamic video making, custom MATH expressions, and Waifu and Robo Diffusion models.
Google Colab Notebook: https://colab.research.google.com/github/deforum/stable-diffusion/blob/main/Deforum_Stable_Diffusion.ipynb
Created by: https://twitter.com/deforum_art
2. Video Killed the Radio Star …Diffusion
This amazing model allows users to create Stable Diffusion animation prompted by the lyrics of any YouTube video. It not only uses Stable Diffusion but also OpenAI’s Whisper, a speech-to-text model.
The features of Video Killed the Radio Star …Diffusion includes an in-notebook spreadsheet for prompt editing, overriding, and flagging images for regeneration, an optional DreamStudio API, and Google Drive connectivity.
Google Colab Notebook: https://colab.research.google.com/github/dmarx/video-killed-the-radio-star/blob/main/Video_Killed_The_Radio_Star_Defusion.ipynb
Created by: https://twitter.com/DigThatData
3. Doohickey Diffusion
Doohickey Diffusion is one model that has all the tools every user dreamt of for Stable Diffusion. It has:
- CLIP guidance
- Perceptual guidance/ Learned Perceptual Image Patch Similarity (LPIPS)
- Perlin initial noise
- Attention slicing for more memory-efficient sampling
- Image inits
- Multiple models to choose from
Google Colab Notebook: https://colab.research.google.com/github/aicrumb/doohickey/blob/main/Doohickey_Diffusion.ipynb
Created by: https://twitter.com/aicrumb
4. Stable Diffusion Interpolation
Stable Diffusion Interpolation is a model that morphs two different images. For example, morphing between “blueberry spaghetti” and “strawberry spaghetti”.
The Stable Diffusion Interpolation V2 includes features such as multiple prompts and multiple seeds.
Google Colab Notebook: https://colab.research.google.com/drive/1EHZtFjQoRr-bns1It5mTcOVyZzZD9bBc?usp=sharing
Created by: https://twitter.com/ygantigravity & https://twitter.com/pharmapsychotic
5. Prompt Parrot 🦜
Prompt Parrot 🦜 is a boon for all who are struggling to create good prompts. It uses a GPT-2 model to train on a list of your prompts. Once the training was over, this model throws multiple tweaked prompts in your own style.
Prompt Parrot has made to create brilliant and hilarious prompts.
The features of Prompt Parrot 2.0 includes integrated Stable Diffusion and pre-loaded 148 signature Kyrick prompts.
Google Colab Notebook: https://colab.research.google.com/drive/1GtyVgVCwnDfRvfsHbeU0AlG-SgQn1p8e?usp=sharing
Created by: https://twitter.com/KyrickYoung
6. Easy Diffusion
Easy Diffusion model notebook is also created to enhance the output of the base Stable Diffusion like Doohickey Diffusion model.
Though the Easy Diffusion is a fork of NOP’s notebook, it has evolved into a whole new model with image upscaling features, such as depth of field, interrogation with CLIP, and more.
Google Colab Notebook: https://colab.research.google.com/github/WASasquatch/easydiffusion/blob/main/Stability_AI_Easy_Diffusion.ipynb
Created by: WASasquatch
Stable-DreamFusion by ashawkey is a cool model for those who love text-to-3D. It is powered by Stable Diffusion and built based on the DreamFusion paper from Google.
A implementation of text-to-3D dreamfusion, powered by stable diffusion
github: https://t.co/1GGjtphz8K pic.twitter.com/FaxIBL5WxE
— AK (@_akhaliq) October 6, 2022
Google Colab Notebook: https://colab.research.google.com/drive/1MXT3yfOFvO0ooKEfiUUvTKwUkrrlCHpF?usp=sharing
Created by: https://twitter.com/_akhaliq
8. Stable Conceptualizer Inference
Stable Conceptualizer Inference is one of the most welcomed models that allows the generation of images with new concepts through Textual Inversion. You can use the already pre-trained concepts as well as teach your own concept.
Google Colab Notebook: https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_conceptualizer_inference.ipynb (pre-trained model) & https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb (To train your own model)
HuggingFace: https://huggingface.co/spaces/sd-concepts-library/stable-diffusion-conceptualizer (check out for pre-trained models)
Created by: https://twitter.com/huggingface
9. Stable Diffusion DreamBooth Inference
Stable Diffusion DreamBooth Inference is a similar kind of “Stable Conceptualizer Inference” model but for Google’s DreamBooth model.
DreamBooth is known for subject-driven AI generation that highly contextualizes the subject. The Stable Diffusion DreamBooth model also inherits this property. Hence, you can even turn yourself into AI art using this model.
Google Colab Notebook: https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_dreambooth_training.ipynb
HuggingFace: https://huggingface.co/sd-dreambooth-library (check out for pre-trained models)
Created by: https://twitter.com/huggingface
10. Stable Craiyon
Stable Craiyon is a model that combines both the Craiyon AI and Stable Diffusion to give great results. Those who want to leverage the features of both models can try this colab notebook.
Just released a Colab notebook that combines Craiyon+Stable Diffusion , to get the best of both worlds. (Works in Colab free tier)
GitHub Repo : https://t.co/RL5FuSlQ8c
(Top image is Craiyon, bottom is stable diffusion's reinterpretation) pic.twitter.com/3jeUHvmoLu
— Geeve George (@GeeveGeorge) September 6, 2022
Google Colab Notebook: https://colab.research.google.com/drive/1EIS7mAu4qcLsoEzQqREuxxaAX9Vmwt7H?usp=sharing
Created by: https://twitter.com/GeeveGeorge
11. Stable Diffusion Videos
Stable Diffusion Videos model makes text-to-video possible in Stable Diffusion. The model generates videos by interpolating between prompts as well as based on audio input.
You can create AI music videos with Stable Diffusion with this model!
It will soon be possible to make custom AI music videos that move to the beat like this using #stablediffusion 🔥
As soon as the code is clean enough, I'll open source it for anyone around the world to use 🤗🌎
Music by @Skrillex @tydollasign @kanyewest @FKAtwigs
More info⤵️ pic.twitter.com/v85UEH32fr
— Nate Raw (@_nateraw) October 6, 2022
Google Colab Notebook: https://colab.research.google.com/github/nateraw/stable-diffusion-videos/blob/main/stable_diffusion_videos.ipynb
Created by: https://twitter.com/_nateraw
12. Seamless Texture Inpainting
As the name suggests, Seamless Texture Inpainting model generates textures of all kinds. This model uses the CompVis SD checkpoint provided by HuggingFace. This model gave me a seam that was 100% perfect.
Seamless textures with #stablediffusion notebook released!
Different than one yesterday that uses convolutions (this cuts the outside, pastes it on the inside, inpaints the outside, and glues them back making a 100% perfect seam)https://t.co/qJHMmx58sn pic.twitter.com/9PFrK0czZZ
— Travis Hoppe (@metasemantic) September 11, 2022
Google Colab Notebook: https://colab.research.google.com/drive/14gt9Z1wqQRS8jVfzJA1K9_PAYlXUAFrR?usp=sharing
Created by: https://twitter.com/metasemantic
In case you want to use AI-generated texture in Blender, a plug-in is available that was created by another AI enthusiast.
13. Stable Worlds
Want to create panoramas in Stable Diffusion? This model has got you covered. Stable Worlds not just create panoramas but immersive panoramas!
Colab to generate panoramas (Stable Worlds v1):https://t.co/mRdNyJW5kY https://t.co/NnnA9fKPjf pic.twitter.com/KUpK0QlwdM
— The AI Dude (@NaxAlpha) October 8, 2022
Google Colab Notebook: https://colab.research.google.com/drive/1RXRrkKUnpNiPCxTJg0Imq7sIM8ltYFz2?usp=sharing
Created by: https://twitter.com/NaxAlpha
14. Grokking Stable Diffusion
Grokking Stable Diffusion is a model that acts as a hands-on experiment on how Stable Diffusion works! By utilizing this model, users will achieve better interaction with Stable Diffusion thus leading to the creation of cool stuff.
Grokking Stable Diffusion covers the diffusion loop, messing with text embeddings, img2img, and arbitrary guidance.
New notebook: "Grokking Stable Diffusion"
Hopefully useful for anyone looking for content that goes deeper than just how to run the model! https://t.co/O7GALcgTxX
– The diffusion loop
– Messing with text embeddings
– arbitrary guidance
More deets in thread 🧵 pic.twitter.com/xQ0ufu8L2Y
— Jonathan Whitaker (@johnowhitaker) September 2, 2022
Google Colab Notebook: https://colab.research.google.com/drive/1dlgggNa5Mz8sEAGU0wFCHhGLFooW_pf1?usp=sharing
Created by: https://twitter.com/johnowhitaker
Openjourney, formerly known as “midjourney-v4-diffusion”, is one of the hottest models that accumulated a slew of attractions just after its release. In fact, it was the #1 trending model on Hugging Face for several weeks, even higher than the Stable Diffusion model itself.
The Openjourney is a custom model trained on the style of Midjourney version 4 by PromptHero, a prompt marketplace. This means you can get Midjourney V4 styles for free in Stable Diffusion. Midjourney V4 is famous for creating ultra-realistic portraits with perfect skin and hair.
To use Openjourney, head to the PromptHero website.
And of course, you can keep using Openjourney on our website, to create images without needing a single line of code nor a GPU. Check it out! 👉 https://t.co/z2ww0hSCkp pic.twitter.com/hI36ml3fgs
— PromptHero – Stable Diffusion, DALL-E & Midjourney (@prompthero) November 18, 2022
Google Colab Notebook: –
Created by: https://twitter.com/prompthero
With these interesting Stable Diffusion open-source models, you can enjoy Stable Diffusion in many forms according to your requirement. Immerse yourself in AI art!
Also, stay away from random anonymous Colab notebooks with mysterious features that you come across.