How to Fix Stable Diffusion 3.5 Performance Bottlenecks and Crashes

Introduction

When you encounter stable diffusion 3.5 performance bottlenecks and crashes, the frustration is immediate. You load Stability AI’s powerful new model expecting stunning image quality, and instead you get “CUDA out of memory” errors, generation taking minutes instead of seconds, or the application crashing entirely on startup.

Stable Diffusion 3.5 represents a major leap forward in AI image generation. With models like SD 3.5 Large (8 billion parameters) and SD 3.5 Medium (2.6 billion parameters), it delivers superior image quality, precise prompt adherence, and excellent text rendering.[reference:0] However, these capabilities come at a cost—significantly higher VRAM requirements and computational demands that can overwhelm consumer GPUs.[reference:1]

This guide provides 10 proven methods to diagnose and fix stable diffusion 3.5 performance bottlenecks and crashes, helping you run SD 3.5 efficiently on your hardware.

For official guidance, Stability AI’s SD 3.5 announcement provides detailed information on model capabilities and requirements.

For broader AI tool troubleshooting, explore our AI Tools Errors hub which covers Stable Diffusion, Midjourney, Flux, and more.

Why SD 3.5 Has Performance Issues and Crashes

Understanding why stable diffusion 3.5 performance bottlenecks and crashes occur helps you choose the right solution. Several factors contribute to these issues:

• Massive model size – SD 3.5 Large has 8 billion parameters, while the Medium version has 2.6 billion.[reference:2] The full FP16 model requires substantial VRAM—some users report that SD 3.5 Large is “more GPU intensive than FLUX dev.”[reference:3]
• High VRAM requirements – SD 3.5 Medium requires approximately 9.9 GB of VRAM (excluding text encoders), while Large can exceed 24GB.[reference:4][reference:5] Users with 24GB RTX 4090s still report OOM errors.[reference:6]
• Text encoder overhead – The T5-XXL text encoder adds significant memory overhead. In some implementations, the T5 encoder alone can exceed available VRAM.
• Slow inference times – With 50-step inference, even high-end GPUs can take significant time. On ROCm 7.0, SD 3.5 Large takes 122 seconds for a 1 Megapixel image.[reference:7]
• Device mismatch errors – Long prompts exceeding the T5 max_sequence_length (256 tokens) can cause crashes when tokenizer IDs are compared across CPU and GPU devices.[reference:8][reference:9]
• Memory fragmentation – PyTorch can fragment CUDA memory, leading to OOM errors even when total VRAM seems sufficient.
• Insufficient system RAM – Some crashes, particularly in ComfyUI, are caused by insufficient system RAM rather than VRAM.[reference:10]

Never assume your hardware is insufficient. Most cases of stable diffusion 3.5 performance bottlenecks and crashes are fixable with the right optimization techniques.

System Requirements and Hardware Considerations

Understanding the system requirements for Stable Diffusion 3.5 is essential for diagnosing stable diffusion 3.5 performance bottlenecks and crashes.

Key considerations:

• GPU support: NVIDIA RTX 40 and 50 Series, as well as Blackwell and Ada Lovelace generation RTX PRO GPUs, benefit from optimized TensorRT support.[reference:11]
• AMD GPUs: SD 3.5 can run on AMD ROCm, but performance may vary. Some users report successful runs on ROCm 6.4.3 while ROCm 7.0 causes crashes.[reference:12]
• MacOS: M-series chips may experience memory management issues when running large models.[reference:13]
• 12GB GPUs: SD 3.5 Medium fits comfortably on 12GB GPUs when optimized.[reference:14] With FP8 quantization, SD 3.5 Large can also run on 12GB GPUs.[reference:15]

Use FP8 Quantization to Reduce VRAM

FP8 quantization is one of the most effective ways to reduce VRAM usage for stable diffusion 3.5 performance bottlenecks and crashes. It reduces VRAM requirements by 40-50% with minimal quality loss.[reference:16]

1. Use the FP8 quantized model from Hugging Face: Comfy-Org/stable-diffusion-3.5-fp8
2. For ComfyUI users, use the fp8_scaled workflow JSON (experimental) and the fp8 scaled model as a low VRAM option.[reference:17]
3. Use t5xxl_fp8_e4m3fn_scaled.safetensors or t5xxl_fp8_e4m3fn.safetensors instead of the full FP16 T5 encoder.[reference:18]
4. FP8 quantization can reduce VRAM usage on a 12GB RTX 3060, allowing SD 3.5 to run smoothly.[reference:19]
5. According to user reports, FP8 quantized models maintain “high visual quality and prompt adherence” with minimal quality loss.[reference:20]

Code example (quantizing T5 Encoder and Transformer):

from diffusers import SD3Transformer2DModel
from transformers import T5EncoderModel
from optimum.quanto import freeze, qint8, quantize
import torch

base_model = "stabilityai/stable-diffusion-3.5-large"
dtype = torch.bfloat16

# Quantize Transformer
transformer = SD3Transformer2DModel.from_pretrained(
    base_model, subfolder="transformer", torch_dtype=dtype
)
quantize(transformer, weights=qint8)
freeze(transformer)
transformer.save_pretrained("./sd35/sd3transformer2dmodel_qint8")

# Quantize T5 Encoder
text_encoder_3 = T5EncoderModel.from_pretrained(
    base_model, subfolder="text_encoder_3", torch_dtype=dtype
)
quantize(text_encoder_3, weights=qint8)
freeze(text_encoder_3)
text_encoder_3.save_pretrained("./sd35/t5encodermodel_qint8")

ComfyUI FP8 Setup:

1. Download the fp8_scaled workflow JSON
2. Download sd3.5_large_fp8_scaled.safetensors
3. Place the model in your ComfyUI models/checkpoint folder
4. Load the workflow in ComfyUI and generate[reference:21]

Enable CPU Offloading and Sequential Offloading

CPU offloading moves model components from GPU VRAM to CPU RAM when not in use, dramatically reducing VRAM usage. This is one of the most reliable fixes for stable diffusion 3.5 performance bottlenecks and crashes.

1. Use pipe.enable_model_cpu_offload() instead of pipe.to("cuda")[reference:22]
2. This moves each model component to the CPU after use, keeping VRAM usage manageable.
3. For maximum VRAM savings, use sequential offloading: pipe.enable_sequential_cpu_offload()
4. Some users report that pipe.enable_model_cpu_offload() allows SD 3.5 Large to run on 24GB GPUs without OOM errors.[reference:23]

Code example:

import torch
from diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3.5-large",
    torch_dtype=torch.bfloat16,
    token=your_hf_token
)

# Enable CPU offloading—moves components to CPU when not in use
pipe.enable_model_cpu_offload()
# Sequential offloading for maximum VRAM savings
pipe.enable_sequential_cpu_offload()

# Generate image
image = pipe(
    prompt="A beautiful landscape with mountains and a lake",
    height=1024,
    width=1024,
    num_inference_steps=30,
    guidance_scale=4.5
).images[0]
image.save("sd35-output.png")

**Note:** CPU offloading trades speed for VRAM savings. Generation will be slower, but the model will run on GPUs with limited VRAM.

Use TensorRT Optimization for 2x Faster Inference

Stability AI has released TensorRT-optimized SD 3.5 models that deliver significantly faster performance. This is the most effective speed optimization for stable diffusion 3.5 performance bottlenecks and crashes.

1. TensorRT-optimized models deliver up to 2.3x faster generation on SD 3.5 Large and 1.7x faster on SD 3.5 Medium.[reference:24]
2. VRAM requirements are reduced by 40% compared to standard models.[reference:25]
3. Supported GPUs include NVIDIA GeForce RTX 50 and 40 Series, as well as NVIDIA Blackwell and Ada Lovelace generation RTX PRO GPUs.[reference:26]
4. Download the TensorRT-optimized models from Stability AI’s official repositories.
5. Follow the TensorRT installation guide for your platform.

Enable Memory Efficient Attention (xFormers)

xFormers reduces memory usage during attention computation, helping with stable diffusion 3.5 performance bottlenecks and crashes.

1. Install xFormers: pip install xformers
2. Enable xFormers in your pipeline: pipe.enable_xformers_memory_efficient_attention()[reference:27]
3. This reduces VRAM usage and can speed up inference by 10-20%.
4. Note: xFormers may not work on all hardware configurations.

Reduce Image Resolution and Inference Steps

If stable diffusion 3.5 performance bottlenecks and crashes persist, reducing image size and steps can significantly lower memory usage and speed up generation.

1. Reduce image dimensions: Try 768×768 or 512×512 instead of 1024×1024. The Medium model supports resolutions from 0.25 to 2 megapixels.[reference:28]
2. Decrease inference steps: For SD 3.5 Large Turbo, use 4 steps (distilled for fast generation without CFG).[reference:29]
3. For SD 3.5 Large, try 20-30 steps instead of 50.
4. Use guidance_scale values between 3.5 and 5.0—lower values use less memory.

Fix T5 Truncation Crashes on Long Prompts

Long prompts can cause stable diffusion 3.5 performance bottlenecks and crashes due to T5 truncation device mismatch errors.

1. When prompts exceed the T5 max_sequence_length (default 256 tokens), a device mismatch can occur between CPU and GPU.[reference:30]
2. Symptom: RuntimeError: Expected all tensors to be on the same device when using long prompts.[reference:31]
3. Fix: Update to a version that keeps tokenizer IDs on CPU for comparison (mirroring Flux pipeline behavior).[reference:32]
4. Short prompts (under 256 tokens) are unaffected because the truncation check is skipped.[reference:33]

Use ComfyUI FP8 Scaled Workflows

For ComfyUI users, the FP8 scaled workflow is the recommended low VRAM solution for stable diffusion 3.5 performance bottlenecks and crashes.

1. Download the fp8_scaled workflow from Hugging Face[reference:34]
2. Download sd3.5_large_fp8_scaled.safetensors[reference:35]
3. Place the model in your ComfyUI models/checkpoint folder
4. Optionally use t5xxl_fp8_e4m3fn_scaled.safetensors for the T5 encoder[reference:36]
5. Load the workflow in ComfyUI and generate

Apply 4-bit Quantization with BitsAndBytes

NF4 (4-bit Normal Float) quantization with BitsAndBytes is another way to reduce VRAM for stable diffusion 3.5 performance bottlenecks and crashes.

1. Use 4-bit quantization with the NF4 format[reference:37]
2. This enables SD 3.5 to run on GPUs with as little as 8-12GB VRAM.
3. NF4 quantization can reduce VRAM usage while maintaining reasonable image quality.
4. Some optimized implementations combine 4-bit quantization with CPU offloading for maximum VRAM savings.[reference:38]

Enable torch.compile for Speed

For users with compatible PyTorch versions, torch.compile can speed up inference by 15-25%.

1. Enable torch.compile on your UNet: pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead")[reference:39]
2. This optimizes the computation graph for faster inference.
3. Compatibility may vary depending on your PyTorch version and GPU.
4. For maximum performance, combine with xFormers and FP8 quantization.

Roll Back ComfyUI for MacOS Performance

MacOS users experiencing stable diffusion 3.5 performance bottlenecks and crashes may benefit from rolling back ComfyUI.

1. Some users report that SD 3.5 performance on MacOS got worse after certain ComfyUI updates.[reference:40]
2. Performance dropped from 7 seconds per iteration to 11.5 seconds per iteration.[reference:41]
3. Fix: Roll back ComfyUI to the latest release version (e.g., v0.2.6).[reference:42]
4. Performance returned to 7 seconds per iteration after rolling back.[reference:43]

Advanced Technical Fixes

For expert users, these advanced methods go beyond standard procedures for stable diffusion 3.5 performance bottlenecks and crashes.

Use DFloat11 for Memory-Constrained Environments

DFloat11 is a memory-efficient implementation that keeps weights compressed in GPU memory and decompresses them just before matrix multiplications.[reference:44] It is “much faster than CPU-offloading approaches” and enables deployment in memory-constrained environments.[reference:45]

Set PyTorch CUDA Memory Configuration

Set os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'expandable_segments:True' to prevent memory fragmentation.

Use vLLM-Omni Cache-DiT Acceleration

vLLM-Omni provides Cache-DiT acceleration for SD 3.5 models, significantly speeding up image generation through caching mechanisms.[reference:46]

Increase System Virtual Memory

If crashes persist due to insufficient system RAM, increase virtual memory (page file) to allow the model to load.[reference:47]

Disable Safety Checker for Faster Inference

Disable the safety checker to speed up inference: pipe.safety_checker = None[reference:48]

For additional AI tool support, see our guide on AI Tools Errors.

Conclusion

To fix stable diffusion 3.5 performance bottlenecks and crashes, start with the most effective optimization: FP8 quantization—this reduces VRAM usage by 40-50% with minimal quality loss and is available as pre-quantized models. For maximum speed, use the TensorRT-optimized models which deliver up to 2.3x faster generation on RTX 40/50 series GPUs.

For users with limited VRAM, CPU offloading (pipe.enable_model_cpu_offload()) allows SD 3.5 to run on GPUs with 8-12GB VRAM. ComfyUI users should use the fp8_scaled workflow as the recommended low VRAM option.

The success of each method depends on your hardware—FP8 quantization and TensorRT work best on NVIDIA GPUs, while CPU offloading works on any system but is slower. Typically, FP8 quantization offers the best balance of speed and VRAM reduction for stable diffusion 3.5 performance bottlenecks and crashes.

If you encounter persistent issues, consider using SD 3.5 Medium instead of Large, or the Turbo version for faster generation with 4 steps. Regular driver updates and staying informed about new SD 3.5 optimizations are the best defenses against performance problems.

For additional AI tool support, explore our AI Tools Errors hub to address other AI image generation issues.

Related Guides

• AI Tools Errors
• Flux.1 Out of Memory OOM and Slow Rendering – 2026 Guide
• Midjourney –cref Character Reference Not Working – 2026 Guide
• Midjourney –p Personalization Code Not Applying – 2026 Guide
• Adobe Firefly Generating Blank Images – 8 Solutions
• Speed Up AI Image Generation on Windows 11 – 8 Tips
• ChatGPT Deep Research Not Working – 2026 Guide
• Top 10 Common ChatGPT Errors and How to Solve Them

Frequently Asked Questions

Why does Stable Diffusion 3.5 keep crashing with CUDA out of memory?

A stable diffusion 3.5 performance bottlenecks and crashes error with OOM is usually caused by insufficient VRAM for the model. SD 3.5 Large requires 18-24GB VRAM in FP16. Use FP8 quantization or CPU offloading to reduce memory usage. The Medium model requires about 9.9 GB of VRAM (excluding text encoders).[reference:49]

What is the minimum VRAM for Stable Diffusion 3.5?

For SD 3.5 Medium: approximately 9.9 GB VRAM (excluding text encoders).[reference:50] With FP8 quantization, SD 3.5 Large can run on 12GB GPUs. Without optimizations, SD 3.5 Large requires 18-24GB VRAM.

How do I make Stable Diffusion 3.5 run faster?

Use the TensorRT-optimized models for up to 2.3x faster generation.[reference:51] Use SD 3.5 Large Turbo with just 4 steps for fast generation without CFG.[reference:52] Enable xFormers and torch.compile for additional speed gains.[reference:53]

What is FP8 quantization and how does it help SD 3.5?

FP8 quantization compresses model weights from 16-bit to 8-bit, reducing VRAM usage by 40-50% and speeding up inference. TensorRT-optimized models deliver up to 2.3x faster generation while reducing VRAM by 40%.[reference:54]

Can I run Stable Diffusion 3.5 on a 12GB GPU?

Yes. With FP8 quantization, SD 3.5 can run on a 12GB RTX 3060.[reference:55] The Medium model fits comfortably on 12GB GPUs.[reference:56] Use CPU offloading for additional memory savings if needed.

Why does SD 3.5 crash on long prompts?

Long prompts exceeding 256 T5 tokens can cause a device mismatch crash where tokenizer IDs are compared across CPU and GPU.[reference:57] Update to a version that keeps tokenizer IDs on CPU for comparison (mirroring Flux pipeline behavior).[reference:58]

Does SD 3.5 work on AMD GPUs?

Yes, SD 3.5 can run on AMD ROCm. Some users report successful runs on ROCm 6.4.3, while ROCm 7.0 may cause crashes.[reference:59] Performance may vary depending on your specific GPU and driver version.

How do I set up SD 3.5 in ComfyUI?

Update ComfyUI to the latest version, download the SD 3.5 model checkpoints and text encoders (clip_g.safetensors, clip_l.safetensors, t5xxl_fp16.safetensors), and place them in the appropriate folders.[reference:60] For low VRAM, use the fp8_scaled workflow.[reference:61]

Why is SD 3.5 slower than SDXL?

SD 3.5 has significantly more parameters (up to 8B vs SDXL’s ~2.6B) and uses a more complex MMDiT (Multimodal Diffusion Transformer) architecture.[reference:62] This results in higher computational demands and longer inference times. The Turbo version and TensorRT optimization can significantly reduce generation time.

What should I do if none of the fixes work?

If all fixes fail, try using SD 3.5 Medium instead of Large, or the Turbo version with 4 steps. Consider upgrading your GPU, using a cloud service, or checking for the latest driver and library updates.

Editorial Team