Skip to main content

​
GPU Setup

ForgeAI uses GPU acceleration for inference (Test module), fine-tuning (Training module), and model conversion (Convert module).

​
Auto-Detection

Go to Settings (07) to see your detected hardware:

​
llama.cpp Variants

For GGUF inference and quantization, install the appropriate llama.cpp variant:
Fastest option for NVIDIA GPUs.Requirements:
  • NVIDIA GPU (GTX 1060+ / RTX series)
  • NVIDIA drivers 515+
In Settings → llama.cpp Tools → select CUDA → DOWNLOAD & INSTALL

​
Python Environments (Training & Convert)

ForgeAI manages two separate Python virtual environments, each with GPU-aware PyTorch:

​
Training Environment

Used by the Training module for LoRA, QLoRA, SFT, DPO, and full fine-tuning:
  • NVIDIA GPU detected: PyTorch is installed with CUDA support automatically during setup
  • No GPU: CPU-only PyTorch (training will be slow)
  • Includes: transformers, peft, trl, bitsandbytes, datasets

​
Convert Environment

Used by the Convert module for SafeTensors-to-GGUF conversion and by Test for SafeTensors inference:
  • NVIDIA GPU detected: PyTorch is installed with CUDA support automatically during setup
  • No GPU: CPU-only PyTorch is installed
  • OOM fallback: If the model doesn’t fit in GPU VRAM, ForgeAI automatically falls back to CPU inference
Both environments can be managed (viewed, cleaned, deleted) in Settings.

​
VRAM Requirements

​
Inference

Approximate VRAM needed to load models on GPU:
If your model exceeds VRAM, GGUF inference via llama.cpp can offload some layers to CPU RAM. SafeTensors inference will auto-fallback to full CPU mode.

​
Training

Approximate VRAM needed for fine-tuning a 7B model:
Layer surgery (remove/duplicate layers) is pure Rust and requires no GPU or Python — it works on any system.

​
Troubleshooting