unsloth-fft

cuba6112's avatarfrom cuba6112

Performing full fine-tuning (FFT) in Unsloth with 100% exact weight updates and optimized gradient checkpointing. Triggers include fft, full fine-tuning, full_finetuning, exact fine-tuning, and weight updates.

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 26, 2025
[Developer Tools]

When & Why to Use This Skill

This Claude skill enables high-performance Full Fine-Tuning (FFT) of Large Language Models using the Unsloth library. It facilitates 100% exact weight updates while optimizing memory efficiency through specialized gradient checkpointing, allowing developers to perform deep model modifications and base model pre-training with significantly reduced VRAM requirements.

Use Cases

  • Base Model Pre-training: Injecting extensive domain-specific knowledge into a base model where low-rank approximations like LoRA are insufficient to capture complex patterns.
  • Memory-Constrained Training: Utilizing Unsloth's optimized gradient checkpointing to fit larger batch sizes and full model gradients on hardware with limited VRAM (e.g., training 7B models on 24GB GPUs).
  • Comprehensive Model Alignment: Modifying 100% of model weights to fundamentally alter model behavior, language style, or core reasoning capabilities for specialized enterprise applications.
  • Efficient Continued Pre-training: Scaling up training workflows on large datasets by leveraging 8-bit AdamW optimizers and Unsloth's 2x speed and memory improvements over standard Hugging Face implementations.
nameunsloth-fft
descriptionPerforming full fine-tuning (FFT) in Unsloth with 100% exact weight updates and optimized gradient checkpointing. Triggers include fft, full fine-tuning, full_finetuning, exact fine-tuning, and weight updates.

Overview

Full Fine-Tuning (FFT) in Unsloth allows for 100% exact weight updates, bypassing the low-rank approximations of LoRA. By utilizing Unsloth's optimized gradient checkpointing, FFT can fit significantly larger batch sizes while ensuring total model modification.

When to Use

  • When performing base model pre-training or continued pre-training on large datasets.
  • When model-wide behaviors need modification that adapters (LoRA) cannot fully capture.
  • When sufficient VRAM is available to handle full model gradients.

Decision Tree

  1. Do you need to modify 100% of the model weights?
    • Yes: Proceed with FFT.
    • No: Use [[unsloth-lora]].
  2. Is VRAM limited (e.g., < 24GB for a 7B model)?
    • Yes: Enable use_gradient_checkpointing = 'unsloth' and adamw_8bit.
    • No: Use standard BF16 and high batch sizes.

Workflows

Initializing Full Fine-tuning

  1. Load the model using FastLanguageModel.from_pretrained with load_in_4bit=False and load_in_8bit=False.
  2. Pass full_finetuning=True in the initialization call to unlock all weight updates.
  3. Apply the 'unsloth' gradient checkpointing via FastLanguageModel.get_peft_model(model, use_gradient_checkpointing='unsloth').

FFT Memory Management

  1. Set per_device_train_batch_size to 1 to accommodate full weight gradients.
  2. Increase gradient_accumulation_steps to simulate effective batch sizes of 16-32.
  3. Use the adamw_8bit optimizer to reduce memory consumption by optimizer states.

Non-Obvious Insights

  • Unsloth's gradient checkpointing implementation is critical for FFT, as it uses 30% less VRAM and allows for 2x larger batch sizes compared to standard Hugging Face implementations.
  • Full fine-tuning is the preferred method for base model pre-training where the objective is knowledge injection rather than task instruction.
  • FFT in Unsloth is strictly 100% exact; there is no numerical drift compared to standard training, only efficiency gains.

Evidence

  • "To enable full fine-tuning (FFT), set full_finetuning = True." Source
  • "use_gradient_checkpointing = 'unsloth' uses 30% less VRAM, fits 2x larger batch sizes!" Source

Scripts

  • scripts/unsloth-fft_tool.py: Script to initialize a full fine-tuning session with Unsloth.
  • scripts/unsloth-fft_tool.js: Node.js utility to generate FFT training parameters.

Dependencies

  • unsloth
  • torch
  • transformers

References

  • [[references/README.md]]