unsloth-cpt

cuba6112's avatarfrom cuba6112

Strategies for continued pretraining and domain adaptation in Unsloth (triggers: continued pretraining, CPT, domain adaptation, lm_head, embed_tokens, rsLoRA, embedding_learning_rate).

⭐0stars🔀0forks📁View on GitHub🕐Updated Dec 26, 2025
[Developer Tools]

When & Why to Use This Skill

This Claude skill provides specialized strategies and optimized workflows for Continued Pretraining (CPT) and domain adaptation using Unsloth. It focuses on stabilizing the training of embedding layers and language modeling heads through Rank Stabilized LoRA (rsLoRA) and precision learning rate management, enabling efficient knowledge transfer to specialized domains while preventing catastrophic forgetting.

Use Cases

  • New Language Adaptation: Efficiently teaching a base model a new language by targeting embed_tokens and lm_head with optimized, differentiated learning rates.
  • Domain-Specific Specialization: Adapting LLMs for highly technical fields such as legal, medical, or scientific research using specialized datasets and stabilized training techniques.
  • High-Rank LoRA Implementation: Utilizing Rank Stabilized LoRA (rsLoRA) to maintain scaling stability when using high LoRA ranks (e.g., r=256) for broad domain adaptation.
  • Embedding Layer Optimization: Preventing model degradation by applying a 2-10x smaller learning rate specifically for embedding updates compared to standard layers.
  • Catastrophic Forgetting Prevention: Enhancing model retention during CPT by including essential modules like gate_proj in the LoRA target modules.
nameunsloth-cpt
descriptionStrategies for continued pretraining and domain adaptation in Unsloth (triggers: continued pretraining, CPT, domain adaptation, lm_head, embed_tokens, rsLoRA, embedding_learning_rate).

Overview

Unsloth-cpt provides specific optimizations for Continued Pretraining (CPT) and domain adaptation. It addresses the critical need for training embedding layers and language modeling heads while stabilizing the training process using Rank Stabilized LoRA (rsLoRA) and differentiated learning rates.

When to Use

  • When teaching a model a new language or highly specialized domain (e.g., legal, medical).
  • When updating the embed_tokens or lm_head layers.
  • When using high LoRA ranks (e.g., r=256) which can become unstable without rsLoRA.

Decision Tree

  1. Are you training on a new domain with unique vocabulary?
    • Yes: Include lm_head and embed_tokens in target_modules.
  2. Are you using a LoRA rank > 64?
    • Yes: Set use_rslora = True.
  3. Are you training embeddings?
    • Yes: Set embedding_learning_rate to 1/10th of the standard learning rate.

Workflows

  1. New Language Adaptation: Load the base model and configure get_peft_model to target lm_head, embed_tokens, and gate_proj with use_rslora = True.
  2. Stabilizing Embedding Updates: Use UnslothTrainer and set learning_rate = 5e-5 with a significantly lower embedding_learning_rate (e.g., 5e-6).
  3. Continued Finetuning from Adapters: Load existing adapters using from_pretrained and resume training on refined domain data.

Non-Obvious Insights

  • Training on lm_head and embed_tokens with the standard learning rate often degrades performance; a 2-10x smaller learning rate is required for stability.
  • Including the gate_proj matrix in LoRA modules is essential for CPT; omitting it leads to significantly faster catastrophic forgetting.
  • Rank Stabilized LoRA (rsLoRA) is mathematically necessary to maintain scaling stability when using very high ranks (r=256) for broad domain adaptation.

Evidence

  • "Blindly training on the lm_head and embed_tokens does even worse! We must use a smaller learning rate for the lm_head and embed_tokens." Source
  • "The paper showed how Llama-2 performed well on maths, but not code - this is because the lm_head & embed_tokens weren't trained." Source

Scripts

  • scripts/unsloth-cpt_tool.py: Configuration for rsLoRA and embedding learning rates.
  • scripts/unsloth-cpt_tool.js: Helper for calculating rank-scaled learning rates.

Dependencies

  • unsloth
  • torch
  • peft

References