unsloth-cpt

majiayu000's avatarfrom majiayu000

Strategies for continued pretraining and domain adaptation in Unsloth (triggers: continued pretraining, CPT, domain adaptation, lm_head, embed_tokens, rsLoRA, embedding_learning_rate).

⭐0stars🔀0forks📁View on GitHub🕐Updated Jan 5, 2026
[Developer Tools]

When & Why to Use This Skill

This Claude skill provides advanced strategies for Continued Pretraining (CPT) and domain adaptation using Unsloth. It optimizes Large Language Model (LLM) training by stabilizing embedding updates with Rank Stabilized LoRA (rsLoRA) and differentiated learning rates, ensuring efficient knowledge transfer to specialized domains like legal or medical fields without performance degradation.

Use Cases

  • Adapting LLMs to specialized domains (e.g., legal, medical, or technical) that require unique vocabulary and deep knowledge integration.
  • Stabilizing high-rank LoRA training (r > 64) using Rank Stabilized LoRA (rsLoRA) to maintain mathematical scaling stability.
  • Safely updating 'embed_tokens' and 'lm_head' layers by applying differentiated learning rates to prevent model degradation.
  • Preventing catastrophic forgetting during continued pretraining by identifying and targeting essential modules like gate_proj.
nameunsloth-cpt
descriptionStrategies for continued pretraining and domain adaptation in Unsloth (triggers: continued pretraining, CPT, domain adaptation, lm_head, embed_tokens, rsLoRA, embedding_learning_rate).

Overview

Unsloth-cpt provides specific optimizations for Continued Pretraining (CPT) and domain adaptation. It addresses the critical need for training embedding layers and language modeling heads while stabilizing the training process using Rank Stabilized LoRA (rsLoRA) and differentiated learning rates.

When to Use

  • When teaching a model a new language or highly specialized domain (e.g., legal, medical).
  • When updating the embed_tokens or lm_head layers.
  • When using high LoRA ranks (e.g., r=256) which can become unstable without rsLoRA.

Decision Tree

  1. Are you training on a new domain with unique vocabulary?
    • Yes: Include lm_head and embed_tokens in target_modules.
  2. Are you using a LoRA rank > 64?
    • Yes: Set use_rslora = True.
  3. Are you training embeddings?
    • Yes: Set embedding_learning_rate to 1/10th of the standard learning rate.

Workflows

  1. New Language Adaptation: Load the base model and configure get_peft_model to target lm_head, embed_tokens, and gate_proj with use_rslora = True.
  2. Stabilizing Embedding Updates: Use UnslothTrainer and set learning_rate = 5e-5 with a significantly lower embedding_learning_rate (e.g., 5e-6).
  3. Continued Finetuning from Adapters: Load existing adapters using from_pretrained and resume training on refined domain data.

Non-Obvious Insights

  • Training on lm_head and embed_tokens with the standard learning rate often degrades performance; a 2-10x smaller learning rate is required for stability.
  • Including the gate_proj matrix in LoRA modules is essential for CPT; omitting it leads to significantly faster catastrophic forgetting.
  • Rank Stabilized LoRA (rsLoRA) is mathematically necessary to maintain scaling stability when using very high ranks (r=256) for broad domain adaptation.

Evidence

  • "Blindly training on the lm_head and embed_tokens does even worse! We must use a smaller learning rate for the lm_head and embed_tokens." Source
  • "The paper showed how Llama-2 performed well on maths, but not code - this is because the lm_head & embed_tokens weren't trained." Source

Scripts

  • scripts/unsloth-cpt_tool.py: Configuration for rsLoRA and embedding learning rates.
  • scripts/unsloth-cpt_tool.js: Helper for calculating rank-scaled learning rates.

Dependencies

  • unsloth
  • torch
  • peft

References