Fine-tune a language model with Unsloth

Prerequisites

Step-by-Step

1. 1

   Install Unsloth

   Unsloth ships a single install command that pulls in compatible torch + bitsandbytes versions.

   pip install 'unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git'

2. 2

   Load with FastLanguageModel

   FastLanguageModel patches transformers under the hood. The API mirrors HuggingFace.

   from unsloth import FastLanguageModel
   model, tok = FastLanguageModel.from_pretrained('unsloth/llama-3-8b-bnb-4bit', max_seq_length=2048, load_in_4bit=True)

3. 3

   Add LoRA adapters

   get_peft_model wires up LoRA with sane defaults for the model.

   model = FastLanguageModel.get_peft_model(model, r=16, target_modules=['q_proj','k_proj','v_proj','o_proj'], lora_alpha=16, use_gradient_checkpointing='unsloth')

4. 4

   Train with SFTTrainer

   Same TRL trainer as vanilla LoRA - Unsloth slots in transparently.

   from trl import SFTTrainer, SFTConfig
   trainer = SFTTrainer(model=model, tokenizer=tok, train_dataset=ds, args=SFTConfig(output_dir='out', max_steps=200, learning_rate=2e-4, bf16=True))
   trainer.train()

5. 5

   Save in GGUF for llama.cpp

   Unsloth's save_pretrained_gguf converts directly to the format Ollama and llama.cpp consume.

   model.save_pretrained_gguf('out-gguf', tok, quantization_method='q4_k_m')

6. 6

   Run locally

   Drop the GGUF into Ollama with a Modelfile and you have a chat-ready model.

   echo 'FROM ./out-gguf/model.Q4_K_M.gguf' > Modelfile
   ollama create my-model -f Modelfile
   ollama run my-model

Common Pitfalls

What's Next

• ->Try Unsloth's continued pretraining mode for domain adaptation.
• ->Push to HuggingFace Hub with model.push_to_hub_gguf.

More Fine-tune a language model