When to fine-tune vs prompt engineer
Fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more → is the right choice when: you have 500+ high-quality examples of the task; the task requires a consistent style or format that prompting cannot reliably produce; you need the lowest possible inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more → cost for a specialised task; or you want to inject domain knowledge not present in the base model.
Fine-tuning is overkill when: a well-crafted system promptSystem PromptA special instruction given to a language model before the user conversation begins, establishing the model's persona, capabilities, constraints, and context.Learn more → and few-shotFew-ShotA prompting technique where you include a small number of input-output examples in the prompt to demonstrate the desired task format and behavior to the model.Learn more → examples achieve 90%+ of the quality you need; you have fewer than 200 training examples; or you are still exploring the problem space. Always try prompt engineeringPrompt EngineeringThe practice of crafting inputs, instructions, examples, and formatting to guide language models toward producing better, more accurate outputs.Learn more → first.
What LoRA does
Low-Rank Adaptation (LoRALoRALow-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains small weight matrices while freezing the base model.Learn more →) fine-tunes a model by adding small trainable adapter matrices to the attention layers, rather than updating all weights. The base model is frozen; only the adapters (0.1–2% of total parametersParametersThe numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.Learn more →) are trained. This drastically reduces VRAMVRAMVideo Random Access Memory (VRAM) is the dedicated memory on graphics cards that stores model weights and the KV cache during LLM inference.Learn more → requirements and training time.
QLoRA combines LoRA with 4-bit quantisation of the base model, enabling fine-tuning of a 7B model on a single 16 GB GPU, or a 13B model on a 24 GB GPU. Unsloth implements an optimised version of QLoRA that is significantly faster than the reference implementation.
Set up Unsloth in Google Colab
Open a new Colab notebook with a T4 (free) or A100 (Pro) GPU. Install Unsloth: `!pip install 'unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git'` followed by `!pip install --no-deps trl peftPEFTParameter-Efficient Fine-Tuning (PEFT) adapts large language models by training only a small subset of parameters, dramatically reducing computational costs while maintaining performance.Learn more → accelerate bitsandbytes`.
Load a model: `from unsloth import FastLanguageModel; model, tokenizer = FastLanguageModel.from_pretrained(model_name='unsloth/Phi-4-mini-instruct', max_seq_length=2048, dtype=None, load_in_4bit=True)`. Unsloth supports Llama 3, Mistral, Phi, Gemma, and Qwen families.
Prepare your dataset
Format your data as a list of instruction-response pairs. Each example should be a dict with an `instruction` (the input) and `output` (the expected response). For chat models, use the model's chat template to format examples correctly.
Use `datasets` to load and process your data: `from datasets import Dataset; data = Dataset.from_list(your_list)`. Add a formatting function that applies the chat template and stores the result in a `text` field. Aim for at least 500 examples; 2000+ is better for reliable results.
Configure and run training
Add LoRA adapters: `model = FastLanguageModel.get_peft_model(model, r=16, target_modules=['q_proj', 'v_proj'], lora_alpha=16, lora_dropout=0)`. `r=16` is a good starting value; higher r means more parameters but better fit.
Set up the Trainer with SFTTrainer from the `trl` library: `trainer = SFTTrainer(model=model, train_dataset=dataset, dataset_text_field='text', max_seq_length=2048, args=TrainingArguments(per_device_train_batch_size=2, num_train_epochs=3, ...))`. Start training with `trainer.train()`. On a T4 GPU, 1000 examples with a 7B model takes about 20–30 minutes.
Save and deploy
Save as GGUFGGUFA file format for distributing quantized language models optimized for efficient local inference, popularized by llama.cpp.Learn more → for local deployment: `model.save_pretrained_gguf('my-model', tokenizer, quantization_method='q4_k_m')`. This exports a GGUF file you can run directly with Ollama or LM Studio.
Push to Hugging Face Hub: `model.push_to_hub_gguf('your-username/my-model', tokenizer, quantization_method='q4_k_m', tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more →='hf_...')`. Once uploaded, you can pull it with Ollama from any machine.