QLoRA Fine-Tuning in 2026: Train a Local LLM with Hugging Face TRL

Hook: A local language model can already answer questions, but getting it to consistently follow your company’s output format, classify support tickets, or write in a defined style is a different problem. You do not always need to train a model from scratch. With QLoRA, you can adapt a supported open-weight model by training a small set of adapter parameters while keeping the base model quantized and frozen.

This guide walks through a reproducible supervised fine-tuning workflow using Hugging Face Transformers, TRL, PEFT, and bitsandbytes. The example adapts a small instruct model to a fictional technical-support response format. It is a practical starting point—not a claim that fine-tuning will automatically improve every task or fit on every GPU.

Who this is for: developers who already understand basic Python and want to customize an open-weight language model for a narrow, measurable task. The walkthrough assumes a Linux environment with an NVIDIA GPU and a CUDA-compatible PyTorch setup. Apple Silicon, AMD, and CPU-only systems may need a different training backend or configuration.

1. Decide whether fine-tuning is actually the right tool

Before installing a training stack, identify what you want the model to learn.

• Use fine-tuning for behaviour: a consistent tone, a specific response structure, domain-specific classification, or repeated transformations that are hard to prompt reliably.
• Use retrieval-augmented generation (RAG) for changing facts: product documentation, internal policies, prices, and source documents that need to be retrieved and cited at answer time.
• Use ordinary prompting first: if a clear instruction and a few examples solve the task, training may add unnecessary cost and maintenance.

Fine-tuning is not a dependable way to load an entire knowledge base into a model. The model can still make facts up, and newly learned behaviour can regress other capabilities. For a retrieval-first architecture, see our production RAG engineering guide and the local document Q&A walkthrough.

2. Understand the moving parts

QLoRA combines two ideas:

• Quantization: load the base model’s weights in a compact 4-bit representation to reduce memory pressure.
• Low-Rank Adaptation (LoRA): freeze the base weights and train small adapter matrices inserted into selected model layers.

During training, the base model is not fully updated; the adapter learns a task-specific change. The adapter is usually much smaller than a complete copy of the model, which makes saving and distributing it easier. Memory is still needed for activations, optimizer state, sequence length, temporary buffers, and the framework itself, so 4-bit weights do not guarantee that a model will fit on a particular GPU.

The implementation below follows the documented integration points in PEFT’s quantization guide, the TRL SFTTrainer documentation, and the PEFT LoRA reference.

3. Prepare a clean Python environment

Use a fresh virtual environment on a supported NVIDIA/CUDA machine. Install a CUDA-compatible PyTorch build that matches your driver and platform first, following the official PyTorch installation selector. Then install the training libraries:

python -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install --upgrade transformers trl peft datasets accelerate bitsandbytes
python -m pip freeze > requirements-lock.txt

On Windows PowerShell, activate the environment with .venv\Scripts\Activate.ps1. The remaining commands assume a Unix-like shell. Package APIs and GPU support evolve, so keep the lock file with your experiment and rerun the small smoke test after changing library versions. Do not blindly combine this environment with an older project that pins incompatible versions.

Check the GPU before downloading a model

python -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('GPU:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'none'); print('BF16:', torch.cuda.is_available() and torch.cuda.is_bf16_supported())"

If CUDA is unavailable, stop here and fix the PyTorch/driver setup or choose a training method supported by your hardware. Do not assume that a package installing successfully means GPU training is ready.

4. Choose a small, license-compatible base model

For a reproducible example, this article uses Qwen/Qwen2.5-1.5B-Instruct. Its model card lists the Apache-2.0 license, but you should still review the model card and license for the exact checkpoint you intend to deploy. This is a demonstration choice, not a claim that it is the best model for every language, task, or GPU.

Before selecting a base model, check:

• the model’s license and any use restrictions;
• whether its tokenizer includes a chat template;
• the language and task performance you need;
• the context length you actually require;
• the model’s hardware footprint and supported quantization path.

For a first experiment, a smaller model is easier to debug. If it cannot learn the desired behaviour from a clean, well-designed dataset, moving to a larger model may increase cost without fixing the underlying data problem.

5. Build a small, high-quality dataset

Dataset quality usually matters more than simply collecting a large number of examples. For supervised fine-tuning (SFT), each example should show the input and the ideal response. Use realistic examples, consistent answers, and a range of normal and edge cases. Remove credentials, personal data, confidential customer records, and material you do not have permission to use.

Create a directory structure like this:

qlora-project/
  data/
    train.jsonl
    eval.jsonl
  train.py
  requirements-lock.txt

Each JSONL line is one JSON object. For this conversational example, each record contains a messages array with user and assistant turns:

{"messages":[{"role":"system","content":"You are a technical support assistant. Be concise, state uncertainty, and never invent commands."},{"role":"user","content":"What does HTTP 502 mean?"},{"role":"assistant","content":"HTTP 502 Bad Gateway means a server acting as a gateway or proxy received an invalid response from an upstream server. Check upstream health, proxy logs, timeouts, and network connectivity before changing configuration."}]}

Put that example in train.jsonl only as a format illustration. Build a meaningful training set with varied examples; a single row is not enough to produce a useful adapter. Keep a separate eval.jsonl containing examples the model will not see during training.

Make the evaluation split honest

• Split examples before training, not after repeatedly tuning on the same holdout set.
• Remove exact duplicates and near-duplicates across training and evaluation data.
• Include failure cases: ambiguous requests, insufficient information, and requests where the correct response is to ask for clarification or say “I don’t know.”
• Write down what a good answer means before looking at results—for example, correct diagnosis, required fields present, no invented commands, and a clear uncertainty statement.

For production work, version the dataset and record where each example came from. If you cannot explain why an answer is correct, do not quietly promote it into training data.

6. Create the QLoRA training script

Save the following as train.py. It uses the current TRL-style SFTConfig and SFTTrainer interfaces, PEFT’s LoRA configuration, and a 4-bit bitsandbytes quantization configuration. It is a starter configuration, not a guaranteed fit for every GPU. If your installed versions expose different arguments, consult their versioned documentation rather than mixing code snippets from unrelated releases.

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
OUTPUT_DIR = "./out/qlora-support-assistant"

if not torch.cuda.is_available():
    raise RuntimeError("This example expects a CUDA-enabled NVIDIA GPU.")

use_bf16 = torch.cuda.is_bf16_supported()
compute_dtype = torch.bfloat16 if use_bf16 else torch.float16

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

data = load_dataset(
    "json",
    data_files={
        "train": "data/train.jsonl",
        "eval": "data/eval.jsonl",
    },
)

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)

lora_config = LoraConfig(
    task_type="CAUSAL_LM",
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    bias="none",
)

training_args = SFTConfig(
    output_dir=OUTPUT_DIR,
    num_train_epochs=2,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    logging_steps=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=2,
    max_length=1024,
    gradient_checkpointing=True,
    bf16=use_bf16,
    fp16=not use_bf16,
    report_to="none",
    seed=42,
)

trainer = SFTTrainer(
    model=MODEL_ID,
    args=training_args,
    train_dataset=data["train"],
    eval_dataset=data["eval"],
    processing_class=tokenizer,
    peft_config=lora_config,
    quantization_config=quantization_config,
)

trainer.train()
trainer.save_model(f"{OUTPUT_DIR}/adapter")
tokenizer.save_pretrained(f"{OUTPUT_DIR}/adapter")
print("Saved adapter to", f"{OUTPUT_DIR}/adapter")

TRL supports conversational datasets and can apply the model’s chat template during preprocessing. Chat templates are not decorative: they define the control tokens and structure the model expects. The Transformers guide explains how chat templates work. Do not manually wrap messages in a different format unless you have verified that it matches the selected model’s template.

7. Understand the important training settings

The values above are example defaults, not universal recommendations. Record your model revision, package versions, dataset version, random seed, configuration, and evaluation results for each run. That record is what makes a fine-tuning experiment reproducible.

8. Run a short smoke test before a full training job

Do not begin with thousands of examples. First use a tiny, representative subset and verify that the complete pipeline works.

1. Confirm that the base model and tokenizer load.
2. Load a handful of JSONL rows and inspect the message structure.
3. Run training for a small number of steps in a disposable output directory.
4. Confirm that the trainer produces finite loss values and writes adapter files.
5. Reload the adapter and generate a response to a held-out prompt.

A decreasing training loss only shows that the training objective is being optimized. It does not prove that the assistant is more accurate, follows the format, or generalizes to new inputs.

9. Load the adapter and generate a response

After training, use the same base model and tokenizer family to load the saved adapter. The following inference example again assumes a supported CUDA environment:

import torch
from peft import PeftModel
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
ADAPTER_DIR = "./out/qlora-support-assistant/adapter"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_DIR)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

use_bf16 = torch.cuda.is_bf16_supported()
compute_dtype = torch.bfloat16 if use_bf16 else torch.float16

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=compute_dtype,
    ),
)

model = PeftModel.from_pretrained(base_model, ADAPTER_DIR)
model.eval()

messages = [
    {"role": "system", "content": "You are a technical support assistant. Be concise and do not invent commands."},
    {"role": "user", "content": "My reverse proxy returns HTTP 502. What should I check first?"},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(
        inputs,
        max_new_tokens=220,
        do_sample=False,
    )

new_tokens = output[0][inputs.shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

If loading fails, confirm that ADAPTER_DIR contains the adapter configuration and weights, that the base model ID is correct, and that the training and inference libraries are compatible. Do not merge an adapter into the base model until you have a reason to do so and have validated the result.

10. Evaluate behaviour—not just loss

Build a small, fixed evaluation set that reflects real usage. Compare the original base model against the fine-tuned adapter on the exact same prompts. A useful evaluation matrix might include:

• Correctness: does the answer contain the right diagnosis or classification?
• Format compliance: are all required fields present and parseable?
• Calibration: does the model acknowledge missing information instead of guessing?
• Regression: did the adapter damage general instructions, refusal behaviour, or unrelated tasks?
• Operational behaviour: are response length and latency acceptable for the intended environment?

Keep examples and scoring criteria stable across runs. For repeatable local-model prompt checks, our Promptfoo and Ollama regression-testing guide explains how to turn prompts into a test suite. Add human review for subjective or high-impact tasks; automated scores can miss important failures.

11. Troubleshooting common failures

CUDA is unavailable or bitsandbytes fails to load

Check the PyTorch build, GPU driver, CUDA compatibility, and the installed bitsandbytes version. Recreate the environment from a clean dependency set if you have mixed incompatible packages. This article’s script is not a universal recipe for Apple Silicon, AMD GPUs, or CPU-only machines.

CUDA out-of-memory during training

Reduce max_length first if your data allows it, keep the per-device batch size small, enable gradient checkpointing, and close other GPU workloads. Gradient accumulation can help effective batch size but does not remove the memory required for a single long sequence. If it still fails, use a smaller model or a backend/configuration designed for your hardware.

Loss becomes NaN or spikes sharply

Inspect malformed records, empty assistant responses, extreme sequence lengths, and mixed-precision support. Lower the learning rate and run a short test on a small, known-good dataset. Do not interpret a single successful step as evidence that the whole training run is stable.

The adapter produces worse answers

Check data quality and leakage, compare with the base model, reduce the number of epochs, and review whether the training examples overemphasize one response pattern. More epochs are not automatically better. If the task needs up-to-date facts, use retrieval rather than expecting the adapter to memorize a changing corpus.

The output format is inconsistent

Include diverse valid examples and negative cases, make the target schema explicit, and validate generated output in application code. If the format must be machine-readable, use constrained decoding or a schema validator where supported instead of relying on fine-tuning alone.

12. Save, version, and deploy responsibly

• Version the experiment: save the base model ID/revision, adapter files, dataset hash, dependency lock file, configuration, and evaluation report.
• Review licensing: check the base model and dataset licenses separately. A model’s license does not grant rights to training data.
• Protect training data: avoid logging private prompts and raw examples unnecessarily. Remove secrets before training and restrict access to checkpoints.
• Validate before release: test regressions, prompt injection, unsafe output, memorization, and accidental disclosure using data appropriate to the application.
• Keep a rollback: retain the base model and previous adapter until the new version passes acceptance checks.

An adapter is not automatically private just because it is small or trained locally. Training can encode distinctive phrases or sensitive examples. Use synthetic or permissioned data where practical, and treat model artifacts as potentially sensitive until reviewed.

If you later serve the fine-tuned model alongside other local models, the deployment problem is separate from training. Our local LLM gateway guide covers a shared API layer for routing requests. For cost planning before scaling up, see cloud GPU versus LLM API economics.

13. Trade-offs: what QLoRA gives you—and what it does not

Frequently asked questions

Is QLoRA the same as full fine-tuning?

No. In QLoRA, the base weights are quantized and frozen while LoRA adapter parameters are trained. Full fine-tuning updates the base model’s parameters and generally has different compute and storage requirements.

How many examples do I need?

There is no universal minimum. A few examples can test the pipeline, but useful results depend on task complexity, example quality, diversity, and evaluation discipline. Start with a small, clean set and measure the difference before expanding it.

Will QLoRA make a small model as capable as a larger model?

Not in general. Fine-tuning can specialize behaviour, but it cannot guarantee that a small model gains the broad reasoning, factual coverage, or reliability of a larger model.

Can I use the adapter with Ollama or LM Studio?

Possibly, but the conversion and loading path depends on the runtime, model architecture, adapter format, and supported quantization workflow. Verify the runtime’s current documentation and test the converted artifact against the original adapter before deploying it. Do not assume a Transformers adapter can be copied directly into every local inference app.

Should I fine-tune for company knowledge or use RAG?

For information that changes and must be traceable to documents, start with RAG. Fine-tuning is more appropriate for repeated behaviour or transformations. Some applications use both: fine-tuning shapes response behaviour while retrieval supplies current, attributable facts.

Does a successful training run prove the model improved?

No. Training loss is only one signal. Compare the base model and adapter on held-out examples, measure task-specific criteria, review failures, and run regression tests before release.

Official documentation and references

• Hugging Face TRL: Supervised Fine-Tuning Trainer — trainer interface and supported dataset formats.
• Hugging Face PEFT: Quantization — QLoRA and 4-bit loading guidance.
• Hugging Face PEFT: LoRA reference — adapter configuration options.
• Hugging Face Transformers: Chat templates — formatting conversational data correctly.
• Hugging Face Datasets: Loading data — JSON/JSONL data loading.
• Qwen2.5-1.5B-Instruct model card — model details and license.
• PyTorch: Get Started Locally — platform-specific installation instructions.

Bottom line: QLoRA is a practical way to experiment with task-specific behaviour without updating every base-model weight, but the training code is only one part of the system. Good data, honest evaluation, reproducible environments, licensing checks, and safe deployment determine whether the adapter is useful.