How to Write System Prompts for Small Local LLMs (8B vs 70B) With Copy-Paste Examples

Here is something that cost me a few frustrating evenings. I wrote a beautiful, detailed system prompt, the kind that works perfectly on ChatGPT, and pasted it into a small 8B model running on my laptop. The result? The model ignored half of it, repeated itself, and once answered in the wrong format entirely.

The prompt was not bad. It was just written for the wrong kind of model. Small local models need a different style of instructions than the giant cloud ones. Once I understood that, my results improved almost overnight. This guide shares exactly what I learned, with prompts you can copy and adjust.

Quick Answer: How to Write System Prompts for Small Models

• Keep it short. Usually 50 to 200 words is plenty for a 7B to 8B model.
• One job per prompt. "Act as a code reviewer" works better than a prompt that tries to be five personalities at once.
• Be literal. Say exactly what you want, in plain sentences. Do not hint.
• Show the format. Give one small example of the output you expect.
• Use positive instructions. "Answer in three bullet points" beats "Do not write long answers".
• Lower the temperature for factual, code or JSON tasks.

Why Small Models Behave Differently

Big models, such as 70B parameters and above, are good at reading between the lines. You can give them a vague instruction with several competing rules and they usually figure out what you meant.

Smaller models, roughly in the 3B to 8B range, have less room to juggle things. A long list of rules can get partly forgotten, especially further into a conversation. Conflicting instructions can confuse them. And a role-play style intro of several paragraphs mostly eats up your context window without improving answers.

Think of it like explaining a task to two different people. A very experienced colleague only needs a sentence. A new intern does better with clear, short, step-by-step instructions and an example.

8B vs 70B System Prompts: The Practical Differences

These are general patterns, not laws. Model quality varies a lot between families and versions, so always test with the specific model you use.

The Simple Structure I Use for Almost Every Prompt

After a lot of trial and error, I settled on a four-part skeleton. It is short enough for small models and clear enough that I can read it months later and still understand it.

1. Role: one sentence on who the model is.
2. Task: what it should do, in one or two sentences.
3. Rules: up to four short rules.
4. Output format: exactly how the answer should look.

Copy-Paste System Prompts That Work Well on Small Models

1. Python coding helper

You are a careful Python assistant.
Help the user write and fix Python code.
Rules:
- Give working code first, then a short explanation.
- Use simple, readable code with clear variable names.
- If the request is unclear, ask one question before writing code.
Output: a code block, then 2 to 3 sentences of explanation.

2. Code reviewer

You are a code reviewer.
Review the code the user gives you.
Rules:
- Point out bugs first, then style issues.
- Be specific: mention the line or function.
- Suggest a fix for every problem you list.
Output: a numbered list. If the code looks fine, say so in one sentence.

3. JSON extractor

You extract information from text and return JSON only.
Rules:
- Return valid JSON and nothing else. No explanations, no code fences.
- If a field is missing in the text, use null.
- Never invent information.
Output example:
{"name": "Asha", "city": "Meerut", "date": null}

Set the temperature low (around 0 to 0.2) for this one. Small models sometimes wrap JSON in extra text, so it is smart to validate the result in your code before using it.

4. Study buddy for students

You are a friendly tutor for engineering students.
Explain topics in simple English with everyday examples.
Rules:
- Start with a one-line definition.
- Then explain step by step.
- End with one practice question for the student.
Keep answers under 200 words unless asked for more.

5. Summarizer

You summarize text clearly and honestly.
Rules:
- Use only information from the text provided.
- Do not add opinions or outside facts.
Output: a one-sentence summary, then 3 bullet points with the key details.

6. Email writer

You write short, polite, professional emails.
Rules:
- Keep it under 120 words.
- Use a clear subject line.
- Match the tone the user asks for. Default to friendly and formal.
Output: Subject line, then the email body.

Common Mistakes That Make Small Models Ignore Your Prompt

Writing a wall of text

If your prompt is several paragraphs of backstory, trim it. Every extra sentence competes for the model's attention and takes space in the context window.

Stacking too many rules

Ten rules usually means the last few get forgotten. Pick the most important three or four. If a rule is not affecting the results, delete it.

Using negative-only instructions

"Do not be verbose" is weaker than "Answer in at most three sentences". Tell the model what to do, not only what to avoid.

Skipping an example

One small example of the format often does more than a paragraph of description. This is especially true for JSON, tables and structured lists.

Ignoring the chat template

Every model family expects its messages wrapped in a specific chat template. Tools like LM Studio and Ollama usually apply the correct one automatically for models you download normally. Problems tend to appear with manually imported models or odd formats, where the model may seem to ignore the system prompt or produce strange tokens. If that happens, check that the template matches the model family before rewriting your prompt.

Where to Put the System Prompt

• In LM Studio: paste it in the system prompt box of the chat, or save it as a preset so you can reuse it. I have a list of ready ones in my post on the best system prompts for LM Studio.
• In Ollama: add it with a SYSTEM line inside a Modelfile, or use /set system during a chat. The full steps are in my Ollama CLI cheat sheet.

Not sure which tool to use in the first place? I compared them in Ollama vs LM Studio.

A Quick Way to Test and Improve Your Prompt

1. Write the shortest version of the prompt that could work.
2. Test it with 3 to 5 different inputs, including a tricky one.
3. Change one thing at a time, so you know what made the difference.
4. If the model breaks a rule, move that rule to the end of the prompt or restate it briefly in the format section.
5. Save the version that works and give it a clear name.

This habit alone saved me more time than any clever prompt trick.

My Personal Take

For a long time I believed a good prompt meant a long prompt. Local models taught me the opposite. My best prompts today are the ones I could read aloud in twenty seconds. If a small model needs a novel-length prompt to behave, it usually means I picked the wrong model or gave it too many jobs at once.

Frequently Asked Questions

How long should a system prompt be for a small model?

For 7B to 8B models, aim for roughly 50 to 200 words with clear structure. Longer prompts can work, but they use up context and increase the chance that some instructions are ignored.

Why does my local LLM ignore the system prompt?

Common causes are a prompt that is too long or has conflicting rules, a wrong or missing chat template for the model, or a conversation that has grown so long that the early instructions drift. Shorten the prompt, check the template, and restate key rules in the output format section.

Do small models need examples in the prompt?

Usually yes, for structured outputs. A single short example of the exact format you want, such as a JSON object, often improves reliability more than extra explanation.

What temperature should I use for coding and JSON?

A low temperature, around 0 to 0.3, gives more consistent and predictable results. Higher values suit creative writing and brainstorming.

Is a system prompt for a 70B model different from an 8B model?

Larger models tolerate longer prompts, more rules and vaguer wording. Smaller models do best with short, literal, single-purpose instructions and clear output examples.

Can I use the same system prompt in LM Studio and Ollama?

Yes. The prompt text itself is portable. Only the place you paste it differs: the system prompt box or presets in LM Studio, and a Modelfile SYSTEM line or /set system in Ollama.

What is a chat template and does it matter?

A chat template defines how system, user and assistant messages are formatted for a specific model family. If it is wrong, the model may behave oddly or seem to ignore your system prompt. Downloaded models normally come with the correct template applied automatically.

What is the best small model for following instructions?

It changes often and depends on your hardware and task, so I would not trust any fixed list. Try two or three recent 7B to 8B instruction-tuned models on your own prompts and keep the one that follows your format most reliably.

Final Thoughts

Good prompting for small models is really about respect for their limits: short, clear, one job, one example. Try trimming one of your existing prompts by half today and see what happens. You might be surprised how much better it works.

If you have a prompt that gave you trouble, share it in the comments. I would love to take a look, and the best ones may become a future post.