Why the system prompt matters
The system promptSystem PromptA special instruction given to a language model before the user conversation begins, establishing the model's persona, capabilities, constraints, and context.Learn more → sets the model's role, constraints, and context for the entire conversation. A well-crafted system prompt can take a general-purpose model and make it behave like a highly specialised expert that gives consistent, formatted, on-brand responses.
Most inconsistency in LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → applications comes from underspecified system prompts. If you have not told the model what tone to use, what to do when it doesn't know something, what format to respond in, and what topics to avoid — it will make its own choices, which may vary between calls.
The five components of a good system prompt
1. Role: Who is the model? 'You are a senior software engineer at a fintech startup specialising in Python and TypeScript.' Be specific — 'helpful assistant' is too generic.
2. Task context: What is the model's purpose in this application? 'You help developers debug their code, explain error messages, and suggest improvements.'
3. Response format: How should responses be structured? 'Always use markdown. For code, include the language tag. For explanations, use a numbered list of steps.'
4. Constraints: What should the model not do? 'Do not provide financial or legal advice. If asked, redirect the user to a qualified professional.' Be explicit about off-topic boundaries.
5. Fallback behaviour: What should happen when the model doesn't know? 'If you are not certain of an answer, say so clearly rather than guessing. Uncertainty is better than confident misinformation.'
Common mistakes
Being too vague: 'You are a helpful assistant' gives the model no actionable guidance. Every instruction should be specific enough that a human new to the task could follow it.
Contradictory instructions: 'Be concise but thorough' gives the model conflicting directives. Choose one: 'Responses should be under 200 words unless the user explicitly asks for a detailed explanation.'
Missing edge cases: Think about the 10% of inputs that are off-topic, ambiguous, or attempts to abuse the system. Your system promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → should specify how to handle each case.
Structured output prompts
If you need consistent JSON or structured outputStructured OutputConstraining a language model to produce output that conforms to a predefined format or schema, such as valid JSON, rather than free-form text.Learn more →, define the exact schema in the system prompt with an example: 'Always respond with a JSON object matching this schema: {"answer": string, "confidence": 0-1, "sources": string[]}. Never include any text outside the JSON object.'
For even better reliability, use the `response_format: {type: 'json_schema', json_schema: {...}}` parameter available in newer OpenAI and Anthropic APIs. This enforces schema compliance at the model output layer.
Test and iterate
Write 10–15 representative test prompts covering normal cases, edge cases, and adversarial inputs. Run them against your system prompt and check for inconsistencies. A good system prompt should produce correct behaviour for 90%+ of your test cases before you ship.
Version your system prompts like code. Small changes can have unexpected effects — keeping a changelog lets you roll back if a seemingly minor edit causes regressions.