Few-Shot, Chain of Thought, Step-by-Step, and Self-Checks; When None of These Work

Today's Goals

1. Judge, for a given task, whether to give examples, how many, and where examples most often bury a trap
2. Explain the difference between chain of thought, step-by-step, and self-checks, and write a usable prompt for each
3. Recognize three typical situations where prompting techniques fail, and name what to switch to instead

Yesterday we turned a fuzzy requirement into a work ticket with all four elements filled in. Today we add four techniques on top of that ticket, but the emphasis is not on the techniques themselves — there is no shortage of articles about those. The emphasis is on what each one solves, what it costs, and when you should not reach for it. When you have read the walkthrough and finished the lab, come back to the top of the page and check off the three goals.

Plain-Language Walkthrough

Few-shot examples: showing the model a couple of finished jobs

Recall the new colleague from Day 1, the one who can only be reached through a written ticket. Ask the new colleague to write a status report your team's way: you could describe the format in 300 words, or just staple two well-written past reports behind the ticket and say "match this." Almost everyone picks the second option, because two samples are much harder to misread than 300 words of specification.

Few-shot examples (few-shot) are exactly that move: put a few input-and-output pairs in the prompt, then hand over the real input and let the model continue the pattern. Why it works follows directly from yesterday's mental model — the model is completing text, and an example directly demonstrates what the continuation should look like, far more precisely than any description of format and edge rules. It shines on the rules that are awkward to state in words. Take "when one piece of feedback contains both a bug and a feature request, classify it by whichever the user mentioned first." One example settles that.

In code, the examples can go straight into the user message, or be assembled as alternating user and assistant turns so the model believes it has already answered this way a few times:

import Anthropic from '@anthropic-ai/sdk'
 
const client = new Anthropic()
 
// Each example covers a different case; the last one is the edge sample
// (mixed feedback goes by whichever was mentioned first).
const examples: Array<[string, string]> = [
  ['Tapping my avatar after login does nothing', 'bug'],
  ['Could you add a dark mode', 'feature'],
  ['Do you support exporting to PDF', 'question'],
  ['The list loads slowly, and I would also like to filter by tag', 'bug'],
]
 
// Assemble alternating user / assistant turns; the model treats them as prior conversation.
const shots = examples.flatMap(([input, label]) => [
  { role: 'user' as const, content: `Feedback: ${input}` },
  { role: 'assistant' as const, content: label },
])
 
const res = await client.messages.create({
  model: 'claude-sonnet-5',
  max_tokens: 10,
  system:
    'Classify user feedback as exactly one of "bug", "feature", or "question". Output only the label. If one piece of feedback covers several, go by whichever was mentioned first.',
  messages: [
    ...shots,
    {
      role: 'user',
      content:
        'Feedback: Excel export misaligns columns when a title contains a comma, and could you support CSV while you are at it?',
    },
  ],
})
console.log(res.content[0].type === 'text' ? res.content[0].text : '')

from anthropic import Anthropic
 
client = Anthropic()
 
# Each example covers a different case; the last one is the edge sample
# (mixed feedback goes by whichever was mentioned first).
examples = [
    ("Tapping my avatar after login does nothing", "bug"),
    ("Could you add a dark mode", "feature"),
    ("Do you support exporting to PDF", "question"),
    ("The list loads slowly, and I would also like to filter by tag", "bug"),
]
 
# Assemble alternating user / assistant turns; the model treats them as prior conversation.
shots = []
for text, label in examples:
    shots.append({"role": "user", "content": f"Feedback: {text}"})
    shots.append({"role": "assistant", "content": label})
 
res = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=10,
    system='Classify user feedback as exactly one of "bug", "feature", or "question". Output only the label. If one piece of feedback covers several, go by whichever was mentioned first.',
    messages=[
        *shots,
        {
            "role": "user",
            "content": "Feedback: Excel export misaligns columns when a title contains a comma, and could you support CSV while you are at it?",
        },
    ],
)
print(res.content[0].text)

Look at the fourth example. The first three only teach what each of the three labels looks like; the fourth teaches what to do when a case is mixed. It is an edge sample, put there deliberately. If your model keeps getting one particular edge case wrong, the problem is usually not too few examples — it is that none of your examples covers that case.

The traps in examples: order, count, overfitting, and bias

Examples work so well that many people's first instinct is to add more. But every example has a cost, and the cost is not only money.

Count. Every example eats context window and gets billed; five examples may run longer than your task description. More importantly, examples exist to cover distinct kinds of cases, not to pile up volume. Two examples that both show a missing required field teach the model that validation has exactly one kind. Four examples covering four kinds of bad input teach it that validation has four. The count follows from how many cases you need to cover, which is usually two to five.

Overfitting. Give too many, and the model starts imitating their surface features — length, phrasing, even punctuation habits — instead of the rule behind them. If all three of your example outputs happen to be two sentences long, the model will lean toward two sentences, even for an input that needs four.

Bias. This is the sneakiest one. If three of your four classification examples are labeled bug, the model will lean toward bug more or less regardless of the input. Keep the label distribution roughly even across examples, or at the very least know which lean you have introduced.

Order. Most models are more sensitive to the later examples, and especially to the last one. So putting the example most like your real input last, and putting the edge sample last, are both well-founded moves. Conversely, if you notice the output always looks like the final example, order is what you are seeing.

There is one more trap, an extension of yesterday's format box: an example is the strongest format signal you can send, stronger than a description. You write "no pleasantries" in the format box, and then your example output opens with "Sure, here is my analysis" — what the model learned is that a pleasantry comes first. The output half of every example must match the format box word for word: section count, code-block count, language tags. Check that before you ship the prompt.

Chain of thought: make it think before it answers

The colleague does the math in their head vs. writes out each step on the back of the ticket — you can catch the mistake if you can see the steps. Say "just tell me the total" and they compute it silently, and if they slip, you will never spot it. Say "write each step on the back of the ticket" and every line has to face the line above it, which makes a slip less likely and, when it happens, immediately visible. Chain of thought is that same request to the model: ask it to lay out its reasoning step by step and only then give the answer.

Why it works ties back to completion again. The model generates token by token, and the intermediate results it writes down become context for everything after, so each step supplies checking material for the next. When it jumps straight to a conclusion, those intermediate results flash by unwritten and never get a chance to be checked.

That makes it clear which tasks it helps: multi-step arithmetic, judgments that require ruling out distractors, decisions that trade off several conditions. Conversely, it adds almost nothing on single-step tasks. Classification, extraction, format conversion — anything where one look gives you the answer — only gets slower, more expensive, and longer if you make the model write three paragraphs of analysis first, because that last step still faces the same single judgment.

Do the arithmetic on the cost: the output goes from one number to seven or eight lines, and both latency and price scale with output length. That is money well spent on a task where a wrong answer is hard to notice, and wasted on a task where a wrong answer is obvious at a glance. There is one more easily overlooked boundary: what the chain of thought writes down is a plausible-looking process, not the model's actual internal computation. It can help you find errors, but it is not evidence that the model really computed things that way.

One usage detail matters: require the model to put the final answer in a fixed position and a fixed format, such as "on its own final line, formatted as Answer: X dollars." Otherwise your program has to hunt for the answer inside a wall of reasoning, which is worse than parsing free text. Reasoning-oriented models from the past couple of years have a thinking process built in, so in most cases you no longer need to write "think step by step" — but you still have to specify the answer's format and position.

Step-by-step and self-checks: several tickets, then a review pass

Hand the new colleague one ticket that says "review this code, fix every problem, add tests, and write a README." What happens? The review is thorough, the fixes are decent, the tests get sloppy, and the README is three lines. They are not slacking. One ticket holds only so much, and four jobs split across it leave none of them enough room. Models behave identically: one response has a length ceiling, and the more tasks you cram in, the smaller the output budget each task gets — detailed at the front, rough at the back.

Step-by-step (multi-call) solves that by splitting it into four separate tickets, four separate hand-offs — each with its own budget, and each one can hand its output to the next. Call one reviews and outputs only a problem list. Call two takes that list and fixes only the severe items. Call three writes a test for each place that changed. Call four writes the documentation. Every step has its own output budget and its own format requirements, and a later step can take an earlier step's output as its input — something a single call cannot do. That is exactly where it differs from chain of thought: chain of thought makes the model think in more detail inside one call, on the same output budget; step-by-step means several calls, so the budget multiplies and intermediate results can travel. The test to apply is whether your task is failing because the thinking is too shallow, or because it does not fit in one response.

import Anthropic from '@anthropic-ai/sdk'
 
const client = new Anthropic()
 
async function ask(system: string, user: string): Promise<string> {
  const res = await client.messages.create({
    model: 'claude-sonnet-5',
    max_tokens: 4096,
    system,
    messages: [{ role: 'user', content: user }],
  })
  return res.content[0].type === 'text' ? res.content[0].text : ''
}
 
// Step one only reviews: its own output budget, its own format.
const issues = await ask(
  'You are a code reviewer. Output only a problem list, one per line as "location: problem: severity", sorted by severity descending. Do not change the code.',
  `Review this TODO API:\n${code}`
)
 
// Step two takes step one's output as input: the essential difference from chain of thought.
const fixed = await ask(
  'You are a backend engineer. Fix only the high-severity items on the list. Output the full revised code in one code block, with no commentary.',
  `Problem list:\n${issues}\n\nOriginal code:\n${code}`
)
console.log(fixed)

from anthropic import Anthropic
 
client = Anthropic()
 
 
def ask(system: str, user: str) -> str:
    res = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=4096,
        system=system,
        messages=[{"role": "user", "content": user}],
    )
    return res.content[0].text
 
 
# Step one only reviews: its own output budget, its own format.
issues = ask(
    'You are a code reviewer. Output only a problem list, one per line as "location: problem: severity", sorted by severity descending. Do not change the code.',
    f"Review this TODO API:\n{code}",
)
 
# Step two takes step one's output as input: the essential difference from chain of thought.
fixed = ask(
    "You are a backend engineer. Fix only the high-severity items on the list. Output the full revised code in one code block, with no commentary.",
    f"Problem list:\n{issues}\n\nOriginal code:\n{code}",
)
print(fixed)

Step-by-step costs you round-trip latency, plus the work of carrying context between calls in your own code. That is precisely what Agent-style tooling automates — handing the splitting, the passing, and the choice of next step to a loop. For today, split by hand, so you can feel where each step's boundary sits.

Self-checks address a different failure. The model invents a plausible helper-function name while writing code and never looks back. A self-check writes the looking back into the task: "when you are done, go through every function name, variable name, and import path referenced in the tests and confirm it genuinely exists in the code given to you; change or remove any that do not, and say which." Two things make it work. First, name what to check — "check for problems" does nothing, "verify one by one that each reference exists" does something. Second, require the check's results as output, for example a reference-audit list, which turns the self-check from a verbal promise into a verifiable artifact. The self-check can itself be wrong, so the real backstop is still running the tests; the self-check only pushes the error rate down.

When none of these work

Now the most important part of the day, and the part almost no tutorial covers: some tasks cannot be done correctly no matter how well the prompt is written. Recognizing them is worth more than any technique, because it saves you hours of thrashing on prompt wording. There are three failure modes.

Missing knowledge. You ask the model whether the version of the validation library in your project has a known vulnerability. Its knowledge has a cutoff date, and it knows nothing about anything disclosed afterward. Your private data and your company's internal conventions are equally invisible. Worse, it may produce a vulnerability identifier in a perfectly correct format that does not exist — the stricter the format, the more convincing the invention looks. The test is one sentence: if this information first appeared yesterday, could the model possibly know it? If not, do not expect a prompt to fix it. The fix lives outside the model: paste the material into the context, or wire up a retrieval system so it can look things up on demand — which is the subject of this platform's RAG course.

Missing tools. The task requires acting on or querying the outside world: running a command, hitting a database, sending a request, reading a file. The model can only produce text, so anything a person who can only type cannot do, it cannot do either. That is the test verbatim: could a person who can only type, and cannot touch anything, complete this? If not, the model needs tools — let it output which tool to call with which arguments, have your code execute that, and feed the result back. That is tool calling, and this platform's MCP course is devoted to it.

The task itself is written wrong. The requirement contradicts itself ("do not change the endpoint, but make the response paginated"), or what you asked for is not what you want (you said "optimize performance" and what you meant was "stop timing out"). The test: would two people reading this requirement do opposite things? If yes, go fix the requirement — no prompt change will help.

The three modes share one property: none of their fixes lives at the prompt layer. That is why yesterday framed prompt engineering as systematically supplying missing context — when what is missing is not context but knowledge, tools, or a coherent requirement, the discipline has reached its boundary. There is a cheap habit for evaluation: put a few samples in your test set that the model could not possibly know, and watch whether it honestly says it does not know or manufactures a respectable-looking answer. You will use that when D4 builds a test set.

Upgrading the running example's prompt one version

Back to the task that runs through the whole course: add input validation and unit tests to the TODO API. Yesterday's ticket already has the four elements, so run it past today's ordered questions. Are examples missing? Yes — the three-section output is described in words, but the model still occasionally slips a transition sentence between sections, and two examples written strictly to the format box suppress that. The two examples should cover different kinds of bad input, say a missing required field and an invalid date, with their output halves matching the format box word for word. Is chain of thought needed? No — writing validation and tests is not multi-step reasoning, and adding it only lengthens the output. Does it fit in one response? Two jobs, four validation cases, five tests: it is close to the ceiling but it fits. If you see the test half getting sloppy, split it into two steps. Are the same factual errors recurring? Yes — the tests occasionally reference a helper function that does not exist, so add a self-check that names the references to verify and asks for the audit list as output.

The upgraded ticket is twice as long as yesterday's, and every added paragraph answers a problem you actually observed. That is the correct use of technique: observe what the output got wrong, then pick the matching technique, rather than listing the techniques first and hunting for somewhere to apply them. Tomorrow we push the ticket's format box to its limit — turning the output into structured data a program can read directly.

Source Reading

Hands-On Lab

Today's lab is document-shaped again. Open labs/prompt-engineering-5days/day-02-before-after-rewrites and do not look at the solution first:

1. Read the five "before" prompts in starter/rewrites.md. For each, decide first whether what it lacks is examples, reasoning room, splitting, or a review pass, and write that decision into the technique column.
2. Write the "after" for each case using exactly one primary technique, then add one or two sentences on why another technique would not work here.
3. After rewriting case five, run it both before and after, confirm it still gets the task wrong, and then write down which failure mode it belongs to and the fix that lives outside the prompt.
4. Fill in part two: add two few-shot examples for the running example task, then check them against D1's format box for section count, code-block count, and language tags.
5. Only once everything is filled in, open solution/rewrites.md and compare. A different choice of technique is not necessarily wrong — check whether your reasoning holds up.

Interview Questions

Today's three questions are in the question bank below, covering the cost of few-shot examples, the difference between chain of thought and step-by-step, and the boundary where prompting fails. Every analysis gives you a reusable test sentence; walk the derivation yourself before reading the answer points.

Checklist and Tomorrow

• Judge, for a given task, whether to give examples, how many, and where examples most often bury a trap
• Explain the difference between chain of thought, step-by-step, and self-checks, and write a usable prompt for each
• Recognize three typical situations where prompting techniques fail, and name what to switch to instead
• All 5 acceptance criteria of the five rewrites pass
• You can answer at least 2 of the 3 interview questions without looking at the points

Tomorrow (D3) we push the ticket's format box to the limit: instead of prose for a human to read, the model outputs structured data a program can consume directly. You will see why the sentence "please output JSON" is not enough, what a schema actually constrains, which limits each provider's structured-output switch carries, and how to fall back when parsing fails. D3 is also this course's first runnable lab — a script that extracts fixed fields out of a requirement description, with an offline mock mode so you can do it without an API key.