Multimodal Prompting (Images & Vision)

Reviewed & published by Brayan K

Multimodal prompting means giving the AI more than text, most often images (and sometimes audio or video). You upload a picture and pair it with a clear instruction: analyze, extract, transcribe, or compare.

Part of the free Prompt Engineering course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

The skill is the same as always: be specific. This lesson shows how to combine images with text instructions and covers common use cases and limitations.

What You'll Learn in This Lesson

• What multimodal prompting is (images, audio, video)
• Describing what to do with an image
• Being specific about the task
• Combining an image with text instructions
• OCR, diagram, and UI / screenshot use cases
• Limitations and when to verify results

1 Image plus a specific instruction

An image alone is not a task. Pair it with a clear instruction about what to do. Same picture, two prompts:

No task, so the model guesses what you want.

Names the task and the output you want.

2 Common image use cases

In each case you combine the picture with a precise instruction about what to identify, extract, or compare.

3 Limitations to keep in mind

Image models are powerful but not perfect. They can misread blurry, low-resolution, or ambiguous pictures and miss small text or fine detail.

• Use clear, high-quality images — better input, better output.
• Verify important numbers — treat receipts, IDs, and legal or medical text as drafts to check.
• Say what to do if unreadable — ask the model to flag anything it cannot read.

Try it for real

There is deliberately no model output in this lesson, because an image reply is not deterministic and any "expected output" here would be invented. What you can check is the request. Run this on both prompts: the bare one leaves four decisions to the model — where to look, what to extract, how to shape it, and what to do when the answer is not there.

# There is no model call here, and there cannot be an "expected output" for one:
# image replies are not deterministic. What IS checkable is the request you send.
#
# A multimodal prompt still has to say what to look at, what to extract, and what
# to do when the image does not show it. This checks a prompt for those parts.

CHECKS = {
    "names the region":  ["top", "bottom", "left", "right", "label", "corner", "row", "column"],
    "names the fields":  ["extract", "return", "list", "read", "total", "date", "amount"],
    "sets the format":   ["json", "csv", "table", "one per line", "sentence"],
    "handles absence":   ["if you cannot", "not visible", "unreadable", "say", "unknown"],
}

prompts = [
    "What's in this image?",
    ("Read the receipt in the photo. From the bottom-right block, extract the total "
     "and the date. Return JSON with keys total and date. If a value is not visible, "
     "use null — do not guess."),
]

for p in prompts:
    low = p.lower()
    print(f"prompt: {p[:70]}{'...' if len(p) > 70 else ''}")
    for label, cues in CHECKS.items():
        hit = any(c in low for c in cues)
        print(f"   {'✓' if hit else '✗'} {label}")
    print()

print("The first prompt is not 'worse writing' — it delegates four decisions to the")
print("model: where to look, what to pull out, how to shape it, and what to do when")
print("the answer is not there. The second makes all four. Its reply is still not")
print("deterministic, which is exactly why the checkable thing is the request.")

# 🎯 YOUR TURN — score a multimodal prompt before you send it.

CHECKS = {
    "names the region": ["top", "bottom", "left", "right", "corner"],
    "sets the format":  ["json", "csv", "table"],
    "handles absence":  ["if you cannot", "not visible", "unknown"],
}

prompt = "From the top-left corner, read the invoice number. Return JSON."

low = prompt.lower()

# 1) A check passes when ANY of its cue phrases appears in the prompt.
passed = [label for label, cues in CHECKS.items() if ___(c in low for c in cues)]
# 👉 replace ___ with  any

# 2) The remaining checks are the decisions still left to the model.
left = [label for label in CHECKS if label ___ passed]
# 👉 replace ___ with  not in

print("specified:", passed)
print("left to the model:", left)

# ✅ Expected output:
# specified: ['names the region', 'sets the format']
# left to the model: ['handles absence']

Common Mistakes (and the fix)

• Sending an image with no instruction. — Always pair it with a specific task.
• Vague requests like 'look at this'. — Name what to analyze, extract, or compare.
• Using a blurry image. — Provide a clear, high-resolution picture.
• Trusting fine details blindly. — Verify important numbers against the original.

⏱ Test Yourself — Timed Quiz

10 quick questions, 12 seconds each. Instant feedback — beat the clock!

🎉 Lesson Complete

• ✅ Multimodal prompting adds images (and audio/video) to text
• ✅ Pair every image with a specific text instruction
• ✅ Common uses: OCR, diagrams, screenshots, comparisons
• ✅ Be specific about the task and the output format
• ✅ Verify important details; image reading is not perfect

Practice quiz

Continue this course

• Previous: Prompting AI Agents
• Next: Prompt Injection & Safety — Defend against jailbreaks and injection when prompts include untrusted text

Frequently asked questions

What does multimodal prompting mean?

Most prompting is text in, text out. Multimodal means you can also give the AI other kinds of input, most commonly images, and sometimes audio or video. You might upload a photo, a screenshot, a chart, or a scanned document and ask the model to analyze, extract, transcribe, or compare what it sees, all guided by a text instruction you send alongside the image.

How do I write a good prompt for an image?

Be specific about the task. 'Look at this' tells the model almost nothing. Instead say exactly what you want: 'Extract every line item and total from this receipt as a list,' or 'Identify the error message in this screenshot and suggest a fix.' Combining the image with a clear, specific text instruction is what produces a useful result.

What are common image use cases?

Reading text out of an image (OCR), like transcribing a handwritten note or a scanned page; analyzing a diagram or chart and explaining what it shows; reviewing a screenshot of an app or an error and suggesting next steps; and comparing two images. In each case you pair the picture with a precise instruction about what to do.

What are the limitations?

Image models can misread blurry, low-resolution, or ambiguous pictures, and they may miss fine details or small text. Treat important results, like numbers off a receipt or a medical or legal document, as a draft to verify rather than ground truth. Clear, high-quality images and specific instructions give the best results.