Prompt engineering remains useful, especially when prompts are tested against a task rather than judged by intuition.
2
Prompt injection cannot be fully prevented with system prompts or guardrails because users can make models reinterpret instructions in unexpected ways.
3
Powerful agents need to resist malicious instructions from websites, advertisements, users, and other external inputs before they can be deployed safely at scale.
Summary
Sander Schulhoff explains prompt engineering as the process of improving a message sent to a generative AI. He separates conversational prompting from prompts built for repeatable API tasks, then reviews techniques from The Prompt Report, including chain of thought, decomposition, ensembling, in-context learning, example selection, label balance, and self-evaluation. His examples show that prompt behavior can be unstable: role prompts may hurt accuracy, example order can change results, and small edits to a prompt can cause large performance changes. He then distinguishes jailbreaking from prompt injection and compares AI security with classical security. SQL injection can be fixed by escaping input, while prompt injection attacks the model's interpretation of combined instructions. Schulhoff argues that agents exposed to the internet or physical world need agentic red teaming because malicious text can arrive through ordinary content such as ads. HackAPrompt's competitions collect these attacks for model testing and improvement.
Prompt engineering includes both conversation and repeatable system prompts
Schulhoff defines a prompt as any message sent to a generative AI, including text, images, or both. Prompt engineering is the process of improving that message. He separates conversational prompt engineering, where someone iterates with ChatGPT, Claude, Perplexity, or Cursor, from the original engineering problem of creating a prompt for a repeatable task such as binary classification. A chatbot conversation can be corrected after each answer. An API prompt may be called millions of times, so it has to work without a human saying, "No, no, that wasn't the right answer." His talk focuses mainly on the second case.
Role prompts are unreliable for accuracy-based tasks
Schulhoff challenges the common advice to tell a model that it is a math professor before asking it to solve a math problem. In The Prompt Report, his team tested many roles on mathematics benchmarks. He created both an intelligent math-professor role and an "idiot can't do math at all" role. The supposedly dumb role performed better. He says role prompting is largely useless when accuracy or F1 can be measured, although it can still help with open-ended writing or summaries. His explanation is tentative: a model that is told it is dumb may write more steps and make fewer mistakes. The follow-up research he cites also found that role prompts did not reliably improve performance.
Prompt performance depends on examples, ordering, labels, and format
Few-shot prompting has many design choices, and the research does not give one stable answer for how many examples to include. Schulhoff has seen papers recommending 10 examples, 80, or thousands, while other work reports degradation after around 40. Example order can change accuracy by roughly 50 percentage points in some results. Label balance may help classification, but matching the real-world distribution can also be reasonable, depending on the task. Incorrect labels can reduce performance, even though some research suggests models may learn the desired output structure without learning the label meanings from the examples. Common prompt formats and examples similar to the test instance may help, but each can also hurt.
Fine-tuning becomes preferable when examples cannot express the task
Schulhoff starts with prompting and switches to fine-tuning when the prompt performs poorly and enough data is available. He gives an example of extracting brand names from transcripts while excluding descriptors such as "old" or "moldy." He could not create enough examples to cover the distinctions he needed, so he considered it a better fine-tuning task. Fine-tuning and prompting can improve performance together, but he generally chooses one or the other. He also describes prompt mining, where a system searches the training corpus for common question and answer structures, such as "question: answer," then uses a familiar structure as the prompt.
Reasoning prompts help, but their visible explanations are not necessarily faithful
Chain of thought prompting asks a model to write steps before giving a final answer, often through phrases such as "Let's think step by step" or "show your work." Schulhoff says it was highly successful on mathematics and logical tasks and helped lead to reasoning models such as OpenAI's o1 and o3. He warns that the written reasoning may not describe the model's actual internal process. A model can produce a plausible sequence while solving the problem through different internal representations. The visible chain can still improve results because it spreads problem solving across more tokens. He says chain of thought is less useful, and often discouraged, with newer reasoning models.
Automated prompt optimization can beat hours of manual prompt work
For an entrapment-detection task involving social media posts about suicidal feelings, Schulhoff spent about 20 hours manually trying prompting techniques. Small changes produced strange swings in F1. Removing an email from the prompt, anonymizing names, or removing a duplicate copy could send performance sharply downward. He eventually found that a colleague using DSPy, an automated prompt engineering library, beat his manually designed prompt with much less effort. A small amount of human work on top of the automated result improved it further. His practical lesson is to understand the task and data first, then use an explicit reward or evaluation function rather than trusting a prompt that simply sounds better.
Prompt injection combines user input with instructions in a way models cannot reliably separate
Schulhoff describes AI red teaming as getting models to say or do harmful things. Jailbreaking usually persuades a model to ignore restrictions, while prompt injection involves user input overriding or altering a developer prompt in a larger system. He traces the term to attacks on a remote-work chatbot that was instructed to respond positively about remote work. Users told it to ignore its instructions and make unrelated statements, and the bot was eventually shut down. He compares this with SQL injection. SQL injection can be prevented by escaping user input, but a language model has to interpret all the text together, so it cannot reliably tell which instructions are trusted and which are attacker-controlled.
Agent security fails when external content can become an instruction
Schulhoff argues that powerful agents will need to operate outside internal tools, whether on the internet or in physical environments. A web-using agent might see malicious text in an advertisement telling it to buy a more expensive flight. A household or restaurant robot might be persuaded to throw food at someone. These examples are difficult because the external content is ordinary text that the model may treat as a legitimate instruction. He says agents cannot work at scale unless they become more resistant to adversarial inputs. HackAPrompt's approach is to collect attacks through competitions and share the data with nonprofit labs, commercial labs, and independent researchers.
"There's no prompt that you can write, no system prompt that you can write that will prevent prompt injection."1:15:21
Who should watch
You are building prompts for production classification, extraction, or other API tasks and need to test them rather than rely on prompt-writing intuition.
You are deciding whether a difficult task belongs in prompting or fine-tuning, especially when examples do not capture the distinctions you need.
You are deploying an agent that reads websites, uses tools, or acts in the physical world and need concrete reasons to run adversarial tests.
