I started by listing every chore I wanted the agent to take off my plate before I looked at a single platform. The shortlist forced a real conversation about which jobs a chatbot can own and which ones only a person can answer.
What I Needed the Agent to Do
- Answer the same shipping, return, and product questions a support rep answers all day.
- Qualify leads who land on the pricing page and hand the warm ones to sales.
- Open tickets, pull tracking, and book a slot through tools the team already pays for.
- Stay quiet when the question lands outside its lane, instead of inventing a confident answer.
Training
Training is the part everyone underestimates. I uploaded every URL, the Notion page, the PDF return policy, and the CSV catalog, then watched the agent ace my own test questions and felt briefly proud. The first real visitor broke that feeling in about ten minutes by asking something I had not thought to ask. That was the day I learned to test against real transcripts, not the questions I wished people would ask.
Guardrails I Write Into the Prompt
The guardrails I write into the prompt are short and blunt: do not promise refunds above the published threshold, do not invent product names that are not in the catalog, do not speak for legal or medical, escalate when the customer asks for a human.
The Four-Line Prompt Cheat Sheet
The four lines I keep rewriting until they hold:
| Line | What it says | What it stops |
|---|---|---|
| Role | You are the site's support agent. | Friendly improvisation in a stranger's voice. |
| Voice | Match the brand, plain language, short sentences. | Marketing copy leaking into answers. |
| Procedure | Greet, ground in sources, act within limits, hand off when asked. | Skipping the source check on hard questions. |
| Guardrails | No refunds above threshold, no invented SKUs, no legal or medical advice. | The confident wrong answer. |
I read those four lines out loud before every deploy.
I hit publish on a Thursday morning. By lunch the agent had handled eleven chats, misread two shipping windows, and confidently invented a discount code. The fix was quieter than the launch. Going live is the first round of edits, and the rhythm I settle into now is watch the first hundred, patch what breaks, then keep watching.
What the First Day Taught Me
Three things broke that first week, and each one taught me a different lesson. A missing PDF in the training set made the shipping page send the default window for every postcode until I uploaded the exceptions PDF and re-crawled. One prompt line about not inventing promos closed the discount-code hole, because the visitor believed the code the agent made up. A Slack alert threshold I had set too low turned the channel into noise by Friday, so I tightened it to a small handful of trigger conditions. I rewrote the prompt twice, added three sources, and lowered the Slack volume.
The Patch Log I Keep on Hand
| What broke | Where it lived | The patch |
|---|---|---|
| Wrong shipping window | Source list | Uploaded the missing PDF and re-crawled |
| Invented discount code | Prompt file | Added a never-invent rule, redeployed |
| Alert fatigue | Slack connector | Tightened the trigger conditions |
How I Read the First Hundred Chats
After the first week, the next ninety chats landed cleanly. I read every escalation transcript the same way: where did the agent skip the source check, where did the guardrails hold, and where did it invent something the catalog never said. Anything that fails any of those three goes back into the prompt or the source list, and the cycle runs again.
Tests That Actually Catch Things
My test set looks like three passes, and I run them in the same order every build. Adversarial questions come first: ask the agent things the docs do not cover and confirm it says it does not know, instead of inventing a confident answer. Real customer transcripts come second: paste ten anonymized chats from last month and watch where the agent stumbles on phrasing the catalog never used. Action edge cases come third, and they are the sharpest pass of the three.
Action Edge Cases
The action pass tries to make the agent do something it should not, and confirms the guardrails hold under pressure. Try to make it refund an order above the published threshold, book a Calendly slot that does not exist, or pull another customer's invoice through Stripe. If the guardrails hold, the build ships. If any one of those three fails, the prompt and the source list both go back into the loop. The test loop that actually catches things runs against what visitors typed last month, not what I wished they would type, and that one shift is why the prompt gets shorter and the source list gets longer every week.