Home Field note
Proposed approach
Getting a startup's first AI workflow into production
This is how I would run the first few weeks with a small team that has a promising demo and no AI feature in production yet. It is a proposed approach, drawn from shipping my own products and from deploying AI inside a regulated business. I have not yet run it end to end with an outside startup.
The aim is modest on purpose: one workflow, live with real users, with a test the team can rerun and a clear owner for what comes next.
1 Choose one workflow
Founders usually arrive with five ideas. I would ask them to list each one as a sentence in this form: when X happens, the system produces Y, and Z checks it. Then pick the one that passes the most of these tests.
- It happens often. At least weekly, ideally daily. A rare workflow gives you nothing to learn from.
- A person can check the output in under a minute. If judging one answer takes an expert half an hour, you cannot afford to evaluate it yet.
- The inputs already exist. Tickets, transcripts, records, documents. If the data has to be created first, that is the project.
- A wrong answer is recoverable. A draft that a person approves is a good first workflow. An irreversible action is a poor one.
- Someone on the team owns the outcome and will look at the results every week.
Write down what is out of scope. “Drafts a reply to a billing question for an agent to approve” is a workflow. “Handles support” is a department.
2 Assemble a small, representative eval set
Thirty cases is enough to start, provided they look like real traffic. I would build the set with the workflow's owner in one sitting.
| Cases | What they are | Why |
|---|---|---|
| 15 | Ordinary requests, sampled from the last month and not hand-picked | This is what the system will mostly see |
| 8 | Awkward ones: missing fields, conflicting records, two questions in one message | Where quality differences between approaches show up |
| 4 | Cases where the right answer is “I can't tell” or “hand this to a person” | Tests abstention, which demos never exercise |
| 3 | Cases that try to push the system off course, such as instructions hidden in the input | Tests the boundary you will be asked about later |
For each case, record the input, what a good output must contain, and what it must not do. Keep ten of the thirty aside and do not look at them while you iterate. Remove anything private or replace it with a synthetic equivalent before the file leaves the company.
The four abstention cases matter more than their number suggests. At the 2023 Anthropic London Hackathon my team's agent invented support email addresses whenever the search results did not contain one. Testing only the cases that have an answer would not have caught it. The case study has the detail.
3 Define the launch-blocking failure
Before running anything, agree one sentence that describes the failure that stops the launch. It should be binary, and somebody outside the team should be able to judge it from a transcript.
- “States a fact about the customer's account that is not in the retrieved record.”
- “Shows one customer's data in a reply to another.”
- “Takes an action, such as a refund, without a person approving it.”
The threshold for this failure is zero on the test set. Everything else is a target with a number: how many outputs the owner would send unedited, the slowest acceptable response, the cost per completed task including retries. Set those with the owner before the first run so the results cannot move the goalposts.
Keep the blocking rule out of the prompt where you can. Permissions, approvals and writes belong in ordinary application code that the model cannot talk its way past. The model drafts and classifies. The code decides what is allowed to happen.
4 Hand off the next iteration
The first version is finished when someone else can improve it without you. I would leave the team with a single page and a single command.
- What ran: model, prompt and code versions, the date, and the eval file used.
- What happened: results against each target, and every failure with its trace.
- The next experiment: one change, the result you expect, and the result that would make you drop it.
- Who owns it: a named person, and the day they will rerun the set.
- How to rerun: one command that reproduces the numbers from a clean checkout.
When real users surface a new failure, it goes into the eval set as a case that same week. That habit, more than the model choice, is what keeps the second and third iterations honest.
If you are a founder working through this and want a second pair of eyes on your first workflow, message me on LinkedIn .