Selected work Case study
Team prototype · Anthropic London Hackathon · November 2023
Agent of Death: a weekend agent and the failure that mattered
When someone dies, the job of closing their accounts falls to family. Every provider handles it differently. We built a prototype that does the finding and the drafting.
- My role
- Co-builder, team of four
- Event
- Anthropic London Hackathon, 4–5 November 2023
- Team
- Gerard Cardoso, Vinayak Athavale, matsoob, Uzair Minhas
- Stack
- Claude 2, Brave Search, Playwright, FastAPI, Next.js
Frame from the team's demo recording by Gerard Cardoso. Watch the demo on Devpost
The problem
People hold dozens of subscriptions and online accounts. After a death, a relative has to discover which ones exist and then work out how each provider wants to be told: a form, a support email, a phone line, a live chat. It is slow work at a hard time.
The prototype takes a bank statement, extracts the recurring payments, and for each one either finds a support email and drafts the message, or returns the simplest set of steps to cancel. A browsing agent that could log in and cancel directly was built as an experiment and was not reliable enough to switch on.
My contribution
This was a team build and Devpost credits all four of us. My code in the public repository is on the onboarding step: a name entry on the main page whose value is passed into the prompts sent to Claude. The statement parser, cancellation chain and browsing agent were written by my teammates.
One decision
The support-email step was allowed to return nothing. If the address was not present in the search results, the model had to say so, and the product fell back to step-by-step cancellation instructions for that provider.
The team added this after seeing the failure. Our write-up put it plainly: Claude 2 was “very good at hallucinating emails”. A drafted message to an invented address looks like a finished task, and a grieving user would only find out when nothing happened.
The lesson about evaluating an agent
This failure only appears on inputs where the correct answer is “there isn't one”. A test set made only of providers with a published support email would never have surfaced it.
I now treat that as a rule for any agent evaluation. Include cases where the right output is an abstention, score them separately, and check that any extracted value appears in the retrieved source. If every case in the set has an answer, the set can't tell you what the agent does when there isn't one.
Evidence
The Devpost entry lists the team, the stack and the challenges, and carries a two-minute demo recording. The repository is public. The entry also says what was not finished: the browsing agent is described there as experimental and not live.