Article · 2 min
Getting LLMs past the demo
Calling an AI API is easy. Making it a reliable production feature is the actual job — and it's ordinary software engineering, applied without exceptions.
There’s a category of software that works beautifully in a demo and quietly falls apart in production, and right now “AI features” dominate it. The gap isn’t the model. The gap is everything around the model.
At House Edge I worked on bringing LLM capabilities — the Claude API and the Model Context Protocol — into production applications. My ownership was the application-layer integration and the production software engineering around it; AI specialists owned prompting strategies and model behavior. That boundary turned out to be clarifying, because it made the actual engineering problem visible.
An LLM is a dependency with attitude
Treat the model as what it is: an external dependency that is nondeterministic, can fail or time out, costs money per request, and sits in the path of real user data. Write that sentence about any other dependency and the engineering requirements fall out immediately. Somehow, when the dependency is an LLM, teams forget to apply them.
The integration work that matters:
- Error handling and fallbacks. The call will fail. The feature has to degrade into something sensible, not a spinner or a stack trace.
- Validation on both sides. Inputs before the call, outputs after it. A nondeterministic dependency means the response shape is a hope, not a guarantee, until you’ve validated it.
- Logging and observability. When the feature behaves oddly in production — and it will — you need to be able to see what happened.
- Testing that accounts for variability. You can’t snapshot-test your way through nondeterminism; the testing approach has to be built for it.
- Maintainability. Models and requirements change. An integration built as a brittle one-off gets rebuilt; one built as a clean application-layer service evolves.
Same standards, no exceptions
The principle I came away with: AI has to be integrated as a product capability, not bolted on as a novelty. Observability, error handling, security, maintainability, and UX apply to an AI feature exactly as they apply to anything else. The moment a team lowers the bar “because it’s AI,” they’re building the demo, not the product.
This wasn’t my first contact with AI — I’d worked on AI chatbot projects years earlier, back when this was much less fashionable. But production LLM work is where the lesson crystallized: the hard, valuable part is the engineering around the model, not the call to it.