Building an LLM demo takes an afternoon. Building an LLM product takes considerably longer, and almost all of the extra time goes into problems that are invisible until real users arrive. Here are the ones we keep meeting, roughly in the order they tend to bite.
1. The cost curve is not the one in your spreadsheet
Early cost estimates are almost always built from a short prompt and a short answer. Then reality adds retrieved documents, conversation history, a system prompt that has grown to two thousand tokens through accumulated bug fixes, and users who send photographs. Input tokens dominate, and input tokens are the ones that grow silently.
The single most effective control is routing. Most queries in most products are easy. Send those to a small, fast model and reserve the expensive one for cases that genuinely need it — a classifier deciding which is a cheap call in itself. Beyond that: cache aggressively, because a stable system prompt across many requests is a large and repeated cost that caching largely eliminates; trim conversation history to what is actually load-bearing rather than sending everything; and cap output length, because an unconstrained model will happily write six paragraphs where one would do.
2. Quality drifts and nobody notices
You change a prompt to fix one bad case. It fixes that case. It also quietly breaks four others that nobody tests, because nobody has a list of what to test. Two months later the feature is measurably worse than at launch and the team cannot point to when it happened.
The fix is unglamorous and non-negotiable: an evaluation set. Fifty to two hundred real inputs with expected properties for each — not exact expected strings, which are too brittle, but assertions. Does the answer cite a source. Does it stay under the length limit. Does it refuse when it should. Does it respond in the language the user wrote in. Run it in CI on every prompt change, and treat a drop the way you treat a failing test.
Prompts belong in version control alongside the code, with the same review process. A prompt edited in a web console by whoever was on call at the time is an untracked production change to your most user-visible logic.
3. Users do not stay in one language
This one is specific to markets like India and it surprises teams every time. A user writes in Hindi, the model answers in English. Or the user writes Hinglish in Latin script and the model answers in Devanagari. Or half the reply is in one language and half in another.
Instructing the model to reply in the user's language is necessary and not sufficient. In KisanGyan we ended up adding an explicit post-generation language check, because in a voice product replying in the wrong language is not a cosmetic bug — the text-to-speech engine reads it in the wrong voice and the output is unintelligible. Detect, and if the language is wrong, regenerate with a stronger constraint rather than shipping it.
4. Retrieval is where accuracy actually lives
When a grounded system gives a wrong answer, the model is usually not the culprit. Retrieval is. It fetched the wrong three chunks, and the model faithfully answered from them.
- Log retrieved chunks with every response. Debugging generation without seeing the context is guesswork.
- Chunk on document structure — sections, headings — not on a fixed character count that cuts sentences in half.
- Combine keyword and vector search. Pure vector search is oddly bad at exact identifiers, product codes and proper nouns.
- Rerank the top candidates. Retrieving twenty and reranking to five reliably beats retrieving five.
- In multilingual products use one multilingual embedding space rather than separate per-language indices, so a question in one language can reach knowledge written in another.
5. Prompt injection is a real vulnerability, not a curiosity
The moment your model reads text it did not author — a user upload, a scraped page, an email, a review — that text can attempt to instruct it. Defences are layered, and none of them is complete on its own: mark untrusted content clearly in the prompt and instruct the model to treat it as data; never let the model's raw output trigger an irreversible action without a schema and a validation step; put human approval in front of anything consequential; and give the model's execution context the least privilege that lets it do its job.
Treat model output the way you treat user input: never trusted, always validated, never directly executed.
6. Refusals need designing
Every product has questions it should not answer. Medical dosages. Legal advice. Anything where being confidently wrong causes real harm — in our case, telling a farmer to apply a pesticide at a concentration that would damage their crop.
A refusal is a product surface. It should say what it cannot help with, why, and what to do instead — ideally routing to a human. Left to itself the model will produce a bland apology, which reads as a broken feature rather than a considered boundary. Design the refusal, put examples of it in the evaluation set, and check that it actually fires.
7. Providers have outages
Rate limits, timeouts, regional degradation, a deprecated model version. If your product's core interaction is a model call, then your uptime is bounded by someone else's. Put every provider behind an interface, keep a second provider configured and tested, and decide in advance what the degraded experience is — a cached answer, a simpler non-AI path, or an honest message. Decide it before the incident, not during it.
The short version
- Build the evaluation harness before the feature ships, not after the first complaint.
- Version prompts like code and review them like code.
- Log tokens, latency and retrieved context on every single call.
- Route cheap queries to cheap models; escalation should be the exception.
- Design refusals and language handling deliberately.
- Assume the provider will fail and decide now what happens when it does.
None of this is exotic. It is the same discipline any other production system gets — tests, observability, version control, graceful degradation. The novelty of the technology tempts teams to skip it, and the skipping is what makes AI features feel unreliable.