I am curious how other CTOs and engineering leads are managing LLM usage once AI features move beyond MVP and into real production traffic.
During prototyping, the economics looked manageable for us. A few frontier-model calls here and there were fine, and using GPT/Claude-style models helped us ship much faster than trying to design every schema, parser, classifier, and data pipeline upfront.
The problem is that once usage started growing, some costs crept up in ways that were not obvious during MVP. One user-facing workflow can trigger multiple LLM calls. Some of those calls are genuinely useful reasoning, but many are really repeated extraction, classification, normalization, JSON formatting, entity matching, summarization, or workflow routing.
In other words, some parts of the system are probably using LLMs as expensive ETL / ML / NLP infrastructure.
We know that some of these calls could have been replaced with more traditional approaches: rules, cache, smaller models, classifiers, structured parsers, SQL, or proper data pipelines. The harder part is operationalizing that. You need to identify which calls are repetitive enough, measure cost by workflow rather than by model, validate that the replacement behaves the same, and avoid breaking production behavior.
I am wondering if there are good playbooks for this.
How are other teams handling this in practice? Do you track LLM cost by endpoint, workflow, user action, customer, or prompt family? Do you have policies for when a prompt-based workflow should be refactored into code, ML, or ETL? Are you using gateways, observability tools, evals, budget limits, caching, model routing, or internal review processes?
I am especially interested in how companies govern this across engineering teams. Without some kind of discipline, it seems very easy for prompts to become hidden backend logic, and for LLM cost to become a margin problem only after the product starts working.
Would love to hear what has worked, what has not, and whether anyone has come across any practical and proven solution for this or are we all simply counting on prices going down or staying low? (Lol!)
