Data
Inference
Inference is the process of running a trained AI model to produce outputs from inputs. Inference cost dominates the operating budget of most LLM deployments and scales with input length, output length, and model size.
Inference in practice
API providers bill inference per token, and output tokens usually cost more than input tokens, so long prompts and verbose answers drive the bill. Common controls are caching repeated context, sending simple tasks to a smaller model, capping output length and setting a monthly spend limit per application. Cost and rate limits are part of every LLM API integration Remolda builds.
Related terms
- LLM (Large Language Model) · Data
- AI ROI · Strategy