Inference

What it is

IBM defines AI inference as using a trained model to make predictions on new data. Training builds the model once. Inference happens every time someone uses it.

How it works

1. Your input is converted to tokens and sent to the model.
2. The model computes the output on specialized hardware such as GPUs.
3. Hosted APIs usually bill per million input and output tokens. Self-hosted setups pay for hardware time instead. Services such as Cloudflare Workers AI run open-source models on the provider’s serverless GPUs, so you don’t pay for idle hardware.

A worked example, using published list prices (checked 2026-10-17). Anthropic lists Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens. A request with 5,000 input tokens and 1,000 output tokens costs about $0.01 + $0.01 = $0.02. Run it 1,000 times a day and that’s about $20 a day. Prices change often, so re-check the vendor page before budgeting.

Benefits

• Understanding inference cost helps you choose the right model tier.
• Explains latency differences between models.
• Frames the build-versus-buy choice for hosting.

Limits and risks

• Costs scale with usage, so heavy automations need budgets.
• Long contexts raise both cost and latency.
• Self-hosted inference needs capacity planning.