LLM Usage Report
The LLM Usage report answers "what are we spending AI tokens on?" It aggregates every AI call the platform makes — token counts, estimated cost, latency, and success — and lets you slice the totals by feature, model, user, project, and time.
It is a cross-project Report Builder data source, available to global admins under Administration → Reports, in the Custom Reports tab as LLM Usage. There is no per-project variant: AI calls made outside a project context (for example, connection tests from the admin AI Models page) belong to no project, and a portfolio cost report has to include them.
What Gets Counted
Every AI call records one usage row at the moment it completes — successful or failed — with the tokens consumed, the estimated provider cost, and the round-trip latency. Costs are in USD, derived from the provider's pricing for the model used; locally hosted models (Ollama) record zero cost.
Dimensions
| Dimension | Groups by |
|---|---|
| Project | The project the call was made from. Calls with no project context group under None. |
| User | The user whose action triggered the call. |
| Feature | The AI feature that made the call (Test Case Generation, Auto Tag, Writing Assistant, …). |
| Model | The model that served the call. |
| Integration | The configured AI Model integration that handled the call. Calls whose integration has since been removed group under None. |
| Outcome | Success vs. Failed. |
| Date | The day the call was made, with the builder's daily / weekly / monthly / quarterly / annual grouping. |
Metrics
| Metric | What it measures |
|---|---|
| LLM Calls | Number of calls, including failed ones. |
| Prompt Tokens | Tokens sent to the model as input. |
| Completion Tokens | Tokens generated by the model. |
| Total Tokens | Prompt + completion tokens. |
| Total Cost | Estimated provider cost in USD. Displays up to four decimal places so sub-cent groups don't collapse to $0.00. |
| Avg. Latency | Average provider round-trip time per call. |
| Success Rate (%) | Percentage of calls that completed without an error. |
| Failed Calls | Number of calls that ended in an error. |
Common Questions
- What were tokens spent on? — Dimension: Feature. Metrics: Total Tokens, Total Cost.
- How is spend trending? — Dimension: Date (monthly grouping). Metric: Total Cost.
- Who are the heaviest users? — Dimension: User. Metrics: LLM Calls, Total Cost.
- Is one model slower or flakier than another? — Dimension: Model. Metrics: Avg. Latency, Success Rate.
- Is a feature burning money on failures? — Dimensions: Feature, Outcome. Metrics: LLM Calls, Total Cost.
Notes
- Metric cells are not clickable — this report does not support the builder's drill-down.
- The Export CSV button exports the full result set, with cost as a plain number for spreadsheet use.
- Reports can be shared via Share Links. On Public and Password-Protected shares, user emails are stripped from the results.
- The admin AI Models page shows each integration's spend for the current billing period and drives budget alerts. This report is the historical view: it spans any date range and breaks spend down by feature, model, user, and project.