AI agent cost becomes hard to control when a simple Slack request turns into several model calls, searches, tool actions, and retries. A monthly token total tells an operations owner that money was spent, but not whether the workflow finished, which branch consumed the budget, or why a failed delivery was billed. That gap makes budgets reactive and leaves admins reconciling provider invoices by hand.
This runbook gives business operations platform owners a per-run cost contract. It covers admission, reservation, cache-aware accounting, retry control, outcome receipts, and reconciliation for scheduled or thread-initiated Slack work. The payoff is a defensible cost per completed outcome and a clear stop condition before a workflow overspends.
Define the unit you are buying
Start with the operational outcome, not a token ceiling. For a weekly metrics workflow, the outcome might be one report posted to the correct Slack thread with every required source marked available, partial, or failed. For a support workflow, it might be one evidence-backed draft that an authorized person can approve.
A budget attached only to the model or API key cannot distinguish those outcomes. Bind each run to:
- the Slack workspace, channel, and thread;
- the requesting user and permission snapshot;
- the workflow type and version;
- the required sources and tools;
- the maximum estimated cost;
- the approval policy for exceeding that limit;
- the final generation and delivery states.
Kipwise Agent provides the product context for this contract: work continues in Slack threads, scheduled requests can use connected tools, permissions follow the requesting user where applicable, sources are cited, and sensitive actions stay explicit. A cost control should preserve the same thread and authorization boundaries.
Choose one currency for the internal ledger. Store the original provider currency and amount as well, then convert through a dated rate owned by finance. Never mix token counts, provider currency, and converted currency in one numeric field.
Use reservation and reconciliation
Reserve estimated cost before each expensive branch, then reconcile it against actual provider usage after the branch reaches a terminal state.
A post-run alert cannot prevent overspend. The workflow needs an admission check before every model call or costly tool branch. Estimate the upper bound, reserve that amount against the run budget, and release the difference when actual usage arrives.
Reservation fails when an unknown estimate is treated as zero. A LiteLLM user report describes budget reservation being skipped when a request cost could not be estimated. That report applies to a specific implementation. The practical rule is still clear: an unknown estimate needs an explicit policy. Treating unknown as free defeats the budget.
Use one of three rules when estimation fails:
- Fail closed: stop the branch and tell the requester which price or usage field is missing.
- Use a conservative ceiling: reserve an admin-approved maximum for that model and operation.
- Require approval: show the missing estimate and ask an authorized owner to accept a bounded fallback limit.
Do not silently use zero. Do not launch the call and hope the monthly dashboard catches it.
A minimal ledger can use this shape:
run_id: run_01_COST_CONTROL
thread:
workspace_id: T_WORKSPACE
channel_id: C_OPERATIONS
thread_ts: "1789384100.001200"
requester_id: U_OPERATOR
workflow: weekly-search-report
currency: USD
budget:
hard_limit: 3.00
reserved: 0.84
actual: 0.61
steps:
- step_id: search_console_query
estimate: 0.00
reserved: 0.00
actual: 0.00
state: succeeded
- step_id: evidence_summary
estimate: 0.84
reserved: 0.84
actual: 0.61
state: succeeded
outcome:
generation: succeeded
delivery: pendingThis is an implementation recommendation, not a provider schema. The important fields are the reservation, actual charge, terminal state, and Slack run identity.
Keep the usage dimensions that change the bill
Track cache reads, cache writes, retries, and tool calls separately because a single total-token field can misstate where workflow spend came from.
The OpenTelemetry generative AI metrics specification defines token-usage and operation-duration measures. Use those common measures where they fit. Keep the provider-specific usage dimensions needed to calculate cost accurately.
Cache accounting is one example. A Langfuse issue about Anthropic cost calculation asks for separate cache-read and cache-creation token fields in OpenTelemetry generation spans. The author reports that omitting them makes cost calculation inaccurate. Operations teams should preserve those categories rather than flattening them into input tokens.
Record at least:
- uncached input tokens;
- cache creation or write tokens;
- cache read tokens;
- output tokens;
- model and provider price version;
- model call attempt number;
- tool call count and duration;
- retry reason and original attempt identity;
- reserved and actual cost for each branch.
The Langfuse token and cost tracking guide explains current usage ingestion, model definitions, and cost calculation. It is useful for observing spend, but your workflow admission rule still needs a local budget decision before the next call starts.
Tool calls deserve their own dimensions even when the tool has no direct per-call charge. A tool can return a large payload that increases the next model input, time out and trigger a retry, or fan out into several searches. Attribute the resulting model usage to the branch that caused it. That gives the workflow owner a practical optimization target.
Put the budget guard inside the run loop
A budget checked only at the beginning misses branching. Recalculate available budget before each call using this sequence:
available = hard_limit - actual_cost - active_reservations
estimate = upper_bound(next_step)
if estimate is unknown:
apply unknown_estimate_policy()
elif estimate > available:
stop_or_request_approval()
else:
reserve(estimate)
execute(next_step)
reconcile(provider_usage)A current OpenAI Agents SDK proposal requests per-run token, request, and cost limits checked before model calls. It is proposal evidence, not a current SDK guarantee. The proposed placement is sound: the guard belongs in the loop where the next action can still be denied.
Use both hard and soft limits. A soft limit can switch to a cheaper model, reduce optional searches, or ask the requester whether a lower-detail result is acceptable. A hard limit stops the run unless an authorized owner explicitly expands it. Never let the agent approve its own expansion.
The approval record should include the old limit, new limit, reason, approving user, permission check, expiry, and payload digest. If the workflow inputs or requested outcome change, invalidate the approval and estimate again.
Control retries and Slack delivery
Measure cost per delivered Slack outcome, not cost per generated answer, so failed posting and duplicate delivery cannot hide inside a successful model call.
Generation and delivery are separate terminal states. A model can finish successfully while Slack posting is delayed, rejected, rate-limited, or left uncertain after a timeout. The Slack Web API rate-limit guide documents per-method and per-workspace limits, plus special limits for message posting. Design for those constraints without regenerating the answer.
Use these states:
generation: not_started | running | succeeded | partial | failed
posting: not_attempted | pending | delivered | failed | unknown
run: running | completed | partial | failed | cancelled | budget_blockedIf posting fails, retry the delivery with the same output digest and idempotency key. Do not rerun source retrieval and generation unless evidence expired or the requester changed the task. If the posting result is unknown, reconcile Slack history before another attempt.
Charge every attempt to the same logical run while keeping attempt-level records. This lets the owner see both total cost per outcome and the branch that caused it. A run with successful generation and failed posting belongs in the failed or partial outcome denominator, not the completed one.
Work through a realistic scheduled report
Assume an operations lead schedules a Monday report with a $2.00 hard limit. The agent reads Search Console and Analytics, summarizes changes, and posts the result to a Slack thread.
The source reads have no model charge in this example. The summary call reserves $0.80 and reconciles to $0.54. A second call to explain an anomaly reserves $0.60 and costs $0.47. The draft is ready at $1.01 actual cost.
Slack posting times out. The workflow marks delivery unknown and checks the thread using the original idempotency key. It finds no message, retries only the posting operation, and records delivery as successful. The completed outcome costs $1.01, not the $1.61 of active reservations and not the cost of a needless second generation.
If the anomaly branch cannot identify the selected model's price, the workflow does not reserve zero. It pauses with $1.46 available and asks the authorized owner to approve a $0.75 conservative ceiling or skip the optional explanation. The owner can make a bounded decision before spend occurs.
Decide what to show in Slack
The run receipt should help the requester without exposing provider internals or sensitive content. Post a concise summary in the task thread:
- workflow and run ID;
- outcome and delivery status;
- budget limit, reserved amount, and reconciled actual cost;
- number of model calls, tool calls, and retries;
- blocked or skipped branches;
- approval changes;
- a restricted link to detailed traces for authorized admins.
Do not put prompts, customer messages, document bodies, credentials, or raw tool payloads in the receipt. Cost attribution needs identifiers, usage categories, and outcomes. It does not need a second copy of the business data.
Handle failure without losing the ledger
Close every reservation. A crashed worker, cancelled run, provider timeout, or missing usage response should not leave money reserved forever or mark actual cost as zero.
Apply these rules:
- Mark unavailable actual usage as
pending_reconciliation, not zero. - Release a reservation only after the branch is confirmed unexecuted or actual usage replaces it.
- Reconcile provider usage by request ID where the provider exposes one.
- Keep one run identity across retries and worker restarts.
- Block duplicate external actions independently of the cost guard.
- Alert when active reservations exceed their expected lifetime.
- Preserve price-table version and exchange-rate date for later audits.
A provider bill can arrive after the Slack workflow finishes. Run a daily or monthly reconciliation that compares internal actuals with provider totals by model, account, and period. Flag missing request IDs, unknown models, stale price definitions, currency differences, and repeated usage records. The goal is to explain the variance, not force both systems to agree by overwriting history.
Verify the controls before rollout
Test one recurring, read-only Slack workflow in a non-production channel:
- Set a low hard limit and confirm an expensive branch is blocked before execution.
- Remove a model price and confirm the unknown-estimate policy runs instead of reserving zero.
- Return cache read and cache write usage and confirm each category is priced separately.
- Force a model retry and confirm both attempts map to one logical run.
- Force Slack posting to fail and confirm generation is not repeated.
- Return an unknown posting result and confirm reconciliation happens before retry.
- Change the requested outcome after approval and confirm the old budget expansion expires.
- Crash a worker after reservation and confirm the ledger reaches a terminal or reconciliation state.
- Compare one known provider request with the internal actual-cost record.
- Verify the Slack receipt contains cost and outcome data but no sensitive source content.
- Replay the same Slack event and confirm no duplicate model or external action occurs.
- Produce a report of cost per delivered outcome, failed outcome, workflow, and requester group.
Track budget-blocked branches, unknown estimates, stale reservations, duplicate attempts, generated-but-undelivered runs, reconciliation variance, and cost per completed outcome. These measures tell an operations owner whether the guard works and where spend can be reduced without cutting useful work.
Pick one scheduled Slack workflow, define its completed outcome and hard limit, and implement reservation plus reconciliation before enabling additional tools or consequential writes.
References
- Kipwise Agent supports the thread-native, scheduled, permission-aware, source-citing product workflow used in this runbook.
- OpenTelemetry generative AI metrics supports common token-usage and operation-duration measures.
- Langfuse token and cost tracking supports current usage ingestion, model pricing definitions, and cost calculation features.
- Slack Web API rate limits supports the delivery constraints that require retry and outcome accounting.
- LiteLLM budget-reservation issue provides the practitioner-reported unknown-estimate failure mode.
- Langfuse cache-token accounting issue provides the practitioner-reported cache accounting gap.
- OpenAI Agents SDK budget-guard proposal provides practitioner demand for per-run limits checked before model calls.
