A habit I’ve been building as more of our pipeline started running LLM agents, and I’m curious how people here handle it.
Build time, bundle size, test coverage — we gate all of those in CI and fail the PR when they regress. But the token/compute cost of the agent steps themselves? For a long time we only saw that on the monthly bill, well after the fact. A refactor that quietly ~5x’d the tokens a review-agent burned per PR was completely invisible until someone squinted at the invoice.
A few tool-agnostic habits that have helped:
-
Treat token cost as a CI metric, not an ops afterthought. The same way you’d fail a build that balloons bundle size, you can flag a PR whose agent step jumps in tokens/cost versus the base branch. Making it a check turns a surprise bill into a surprise review comment — much cheaper to act on.
-
Attribute cost to a diff, not a month. “This PR adds 40k tokens/run to the review agent because it dumped a 900-line fixture into context” is actionable; “we spent $X in July” is not.
-
Budget before merge, not after deploy. A cheap pre-run estimate over just the changed files catches the 10x cases before they land.
The CI-gate part is the one I still wire up by hand every time. For folks running agents in CI or pipelines: do you gate the token/dollar cost of an agent step the way you’d gate build time or bundle size — and if so, how do you wire it up? Or is it still a monthly-invoice surprise for everyone else too?
Totally fair — for a one-off question the website is unbeatable: no infra, no bill to think about. The case I’m describing is narrower: agents running unattended in a pipeline — a review bot on every PR, a nightly batch, a CI step — where there’s no human in the loop to notice that a prompt tweak quietly 5x’d the tokens per run. Interactive chat, you feel the cost immediately; automated steps, you only see it on the invoice weeks later. If you’re not running agents in automation, you genuinely don’t have this problem. :)