- AT&T is betting on open AI models to control GenAI costs as usage surges from 8 billion to 45 billion tokens per day
- The operator is shifting older AI use cases from closed models to open alternatives
- Smart model routing is central to AT&T’s AI cost strategy, with cache-aware routing helping decide when to use open models or premium options
AT&T is leaning hard into open AI models from American companies as it looks to keep AI costs in check despite rising usage, pursing a “tokenomics” strategy that aims to serve 65% to 70% of its AI tokens through open models instead of pricier closed systems.
Mark Austin, VP in AT&T’s Data Office, told Fierce the operator expects to route 25% to 40% of its AI tokens through open models this year.
“We’re trying to keep our overall cost in control,” he said of its motivation. Last year AT&T was generating around 8 billion tokens per day. That figure is now upwards of 45 billion. But by leveraging open models – along with smart model routing and fine tuning – AT&T has been able to keep costs relatively flat, he said.
Opting for open models
AT&T made waves in the open AI realm with the release of its OTel 1.0 and 2.0 models earlier this year, which are fine-tuned iterations of Google’s Gemma 4 model specifically designed to serve telcos. Austin said AT&T was driven to create its own models because it found that most big large language models just didn’t speak enough telco to get things right.
The difference between the OTel 1.0 model released in March and the 2.0 version that came out in July is that the former took more of a RAG-style approach while the latter is a true question and answer model. Austin said the 2.0 model benefits from the input of 15 billion raw tokens from GSMA, giving it a better grounding in telco standards. Already OTel 1.0 has been downloaded around 28 million times and 2.0 has been downloaded around 5 million times, he said.
Austin noted that beyond wanting a more telco-specifc model, AT&T created OTel to gain more hands-on experience training on large GPU clusters, building models and fine tuning. That experience will be critical as it continues its tokenomics approach.
In addition to Google’s Gemma, Austin said AT&T has deployed Nvidia Nemotron models. It also plans to “kick the tires” on Meta’s new Glimmer model family. Asked if it has considered other options like DeepSeek or Kimi AI’s namesake model, Austin said “we look at them all,” but added “we’ve only kind of mostly focused on the American.”
Here’s where its tokenomics game gets interesting: AT&T isn’t just using open models to deploy new use cases, it’s actively shifting existing use cases to open options.
Austin noted that open models are generally considered to be six to 10 months behind proprietary frontier models, with open models increasingly closing the gap. For use cases that are over a year old, that means open models have likely caught up enough to take over. So, AT&T is scanning its 150 or so AI use cases in production to see what can be shifted to a lower-cost option.
“What we do is go across all of those and we’ll look at what are all the closed source models that I’ve deployed that are more than a year old. I should technically be able to take all of those and I should be able to shift those to open models – and possibly tweak the prompt a bit and get them to work,” he explained. “We looked at it and that’s about 25% of our tokens. So, we’re going after that to get all of those redeployed on open.”
Austin added there are a handful of use cases that AT&T has found don't require generative AI at all. “Some things you can actually build with GenAI to get there fast, but you can convert them to traditional ML afterwards and harden it,” he said.
Thus far, AT&T has found seven such use cases. Austin said these are mostly use cases involving classification of things. If the answer can only be A, B or C, you might as well “not burn tokens on that if you don’t have to,” he said.
Getting smart on model routing
The other key part of AT&T’s tokenomics strategy is model routing. The company has designed its own smart router – which lives in a LiteLLM gateway – to decide which model is needed to handle which tasks.
“It can figure out when you need to escalate [from an open model] up to a premium model,” Austin said. “Doing that smartly is key…when we’re doing routing decisions, we’re scoring it on thousands of benchmark tasks.”
AT&T’s smart router doesn’t just look at model capabilities. Austin said it also takes into account model pricing across different cloud environments as well as cache issues. The latter factor is surprisingly critical.
“When you switch models, you technically lose the cache, and the cache is one-tenth the cost,” Austin explained. “We try to be cache aware. Even though you’re on a premium model, you might be able to serve it with a cheaper one. If you have a very long set of agentic coding back and forth you’ve been doing, all that’s in your context. So, you may be better off just staying there because all of that’s in cache.”
Austin said there are off the shelf options for smart routing, but these are usually not cache aware.
Getting started on open models
Looking ahead, Austin said AT&T is actively continuing development efforts on the open model front, though he couldn’t promise future efforts would be made public like OTel was.
Asked what advice he’d give to other operators getting started on similar journeys, he said it’s best to start with the “most obvious stuff and start small and then grow from there.”
“We stack rank the tokens and the cost of all our use cases. Then we look at ‘what did we use?’ That’s how you find the opportunities,” he said. “We go through all the use cases and we figure out what’s the right strategy for it. Do we just optimize the prompt? Do we shift it to open? Do we fine tune the model? Do we stick it on the smart router and break it up to a couple models? Just coming up with your plan is probably the most important thing.”
Read more about AT&T's AI efforts here:
AT&T's AI advice: Stop 'racing from stoplight to stoplight'
Learner’s permits are the secret to AT&T’s internal AI rollout
AT&T’s network foundation models take telco ML to the next level