- Telecom data quality problems can undermine AI accuracy
- Reliable telco AI requires stronger feature engineering and model governance
- Something called "train-serve skew" is a hidden risk for telco AI
Mind the gap. That famous warning is closely associated with the London Underground, but it’s also a pertinent caution for telco AI deployments. Why? Because one of the biggest risks to real-world deployments is the gap between the data a model was trained on and the data it encounters in production, Priyank Jain, a data scientist working on AI behind the scenes at Boost Mobile, told Fierce Network.
The resulting performance gap, known as "train-serve skew," could be a silent killer for telco AI deployments, according to Jain.
“What makes it dangerous rather than merely annoying is that nothing breaks. No error, no failed job, no alert. The model just gets quietly worse,” Jain said.
GSMA’s Director of AI Technologies Louis Powell agreed train-serve skew is an "easily overlooked pitfall" for telcos.
Train-serve skew happens when the data a model is trained on differs from the data that it sees in production. There are couple different reasons this can happen. The first and most obvious is when a mainstream LLM is deployed in a telco environment.
"Frontier models are trained on internet data, which looks very different from what they meet inside an operator," Powell explained. "The challenge is that models haven't seen network-specific data formats and vendor taxonomies. Networks also run on a patchwork of systems full of workarounds and legacy code. Plus, datasets are highly complex – for example, with spectrum interference data running to over 100 columns, often with custom parameters. The result is models that don't understand telecoms and frequently hallucinate answers, which is a real risk in an enterprise setting."
But a more insidious form of train-serve skew can happen even to models trained specifically on telecom data. Jain explained telecom usage, payment and care interaction records all settle over different periods of time and can arrive late or out of order. That means some relevant data may not be captured in the training window. The result is that you can get slightly different readings (in training and in production) from what is supposed to be the same model.
Mark Austin, VP in AT&T's Data Office, told Fierce something similar: "Telco data inherently has a lot of variation, which can make production look different from testing."
What does 'train-serve skew' look like in production?
For operators developing their own models or post-training LLMs, the difference between the training and production data can be subtle. But it matters.
Here’s what it could look like in practice: a model uses a rolling count of recent customer interactions as an input. But because the model was trained on settled history, some interactions that had not yet settled may have been excluded from the training data. If the production model is reading off the same metric, but can see all interactions regardless of whether they're settled, it may give a different reading than the model showed in training.
“Both are correct readings of ‘interactions in the last seven days,’" Jain said. “They simply disagree, and they disagree most for high-contact accounts, which are exactly the ones you care about.”
Why are high-contact accounts the ones most affected? Because “they are the ones generating the most late-arriving records,” Jain explained.
The breakdown can be difficult to diagnose because the model may still look healthy overall. Jain said validation can look fine, accuracy can look fine and input distributions may not move enough to trip monitoring thresholds.
The fix, in his view, is direct comparison: computing the same feature through both the training and serving paths and checking whether the results match.
Model skew is falling through the cracks
Jain said operators are not paying enough attention to train-serve skew, data quality and feature engineering — and that the excitement around foundation models may be making the problem worse.
“There is an asymmetry in how this work gets rewarded,” Jain said. “Shipping a model gets you a demo. Verifying that your serving features match your training features gets you nothing visible.”
The challenge also cuts across organizational boundaries. Jain said the platform team may own the pipeline while the data science team owns the model, leaving responsibility for verifying outputs somewhere in between.
The threat from ignoring the issue is real. A model that performs well in a lab but degrades silently in production could undermine the customer experience, misdirect human intervention or create false confidence in automated decisions.
AT&T's Austin said there are a few things operators can do to avoid the skew.
- First, they should ensure that their training data is representative of production – that is, they should use the same source and check for seasonality and other issues that could impact the data.
- Second, he said operators can run their models in silent mode (i.e., they ingest and process data without outputs being shown to external users or resulting in final decisions) to fully validate and test the model prior to full deployment.
- And third, like Jain, he said they should test the model, both immediately after deployment and periodically to spot issues and make any adjustments needed.