Many of us think about AI in terms of individual interactions, such as with customer service chatbots or copilots and other productivity tools at work. But we are moving into a different phase that has two major, parallel trends. The first is that inference is overtaking the training of models.
The second is that the most impactful AI systems will run close to where data is created – whether by devices, within infrastructure or operational environments – to interact with the physical world. Some people call this physical AI, others edge AI because interacting with the physical means embedding AI at the edge.
Cloud versus the edge
This is because AI in the cloud excels at the large-scale model training and handling heavy inferencing workloads but is entirely dependent on network connectivity and has higher latency levels.
Edge AI processes data locally, reducing reliance on the cloud and lowering latency levels. In turn, this enables faster, more resilient decision making which is essential in real-world systems such as transportation, utilities, manufacturing, mining and public safety where delays and outages are almost certainly going to have serious consequences.
They range from endangering life to damaging businesses. For example, research by IT Intelligence Consulting in 2024 found that downtime costs 90% of businesses more than $300,000 an hour. A fifth of those surveyed said an hour’s downtime costs them more than $5 million.
Fundamental changes
Real-time decision loops fundamentally change the requirements for AI. Embedded AI systems generate outputs, but they also constantly observe, decide and act. They interpret sensor data and respond in milliseconds, not seconds, and mostly operate autonomously, only escalating decisions to humans as necessary.
Autonomous operations are key to speed and consistency, underpinning the reliable, predictable performance that is essential in all kinds of safety and operational contexts. The emphasis is always on dependability and immediacy, not the size or complexity of the language model. The result, not the technology, is what matters.
Another change brought about by edge AI is the processing. Data centers are optimized for GPU acceleration, but the edge prioritizes efficiency, durability and continuous operation. CPUs are ideal for distributed, always-on deployment at the edge: their hardware is simpler and therefore more compact, they are more power efficient, and have more flexible thermal limits. They can process sensors’ data and control logic, and coordinate systems – interacting with robotics, automotive systems, autonomous machines and more – alongside the AI workloads.
Handling the fallout from hurricanes
Let’s look at a slightly futuristic example to bring all this to life. Florida has hurricanes that can result in palm fronds and other debris falling on railway lines. The debris could derail a train, some of which are up to 2 miles long and carry all sorts of sensitive cargo through major metropolitan areas. After a hurricane, safety authorities could fly a drone the length of the line to inspect the tracks using AI robotics and real-time inferencing.
The drone has to fly at a certain rate, say 100 miles an hour, starting from Key West. It must pick up and process information instantaneously, using computer vision on the device itself. The inferencing runs a very small model because the device installed on the drone has very limited memory.
The model is looking at the size of objects on the track. If anything is detected that exceeds the set threshold, the image is sent to a local data center, which has greater processing and memory resources. Here, further inferencing decides if the image shows a car or person legitimately crossing the track, which can be ignored, or a tree on the line.
If it’s the latter, the model feeds the information to another specialized model that deduces the priority for moving the tree – could the tree cause minor damage to the train but allow it to pass or is it likely to derail the train?
If it’s the latter, the situation is given priority and passed to a third agent which figures out the most efficient and effective way to get a crew to the scene, along with the right gear. Then the most appropriate crew receives the service order and is dispatched. That crew might be based in Florida or outside the state, depending on location and requirements.
This example gives some idea of how many layers of decision making are involved to carry out a deeper and deeper inspection to trigger the best course of action, passing data up the chain in real time.
The success of AI at the edge and its impact on the physical world depends on practical, efficient compute. Stake holders who gain the greatest benefit from physical AI will be those who build efficient, integrated systems that can handle real-world demands and scalability, at a viable cost, in real time. In other words, CPUs and the system intelligence they make possible.
Learn more about how Vultr VX1 Cloud Compute is designed to deliver affordable, enterprise-grade performance for modern cloud-native workloads.
The editorial staff had no role in this post's creation.