- AWS puts reserve failover capacity to work carrying duplicate network traffic with Multipath Resilience
- Packets are copied and sent across as many as three physically diverse routes
- A fail-open mechanism lets packets pass through even if AWS’ packet processors were to malfunction
Two minutes is too long to wait for a network to failover. So, AWS built Multipath Resilience, a new capability that sends duplicate packets across as many as three physically independent paths to keep critical traffic moving when a single route fails.
“The challenge with traditional networks is that you can build all the resiliency and functionality that you want, but when the network doesn’t fail in a clean manner you are left with detecting that as rapidly as you can and then mitigating it,” AWS VP of Network Services Rob Kennedy told Fierce.
Fixing the problem, of course, takes time. Time that customers with critical workloads can’t afford. “A couple minutes isn’t good enough when you’re running a real-time financial transaction,” he explained.
Kennedy said Multipath Resilience was born out of development work AWS did with payments vendor Stripe. When the pair discovered that existing network technology couldn’t solve for the level of resilience Stripe was after, AWS set about building a solution.
Making traffic failure-proof
Here’s how it works. When a packet is sent out of one region, rather than flowing down one primary path, it is copied and sent down three physically diverse paths. The first of those packets to reach the other side is used and the spares are discarded.
This raises a few questions: What counts as a “diverse” path? Doesn’t sending extra packets clog up the network? And how do you sort through all those packets on the other side?
According to Kennedy, AWS actually has strict guidelines around what counts – and doesn’t count – as a diverse path. If two fiber paths cross at even one intersection, that means they have a shared potential point of failure.
“We are not allowed to use those for a failure. We don’t believe that those are failover paths for each other,” he said. “We create these shared risk link groups in terms of understanding exactly where every single fiber path goes…we store all of that information and that system generates these shared risk groups which then are used by the routers in terms of programming various paths.”
Asked whether duplicating packets over so many routes comes with overhead costs, Kennedy said yes.
For instance, he noted that Multipath Resilience requires a “fleet of packet processors on either end” that handle packet copying and deduplication. Given that there are 16 of these devices at every hop, that’s a cost, he said.
And while adding copies does indeed require more capacity, Kennedy put a positive spin on it.
“The copy that’s going down the second and third path is actually the capacity that we’ve already built for failover. It’s just now being used instead of waiting to be used of a path fails,” he explained.
What Multipath Resilience can’t protect against
Fierce pressed Kennedy on the limits of Multipath Resilience, asking if there’s a failure mode that it can’t protect against. Turns out, there are. Kind of.
First, Kennedy said Multipath Resilience wouldn’t help if a region suddenly became truly isolated with no connectivity due to fiber failures. But that has never happened to date, he added.
Second, Kennedy said the system itself – those aforementioned packet processors doing all the duplicating and sorting – could fail. But even if that were to happen, he said, AWS has a “fail open mechanism.”
That just means that if there was a bug or some other cause of failure in the packet processors, the packets would simply flow through them to the host (aka, the customer instance).
“So, there is a potential that there are too many packets coming, but generally speaking , these are not super high-bandwidth use cases. They’re just super sensitive use cases,” Kennedy said.
Multipath Resilience is now generally available, with initial support spanning four North American regions: San Francisco, Northern Virginia, Oregon, and Columbus.