Has cloud cracked the multicast ‘holy grail’ for exchanges?

Peter Williams recalls how the advent of IP multicast technology changed the face of market data distribution around the start of the new millennium. During that time, Williams held a series of market data engineering support roles at firms such as interdealer broker Tullett Prebon, and investment banks Credit Suisse and BNP Paribas. Back then, firms used broadcast, a legacy distribution protocol for distributing data from a source to multiple points on a local area network.

“I remember going from broadcast to multicast on the backbone. Broadcast was great, but then message rates went up and everything would fall over,” says Williams, who today is CTO at UK-based consultancy and managed services provider CJC. “So, multicast was fantastic. It was the right tool for the job at the time.”

But times change. And technology changes quickly. In the early 2000s, every company involved in the financial markets—from banks and asset managers to exchange operators and data and technology providers—invested small fortunes in buying and deploying on-premises hardware to run software in-house.

Over the past two decades, multicast has proved essential for the timely and efficient distribution of market data from exchanges to market participants. But now, as all parties start moving significant chunks of their technology infrastructures—from hardware stacks to applications—to the cloud, the ability of exchanges and cloud providers to support multicast in cloud environments has proved a barrier to adoption in some areas.

Using IP multicast allows a data source, such as an exchange, to send the same message to all recipients at the same time, rather than sending them one after the other using serial unicast, and requiring dedicated channels for each subscribing entity. Unicast is considered more reliable—with multicast sometimes affectionally dubbed “spray and pray”—though multicast has made strides in its 35-plus years of existence, and is widely used beyond finance for functions such as live video broadcast.

“Multicast solves a difficult problem—making sure that multiple recipients get the same data at precisely the same time,” to ensure that exchanges meet their obligations of distributing data fairly, says Alex Wolcough, CEO of UK-based consultancy GreenBirch.

Therefore, exchanges such as Nasdaq, CME Group and the London Stock Exchange Group (LSEG), which have all announced major cloud partnerships with Amazon Web Services, Google, and Microsoft, respectively, must either find a way to implement or replicate multicast in the cloud. Otherwise, they will have to find another technology that can achieve similar results, or keep more sensitive processes on dedicated hardware while offloading selected other areas of their businesses to the cloud.

Exchanges already use the cloud widely for storage of large datasets of market data—for example, AWS has been hosting Nasdaq’s Market Replay analytics service since 2008—and are closely investigating how to use its elastic compute power and the analytical tools built by the major cloud providers to deliver more support capabilities on-demand.

“Any function that has a timeframe of one millisecond after a trade to one day after, from surveillance to settlement—those areas are easy to move to the cloud,” Wolcough says. “Moving all the non-matching engine areas into the cloud will probably happen, but maybe matching engines will prove too hard,” not least for those who haven’t solved the multicast issue.

Multicast by any other name

Three years ago, Stephane Dubois—then CEO of web services data provider Xignite, and now chief product officer at desktop and feeds provider Quodd, which bought Xignite in February—told WatersTechnology that he was skeptical whether cloud providers and exchanges could solve the multicast challenge in a timely manner. Now, though he still predicts it will take time to fully implement, he seems cautiously optimistic that the industry is making strides toward overcoming the challenge.

“Multicast feeds are still an issue, but it’s getting easier and easier to connect to datafeeds in the cloud,” Dubois says. “It’s still a challenging issue to address, but AWS has done it, the other players will address it, and it will become a non-issue.”

Even three years ago, AWS had already built a way to emulate multicast in the cloud within its Transit Gateways—hubs that connect AWS clouds to companies’ on-premises networks.

“The only difference is how the multicast occurs and the performance of it,” says John Kain, head of worldwide financial services market development at AWS.

“If multicast is done on a network switch in a datacenter, it’s running on hardware. In the cloud we use a software process—essentially an intelligent network—and the process of copying a message to multiple participants simultaneously happens in microseconds rather than nanoseconds. But functionally, it’s still multicast,” Kain says.

The lessons we’ve learned and the results of that collaboration will benefit the industry as a whole

John Kain, AWS

However, while this way of emulating multicast is perfectly satisfactory for most use cases, the ultra-low latency requirements of a top-tier exchange required a different approach.

“For an exchange with the high-performance, low-latency characteristics of Nasdaq, that wouldn’t be an optimal solution,” Kain says. “You would notice the time to copy the messages compared to the overall performance of the exchange. If you’re trading corporate bonds or off-the-run treasuries, you probably wouldn’t notice it.”

A lot of capital markets infrastructure outside of the low-latency, highest-volume markets don’t necessarily need multicast, such as fixed income or cryptocurrency markets, or many asset management functions. But for fast-moving equities markets, exchanges need order-to-execution times in the tens of microseconds, and members need to be able to receive updates and place orders at the same time.

So in partnership with Nasdaq and datacenter operator Equinix, AWS built a “hybrid Edge” solution that saw Equinix construct a new facility next to Nasdaq’s existing datacenter in Carteret, New Jersey. That facility was then populated with AWS Outposts servers that enabled the creation of “Local Zones”—local, specialized cloud instances for a region or industry that connect to the larger, broader cloud.

“The product we launched with Nasdaq was optimized for the exchange segment and was designed to address those needs specific to financial infrastructures that require fine-grained timestamping, multicast capability, and low latency from applications down to the network card,” Kain says.

He adds that AWS is undertaking similar projects with other, unnamed exchange operators, and that Nasdaq’s configuration is not proprietary to the exchange.

“The lessons we’ve learned and the results of that collaboration will benefit the industry as a whole,” Kain says.

In addition, this summer, AWS announced an initiative with network provider Colt earlier this year that allows clients to subscribe to multicast feeds from exchanges and use them in cloud-hosted applications that are accustomed to receiving data in multicast format.

The long and winding road to the cloud

Some see the AWS–Nasdaq approach as bringing the cloud to the exchange rather than bringing the exchange to the cloud. But it’s a first step on a longer and broader migration path—albeit one that allows Nasdaq to maintain control over some elements of how the hardware is used and how much is dedicated to specific processes, such as its matching engines, Wolcough says.

And traversing that entire migration path could take several years to complete, and even longer to migrate all client connections, Quodd’s Dubois warns.

“Think about a matching engine: It’s a very complicated piece of technology. So migrating a matching engine to the cloud is a multi-year project, and you’re betting the farm on it,” he says. “Ultimately, the cost benefits will prove massive and all the exchanges will have to do it. But you have market data infrastructures feeding off those matching engines and distributing data to lots of customers. And, for example, if multicast doesn’t work in the cloud, that could hold things up.”

Those clients who subscribe to the exchanges’ data could also benefit from migrating their connectivity to the cloud, but that will likely be led by the exchanges and by their willingness to migrate datafeeds, client connectivity, and feed handlers to the cloud without any degradation in performance, and to support existing connectivity options for years to come for any technical holdouts. That could add another two to three years after migrating an exchange’s core infrastructure, Dubois estimates.

On the performance front, there are already encouraging signs. Nasdaq, for example, reported a 10% performance improvement after migrating its first matching engine to the cloud—that of its MRX options market in December last year. Since then, the exchange has migrated its Nasdaq Bond Exchange non-convertible bonds market, and plans to migrate its GemX options market (the former ISE Gemini exchange platform) to AWS’ cloud by the end of this year.

To achieve this, Nasdaq optimized its matching engine software, while AWS tuned the hardware of its Outpost servers to meet the exchange’s latency requirements.

But operating an exchange marketplace isn’t just about speed and performance: it’s also about fairness and equality. That’s where multicast has proved so critical in the past: ensuring that regulated exchange data could be distributed simultaneously and without preference (and mostly reliably) to thousands of market participants.

“It’s not just about the lowest latency; it’s about reliable latency,” Wolcough says. “So, the most important thing is reliable and predictable latency. You can have a slower matching engine, and that’s fine, so long as the latency is predictable.”

The theme of reliability and parity is one that recurs throughout any discussion about cloud and multicast.

Rohit Bhat, senior director for capital markets, digital assets and exchanges at Google Cloud, says that while multicast presented an initial roadblock, the provider has worked closely with market participants and trading venues to develop a deeper understanding of industry needs.

“We’ve discovered software- and partner-based approaches to enable applications that are dependent on multicast to leverage cloud and simplify distribution of information at scale,” Bhat says. “This is especially useful for use cases that leverage multicast but don’t have low-latency requirements, such as passive order flow. In addition, there now exist adapters, which enable popular data distribution platforms like Kafka and Google Pub/Sub to dramatically simplify operations, while gaining the benefit of cloud resilience and global connectivity.”

I don’t think we’ll ever have true multicast and matching engines in the cloud as we know it

Lee Bressler, Spring Close Advisors, formerly with Microsoft

But tackling the broader requirements of exchanges, including low-latency data distribution and trade messages is no easy task. In fact, it’s not even one task, but three, as Bhat sees it. First, exchanges need to be able to provide low-latency access to their markets. Second, that low latency must demonstrate deterministic and low variance or jitter. And third, they must be able to provide that access at scale in a fair and equal manner, similar to the exchange gateways used by firms to connect to markets today.

“Each of these requirements on their own are difficult to enable via cloud,” Bhat says. “While bolting on technologies may address some of the requirements, we have to take a thoughtful approach to address all three in order to enable efficient markets and participants.”

And because exchanges have a broad array of customers from banks to market makers with differing requirements—whether it’s being able to offer more decision support, analytics and risk management services on a single platform to simplify access, or providing drop copy and testing capabilities—there’s no one-size-fits-all approach.

“The industry has seen deployment of disparate infrastructure that could be ideal for point solutions. These types of deployments can solve one of the three requirements, but unless you solve all three, you can’t operate or make an efficient market,” Bhat says.

Technical difficulties

That disparate infrastructure creates further challenges of its own. Lee Bressler was one of the executives interviewed for WatersTechnology’s original story on exchange multicasting three years ago. The former US capital markets lead at Microsoft, who now runs Spring Close Advisors, a consultancy that helps different parties communicate about and coordinate cloud projects, acknowledges that progress has been made over the intervening years, but questions whether that progress is enough to support more exchanges initiating large migration projects.

“I’m slightly more bearish about the entire concept than many other folks,” he says. “I don’t think we’ll ever have true multicast and matching engines in the cloud as we know it.”

The nirvana of “true” multicast may not matter if existing solutions prove sufficient. “We can come up with half-measures,” Bressler says, referring to hybrid and edge cloud offerings. “But I don’t see a competing technology that gets us all of the way there, yet.”

It’s possible from a technology perspective to get hung up on purist definitions, rather than functional capabilities. To put it another way, who cares whether you can achieve multicast if you can achieve something that does what multicast does?

For example, in tests run by CJC, AWS’ Transit Gateways performed “very well compared to bare metal,” Williams says. “I’m not worried about ‘multicast’—if the cloud providers can engineer something like multicast, or a facsimile of multicast, that’s fine.”

But, while the holy grail of multicast in the cloud remains elusive, there’s definitely a sense that, three years on, as new solutions emerge to bridge the technological gaps, there are fewer explorers hunting—and with less urgency—for the holy grail. Multicast now seems less “must-have” and more “nice-to-have.” Maybe the industry will eventually get there in the end; maybe it won’t. But along the way, it can take more steps to get ever closer, and these incremental steps may even prove more beneficial.

Quodd’s Dubois, for example, says the vendor has been able to reduce the number of physical co-location facilities it rents space in as a result of using the cloud. And he’s not sweating the multicast issue.

“Technical problems tend to have a way of resolving themselves over time,” he says.

While some can afford to be whimsical about the technical challenges, exchange operators—and particularly those who sell exchange technology and matching engines to other marketplaces—have to move quickly to respond to demand and future-proof their business models.

“Nasdaq, for example, can’t afford to not be in the cloud because they’re not just an exchange; they’re an exchange technology provider, and new exchanges and their customers want to be in the cloud,” Dubois says.

It’s not just Nasdaq in that position. When Hirander Misra, CEO of exchange operator and exchange technology provider GMEX Group spoke to WatersTechnology three years ago about the prospect of exchanges running in the cloud, he noted that many crypto exchanges already ran in the cloud and were not subject to the same considerations as regulated financial exchanges, adding that the barriers to adopting the cloud were more commercial in nature than technical.

But that was before GMEX underwent its own cloud transformation, updating its technology and re-platforming acquisitions onto the same cloud environment. The program began with an initiative to update its core exchange technology, which was already more than 10 years old. For that, GMEX looked at how it could rewrite components to be cloud-native, and over the past three years did that for all components except its matching engine, Misra says.

“With a technology stack [that’s over 10 years old], the challenge is how to rewrite that to work in the cloud. So, we had the opportunity to build something cloud-native from the ground up,” Misra says.

When GMEX launched its Multihub platform-as-a-service offering in 2021, and sought to make it a many-to-many infrastructure fabric, the company realized that should also be cloud-native, so it approached AWS to help it create a scalable hub. Then a year ago, GMEX acquired the Pyctor digital assets post-trade infrastructure platform from Dutch banking group ING. Pyctor originally ran on Microsoft’s Azure cloud, but GMEX ported that to AWS’ cloud and now runs it on the same cluster as Multihub.

The third phase of GMEX’s cloud transformation will be the launch of a cloud-native variant of its matching engine.

“It lowers the cost of ownership, and from our standpoint, we can see massive cost efficiencies in the near term, especially for small and medium-sized exchanges,” Misra says. “We’re getting a lot of requests for an ‘exchange in a box.’ Time to market is quicker, it’s more configurable, and we see this as a big opportunity to white-label climate markets with a cloud-based, decentralized repository at low cost.”

For larger markets with low-latency data distribution and liquidity provision requirements, multicast is still the elephant in the room, Misra warns, but adds that starting with smaller and less liquid markets can help broader exchange groups mitigate the risks of cloud migration.

Beyond multicast

The technical challenges of how to successfully migrate to the cloud is just one issue that exchanges face: Another is “Why?”—or rather, “What do you do with the cloud once you have it?” Initial drivers for cloud migration were around moving from buying and running hardware in-house to leased services and elastic infrastructure—both essentially cost drivers. The next step, officials say, is how to turn those drivers into revenue generators and how to use the cloud to better monetize assets and services.

AWS’ Kain says the next challenge facing exchanges seeking to leverage the potential of the cloud may not be technical in nature, like multicast, but more strategic: what services to offer over and above traditional marketplace operations that will allow them to monetize the rich environments of data and analytics being generated by exchanges.

And the first step in that is to align what you want to achieve with technology to what you want to achieve on the business side for a set period.

“You may want to do this today, but what will you want to do in five years? What if you acquire a rival and triple in size?” Williams says, adding that the current generation of cloud migrations will likely be more successful than early projects where CJC witnessed companies abandon cloud projects because of unrealistic expectations and a lack of understanding and planning. “What do you want the outcome to be? Start there, then find the right solutions and technologies, and identify any gaps.”

And that may be where the debate moves beyond technical issues. Once those are no longer barriers, what remains is the human element: culture, strategy and understanding of both the technology and the markets it serves.

Key to this is finding the right balance of “industry people” who understand the financial markets and “tech people,” who understand the technology, and then finding a way for those to meet in the middle so both sides understand each other’s needs and how to achieve them, says Bressler. A large part of his business is acting as a go-between, or “translator” between systems integrators, banks, and cloud providers.

“I spent 15 years in the industry and then leading solutions at Microsoft … and I can tell you it’s very hard to hire the people you need to do that job,” Bressler says.

In some ways, while the next steps on exchanges’ cloud journeys offer greater potential, the human and strategic issues that must be overcome may prove to be even greater challenges than multicast itself.