Nasdaq Quiet on New FPGA Feed Outages

Nasdaq OMX took its new hardware-enabled TotalView-ITCH FPGA datafeed offline for periods of time on consecutive days last week, after the feed experienced undisclosed issues and interruptions.

At 2:55pm last Tuesday, Aug. 21, Nasdaq alerted data subscribers and traders that the feed was experiencing an issue, and advised firms to use their multicast ITCH feed as a backup, later advising that the feed would likely remain offline for the remainder of the day.

At 10:23am on Wednesday, Aug. 22, Nasdaq suffered another undisclosed issue that interrupted the primary version of the FPGA feed, forcing the exchange to fail over to a secondary version of the FPGA feed shortly thereafter.

A spokesperson tells Inside Market Data that the FPGA feed was intermittent on Wednesday, prompting Nasdaq to use a backup version for reliability, but was fully restored on Thursday. The spokesperson says the exchange has identified the cause of the failures, but declines to provide specific details, citing security reasons.

Nasdaq launched the feed on Aug. 1 to provide more deterministic latency under peak message volume conditions (IMD, Aug. 20), though—according to data traffic monitoring website marketdatapeaks.com—message volumes were not excessive around the outages.

Sources familiar with FPGA usage contacted by IMD were baffled as to the cause of the problems. “FPGAs are great for dealing with the volume problems that the markets are experiencing today, with multiple marketplaces, voluminous order books, and the need to consolidate all that information back together,” says a marketing executive at one technology vendor, adding that these benefits have led to FPGAs being embraced widely for processing incoming data—though not as much for optimizing data distribution in the way that Nasdaq is using its FPGA feed. “But they are very hard to work with,” he adds, citing the complex calculations required to create the processes that will run on the FPGA processor.

In addition to being complex, FPGAs are also more rigid than working with software, says Adam Honoré, research director at Aite Group. “The problem with FPGAs is that you can’t just register a new DLL (Dynamic Link Library) or throw a new Java object out there when you need to change something. You have to code it, test it, and flash the cards, which means you can’t do anything to fix a live feed without bringing it down.  Further, it’s not easy to test in the first place, and it takes forever to expand usage,” Honoré adds.

And just as the process of coding for FPGAs takes much longer than software because the standard libraries of functions that software programmers take for granted—such as the ability to write to and read memory—don’t exist in hardware, finding out what causes problems can be equally difficult because FPGAs do not generate core dump files when an error occurs, as CPU operating systems do, says one data technologist. “Nasdaq is wise to get into the FPGA space… but it takes a long time, and can be considerably harder to identify problems,” he says. “You can’t hardware-accelerate a plate of spaghetti, which is what some platforms look like—you have to design the platform to be accelerated with very simple flows that don’t need changing constantly.”