Run the bank, change the bank: CTOs juggle needs and wants

A few years ago, a tier-one US bank invested $2 million—allocated over four years—to build an electronic trading platform. It was an efficiency play. Essentially, operations teams were having to look at five different screens to research trades. The goal was to get that number down to one—the much-desired one-stop shop of information. 

As a global head of technology at the bank tells it, a lot of money went into figuring out regulatory control functions before any could be spent on development and testing. The issue was that this new system needed to plug into multiple legacy platforms—hence the five screens—and there were concerns over data licensing and disrupting reporting processes. 

Once they got to the testing stage, they found databases using old technology became unstable when they added new technology on top. As the bank worked through these issues, “a lot of money” was spent configuring the new platform for individual products (such as non-deliverable forwards) and clients (extending corporate services to smaller clients). From there, the bank had to decommission those older platforms, with each instance creating its own ripple effects. 

But at least at the end of the day they had an all-singing, all-dancing, brand new platform… right? Well, not according to the head of technology. “Very little was delivered,” they tell WatersTechnology.

This is a scenario that every technologist can appreciate—there’s an inexorable push coming from the top to be “data-driven” and “cloud-native”. These are catchy phrases for an earnings call, but they ignore the reality of modernizing and automating legacy systems, and the technical debt they accrue.

Slow train coming

There was a time not so long ago when the largest banks and asset managers preferred to build their own platforms, rather than buy off the shelf—think Goldman Sachs and SecDB or BlackRock and Aladdin. And it was only 15 years ago that financial institutions began tinkering with the cloud in earnest, with most of the work done via a private or hybrid model. “I don’t think an extreme will ever happen. Hybrid is not a path toward full public cloud adoption. There will always be some amount of competitive edge involved with resources you have complete control over,” one technologist told WatersTechnology back in 2010.

While there’s still some truth to that statement, the needle is rapidly moving toward partnerships with the four major cloud providers in the capital markets: Amazon Web Services, Google Cloud, IBM Cloud, and Microsoft Azure.

Cloud has been the great disruptor in the 21^st century. It has ushered in the era of Big Data, software-as-a-service (and [fill-in-the-blank]-as-a-service), which gave rise to managed services and wider adoption of APIs. Cloud played a major role in democratizing AI tools and pushed firms to be more open to the ethos of open source. And, while Wall Street was once all about building platforms internally, these shifts have completely reshaped the debate around buy versus build. 

But if banks want to truly embrace these cutting-edge new tools, they first need to deal with their legacy systems. 

“Things like quantum and DLT, we very much keep an eye on. But they’re not really a big deal at all—it’s more about stability,” says the chief information officer at a second global systemically important bank (G-Sib). “How do we get away from our legacy environments and still provide a safe environment? In turn, it might cost more, but we can worry less about infrastructure failure.” 

The evolution of trading technology has been dizzying over the past decade. A little over a year ago, generative AI wasn’t a point of discussion at banks’ board meetings—now, it’s everywhere. But the CIO says the main priority for bank technology teams today is flexibility. 

The financial crisis and the deluge of regulation it brought in its wake taught a lesson—banks were too siloed; data was scattered and pinning down a dataset’s lineage was often impossible. This period also forced banks to look to vendors for help, as their own internal technology teams had been cut to the bone. “So it’s really about cloud, AI, and shifts in development. It all underlines the notion of doing more for less and reducing costs,” says the CIO. 

They add that they are hopeful that as capital markets firms increasingly rely on technology and automation, it will lead to conversations between the business stakeholders and technologists: What should be kept on-prem? What are some of the burdens of client data? How much control do you have over data if it’s off-site versus in-house? And, when it comes to budgeting, how much can be directed toward more forward-looking projects versus modernizing (or decommissioning) legacy systems?

Unfortunately, the ebb and flow of those conversations changes from year to year, they say. “If you’re in cost-reduction mode, you want technologists to be in control, right? Tell us how much to cut and have us figure it out. But when you’re in growth mode, technologists aren’t allowed on the business side; they’re told where to direct their attention. So you go through these natural cycles of cloud, on-prem, distributed, mainframe, and so on. Right now, we’re [being told what to do]—how many years until we’re back in an aligned phase? I don’t know: one year, two years, three years? At least this keeps the consultants happy.”

‘And I’m sayin’, I’m ready to run now’

“Run the organization; change the organization” is a common term used in technology circles—it’s the balancing act that takes place between day-to-day, keep-the-lights-on needs and innovative (potentially alpha-driving) tools. The chief operating officer at a third bank says the budgeting process is split roughly 50/50 between these dichotomies. 

One truth that is often not acknowledged, says the COO, is that systems described as “legacy” today were once considered innovative. Banks have dug themselves a hole in the past by getting, say, a couple decades of use out of a system, but if it was designed for a singular purpose, then the technical debt costs grew each year. As a platform nears the end of its life, technical debt can cost a firm a third of its overall software development budget, if not more, they say.

“You will see some institutions be lopsided toward one or the other [run, or change] depending on where they are on their journey. If you rewind five years, we did an incredible amount of infrastructure modernization. We had end-of-life across our fleet of compute—50,000 to 60,000 servers.” 

Right now, we’re [being told what to do]—how many years until we’re back in an aligned phase? I don’t know: one year, two years, three years? At least this keeps the consultants happy.

This led the bank to build a redundant datacenter to improve its resiliency, because the plan was to invest in more modern tools while decommissioning older ones. For its most critical workloads, it now runs a hot-hot workload configuration, where each environment actively handles requests and transactions at the same time.

“As a result, over the last couple of years the pendulum has swung as we’ve completed a lot of that [infrastructure] work, and we have made sure to swing that capacity toward modernizing our trading and risk systems,” they say. 

The CTO at a fourth tier-one bank notes, though, that getting the business to understand the importance of managing technical debt and spending budget on modernizing aging systems is often difficult. Cloud, machine learning, and cutting-edge analytics systems can improve trading profits, help firms to better manage risk, and reduce staffing costs in the back office—they’re easy sells, even if sometimes overused buzzwords. 

Technical debt, even though it can eat up millions of dollars of budgetary spend, is a boring topic for the business side of the house, much like data governance and management (more on that next week in part three of this series).

“Why I’m calling out technical debt is because we find a lot of the time for these kinds of more forward initiatives that there’s some degree of retrofitting that has to take place,” says the fourth bank technologist. “So, there are three levels, really: it’s the actual build, the trying to pre-empt those unpleasant surprises—‘Oh shit, we didn’t know! We didn’t realize we had to do that!’—and managing assumptions and whatnot. That’s your technical debt, and that’s not even including your downstream maintenance.”

Sources say that for any modernization project, technologists need to be able to articulate the drivers, cost, and ROI for these endeavors: Cost control versus business drivers? How will this help the bank to react to market shifts better, faster? How does it help the bank make money? How does it help protect money?

As the first bank technologist learned from the building of a unified trading system that hasn’t yet delivered on its intended ROI, firms should first address downstream legacy systems before creating a one-stop-shop platform; they need to build in regulatory controls so that they do not always have to readjust to new requirements, or risk disrupting current reporting workflows; and they need to have a clear picture as to how the project will create new business and improve the client experience. 

Otherwise, you’re simply building a system that will become another technical debt black hole that will need to be re-platformed at some point in the future. All while competitors are exploring new ways to, say, use AI and other innovations in the investment process.