A.I. has a discrimination problem. In banking, the consequences can be severe
Artificial intelligence algorithms are increasingly being used in financial services — but they come with some serious risks around discrimination.
Sadik Demiroz | Photodisc | Getty Images
AMSTERDAM — Artificial intelligence has a racial bias problem.
From biometric identification systems that disproportionately misidentify the faces of Black people and minorities, to applications of voice recognition software that fail to distinguish voices with distinct regional accents, AI has a lot to work on when it comes to discrimination.
And the problem of amplifying existing biases can be even more severe when it comes to banking and financial services.
Deloitte notes that AI systems are ultimately only as good as the data they’re given: Incomplete or unrepresentative datasets could limit AI’s objectivity, while biases in development teams that train such systems could perpetuate that cycle of bias.
A.I. can be dumb
Nabil Manji, head of crypto and Web3 at Worldpay by FIS, said a key thing to understand about AI products is that the strength of the technology depends a lot on the source material used to train it.
“The thing about how good an AI product is, there’s kind of two variables,” Manji told CNBC in an interview. “One is the data it has access to, and second is how good the large language model is. That’s why the data side, you see companies like Reddit and others, they’ve come out publicly and said we’re not going to allow companies to scrape our data, you’re going to have to pay us for that.”
As for financial services, Manji said a lot of the backend data systems are fragmented in different languages and formats.
“None of it is consolidated or harmonized,” he added. “That is going to cause AI-driven products to be a lot less effective in financial services than it might be in other verticals or other companies where they have uniformity and more modern systems or access to data.”
Manji suggested that blockchain, or distributed ledger technology, could serve as a way to get a clearer view of the disparate data tucked away in the cluttered systems of traditional banks.
However, he added that banks — being the heavily regulated, slow-moving institutions that they are — are unlikely to move with the same speed as their more nimble tech counterparts in adopting new AI tools.
“You’ve got Microsoft and Google, who like over the last decade or two have been seen as driving innovation. They can’t keep up with that speed. And then you think about financial services. Banks are not known for being fast,” Manji said.
Banking’s A.I. problem
Rumman Chowdhury, Twitter’s former head of machine learning ethics, transparency and accountability, said that lending is a prime example of how an AI system’s bias against marginalized communities can rear its head.
“Algorithmic discrimination is actually very tangible in lending,” Chowdhury said on a panel at Money20/20 in Amsterdam. “Chicago had a history of literally denying those [loans] to primarily Black neighborhoods.”
In the 1930s, Chicago was known for the discriminatory practice of “redlining,” in which the creditworthiness of properties was heavily determined by the racial demographics of a given neighborhood.
“There would be a giant map on the wall of all the districts in Chicago, and they would draw red lines through all of the districts that were primarily African American, and not give them loans,” she added.
“Fast forward a few decades later, and you are developing algorithms to determine the riskiness of different districts and individuals. And while you may not include the data point of someone’s race, it is implicitly picked up.”
Indeed, Angle Bush, founder of Black Women in Artificial Intelligence, an organization aiming to empower Black women in the AI sector, tells CNBC that when AI systems are specifically used for loan approval decisions, she has found that there is a risk of replicating existing biases present in historical data used to train the algorithms.
“This can result in automatic loan denials for individuals from marginalized communities, reinforcing racial or gender disparities,” Bush added.
“It is crucial for banks to acknowledge that implementing AI as a solution may inadvertently perpetuate discrimination,” she said.
Frost Li, a developer who has been working in AI and machine learning for over a decade, told CNBC that the “personalization” dimension of AI integration can also be problematic.
“What’s interesting in AI is how we select the ‘core features’ for training,” said Li, who founded and runs Loup, a company that helps online retailers integrate AI into their platforms. “Sometimes, we select features unrelated to the results we want to predict.”
When AI is applied to banking, Li says, it’s harder to identify the “culprit” in biases when everything is convoluted in the calculation.
“A good example is how many fintech startups are especially for foreigners, because a Tokyo University graduate won’t be able to get any credit cards even if he works at Google; yet a person can easily get one from community college credit union because bankers know the local schools better,” Li added.
Generative AI is not usually used for creating credit scores or in the risk-scoring of consumers.
“That is not what the tool was built for,” said Niklas Guske, chief operating officer at Taktile, a startup that helps fintechs automate decision-making.
Instead, Guske said the most powerful applications are in pre-processing unstructured data such as text files — like classifying transactions.
“Those signals can then be fed into a more traditional underwriting model,” said Guske. “Therefore, Generative AI will improve the underlying data quality for such decisions rather than replace common scoring processes.”
But it’s also difficult to prove. Apple and Goldman Sachs, for example, were accused of giving women lower limits for the Apple Card. But these claims were dismissed by the New York Department of Financial Services after the regulator found no evidence of discrimination based on sex.
The problem, according to Kim Smouter, director of anti-racism group European Network Against Racism, is that it can be challenging to substantiate whether AI-based discrimination has actually taken place.
“One of the difficulties in the mass deployment of AI,” he said, “is the opacity in how these decisions come about and what redress mechanisms exist were a racialized individual to even notice that there is discrimination.”
“Individuals have little knowledge of how AI systems work and that their individual case may, in fact, be the tip of a systems-wide iceberg. Accordingly, it’s also difficult to detect specific instances where things have gone wrong,” he added.
Smouter cited the example of the Dutch child welfare scandal, in which thousands of benefit claims were wrongfully accused of being fraudulent. The Dutch government was forced to resign after a 2020 report found that victims were “treated with an institutional bias.”
This, Smouter said, “demonstrates how quickly such disfunctions can spread and how difficult it is to prove them and get redress once they are discovered and in the meantime significant, often irreversible damage is done.”
Policing A.I.’s biases
Chowdhury says there is a need for a global regulatory body, like the United Nations, to address some of the risks surrounding AI.
Though AI has proven to be an innovative tool, some technologists and ethicists have expressed doubts about the technology’s moral and ethical soundness. Among the top worries industry insiders expressed are misinformation; racial and gender bias embedded in AI algorithms; and “hallucinations” generated by ChatGPT-like tools.
“I worry quite a bit that, due to generative AI, we are entering this post-truth world where nothing we see online is trustworthy — not any of the text, not any of the video, not any of the audio, but then how do we get our information? And how do we ensure that information has a high amount of integrity?” Chowdhury said.
Now is the time for meaningful regulation of AI to come into force — but knowing the amount of time it will take regulatory proposals like the European Union’s AI Act to take effect, some are concerned this won’t happen fast enough.
“We call upon more transparency and accountability of algorithms and how they operate and a layman’s declaration that allows individuals who are not AI experts to judge for themselves, proof of testing and publication of results, independent complaints process, periodic audits and reporting, involvement of racialized communities when tech is being designed and considered for deployment,” Smouter said.
The AI Act, the first regulatory framework of its kind, has incorporated a fundamental rights approach and concepts like redress, according to Smouter, adding that the regulation will be enforced in approximately two years.
“It would be great if this period can be shortened to make sure transparency and accountability are in the core of innovation,” he said.
Microsoft CEO Satya Nadella speaks during the Microsoft Build conference at Microsoft headquarters in Redmond, Washington, on May 21, 2024.
Jason Redmond | AFP | Getty Images
A half-century ago, childhood friends Bill Gates and Paul Allen started Microsoft from a strip mall in Albuquerque, New Mexico. Five decades and almost $3 trillion later, the company celebrates its 50th birthday on Friday from its sprawling campus in Redmond, Washington.
Now the second most valuable publicly traded company in the world, Microsoft has only had three CEOs in its history, and all of them are in attendance for the monumental event. One is current CEO Satya Nadella. The other two are Gates and Steve Ballmer, both among the 11 richest people in the world due to their Microsoft fortunes.
While Microsoft has mostly been on the ascent of late, with Nadella turning the company into a major power player in cloud computing and artificial intelligence, the birthday party lands at an awkward moment.
The company’s stock price has dropped for four consecutive months for the first time since 2009 and just suffered its steepest quarterly drop in three years. That was all before President Donald Trump’s announcement this week of sweeping tariffs, which sent the Nasdaq tumbling on Thursday and Microsoft down another 2.4%.
Cloud computing has been Microsoft’s main source of new revenue since Nadella took over from Ballmer as CEO in 2014. But the Azure cloud reported disappointing revenue in the latest quarter, a miss that finance chief Amy Hood attributed in January to power and space shortages and a sales posture that focused too much on AI. Hood said revenue growth in the current quarter will fall to 10% from 17% a year earlier
Nadella said management is refining sales incentives to maximize revenue from traditional workloads, while positioning the company to benefit from the ongoing AI boom.
“You would rather win the new than just protect the past,” Nadella told analysts on a conference call.
The past remains healthy. Microsoft still generates around one-fifth of its roughly $262 billion in annual revenue from productivity software, mostly from commercial clients. Windows makes up around 10% of sales.
Meanwhile, the company has used its massive cash pile to orchestrate its three largest acquisitions on record in a little over eight years, snapping up LinkedIn in late 2016, Nuance Communications in 2022 and Activision Blizzard in 2023, for a combined $121 billion.
“Microsoft has figured out how to stay ahead of the curve, and 50 years later, this is a company that can still be on the forefront of technology innovation,” said Soma Somasegar, a former Microsoft executive who now invests in startups at venture firm Madrona. “That’s a commendable place for the company to be in.”
When Somasegar gave up his corporate vice president position at Microsoft in 2015, the company was fresh off a $7.6 billion write-down from Ballmer’s ill-timed purchase of Nokia’s devices and services business.
Microsoft is now in a historic phase of investment. The company has built a $13.8 billion stake in OpenAI and last year spent almost $76 billion on capital expenditures and finance leases, up 83% from a year prior, partly to enable the use of AI models in the Azure cloud. In January, Nadella said Microsoft has $13 billion in annualized AI revenue, more even than OpenAI, which just closed a financing round valuing the company at $300 billion.
Microsoft’s spending spree has constrained free cash flow growth. Guggenheim analysts wrote in a note after the company’s earnings report in January, “You just have to believe in the future.”
Of the 35 Microsoft analysts tracked by FactSet, 32 recommend buying the stock, which has appreciated tenfold since Nadella became CEO. Azure has become a fearsome threat to Amazon Web Services, which pioneered the cloud market in the 2000s, and startups as well as enterprises are flocking to its cloud technology.
Winston Weinberg, CEO of legal AI startup Harvey, uses OpenAI models through Azure. Weinberg lauded Nadella’s focus on customers of all sizes.
“Satya has literally responded to emails within 15 minutes of us having a technical problem, and he’ll route it to the right person,” Weinberg said.
Still, technology is moving at an increasingly rapid pace and Microsoft’s ability to stay on top is far from guaranteed. Industry experts highlighted four key issues the company has to address as it pushes into its next half-century.
Microsoft didn’t respond to a request for comment.
Regulation
There’s some optimism that the Trump administration and a new head of the Federal Trade Commission will open up the door to the kinds of deal-making that proved very challenging during Joe Biden’s presidency, when Lina Khan headed the FTC.
But regulatory uncertainty remains.
It’s not a new risk for Microsoft. In 1995, the company paid a $46 million breakup fee to tax software maker Intuit after the Justice Department filed suit to block the proposed deal. Years later, the DOJ got Microsoft to revamp some of its practices after a landmark antitrust case.
Microsoft pushed through its largest acquisition ever, the $75 billion purchase of video game publisher Activision, during Biden’s term. But only after a protracted legal battle with the FTC.
At the very end of Biden’s time in office, the FTC opened an antitrust investigation on Microsoft. That probe is ongoing, Bloomberg reported in March.
Nadella has cultivated a relationship with Trump. In January, the two reportedly met for lunch at Trump’s Mar-a-Lago resort in Florida, alongside Tesla CEO Elon Musk.
President Donald Trump shakes hands with Microsoft CEO Satya Nadella during an American Technology Council roundtable at the White House in Washington on June 19, 2017.
Nicholas Kamm | AFP | Getty Images
The U.S. isn’t the only concern. The U.K.’s Competition and Markets Authority said in January that an independent inquiry found that “Microsoft is using its strong position in software to make it harder for AWS and Google to compete effectively for cloud customers that wish to use Microsoft software on the cloud.”
Microsoft last year committed to unbundling Teams from Microsoft 365 productivity software subscriptions globally to address concerns from the European Union’s executive arm, the European Commission.
Noncore markets
Fairly early in Microsoft’s history the company became the world’s largest software maker. And in cloud, Microsoft is the biggest challenger to AWS. Most of the company’s revenue comes from corporations, schools and governments.
But Microsoft is in other markets where its position is weaker. Those include video games, laptops and search advertising.
Mary Jo Foley, editor in chief at advisory group Directions on Microsoft, said the company may be better off focusing on what it does best, rather than continuing to offer Xbox consoles and Surface tablets.
“Microsoft is not good at anything in the consumer space (with the possible exception of gaming),” wrote Foley, who has covered the company on and off since 1984. “You’re wasting time and money on trying to figure it out. Microsoft is an enterprise company — and that is more than OK.”
It’s unlikely Microsoft will back away from games, particularly after the Activision deal. Nearly $12 billion of Microsoft’s $69.6 billion in fourth-quarter revenue came from gaming, search and news advertising, and consumer subscriptions to the Microsoft 365 productivity bundle. That doesn’t include sales of devices, Windows licenses or advertising on LinkedIn.
“As a company, Microsoft’s all-in on gaming,” Nadella said in 2021 in an appearance alongside gaming unit head Phil Spencer. “We believe we can play a leading role in democratizing gaming and defining that future of interactive entertainment, quite frankly, at scale.”
AI pressure
Microsoft has an unquestionably strong position in AI today, thanks in no small part to its early alliance with OpenAI. Microsoft has added the startup’s AI models to Windows, Excel, Bing and other products.
The breakout has been GitHub Copilot, which generates source code and answers developers’ questions. GitHub reached $2 billion in annualized revenue last year, with Copilot accounting for more than 40% of sales growth for the business. Microsoft bought GitHub in 2018 for $7.5 billion.
Microsoft CEO Satya Nadella, right, speaks as OpenAI CEO Sam Altman looks on during the OpenAI DevDay event in San Francisco on Nov. 6, 2023.
Justin Sullivan | Getty Images
But speedy deployment in AI can be worrisome.
The company is “not providing the underpinnings needed to deploy AI properly, in terms of security and governance — all because they care more about being ‘first,'” Foley wrote. Microsoft also hasn’t been great at helping customers understand the return on investment, she wrote.
AI-ready Copilot+ PCs, which Microsoft introduced last year, aren’t gaining much traction. The company had to delay the release of the Recall search feature to prevent data breaches. And the Copilot assistant subscription, at $30 a month for customers of the Microsoft 365 productivity suite, hasn’t become pervasive in the business world.
“Copilot was really their chance to take the lead,” said Jason Wong, an analyst at technology industry researcher Gartner. “But increasingly, what it’s seeming like is Copilot is just an add-on and not like a net-new thing to drive AI.”
Innovation
At 50, the biggest question facing Microsoft is whether it can still build impressive technology on its own. Products like the Surface and HoloLens augmented reality headset generated buzz, but they hit the market years ago.
Teams was a novel addition to its software bundle, though the app’s success came during the Covid pandemic after the explosive growth in products like Zoom and Slack, which Salesforce acquired. And Microsoft is still researching quantum computing.
In AI, Microsoft’s best bet so far was its investment in OpenAI. Somasegar said Microsoft is in prime position to be a big player in the market.
“To me, it’s been 2½ years since ChatGPT showed up, and we are not even at the Uber and Airbnb moment,” Somasegar said. “There is a tremendous amount of value creation that needs to happen in AI. Microsoft as much as everybody else is thinking, ‘What does that mean? How do we get there?'”
Technology
Nvidia positioned to weather Trump tariffs, chip demand ‘off the charts,’ says Altimeter’s Gerstner
Altimeter Capital CEO Brad Gerstner said Thursday that he’s moving out of the “bomb shelter” with Nvidia and into a position of safety, expecting that the chipmaker is positioned to withstand President Donald Trump’s widespread tariffs.
“The growth and the demand for GPUs is off the charts,” he told CNBC’s “Fast Money Halftime Report,” referring to Nvidia’s graphics processing units that are powering the artificial intelligence boom. He said investors just need to listen to commentary from OpenAI, Google and Elon Musk.
President Trump announced an expansive and aggressive “reciprocal tariff” policy in a ceremony at the White House on Wednesday. The plan established a 10% baseline tariff, though many countries like China, Vietnam and Taiwan are subject to steeper rates. The announcement sent stocks tumbling on Thursday, with the tech-heavy Nasdaq down more than 5%, headed for its worst day since 2022.
The big reason Nvidia may be better positioned to withstand Trump’s tariff hikes is because semiconductors are on the list of exceptions, which Gerstner called a “wise exception” due to the importance of AI.
Nvidia’s business has exploded since the release of OpenAI’s ChatGPT in 2022, and annual revenue has more than doubled in each of the past two fiscal years. After a massive rally, Nvidia’s stock price has dropped by more than 20% this year and was down almost 7% on Thursday.
Gerstner is concerned about the potential of a recession due to the tariffs, but is relatively bullish on Nvidia, and said the “negative impact from tariffs will be much less than in other areas.”
He said it’s key for the U.S. to stay competitive in AI. And while the company’s chips are designed domestically, they’re manufactured in Taiwan “because they can’t be fabricated in the U.S.” Higher tariffs would punish companies like Meta and Microsoft, he said.
“We’re in a global race in AI,” Gerstner said. “We can’t hamper our ability to win that race.”
-
Sports2 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports12 months agoStory injured on diving stop, exits Red Sox game
-
Sports1 year agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike
-
Business3 years agoBank of England’s extraordinary response to government policy is almost unthinkable | Ed Conway