A.I. has a discrimination problem. In banking, the consequences can be severe
Artificial intelligence algorithms are increasingly being used in financial services — but they come with some serious risks around discrimination.
Sadik Demiroz | Photodisc | Getty Images
AMSTERDAM — Artificial intelligence has a racial bias problem.
From biometric identification systems that disproportionately misidentify the faces of Black people and minorities, to applications of voice recognition software that fail to distinguish voices with distinct regional accents, AI has a lot to work on when it comes to discrimination.
And the problem of amplifying existing biases can be even more severe when it comes to banking and financial services.
Deloitte notes that AI systems are ultimately only as good as the data they’re given: Incomplete or unrepresentative datasets could limit AI’s objectivity, while biases in development teams that train such systems could perpetuate that cycle of bias.
A.I. can be dumb
Nabil Manji, head of crypto and Web3 at Worldpay by FIS, said a key thing to understand about AI products is that the strength of the technology depends a lot on the source material used to train it.
“The thing about how good an AI product is, there’s kind of two variables,” Manji told CNBC in an interview. “One is the data it has access to, and second is how good the large language model is. That’s why the data side, you see companies like Reddit and others, they’ve come out publicly and said we’re not going to allow companies to scrape our data, you’re going to have to pay us for that.”
As for financial services, Manji said a lot of the backend data systems are fragmented in different languages and formats.
“None of it is consolidated or harmonized,” he added. “That is going to cause AI-driven products to be a lot less effective in financial services than it might be in other verticals or other companies where they have uniformity and more modern systems or access to data.”
Manji suggested that blockchain, or distributed ledger technology, could serve as a way to get a clearer view of the disparate data tucked away in the cluttered systems of traditional banks.
However, he added that banks — being the heavily regulated, slow-moving institutions that they are — are unlikely to move with the same speed as their more nimble tech counterparts in adopting new AI tools.
“You’ve got Microsoft and Google, who like over the last decade or two have been seen as driving innovation. They can’t keep up with that speed. And then you think about financial services. Banks are not known for being fast,” Manji said.
Banking’s A.I. problem
Rumman Chowdhury, Twitter’s former head of machine learning ethics, transparency and accountability, said that lending is a prime example of how an AI system’s bias against marginalized communities can rear its head.
“Algorithmic discrimination is actually very tangible in lending,” Chowdhury said on a panel at Money20/20 in Amsterdam. “Chicago had a history of literally denying those [loans] to primarily Black neighborhoods.”
In the 1930s, Chicago was known for the discriminatory practice of “redlining,” in which the creditworthiness of properties was heavily determined by the racial demographics of a given neighborhood.
“There would be a giant map on the wall of all the districts in Chicago, and they would draw red lines through all of the districts that were primarily African American, and not give them loans,” she added.
“Fast forward a few decades later, and you are developing algorithms to determine the riskiness of different districts and individuals. And while you may not include the data point of someone’s race, it is implicitly picked up.”
Indeed, Angle Bush, founder of Black Women in Artificial Intelligence, an organization aiming to empower Black women in the AI sector, tells CNBC that when AI systems are specifically used for loan approval decisions, she has found that there is a risk of replicating existing biases present in historical data used to train the algorithms.
“This can result in automatic loan denials for individuals from marginalized communities, reinforcing racial or gender disparities,” Bush added.
“It is crucial for banks to acknowledge that implementing AI as a solution may inadvertently perpetuate discrimination,” she said.
Frost Li, a developer who has been working in AI and machine learning for over a decade, told CNBC that the “personalization” dimension of AI integration can also be problematic.
“What’s interesting in AI is how we select the ‘core features’ for training,” said Li, who founded and runs Loup, a company that helps online retailers integrate AI into their platforms. “Sometimes, we select features unrelated to the results we want to predict.”
When AI is applied to banking, Li says, it’s harder to identify the “culprit” in biases when everything is convoluted in the calculation.
“A good example is how many fintech startups are especially for foreigners, because a Tokyo University graduate won’t be able to get any credit cards even if he works at Google; yet a person can easily get one from community college credit union because bankers know the local schools better,” Li added.
Generative AI is not usually used for creating credit scores or in the risk-scoring of consumers.
“That is not what the tool was built for,” said Niklas Guske, chief operating officer at Taktile, a startup that helps fintechs automate decision-making.
Instead, Guske said the most powerful applications are in pre-processing unstructured data such as text files — like classifying transactions.
“Those signals can then be fed into a more traditional underwriting model,” said Guske. “Therefore, Generative AI will improve the underlying data quality for such decisions rather than replace common scoring processes.”
But it’s also difficult to prove. Apple and Goldman Sachs, for example, were accused of giving women lower limits for the Apple Card. But these claims were dismissed by the New York Department of Financial Services after the regulator found no evidence of discrimination based on sex.
The problem, according to Kim Smouter, director of anti-racism group European Network Against Racism, is that it can be challenging to substantiate whether AI-based discrimination has actually taken place.
“One of the difficulties in the mass deployment of AI,” he said, “is the opacity in how these decisions come about and what redress mechanisms exist were a racialized individual to even notice that there is discrimination.”
“Individuals have little knowledge of how AI systems work and that their individual case may, in fact, be the tip of a systems-wide iceberg. Accordingly, it’s also difficult to detect specific instances where things have gone wrong,” he added.
Smouter cited the example of the Dutch child welfare scandal, in which thousands of benefit claims were wrongfully accused of being fraudulent. The Dutch government was forced to resign after a 2020 report found that victims were “treated with an institutional bias.”
This, Smouter said, “demonstrates how quickly such disfunctions can spread and how difficult it is to prove them and get redress once they are discovered and in the meantime significant, often irreversible damage is done.”
Policing A.I.’s biases
Chowdhury says there is a need for a global regulatory body, like the United Nations, to address some of the risks surrounding AI.
Though AI has proven to be an innovative tool, some technologists and ethicists have expressed doubts about the technology’s moral and ethical soundness. Among the top worries industry insiders expressed are misinformation; racial and gender bias embedded in AI algorithms; and “hallucinations” generated by ChatGPT-like tools.
“I worry quite a bit that, due to generative AI, we are entering this post-truth world where nothing we see online is trustworthy — not any of the text, not any of the video, not any of the audio, but then how do we get our information? And how do we ensure that information has a high amount of integrity?” Chowdhury said.
Now is the time for meaningful regulation of AI to come into force — but knowing the amount of time it will take regulatory proposals like the European Union’s AI Act to take effect, some are concerned this won’t happen fast enough.
“We call upon more transparency and accountability of algorithms and how they operate and a layman’s declaration that allows individuals who are not AI experts to judge for themselves, proof of testing and publication of results, independent complaints process, periodic audits and reporting, involvement of racialized communities when tech is being designed and considered for deployment,” Smouter said.
The AI Act, the first regulatory framework of its kind, has incorporated a fundamental rights approach and concepts like redress, according to Smouter, adding that the regulation will be enforced in approximately two years.
“It would be great if this period can be shortened to make sure transparency and accountability are in the core of innovation,” he said.
Technology
A ‘seismic’ Nvidia shift, AI chip shortages and how it’s threatening to hike gadget prices
The logo of an Apple Store is seen reflected on the glass exterior of a Samsung flagship store in Shanghai, China Monday, Oct. 20, 2025.
Wang Gang | Feature China | Future Publishing | Getty Images
The cost of your smartphone might rise, analysts are warning, as the AI boom clogs up supply chains and a recent change by Nvidia to its products could make it worse.
AI data centers, on which tech giants globally are spending hundreds of billions of dollars, require chips from suppliers, like Nvidia, which relies on many different components and companies to create its coveted graphics processing units.
But other companies like AMD, the hyperscalers like Google and Microsoft, and other component suppliers all rely on this supply chain.
Many parts of the supply chain can’t keep up with demand, and it’s slowing down components that are critical for some of the world’s most popular consumer electronics. Those components are seeing huge spikes in prices, threatening price rises for the end product and could even lead to shortages of some devices.
“We see the rapid increase in demand for AI in data centers driving bottlenecks in many areas,” Peter Hanbury, partner in the technology practice at Bain & Company, told CNBC.
Where is the supply chain clogged?
One of the starkest assessments came from Alibaba CEO Eddie Wu, CEO of Chinese tech giant Alibaba.
Wu, whose company is building its own AI infrastructure and designs its own chips, said last week that there are shortages across semiconductor manufacturers, memory chips and storage devices like hard drives.
“There is a situation of undersupply,” Wu said, adding that the “supply side is going to be a relatively large bottleneck.” He added this could last two to three years.
Bain and Co.’s Hanbury said there are shortages of hard disk drives, or HDDs, which store data. HDDs are used in the data center. These are preferred by hyperscalers,: big companies like Microsoft and Google. But, with HDDs at capacity, these firms have shifted to using solid-state drives, or SSDs, another type of storage device.
However, these SSDs are key components for consumer electronics.
The other big focus is on a type of chip under the umbrella of memory called dynamic random-access memory or DRAM. Nvidia’s chips use high-bandwidth memory which is a type of chip that stacks multiple DRAM semiconductors.
Memory prices have surged as a result of the huge demand and lack of supply. Counterpoint Research said it expects memory prices to rise 30% in the fourth quarter of this year and another 20% in early 2026. Even small imbalances in supply and demand can have major knock on effects on memory pricing. And because of the demand for HBM and GPUs, chipmakers are prioritizing these over other types of semiconductors.
“DRAM is certainly a bottleneck as AI investments continue to feed the imbalance between demand and supply with HBM for AI being prioritized by chipmakers,” MS Hwang, research director at Counterpoint Research, told CNBC.
“Imbalances of 1-2% can trigger sharp price increases and we’re seeing that figure hitting 3% levels at the moment – this is very significant.”
Why are there issues?
Building up capacity in various areas of the semiconductor supply chain can be capital-intensive. And it’s an industry that’s known to be risk-averse and did not add the capacity necessary to meet the projections provided by key industry players, Bain & Co.’s Hanbur said.
“The direct cause of the shortage is the rapid increase in demand for data center chips,” Hanbury said.
“Basically, the suppliers worried the market was too optimistic and they did not want to overbuild very expensive capacity so they did not build to the estimates provided by their customers. Now, the suppliers need to add capacity quickly but as we know, it takes 2-3 years to add semiconductor manufacturing fabs.”
Nvidia at the center
A lot of attention is on Nvidia given it dominates when it comes to the chips that are being put into AI data centers.
It is a huge customer of high bandwidth memory, for example. And its products are manufactured by TSMC which also has other major customers like Apple.
But analysts are focused on a change Nvidia has made to its products that has the potential to add major pressure to consumer electronics supply chains. The U.S. giant is increasingly shifting toward using a type of memory in its products called Low-Power Double Data Rate (LPDDR). This is seen as more power efficient than the previous Double Data Rate, or DDR memory.
The problem is, Nvidia is increasingly using the latest generation of LPDDR memory, which is also used by high-end consumer electronics makers such as Samsung and Apple.
Typically, the industry would just be dealing with demand for this product from a handful of big electronics players. But now Nvidia, which has huge scale, is entering the mix.
“We also see a bigger risk on the horizon is with advanced memory as Nvidia’s recent pivot to LPDDR means they’re a customer on the scale of a major smartphone maker — a seismic shift for the supply chain which can’t easily absorb this scale of demand,” Hwang from Counterpoint Research said.
How AI boom is impacting consumer electronics
Here’s the link between all of this.
From chip manufacturers like TSMC, Intel and Samsung, there is only so much capacity. If there is huge demand for certain types of chips, then these companies will prioritize those, especially from their larger customers. That can lead to shortages of other types of semiconductors elsewhere.
Memory chips, in particular DRAM which has seen prices shoot up, is of particular concern because it’s used in so many devices from smartphones to laptops. And this could lead to price rises in the world’s favorite electronics.
DRAM and storage represent around 10% to 25% of the bill of materials for a typical PC or smartphone, according to Hanbury of Bain & Co. A price increase of 20% to 30% in these components would increase the total bill of materials costs by 5% to 10%.
“In terms of timing, the impact will likely start shortly as component costs are already increasing and likely accelerate into next year,” Hanbury said.
On top of this, there is now demand from players involved in AI data centers like Nvidia, for components that would have typically been used for consumer devices such as LPDDR which adds more demand to a supply constrained market.
If electronics firms can’t get their hands on the components needed for their devices because they’re in short supply or going toward AI data centers, then there could be shortages of the world’s most popular gadgets.
“Beyond the rise in cost there’s a second issue and that’s the inability to secure enough components, which constrains the production of electronic devices,” Counterpoint Research’s Hwang said.
What are tech firms saying?
A number of electronics companies have warned about the impact they are seeing from all of this.
Xiaomi, the third-biggest smartphone vendor globally, said it expects that consumers will see “a sizeable rise in product retail prices,” according to a Reuters reported this month.
Jeff Clark, chief operating officer at Dell, this month said the price rises of components is “unprecedented.”
“We have not seen costs move at the rate that we’ve seen,” Clark said on an earnings call, adding that the pressure is seen across various types of memory chips and storage hard drives.
The unintended consequences
The AI infrastructure players are using similar chips to those being used in consumer electronics. These are often some of the more advanced semiconductors on the market.
But there are legacy chips which are manufactured by the same companies that the AI market is relying on. As these manufacturers shift attention to serving their AI customers, there could be unintended consequences for other industries.
“For example, many other markets depend on the same underlying semiconductor manufacturing capabilities as the data center market” including automobiles, industrials and aerospace and defense, which “will likely see some impact from these price increases as well,” Hanbury said.
Technology
Samsung launches its first multi-folding phone as competition from Chinese brands intensifies
Technology
Nvidia CEO to Cramer: Synopsys deal is ‘culmination of everything I showed you’ over the years
-
Sports2 years agoStory injured on diving stop, exits Red Sox game
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports3 years agoButton battles heat exhaustion in NASCAR debut
-
Sports3 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment3 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment1 year agoHere are the best electric bikes you can buy at every price level in October 2024