A.I. has a discrimination problem. In banking, the consequences can be severe
Artificial intelligence algorithms are increasingly being used in financial services — but they come with some serious risks around discrimination.
Sadik Demiroz | Photodisc | Getty Images
AMSTERDAM — Artificial intelligence has a racial bias problem.
From biometric identification systems that disproportionately misidentify the faces of Black people and minorities, to applications of voice recognition software that fail to distinguish voices with distinct regional accents, AI has a lot to work on when it comes to discrimination.
And the problem of amplifying existing biases can be even more severe when it comes to banking and financial services.
Deloitte notes that AI systems are ultimately only as good as the data they’re given: Incomplete or unrepresentative datasets could limit AI’s objectivity, while biases in development teams that train such systems could perpetuate that cycle of bias.
A.I. can be dumb
Nabil Manji, head of crypto and Web3 at Worldpay by FIS, said a key thing to understand about AI products is that the strength of the technology depends a lot on the source material used to train it.
“The thing about how good an AI product is, there’s kind of two variables,” Manji told CNBC in an interview. “One is the data it has access to, and second is how good the large language model is. That’s why the data side, you see companies like Reddit and others, they’ve come out publicly and said we’re not going to allow companies to scrape our data, you’re going to have to pay us for that.”
As for financial services, Manji said a lot of the backend data systems are fragmented in different languages and formats.
“None of it is consolidated or harmonized,” he added. “That is going to cause AI-driven products to be a lot less effective in financial services than it might be in other verticals or other companies where they have uniformity and more modern systems or access to data.”
Manji suggested that blockchain, or distributed ledger technology, could serve as a way to get a clearer view of the disparate data tucked away in the cluttered systems of traditional banks.
However, he added that banks — being the heavily regulated, slow-moving institutions that they are — are unlikely to move with the same speed as their more nimble tech counterparts in adopting new AI tools.
“You’ve got Microsoft and Google, who like over the last decade or two have been seen as driving innovation. They can’t keep up with that speed. And then you think about financial services. Banks are not known for being fast,” Manji said.
Banking’s A.I. problem
Rumman Chowdhury, Twitter’s former head of machine learning ethics, transparency and accountability, said that lending is a prime example of how an AI system’s bias against marginalized communities can rear its head.
“Algorithmic discrimination is actually very tangible in lending,” Chowdhury said on a panel at Money20/20 in Amsterdam. “Chicago had a history of literally denying those [loans] to primarily Black neighborhoods.”
In the 1930s, Chicago was known for the discriminatory practice of “redlining,” in which the creditworthiness of properties was heavily determined by the racial demographics of a given neighborhood.
“There would be a giant map on the wall of all the districts in Chicago, and they would draw red lines through all of the districts that were primarily African American, and not give them loans,” she added.
“Fast forward a few decades later, and you are developing algorithms to determine the riskiness of different districts and individuals. And while you may not include the data point of someone’s race, it is implicitly picked up.”
Indeed, Angle Bush, founder of Black Women in Artificial Intelligence, an organization aiming to empower Black women in the AI sector, tells CNBC that when AI systems are specifically used for loan approval decisions, she has found that there is a risk of replicating existing biases present in historical data used to train the algorithms.
“This can result in automatic loan denials for individuals from marginalized communities, reinforcing racial or gender disparities,” Bush added.
“It is crucial for banks to acknowledge that implementing AI as a solution may inadvertently perpetuate discrimination,” she said.
Frost Li, a developer who has been working in AI and machine learning for over a decade, told CNBC that the “personalization” dimension of AI integration can also be problematic.
“What’s interesting in AI is how we select the ‘core features’ for training,” said Li, who founded and runs Loup, a company that helps online retailers integrate AI into their platforms. “Sometimes, we select features unrelated to the results we want to predict.”
When AI is applied to banking, Li says, it’s harder to identify the “culprit” in biases when everything is convoluted in the calculation.
“A good example is how many fintech startups are especially for foreigners, because a Tokyo University graduate won’t be able to get any credit cards even if he works at Google; yet a person can easily get one from community college credit union because bankers know the local schools better,” Li added.
Generative AI is not usually used for creating credit scores or in the risk-scoring of consumers.
“That is not what the tool was built for,” said Niklas Guske, chief operating officer at Taktile, a startup that helps fintechs automate decision-making.
Instead, Guske said the most powerful applications are in pre-processing unstructured data such as text files — like classifying transactions.
“Those signals can then be fed into a more traditional underwriting model,” said Guske. “Therefore, Generative AI will improve the underlying data quality for such decisions rather than replace common scoring processes.”
But it’s also difficult to prove. Apple and Goldman Sachs, for example, were accused of giving women lower limits for the Apple Card. But these claims were dismissed by the New York Department of Financial Services after the regulator found no evidence of discrimination based on sex.
The problem, according to Kim Smouter, director of anti-racism group European Network Against Racism, is that it can be challenging to substantiate whether AI-based discrimination has actually taken place.
“One of the difficulties in the mass deployment of AI,” he said, “is the opacity in how these decisions come about and what redress mechanisms exist were a racialized individual to even notice that there is discrimination.”
“Individuals have little knowledge of how AI systems work and that their individual case may, in fact, be the tip of a systems-wide iceberg. Accordingly, it’s also difficult to detect specific instances where things have gone wrong,” he added.
Smouter cited the example of the Dutch child welfare scandal, in which thousands of benefit claims were wrongfully accused of being fraudulent. The Dutch government was forced to resign after a 2020 report found that victims were “treated with an institutional bias.”
This, Smouter said, “demonstrates how quickly such disfunctions can spread and how difficult it is to prove them and get redress once they are discovered and in the meantime significant, often irreversible damage is done.”
Policing A.I.’s biases
Chowdhury says there is a need for a global regulatory body, like the United Nations, to address some of the risks surrounding AI.
Though AI has proven to be an innovative tool, some technologists and ethicists have expressed doubts about the technology’s moral and ethical soundness. Among the top worries industry insiders expressed are misinformation; racial and gender bias embedded in AI algorithms; and “hallucinations” generated by ChatGPT-like tools.
“I worry quite a bit that, due to generative AI, we are entering this post-truth world where nothing we see online is trustworthy — not any of the text, not any of the video, not any of the audio, but then how do we get our information? And how do we ensure that information has a high amount of integrity?” Chowdhury said.
Now is the time for meaningful regulation of AI to come into force — but knowing the amount of time it will take regulatory proposals like the European Union’s AI Act to take effect, some are concerned this won’t happen fast enough.
“We call upon more transparency and accountability of algorithms and how they operate and a layman’s declaration that allows individuals who are not AI experts to judge for themselves, proof of testing and publication of results, independent complaints process, periodic audits and reporting, involvement of racialized communities when tech is being designed and considered for deployment,” Smouter said.
The AI Act, the first regulatory framework of its kind, has incorporated a fundamental rights approach and concepts like redress, according to Smouter, adding that the regulation will be enforced in approximately two years.
“It would be great if this period can be shortened to make sure transparency and accountability are in the core of innovation,” he said.
Technology
Blockchain-driven platform to mimic stock trading, allowing users to buy shares of SpaceX, other hot private companies
Artistgndphotography | E+ | Getty Images
Republic, a New York-based investment startup, is offering users exposure to SpaceX by issuing a “tokenized” representation of its shares.
The company will begin selling the digital tokens this week and eventually plans to expand the offering to other private companies like artificial intelligence darlings OpenAI and Anthropic, as well as Stripe, X, Waymo, Epic Games and more. The Wall Street Journal first reported the story Wednesday.
“We’re talking about delivering products to retail investors that they’ve have been held out of previously,” Republic co-CEO Andrew Durgee told CNBC. “The fact that retail investors couldn’t own pre-IPO SpaceX has always been crazy to us. Now that’s going to be attached to the upside of these pre-IPO businesses. The businesses that we target out of the gate we want to have a retail focus, or at least significant retail following.”
In the crypto world, tokenization is the process of issuing digital representations on a blockchain network of publicly traded securities, real world assets or any other form of value. Holders of tokenized assets don’t have outright ownership of the assets themselves.
The move comes as the U.S. crypto industry is testing new regulatory boundaries under President Donald Trump’s pro-crypto administration. Since he took office, the Securities and Exchange Commission has moved swiftly to loosen the restraints left on the crypto industry by the previous administration, ending an enforcement case against Coinbase; closing investigations into Robinhood Crypto, Uniswap, Gemini and Consensys without enforcement action; scaling back its crypto enforcement unit; declaring meme coins are not securities and launching a Crypto Task Force that’s been holding a series of roundtables on crypto asset regulation.
“If you take a step back and look at what the last four to eight years looked like in the space, innovation was very stifled,” Durgee said. “The reality is the space was just difficult for most to understand and consume. Now we’ve gotten to a point where it’s certainly become more mainstay.”
“We’ve moved from what was ultimately … nothing but headwinds,” he added. “And now we’re finally in a place industrywide, where we actually have tailwinds and we have some room to really innovate.”
Republic will allow investors to invest between $50 and $5,000 in the tokens. Typically, those wanting to invest in private companies are required to meet a minimum closer to $10,000 and need to meet specific income or net-worth requirements. Shares of private company can be exchanged by accredited investors in secondary markets; Republic will initially price SpaceX tokens based on how the company’s shares are performing there.
Tokenized private equity is new territory for regulators and the underlying companies being digitally represented. There are outstanding questions about the legality of the tokens, how Republic will give financial information to investors as required, and how selling private investments to retail investors could provoke stress in the financial markets.
“We don’t need a company’s approval to be able to do these types of offerings, and I do think there will be some companies that will want more control over something like that,” Durgee said. “The reality is the structure that we’re using, which was built on securities law from the 1930s, in a lot of instances allows us the leeway to give these types of offerings. People are going to really have to start to question how they’re going to approach some of these innovations, and how far they will want to push that risk envelope.”
Financial institutions are becoming increasingly interested in tokenizing traditional assets because of the often-touted benefits of blockchain technology: lower costs, faster settlement times, greater transparency about ownership and performance and programmable terms, as well as increased accessibility for retail investors and global reach.
The announcement comes about a week after Coinbase said it’s pushing for SEC approval to offer trading of tokenized public stocks, which would give the crypto services provider an additional revenue stream and put it in closer competition with brokerages like Robinhood and eToro.
Competing crypto exchange Kraken recently said it’ll offer tokens of U.S. stocks for 24/7 trading in unspecified markets abroad.
BlackRock CEO Larry Fink has said he sees the “tokenization of every financial asset” as an important step in “the technological revolution in the financial markets.”
An unmanned aerial vehicle (UAV) at the AeroVironment Inc. booth during the Special Operations Forces Industry Conference (SOFIC) in Tampa, Florida, US, on Tuesday, May 17, 2022.
Luke Sharrett | Bloomberg | Getty Images
AeroVironment stock rocketed more than 24% higher Wednesday as the drone maker beat fourth quarter expectations on the top and bottom lines.
Here’s how the company did compared to analyst expectations:
- Earnings: $1.61 per share adjusted vs $1.39 per share expected
- Revenue: $275 million vs $242 million expected
The company reported financial results after market close Tuesday and logged record fiscal year revenue of $820.6 million, up 14% over the prior period.
AeroVironment reported net income of $16.66 million for the fourth quarter, or 59 cents per share, compared to net income of $6.05 million, or 22 cents per share, last year.
The company closed the $4.1 billion acquisition of defense tech company BlueHalo on May 1. BlueHalo makes drone and defense technology such as laser weapon systems, with a focus on space tech.
“Our acquisition of BlueHalo further advances our leadership position within the defense-technology sector by adding a complementary portfolio of innovative products and capabilities aligned to our customers’ highest priorities,” AeroVironment CEO Wahid Nawabi said in a statement.
For the new fiscal year, the company said it expects revenues to range between $1.9 billion and $2 billion. The company forecast earnings between $2.80 and $3.00 per share.
WATCH: AeroVironment CEO on European defense spending boost, U.S. defense spending and Trump
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports1 year agoStory injured on diving stop, exits Red Sox game
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Sports2 years agoButton battles heat exhaustion in NASCAR debut
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike