A.I. has a discrimination problem. In banking, the consequences can be severe
Artificial intelligence algorithms are increasingly being used in financial services — but they come with some serious risks around discrimination.
Sadik Demiroz | Photodisc | Getty Images
AMSTERDAM — Artificial intelligence has a racial bias problem.
From biometric identification systems that disproportionately misidentify the faces of Black people and minorities, to applications of voice recognition software that fail to distinguish voices with distinct regional accents, AI has a lot to work on when it comes to discrimination.
And the problem of amplifying existing biases can be even more severe when it comes to banking and financial services.
Deloitte notes that AI systems are ultimately only as good as the data they’re given: Incomplete or unrepresentative datasets could limit AI’s objectivity, while biases in development teams that train such systems could perpetuate that cycle of bias.
A.I. can be dumb
Nabil Manji, head of crypto and Web3 at Worldpay by FIS, said a key thing to understand about AI products is that the strength of the technology depends a lot on the source material used to train it.
“The thing about how good an AI product is, there’s kind of two variables,” Manji told CNBC in an interview. “One is the data it has access to, and second is how good the large language model is. That’s why the data side, you see companies like Reddit and others, they’ve come out publicly and said we’re not going to allow companies to scrape our data, you’re going to have to pay us for that.”
As for financial services, Manji said a lot of the backend data systems are fragmented in different languages and formats.
“None of it is consolidated or harmonized,” he added. “That is going to cause AI-driven products to be a lot less effective in financial services than it might be in other verticals or other companies where they have uniformity and more modern systems or access to data.”
Manji suggested that blockchain, or distributed ledger technology, could serve as a way to get a clearer view of the disparate data tucked away in the cluttered systems of traditional banks.
However, he added that banks — being the heavily regulated, slow-moving institutions that they are — are unlikely to move with the same speed as their more nimble tech counterparts in adopting new AI tools.
“You’ve got Microsoft and Google, who like over the last decade or two have been seen as driving innovation. They can’t keep up with that speed. And then you think about financial services. Banks are not known for being fast,” Manji said.
Banking’s A.I. problem
Rumman Chowdhury, Twitter’s former head of machine learning ethics, transparency and accountability, said that lending is a prime example of how an AI system’s bias against marginalized communities can rear its head.
“Algorithmic discrimination is actually very tangible in lending,” Chowdhury said on a panel at Money20/20 in Amsterdam. “Chicago had a history of literally denying those [loans] to primarily Black neighborhoods.”
In the 1930s, Chicago was known for the discriminatory practice of “redlining,” in which the creditworthiness of properties was heavily determined by the racial demographics of a given neighborhood.
“There would be a giant map on the wall of all the districts in Chicago, and they would draw red lines through all of the districts that were primarily African American, and not give them loans,” she added.
“Fast forward a few decades later, and you are developing algorithms to determine the riskiness of different districts and individuals. And while you may not include the data point of someone’s race, it is implicitly picked up.”
Indeed, Angle Bush, founder of Black Women in Artificial Intelligence, an organization aiming to empower Black women in the AI sector, tells CNBC that when AI systems are specifically used for loan approval decisions, she has found that there is a risk of replicating existing biases present in historical data used to train the algorithms.
“This can result in automatic loan denials for individuals from marginalized communities, reinforcing racial or gender disparities,” Bush added.
“It is crucial for banks to acknowledge that implementing AI as a solution may inadvertently perpetuate discrimination,” she said.
Frost Li, a developer who has been working in AI and machine learning for over a decade, told CNBC that the “personalization” dimension of AI integration can also be problematic.
“What’s interesting in AI is how we select the ‘core features’ for training,” said Li, who founded and runs Loup, a company that helps online retailers integrate AI into their platforms. “Sometimes, we select features unrelated to the results we want to predict.”
When AI is applied to banking, Li says, it’s harder to identify the “culprit” in biases when everything is convoluted in the calculation.
“A good example is how many fintech startups are especially for foreigners, because a Tokyo University graduate won’t be able to get any credit cards even if he works at Google; yet a person can easily get one from community college credit union because bankers know the local schools better,” Li added.
Generative AI is not usually used for creating credit scores or in the risk-scoring of consumers.
“That is not what the tool was built for,” said Niklas Guske, chief operating officer at Taktile, a startup that helps fintechs automate decision-making.
Instead, Guske said the most powerful applications are in pre-processing unstructured data such as text files — like classifying transactions.
“Those signals can then be fed into a more traditional underwriting model,” said Guske. “Therefore, Generative AI will improve the underlying data quality for such decisions rather than replace common scoring processes.”
But it’s also difficult to prove. Apple and Goldman Sachs, for example, were accused of giving women lower limits for the Apple Card. But these claims were dismissed by the New York Department of Financial Services after the regulator found no evidence of discrimination based on sex.
The problem, according to Kim Smouter, director of anti-racism group European Network Against Racism, is that it can be challenging to substantiate whether AI-based discrimination has actually taken place.
“One of the difficulties in the mass deployment of AI,” he said, “is the opacity in how these decisions come about and what redress mechanisms exist were a racialized individual to even notice that there is discrimination.”
“Individuals have little knowledge of how AI systems work and that their individual case may, in fact, be the tip of a systems-wide iceberg. Accordingly, it’s also difficult to detect specific instances where things have gone wrong,” he added.
Smouter cited the example of the Dutch child welfare scandal, in which thousands of benefit claims were wrongfully accused of being fraudulent. The Dutch government was forced to resign after a 2020 report found that victims were “treated with an institutional bias.”
This, Smouter said, “demonstrates how quickly such disfunctions can spread and how difficult it is to prove them and get redress once they are discovered and in the meantime significant, often irreversible damage is done.”
Policing A.I.’s biases
Chowdhury says there is a need for a global regulatory body, like the United Nations, to address some of the risks surrounding AI.
Though AI has proven to be an innovative tool, some technologists and ethicists have expressed doubts about the technology’s moral and ethical soundness. Among the top worries industry insiders expressed are misinformation; racial and gender bias embedded in AI algorithms; and “hallucinations” generated by ChatGPT-like tools.
“I worry quite a bit that, due to generative AI, we are entering this post-truth world where nothing we see online is trustworthy — not any of the text, not any of the video, not any of the audio, but then how do we get our information? And how do we ensure that information has a high amount of integrity?” Chowdhury said.
Now is the time for meaningful regulation of AI to come into force — but knowing the amount of time it will take regulatory proposals like the European Union’s AI Act to take effect, some are concerned this won’t happen fast enough.
“We call upon more transparency and accountability of algorithms and how they operate and a layman’s declaration that allows individuals who are not AI experts to judge for themselves, proof of testing and publication of results, independent complaints process, periodic audits and reporting, involvement of racialized communities when tech is being designed and considered for deployment,” Smouter said.
The AI Act, the first regulatory framework of its kind, has incorporated a fundamental rights approach and concepts like redress, according to Smouter, adding that the regulation will be enforced in approximately two years.
“It would be great if this period can be shortened to make sure transparency and accountability are in the core of innovation,” he said.
Alphabet CEO Sundar Pichai during the Google I/O developers conference in Mountain View, California, on May 10, 2023.
David Paul Morris | Bloomberg | Getty Images
Alphabet‘s stock gained 3% Friday after signaling strong growth in its search and advertising businesses amid a competitive artificial intelligence environment and uncertain macro backdrop.
“GOOGL‘s pace of GenAI product roll-out is accelerating with multiple encouraging signals,” wrote Morgan Stanley‘s Brian Nowak. “Macro uncertainty still exists but we remain [overweight] given GOOGL’s still strong relative position and improving pace of GenAI enabled product roll-out.”
The search giant posted earnings of $2.81 per share on $90.23 billion in revenues. That topped the $89.12 billion in sales and $2.01 in EPS expected by LSEG analysts. Revenues grew 12% year-over-year and ahead of the 10% anticipated by Wall Street.
Net income rose 46% to $34.54 billion, or $2.81 per share. That’s up from $23.66 billion, or $1.89 per share, in the year-ago period. Alphabet said the figure included $8 billion in unrealized gains on its nonmarketable equity securities connected to its investment in a private company.
Adjusted earnings, excluding that gain, were $2.27 per share, according to LSEG, and topped analyst expectations.
Alphabet shares have pulled back about 16% this year as it battles volatility spurred by mounting trade war fears and worries that President Donald Trump‘s tariffs could crush the global economy. That would make it more difficult for Alphabet to potentially acquire infrastructure for data centers powering AI models as it faces off against competitors such as OpenAI and Anthropic to develop largely language models.
During Thursday’s call with investors, Alphabet suggested that it’s too soon to tally the total impact of tariffs. However, Google’s business chief Philipp Schindler said that ending the de minimis trade exemption in May, which created a loophole benefitting many Chinese e-commerce retailers, could create a “slight headwind” for the company’s ads business, specifically in the Asia-Pacific region. The loophole allows shipments under $800 to come into the U.S. duty-free.
Despite this backdrop, Alphabet showed steady growth in its advertising and search business, reporting $66.89 billion in revenues for its advertising unit. That reflected 8.5% growth from the year-ago period. The company reported $8.93 billion in advertising revenue for its YouTube business, shy of an $8.97 billion estimate from StreetAccount.
Alphabet’s “Search and other” unit rose 9.8% to $50.7 billion, up from $46.16 billion last year. The company said that its AI Overviews tool used in its Google search results page has accumulated 1.5 billion monthly users from a billion in October.
Bank of America analyst Justin Post said that Wall Street is underestimating the upside potential and “monetization ramp” from this tool and cloud demand fueled by AI.
“The strong 1Q search performance, along with constructive comments on Gemini [large language model] performance and [AI Overviews] adoption could help alleviate some investor concerns on AI competition,” Post wrote in a note.
WATCH: Gemini delivering well for Google, says Check Capital’s Chris Ballard
CNBC’s Jennifer Elias contributed to this report.
An Amazon employee works to fulfill same-day orders during Cyber Monday, one of the company’s busiest days at an Amazon fulfillment center on December 2, 2024 in Orlando, Florida.
Miguel J. Rodriguez Carrillo | Getty Images
For 10 years, Aaron Cordovez has been selling kitchen appliances on Amazon. Now he’s in a bind, because most of his products are manufactured in China.
Cordovez, co-founder of Zulay Kitchen, said his company is moving “as fast as we can” to move production to India, Mexico and other markets, where tariffs are increasing under President Donald Trump, but are mild compared with the levies imposed on goods from China. That process will likely take at least a year or two to complete, he said.
“We’re making our inventory last as long as we can,” Cordovez said in an email.
Zulay is also temporarily raising the price of some of its milk frothers, smores roasting sticks and other products. The company’s popular kitchen strainer now costs $12.99, up from $9.99 before Trump announced his sweeping tariff proposal earlier this month.
Amazon merchants are hiking prices for everything from diaper bags and refrigerator magnets to charm necklaces and other top-selling items as they confront higher import costs. E-commerce software company SmartScout tracked 930 products on Amazon that have seen increased prices since April 9, with an average jump of 29%.
The price hikes affect a range of categories, including clothing, jewelry, household items, office supplies, electronics and toys.
The trade war with China has threatened to upend sellers on Amazon’s third-party marketplace, which accounts for about 60% of the company’s online sales. Many merchants are based in China or rely on the world’s second-largest economy to source and assemble their products.
Sellers are now faced with the conundrum of raising prices or eating the extra costs associated with Trump’s new tariffs. It’s an existential threat for many sellers, who subsist on razor-thin margins and have, for the last several years, dealt with rising costs on Amazon tied to storage, fulfillment, shipping and advertising fees along with pricing pressure from increased competition.
CEO Andy Jassy told CNBC earlier this month that the company was “going to try and do everything we can” to keep prices low for shoppers, including renegotiating terms with some of its suppliers. But he acknowledged some third-party sellers will “need to pass that cost” of tariffs on to consumers.
Amazon’s stock price is down 15% so far this year, sliding along with the broader market. The company reports first-quarter earnings next week.
Goods imported from China now face import duties of 145%, though Trump said Wednesday his administration is “actively” talking with China about a potential deal to lower tariffs. Chinese officials on Thursday denied that trade talks are taking place.
About 25% of the price increases observed by SmartScout were initiated by sellers based in China, said Scott Needham, the company’s CEO. Last week, stainless steel jewelry maker Ursteel hiked prices on four of its products by $6.50, while apparel brand Chouyatou raised the price of some of its dresses by $2. Both businesses are based in China’s Zhejiang province.
Anker, a Chinese electronics brand and one of Amazon’s largest sellers, has raised prices on one-fifth of its products sold in the U.S., including a portable power bank, which went up to $135 from $110, SmartScout data shows.
Representatives from Anker, Ursteel and Chouyatou didn’t respond to requests for comment.
Zulay, headquartered in Florida, is one of many U.S.-based sellers raising prices. The company is also cutting costs. Cordovez said he’s been forced to lay off 19% of his workforce and slash online ad spending by 85%.
Desert Cactus, based in Illinois, is also taking action. Joe Stefani, the company’s president, has been looking to move production of some of his brand’s college-themed merchandise out of China and into Mexico, India and Vietnam. About half of Desert Cactus’ goods come from China, while the rest are made in the U.S., Stefani said.
An Amazon worker moves a cart filled with packages at an Amazon delivery station in Alpharetta, Georgia, on Nov. 28, 2022.
Justin Sullivan | Getty Images
One of the company’s top products is a customizable license plate frame that’s manufactured in China. At the start of Trump’s first term in 2016, Stefani’s company paid import and shipping fees of 4% on the license plates. That rate has since skyrocketed to 170%, he said.
“The tariffs can’t stay this high,” Stefani said. “There’s so many people that just aren’t going to make it.”
Stefani said he expects Desert Cactus will end up raising prices on some products, though he’s worried shoppers might be put off by sticker shock.
“Will someone be willing to pay $50 for a hat on Amazon?” Stefani said. “You know it’s going to be expensive at the ballpark, but on Amazon we don’t know.”
Dave Dama, co-founder of health and beauty business Pure Daily Care, said the price to manufacture one of his skin-care products in China jumped to $25 from $10. Most Amazon sellers will have no choice but to raise prices, he said.
“If you were selling something for $40 and making a $7 or $8 profit at the end of the day, with these tariffs, those days are gone,” Dama said. “You can’t do that anymore. It’s unsustainable.”
Pure Daily Care plans to stagger price increases over several weeks, and only on products “we absolutely need to,” to keep Amazon’s algorithms from ranking it lower in search results or losing the valuable buy box, he said. The buy box determines which listing pops up first when a shopper clicks on a particular product, and the one that gets purchased when they tap “Add to Cart.”
An Amazon spokesperson said the company’s pricing policies continue to apply.
“As always, sellers set their own prices, and we regularly monitor how we highlight great prices as Featured Offers to provide customers with low prices across a wide selection,” the spokesperson said in a statement.
Dama said his company has enough inventory for some products to last up to six months, which it aims to “stretch as long as possible” in the hope that China and the U.S. can reach a trade deal. The company is also forgoing some sales promotions and discounts, while pausing spend on some display and video ads.
Regarding his inventory, Dama said, “We can try to stretch that seven, eight, nine months, which buys us a lot more time for this thing to work out, hopefully.”
A Pony.ai autonomous car.
Pony.ai
Chinese start-up Pony.ai said Friday it will develop autonomous driving technology in partnership with Tencent Cloud and deploy robotaxi services on tech giant Tencent’s WeChat and other applications.
The Nasdaq-listed company which specializes in autonomous vehicle technology, particularly robotaxis and robotrucks, said in a press release that the deal will include cooperation in areas such as cloud services, map data, information security and intelligent cockpit ecosystems.
The arrangement will also see the two companies integrate Pony.ai’s robotaxi ride-hailing services within Tencent’s popular WeChat app as well as other applications like Tencent Maps.
Both companies had been in talks “for quite some time,” Pony.ai CEO James Peng told CNBC on the sidelines of the Shanghai Auto Show on Friday. He cited Tencent’s huge user base and its cloud offerings as factors supporting the “win-win” collaboration as the start-up continues to scale up.
Following the partnership, Peng said that “hopefully in the near future,” users would be able to call Pony.ai robotaxi rides straight through the WeChat app.
WeChat is known as the world’s most popular ‘super app,’ housing everything from messaging to payment transactions to food delivery services, with a monthly user base of over 1 billion people.
“Pony.ai possesses industry-leading autonomous driving technology accumulations, while Tencent excels in cloud services, mapping, and cockpit ecosystem technologies,” Vice President of Tencent Group and President of Tencent Smart Mobility Zhong Xiangping was quoted as saying in the Friday release.
“This strategic partnership between the two parties is not only about complementing each other’s technologies and resources but also marks a new starting point for collaborative innovation,” he added.
The release said that the partnership would also see both companies collaborate on the development, testing, and operation of Robotaxis, particularly in L4-level autonomous driving.
According to SAE International, L4 is a type of autonomous driving that allows drivers to take their eyes off the road in designated areas. For comparison, L3 is considered a hands-off system, but drivers must actively monitor the vehicle and be ready to take over the wheel.
The Tencent Cloud agreement comes a day after it was reported that Pony.ai unveiled its L4, seventh-generation robotaxi solution at the Shanghai Auto Show on Wednesday. The company’s shares surged about 40% in the U.S. on Thursday.
The start-up continues to establish itself as a prominent player in China’s autonomous driving industry. The company obtained China’s first permit to charge fares for fully driverless taxis in core parts of a business district of Shenzhen, where Tencent is headquartered.
However, the firm may be implicated in increasing trade tensions between China and the U.S. as the latter is a market Pony.ai considers “hugely important” to its expansion plans.
James Peng, co-founder and chief executive of Pony.ai this week reportedly told the Financial Times that the company is considering a secondary listing outside the U.S. amid mounting concerns that Washington will push for the delisting of Chinese companies off the New York Stock Exchange.
If this were to happen, it would come less than six months after the company’s initial public offering in the U.S. Notwithstanding, Peng told FT that a lot of factors need to be considered.
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports1 year agoStory injured on diving stop, exits Red Sox game
-
Sports1 year agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike
-
Business3 years agoBank of England’s extraordinary response to government policy is almost unthinkable | Ed Conway