Microsoft’s Bing A.I. made several factual errors in last week’s launch demo
Microsoft CEO Satya Nadella
Jordan Novet | CNBC
During last week’s chatbot hype, with Microsoft and Google attempting to outduel each other in showcasing early versions of artificial intelligence-powered search, more than 1 million people signed up to try Microsoft’s tool in the first 48 hours, the company said.
Microsoft CEO Satya Nadella told CNBC that the technology, which can spit out complete answers that read like they were written by a human, was “perhaps the industrial revolution brought to knowledge work.”
But for those concerned about accuracy, the AI leaves plenty to be desired.
In Microsoft’s demo in front of reporters, the ChatGPT-like technology embedded in the company’s Bing search engine analyzed earnings reports from Gap and Lululemon. In comparing its answers to the actual reports, the chatbot missed some numbers. Others appear to have been made up.
“Bing AI got some answers completely wrong during their demo. But no one noticed,” wrote independent search researcher Dmitri Brereton in a Substack post on Monday. “Instead, everyone jumped on the Bing hype train.”
Brereton identified possible factual issues in the Microsoft demo in its responses about vacuum cleaner specifications and travel plans to Mexico in addition to the financial errors. He told CNBC he wasn’t initially looking for errors, and only discovered them when he looked more closely to write a comparison of the AI unveilings from Microsoft and Google.
AI experts call the phenomenon “hallucination,” or the propensity of tools based on large language models to simply make stuff up. Last week, Google introduced a competing AI tool that also included factual errors — although the mistakes were quickly called out by viewers.
Both companies are rushing to incorporate new kinds of generative AI into search engines and are eager to show their advancements following the explosion of ChatGPT, which OpenAI introduced to the public in November. OpenAI has raised billions from Microsoft, while competing startups like Stability AI and Hugging Face also have ballooned to billion-dollar valuations in private funding rounds.
While Google has been reluctant to add AI-generated responses into search engines, citing reputational risk and safety concerns, Microsoft, in its announcement last week, stressed the short-term potential of releasing the technology to some of the public.
“I think it’s important not to be in a lab,” Nadella said. “You have to get these things out safely.”
When it came time to demo Bing AI’s response to a query on corporate earnings, there were some problems.
Yusuf Mehdi, a marketing executive at Microsoft, navigated to Gap’s investor relations site, and asked the Bing AI to summarize the “key takeaways” from the retailer’s third-quarter earnings release in November.
“Very cool. A massive time savings,” Mehdi said.
These are screen shots from Microsoft’s demo:
Kif Leswing/CNBC
Kif Leswing/CNBC
Here are some mistakes in the summary:
- Gap’s reported gross margin was 37.4%. But after excluding charges related to Yeezy, the adjusted gross margin was 38.7%.
- Gap operating margin was 4.6%, not 5.9%, a number that can’t be found in the company’s report.
- Adjusted diluted earnings per share was $0.71 adjusted, instead of $0.42, a number that’s not in the report. The figure Gap reported included an adjusted income tax benefit of about $0.33.
- Gap pulled its full-year outlook in August and said in the third-quarter report that “net sales could be down mid-single digits year-over-year in the fourth quarter.” That would imply a decline in revenue for the full year as opposed to “growth in the low double digits.” There is no forecast for operating margin or EPS.
Microsoft said it knows about the errors and that it expects Bing AI to make mistakes.
“We’re aware of this report and have analyzed its findings in our efforts to improve this experience,” a Microsoft spokesperson told CNBC. “We recognize that there is still work to be done and are expecting that the system may make mistakes during this preview period, which is why the feedback is critical so we can learn and help the models get better.”
Microsoft then asked Bing AI to compare Gap’s earnings with Lululemon’s report. Mehdi wanted Bing to pull the information from the two reports into a table.
“Look how amazing this is,” he said. “Just like that, in one table, I can get an answer to this question. Think how much time that would’ve taken otherwise.”
Here’s what the Bing AI tool returned:
Kif Leswing/CNBC
Kif Leswing/CNBC
There are several errors in the table, starting with margins.
- Lululemon’s gross margin was 55.9%, not 58.7%.
- The company’s operating margin was 19%, not 20.7%.
- Lululemon reported diluted EPS of $2, and adjusted EPS of $1.62. Bing showed a diluted EPS number of $1.65.
- Gap had $679 million in cash and cash equivalents, not $1.4 billion.
- Gap had $3.04 billion in inventory, not $1.9 billion.
A driver for an independent contractor to FedEx delivers packages on Cyber Monday in New York, US, on Monday, Nov. 27, 2023.
Stephanie Keith | Bloomberg | Getty Images
President Donald Trump on Wednesday signed an executive order shutting the de minimis trade loophole, effective May 2.
Trump in February abruptly ended the de minimis trade exemption, which allows shipments worth less than $800 to enter the U.S. duty-free. The order overwhelmed U.S. Customs and Border Protection employees and caused the U.S. Postal Service to temporarily halt packages from China and Hong Kong. Within days of its announcement, Trump reversed course and delayed the cancellation of the provision.
Wednesday’s announcement, which came alongside a set of sweeping new tariffs, gives customs officials, retailers and logistics companies more time to prepare. Goods that qualify under the de minimis exemption will be subject to a duty of either 30% of their value, or $25 per item. That rate will increase to $50 per item on June 1, the White House said.
Use of the de minimis provision has exploded in recent years as shoppers flock to Chinese e-commerce companies Temu and Shein, which offer ultra-low cost apparel, electronics and other items. The U.S. Customs and Border Protection has said it processed more than 1.3 billion de minimis shipments in 2024, up from over 1 billion shipments in 2023.
Critics of the provision say it provides an unfair advantage to Chinese e-commerce companies and creates an influx of packages that are “subject to minimal documentation and inspection,” raising concerns around counterfeit and unsafe goods.
The Trump administration has sought to close the loophole over concerns that it facilitates shipments of fentanyl and other illicit substances on the claims that the packages are less likely to be inspected by customs agents.
Temu and Shein have taken steps to grow their operations in the U.S. as the de minimis loophole has come under greater scrutiny. After onboarding sellers with inventory in U.S. warehouses, Temu recently began steering shoppers to those items on its website, allowing it to speed up deliveries. Shein opened distribution centers in states including Illinois and California in 2022, and a supply chain hub in Seattle last year.
WATCH: President Trump signs executive orders for reciprocal tariffs
Apple CEO Tim Cook, center, watches during the inauguration ceremonies for President Donald Trump, right, and Vice President JD Vance, left, in the rotunda of the U.S. Capitol in Washington, Jan. 20, 2025.
Shawn Thew | Afp | Getty Images
Apple slid more than 6% in late trading Wednesday and led a broader decline in tech stocks after President Donald Trump announced new tariffs of between 10% and 49% on imported goods.
The majority of Apple’s revenue comes from devices manufactured primarily in China and a handful of other Asian countries. Nvidia, which manufactures new chips in Taiwan and assembles its artificial intelligence systems in Mexico and elsewhere, fell about 4%, while electric vehicle company Tesla dropped 4.5%.
Across the rest of the megacap universe, Alphabet, Amazon and Meta all dropped between 2.5% and 5%, and Microsoft was down by almost 2%.
If Apple’s postmarket loss is matched in regular trading Thursday, it would be the steepest decline for the stock since September 2020.
Trump on Wednesday afternoon said the new taxes on imported goods would be a “declaration of economic independence” for the country. He announced a 10% blanket tariff on all imports, and higher duties for specific countries, including 34% for China, 20% for European nations, and 24% for Japanese imports, based on what tariffs they charge on U.S. exports, Trump said.
“We will supercharge our domestic industrial base, we will pry open foreign markets and break down foreign trade barriers,” Trump said during his speech. “Ultimately, more production at home will mean stronger competition and lower prices for consumers.”
Stocks broadly got hit by Trump’s announcements. An exchange-traded fund tracking the S&P 500 slid 2.8%, while an ETF following the Nasdaq 100 lost more than 3%.
During his speech, Trump praised Apple, Meta, and Nvidia for spending money and investing in the United States.
“Apple is going to spend $500 billion, they never spent money like that here,” Trump said. “They’re going to build their plants here.”
The Nasdaq just wrapped up its worst quarter since 2022, dropping 10% in the first three months of the year, though the tech-heavy index rose in each of the first two days of the second quarter.
WATCH: President Trump signs executive orders for reciprocal tariffs
Guests including Mark Zuckerberg, Lauren Sanchez, Jeff Bezos, Sundar Pichai and Elon Musk attend the Inauguration of Donald J. Trump in the U.S. Capitol Rotunda on January 20, 2025 in Washington, DC. Donald Trump takes office for his second term as the 47th president of the United States.
Julia Demaree Nikhinson | Getty Images
Amazon submitted a bid to the White House to purchase the social media app TikTok from its Chinese owners, CNBC has confirmed.
The company sent its proposal in a letter this week to Vice President JD Vance and Commerce Secretary Howard Lutnick, according to a source familiar with the matter who asked not to be named because the discussions are confidential. The parties aren’t treating the bid seriously, however, given that it was submitted just days before a deadline staving off a U.S. ban is set to expire, the person said.
Amazon declined to comment.
The e-commerce company’s offer, which was first reported by The New York Times, comes as TikTok’s fate in the U.S. is up in the air. The short-form video app faces another potential shutdown in the U.S. on April 5 if ByteDance, its parent company, can’t reach a deal to divest TikTok’s American operations. Lawmakers passed a bill last year setting a Jan. 19 deadline for the sale, but Trump signed an executive order granting a 75-day extension for a potential deal.
Trump could announce a decision on TikTok’s fate in the U.S. as soon as Wednesday, sources familiar with the situation told CNBC’s David Faber. Mobile technology company AppLovin has also made a bid for TikTok, Faber reported separately, citing sources familiar with the matter.
TikTok has emerged as a major hub for e-commerce as it has poured money into growing its online marketplace, called TikTok Shop. TikTok’s lucrative marketplace, coupled with the app’s more than 170 million users, could be an attractive asset for Amazon. Following TikTok’s success, Amazon launched and then shuttered a short-form video service of its own.
Last August, the two companies formed a partnership that allowed TikTok users to link their account with Amazon and make purchases from the site without leaving the app. The deal attracted scrutiny from lawmakers who were concerned about its potential national security risks.
WATCH: How TikTok Shop is beating Amazon and Temu in social shopping
-
Sports2 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports12 months agoStory injured on diving stop, exits Red Sox game
-
Sports1 year agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike
-
Business3 years agoBank of England’s extraordinary response to government policy is almost unthinkable | Ed Conway