Microsoft’s Bing A.I. made several factual errors in last week’s launch demo
Microsoft CEO Satya Nadella
Jordan Novet | CNBC
During last week’s chatbot hype, with Microsoft and Google attempting to outduel each other in showcasing early versions of artificial intelligence-powered search, more than 1 million people signed up to try Microsoft’s tool in the first 48 hours, the company said.
Microsoft CEO Satya Nadella told CNBC that the technology, which can spit out complete answers that read like they were written by a human, was “perhaps the industrial revolution brought to knowledge work.”
But for those concerned about accuracy, the AI leaves plenty to be desired.
In Microsoft’s demo in front of reporters, the ChatGPT-like technology embedded in the company’s Bing search engine analyzed earnings reports from Gap and Lululemon. In comparing its answers to the actual reports, the chatbot missed some numbers. Others appear to have been made up.
“Bing AI got some answers completely wrong during their demo. But no one noticed,” wrote independent search researcher Dmitri Brereton in a Substack post on Monday. “Instead, everyone jumped on the Bing hype train.”
Brereton identified possible factual issues in the Microsoft demo in its responses about vacuum cleaner specifications and travel plans to Mexico in addition to the financial errors. He told CNBC he wasn’t initially looking for errors, and only discovered them when he looked more closely to write a comparison of the AI unveilings from Microsoft and Google.
AI experts call the phenomenon “hallucination,” or the propensity of tools based on large language models to simply make stuff up. Last week, Google introduced a competing AI tool that also included factual errors — although the mistakes were quickly called out by viewers.
Both companies are rushing to incorporate new kinds of generative AI into search engines and are eager to show their advancements following the explosion of ChatGPT, which OpenAI introduced to the public in November. OpenAI has raised billions from Microsoft, while competing startups like Stability AI and Hugging Face also have ballooned to billion-dollar valuations in private funding rounds.
While Google has been reluctant to add AI-generated responses into search engines, citing reputational risk and safety concerns, Microsoft, in its announcement last week, stressed the short-term potential of releasing the technology to some of the public.
“I think it’s important not to be in a lab,” Nadella said. “You have to get these things out safely.”
When it came time to demo Bing AI’s response to a query on corporate earnings, there were some problems.
Yusuf Mehdi, a marketing executive at Microsoft, navigated to Gap’s investor relations site, and asked the Bing AI to summarize the “key takeaways” from the retailer’s third-quarter earnings release in November.
“Very cool. A massive time savings,” Mehdi said.
These are screen shots from Microsoft’s demo:
Kif Leswing/CNBC
Kif Leswing/CNBC
Here are some mistakes in the summary:
- Gap’s reported gross margin was 37.4%. But after excluding charges related to Yeezy, the adjusted gross margin was 38.7%.
- Gap operating margin was 4.6%, not 5.9%, a number that can’t be found in the company’s report.
- Adjusted diluted earnings per share was $0.71 adjusted, instead of $0.42, a number that’s not in the report. The figure Gap reported included an adjusted income tax benefit of about $0.33.
- Gap pulled its full-year outlook in August and said in the third-quarter report that “net sales could be down mid-single digits year-over-year in the fourth quarter.” That would imply a decline in revenue for the full year as opposed to “growth in the low double digits.” There is no forecast for operating margin or EPS.
Microsoft said it knows about the errors and that it expects Bing AI to make mistakes.
“We’re aware of this report and have analyzed its findings in our efforts to improve this experience,” a Microsoft spokesperson told CNBC. “We recognize that there is still work to be done and are expecting that the system may make mistakes during this preview period, which is why the feedback is critical so we can learn and help the models get better.”
Microsoft then asked Bing AI to compare Gap’s earnings with Lululemon’s report. Mehdi wanted Bing to pull the information from the two reports into a table.
“Look how amazing this is,” he said. “Just like that, in one table, I can get an answer to this question. Think how much time that would’ve taken otherwise.”
Here’s what the Bing AI tool returned:
Kif Leswing/CNBC
Kif Leswing/CNBC
There are several errors in the table, starting with margins.
- Lululemon’s gross margin was 55.9%, not 58.7%.
- The company’s operating margin was 19%, not 20.7%.
- Lululemon reported diluted EPS of $2, and adjusted EPS of $1.62. Bing showed a diluted EPS number of $1.65.
- Gap had $679 million in cash and cash equivalents, not $1.4 billion.
- Gap had $3.04 billion in inventory, not $1.9 billion.
CEO of Palantir Technologies Alex Karp attends the Pennsylvania Energy and Innovation Summit, at Carnegie Mellon University in Pittsburgh, Pennsylvania, U.S., July 15, 2025.
Nathan Howard | Reuters
It’s been a tough November for Palantir.
Shares of the software analytics provider dropped 16% for their worst month since August 2023 as investors dumped AI stocks due to valuation fears. Meanwhile, famed investor Michael Burry doubled down on the artificial intelligence trade and bet against the company.
Palantir started November off on a high note.
The Denver-based company topped Wall Street’s third-quarter earnings and revenue expectations. Palantir also posted its second-straight $1 billion revenue quarter, but high valuation concerns contributed to a post-print selloff.
In a note to clients, Jefferies analysts called Palantir’s valuation “extreme” and argued investors would find better risk-reward in AI names such as Microsoft and Snowflake. Analysts at RBC Capital Markets raised concerns about the company’s “increasingly concentrated growth profile,” while Deutsche Bank called the valuation “very difficult to wrap our heads around.”
Adding fuel to the post-earnings selloff was the revelation that Burry is betting against Palantir and AI chipmaker Nvidia. Burry, who is widely known for predicting the housing crisis that occurred in 2008 and the portrayal of him in the film “The Big Short,” later accused hyperscalers of artificially boosting earnings.
Palantir CEO Alex Karp vocally hit the front lines, appearing twice in one week on CNBC, where he accused Burry of “market manipulation” and called the investor’s actions “egregious.”
“The idea that chips and ontology is what you want to short is bats— crazy,” Karp told CNBC’s “Squawk Box.”
Despite the vicious selloff, Palantir has notched some deal wins this month. That included a multiyear contract with consulting firm PwC to speed up AI adoption in the U.K. and a deal with aircraft engine maintenance company FTAI.
But those announcements did little to shake off valuation worries that have haunted all AI-tied companies in November.
Across the board, investors have viciously ditched the high-priced group, citing fears of stretched valuations and a bubble.
In November, Nvidia pulled back more than 12%, while Microsoft and Amazon dropped about 5% each. Quantum computing names such as Rigetti Computing and D-Wave Quantum have shed more than a third of their value.
Apple and Alphabet were the only Magnificent 7 stocks to end the month with gains.
Sill, questions linger over Palantir’s valuation, and those worries aren’t a new concern.
Even after its steep price drop, the company’s stock trades at 233 times forward earnings. By comparison, Nvidia and Alphabet traded at about 38 times and 30 times, respectively, at Friday’s close.
Karp, who has long defended the company, didn’t miss an opportunity to clap back at his critics, arguing in a letter to shareholders that the company is making it feasible for everyday investors to attain rates of return once “limited to the most successful venture capitalists in Palo Alto.”
“Please turn on the conventional television and see how unhappy those that didn’t invest in us are,” Karp said during an earnings call. “Enjoy, get some popcorn. They’re crying. We are every day making this company better, and we’re doing it for this nation, for allied countries.”
Palantir declined to comment for this story.
WATCH: Palantir CEO Alex Karp: We’ve printed venture results for the average American
-
Sports2 years agoStory injured on diving stop, exits Red Sox game
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports3 years agoButton battles heat exhaustion in NASCAR debut
-
Sports3 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment3 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment1 year agoHere are the best electric bikes you can buy at every price level in October 2024