Microsoft’s Bing A.I. made several factual errors in last week’s launch demo
Microsoft CEO Satya Nadella
Jordan Novet | CNBC
During last week’s chatbot hype, with Microsoft and Google attempting to outduel each other in showcasing early versions of artificial intelligence-powered search, more than 1 million people signed up to try Microsoft’s tool in the first 48 hours, the company said.
Microsoft CEO Satya Nadella told CNBC that the technology, which can spit out complete answers that read like they were written by a human, was “perhaps the industrial revolution brought to knowledge work.”
But for those concerned about accuracy, the AI leaves plenty to be desired.
In Microsoft’s demo in front of reporters, the ChatGPT-like technology embedded in the company’s Bing search engine analyzed earnings reports from Gap and Lululemon. In comparing its answers to the actual reports, the chatbot missed some numbers. Others appear to have been made up.
“Bing AI got some answers completely wrong during their demo. But no one noticed,” wrote independent search researcher Dmitri Brereton in a Substack post on Monday. “Instead, everyone jumped on the Bing hype train.”
Brereton identified possible factual issues in the Microsoft demo in its responses about vacuum cleaner specifications and travel plans to Mexico in addition to the financial errors. He told CNBC he wasn’t initially looking for errors, and only discovered them when he looked more closely to write a comparison of the AI unveilings from Microsoft and Google.
AI experts call the phenomenon “hallucination,” or the propensity of tools based on large language models to simply make stuff up. Last week, Google introduced a competing AI tool that also included factual errors — although the mistakes were quickly called out by viewers.
Both companies are rushing to incorporate new kinds of generative AI into search engines and are eager to show their advancements following the explosion of ChatGPT, which OpenAI introduced to the public in November. OpenAI has raised billions from Microsoft, while competing startups like Stability AI and Hugging Face also have ballooned to billion-dollar valuations in private funding rounds.
While Google has been reluctant to add AI-generated responses into search engines, citing reputational risk and safety concerns, Microsoft, in its announcement last week, stressed the short-term potential of releasing the technology to some of the public.
“I think it’s important not to be in a lab,” Nadella said. “You have to get these things out safely.”
When it came time to demo Bing AI’s response to a query on corporate earnings, there were some problems.
Yusuf Mehdi, a marketing executive at Microsoft, navigated to Gap’s investor relations site, and asked the Bing AI to summarize the “key takeaways” from the retailer’s third-quarter earnings release in November.
“Very cool. A massive time savings,” Mehdi said.
These are screen shots from Microsoft’s demo:
Kif Leswing/CNBC
Kif Leswing/CNBC
Here are some mistakes in the summary:
- Gap’s reported gross margin was 37.4%. But after excluding charges related to Yeezy, the adjusted gross margin was 38.7%.
- Gap operating margin was 4.6%, not 5.9%, a number that can’t be found in the company’s report.
- Adjusted diluted earnings per share was $0.71 adjusted, instead of $0.42, a number that’s not in the report. The figure Gap reported included an adjusted income tax benefit of about $0.33.
- Gap pulled its full-year outlook in August and said in the third-quarter report that “net sales could be down mid-single digits year-over-year in the fourth quarter.” That would imply a decline in revenue for the full year as opposed to “growth in the low double digits.” There is no forecast for operating margin or EPS.
Microsoft said it knows about the errors and that it expects Bing AI to make mistakes.
“We’re aware of this report and have analyzed its findings in our efforts to improve this experience,” a Microsoft spokesperson told CNBC. “We recognize that there is still work to be done and are expecting that the system may make mistakes during this preview period, which is why the feedback is critical so we can learn and help the models get better.”
Microsoft then asked Bing AI to compare Gap’s earnings with Lululemon’s report. Mehdi wanted Bing to pull the information from the two reports into a table.
“Look how amazing this is,” he said. “Just like that, in one table, I can get an answer to this question. Think how much time that would’ve taken otherwise.”
Here’s what the Bing AI tool returned:
Kif Leswing/CNBC
Kif Leswing/CNBC
There are several errors in the table, starting with margins.
- Lululemon’s gross margin was 55.9%, not 58.7%.
- The company’s operating margin was 19%, not 20.7%.
- Lululemon reported diluted EPS of $2, and adjusted EPS of $1.62. Bing showed a diluted EPS number of $1.65.
- Gap had $679 million in cash and cash equivalents, not $1.4 billion.
- Gap had $3.04 billion in inventory, not $1.9 billion.
Vice Chair and President at Microsoft, Brad Smith, participates in the first day of Web Summit in Lisbon, Portugal, on November 12, 2024. The largest technology conference in the world this year has 71,528 attendees from 153 countries and 3,050 companies, with AI emerging as the most represented industry. (Photo by Rita Franca/NurPhoto via Getty Images)
Nurphoto | Nurphoto | Getty Images
Microsoft plans to spend $80 billion in fiscal 2025 on the construction of data centers that can handle artificial intelligence workloads, the company said in a Friday blog post.
Over half of the expected AI infrastructure spending will take place in the U.S., Microsoft Vice Chair and President Brad Smith wrote. Microsoft’s 2025 fiscal year ends in June.
“Today, the United States leads the global AI race thanks to the investment of private capital and innovations by American companies of all sizes, from dynamic start-ups to well-established enterprises,” Smith said. “At Microsoft, we’ve seen this firsthand through our partnership with OpenAI, from rising firms such as Anthropic and xAI, and our own AI-enabled software platforms and applications.”
Several top-tier technology companies are rushing to spend billions on Nvidia graphics processing units for training and running AI models. The fast spread of OpenAI’s ChatGPT assistant, which launched in late 2022, kicked off the AI race for companies to deliver their own generative AI capabilities. Having invested more than $13 billion in OpenAI, Microsoft provides cloud infrastructure to the startup and has incorporated its models into Windows, Teams and other products.
Microsoft reported $20 billion in capital expenditures and assets acquired under finance leases worldwide, with $14.9 billion spent on property and equipment, in the first quarter of fiscal 2025. Capital expenditures will increase sequentially in the fiscal second quarter, Microsoft Chief Financial Officer Amy Hood said in October.
The company’s revenue from Azure and other cloud services grew 33% year over year, with 12 percentage points of that growth stemming from AI services.
Smith called on President-elect Donald Trump‘s incoming administration to protect the country’s leadership in AI through education and the promotion of U.S. AI technologies abroad.
“China is starting to offer developing countries subsidized access to scarce chips, and it’s promising to build local AI data centers,” Smith wrote. “The Chinese wisely recognize that if a country standardizes on China’s AI platform, it likely will continue to rely on that platform in the future.”
He added, “The best response for the United States is not to complain about the competition but to ensure we win the race ahead. This will require that we move quickly and effectively to promote American AI as a superior alternative.”
Technology
Meta replaces Global Affairs President Nick Clegg with Joel Kaplan ahead of Trump inauguration
Facebook vice president of global public policy Joel Kaplan and Facebook CEO Mark Zuckerberg leave the Elysee Presidential Palace after a meeting with French President Emmanuel Macron on May 23, 2018 in Paris, France.
Chesnot | Getty Images
Facebook parent Meta is replacing President of Global Affairs Nick Clegg with Joel Kaplan, the company’s current policy vice president and a former Republican party staffer.
The shake up comes three weeks before President-elect Donald Trump’s inauguration, and it’s the latest sign of how tech companies are positioning themselves for a new administration in Washington.
Clegg, a former British deputy prime minister, said he is stepping down, citing the new year as the right time to move on. He’ll be replaced by Kaplan, who will take on the title of Chief Global Affairs Officer.
Kaplan was a staffer under former President George W. Bush, and he appeared at the NYSE with Vice President-elect J.D. Vance and Trump in December. He also attended Supreme Court Justice Brett Kavanaugh’s confirmation hearing in 2018 as a personal friend, causing a controversy for the social media company.
“I will look forward to spending a few months handing over the reins — and to representing the company at a number of international gatherings in Q1 of this year,” Clegg wrote in a memo to his staff that he shared on Facebook on Thursday.
Clegg joined the company in 2018 after a career in British politics with the Liberal Democrats party, and he helped Meta navigate incredible scrutiny, especially over the company’s influence on elections and its efforts to control harmful content. Clegg also helped steer the company through the Cambridge Analytica scandal, in which Facebook shared user data with third-party political consultants. He also represented the company in Washington and London, frequently at panels for artificial intelligence and at congressional hearings.
“My time at the company coincided with a significant resetting of the relationship between ‘big tech’ and the societal pressures manifested in new laws, institutions and norms affecting the sector,” Clegg wrote.
In his note, Clegg said that former Federal Communications Commission chairman Kevin Martin would replace Kaplan as Meta’s vice president of global policy. He mentioned that Kaplan would work closely with David Ginsburg, the company’s vice president of global communications and public affairs.
“Nick: I’m grateful for everything you’ve done for Meta and the world these past seven years,” Meta CEO Mark Zuckerberg said in a statement. You “built a strong team to carry this work forward. I’m excited for Joel to step into this role next given his deep experience and insight leading our policy work for many years.”
Semafor first reported the news.
WATCH: Meta: Here’s why Rosenblatt Securities has set a price target of $811 for the stock
-
Sports2 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports9 months agoStory injured on diving stop, exits Red Sox game
-
Sports1 year agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports3 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike
-
Business2 years agoBank of England’s extraordinary response to government policy is almost unthinkable | Ed Conway