Google’s newest A.I. model uses nearly five times more text data for training than its predecessor
Sundar Pichai, chief executive officer of Alphabet Inc., during the Google I/O Developers Conference in Mountain View, California, on Wednesday, May 10, 2023.
David Paul Morris | Bloomberg | Getty Images
Google’s new large language model, which the company announced last week, uses almost five times as much training data as its predecessor from 2022, allowing its to perform more advanced coding, math and creative writing tasks, CNBC has learned.
PaLM 2, the company’s new general-use large language model (LLM) that was unveiled at Google I/O, is trained on 3.6 trillion tokens, according to internal documentation viewed by CNBC. Tokens, which are strings of words, are an important building block for training LLMs, because they teach the model to predict the next word that will appear in a sequence.
Google’s previous version of PaLM, which stands for Pathways Language Model, was released in 2022 and trained on 780 billion tokens.
While Google has been eager to showcase the power of its artificial intelligence technology and how it can be embedded into search, emails, word processing and spreadsheets, the company has been unwilling to publish the size or other details of its training data. OpenAI, the Microsoft-backed creator of ChatGPT, has also kept secret the specifics of its latest LLM called GPT-4.
The reason for the lack of disclosure, the companies say, is the competitive nature of the business. Google and OpenAI are rushing to attract users who may want to search for information using conversational chatbots rather than traditional search engines.
But as the AI arms race heats up, the research community is demanding greater transparency.
Since unveiling PaLM 2, Google has said the new model is smaller than prior LLMs, which is significant because it means the company’s technology is becoming more efficient while accomplishing more sophisticated tasks. PaLM 2, according to internal documents, is trained on 340 billion parameters, an indication of the complexity of the model. The initial PaLM was trained on 540 billion parameters.
Google didn’t immediately provide a comment for this story.
Google said in a blog post about PaLM 2 that the model uses a “new technique” called “compute-optimal scaling.” That makes the the LLM “more efficient with overall better performance, including faster inference, fewer parameters to serve, and a lower serving cost.”
In announcing PaLM 2, Google confirmed CNBC’s previous reporting that the model is trained on 100 languages and performs a broad range of tasks. It’s already being used to power 25 features and products, including the company’s experimental chatbot Bard. It’s available in four sizes, from smallest to largest: Gecko, Otter, Bison and Unicorn.
PaLM 2 is more powerful than any existing model, based on public disclosures. Facebook’s LLM called LLaMA, which it announced in February, is trained on 1.4 trillion tokens. The last time OpenAI shared ChatGPT’s training size was with GPT-3, when the company said it was trained on 300 billion tokens at the time. OpenAI released GPT-4 in March, and said it exhibits “human-level performance” on many professional tests.
LaMDA, a conversation LLM that Google introduced two years ago and touted in February alongside Bard, was trained on 1.5 trillion tokens, according to the latest documents viewed by CNBC.
As new AI applications quickly hit the mainstream, controversies surrounding the underlying technology are getting more spirited.
El Mahdi El Mhamdi, a senior Google Research scientist, resigned in February over the company’s lack of transparency. On Tuesday, OpenAI CEO Sam Altman testified at a hearing of the Senate Judiciary subcommittee on privacy and technology, and agreed with lawmakers that a new system to deal with AI is needed.
“For a very new technology we need a new framework,” Altman said. “Certainly companies like ours bear a lot of responsibility for the tools that we put out in the world.”
— CNBC’s Jordan Novet contributed to this report.
Oracle CEO Safra Catz speaks at the FII PRIORITY Summit in Miami Beach, Florida, on Feb. 20, 2025.
Joe Raedle | Getty Images
Oracle shares jumped more than 5% after a recent filing showed a cloud deal that would add over $30 billion annually.
CEO Safra Catz is slated to share the deal news at a company meeting Monday, according to a filing with the Securities and Exchange Commission. The revenues are expected to start hitting in the 2028 fiscal year.
“Oracle is off to a strong start in FY26,” Catz is expected to say, according to the filing. “Our MultiCloud database revenue continues to grow at over 100%, and we signed multiple large cloud services agreements including one that is expected to contribute more than $30 billion in annual revenue starting in FY28.”
The deals revealed Monday by Catz will not affect the company’s 2026 guidance, according to the filing.
U.S. President Donald Trump announced on April 4 that he would again postpone enforcement of a law banning TikTok unless its Chinese owner ByteDance divests from the platform.
Vcg | Visual China Group | Getty Images
U.S. President Donald Trump told Fox News in an interview aired on Sunday that he has a group of “very wealthy people” ready to buy TikTok, whose identities he can reveal in about two weeks.
Trump added that the deal will probably need Beijing’s approval to move forward, but said “I think President Xi will probably do it,” in reference to China’s leader Xi Jinping.
The president made the off-the-cuff remarks while discussing the possibility of another pause of his “reciprocal” tariffs on Fox News’ “Sunday Morning Futures with Maria Bartiromo.”
Tiktok’s fate in the U.S. has been in doubt since the approval of a law in 2024 that sought to ban the platform unless its Chinese owner, ByteDance, divested from it. The legislation was driven by concerns that the Chinese government could manipulate content and access sensitive data from American users.
Earlier this month, Trump extended the deadline for ByteDance to divest from the platform’s U.S. business. It was his third extension since the Supreme Court upheld the TikTok law just a few days before Trump’s second presidential inauguration in January. The new deadline is Sept. 17.
The Protecting Americans from Foreign Adversary Controlled Applications Act, of PAFACA, had originally been set to take effect on Jan. 19, after which app store operators and internet service providers would be penalized for supporting TikTok.
TikTok went dark in the U.S. ahead of the original deadline, but was restored after Trump provided it with assurances on the extension.
Trump, who credited the app with boosting his support among young voters in the last presidential election, has maintained that he would like to see the platform stay afloat under new ownership.
Potential buyers that have voiced interest in the app include Trump insiders such as Oracle’s Larry Ellison to firms like AppLovin and Perplexity AI.
However, it’s unclear if ByteDance would be willing to sell the company. Any potential divestiture is likely to require approval from the Chinese government.
A deal that would have spun off TikTok’s U.S. operations and allowed ByteDance to retain a minority position had been in the works in April, but was derailed by the announcement of Donald Trump’s tariffs on China, Reuters reported that month.
The president previously floated a proposal for American stakeholders to buy the company and then sell a 50% stake to the U.S. government as part of a joint venture.
Experts have previously told CNBC that any potential deal could face legal challenges in the U.S., depending on whether it complies with PAFACA.
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports1 year agoStory injured on diving stop, exits Red Sox game
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Sports2 years agoButton battles heat exhaustion in NASCAR debut
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike