Google’s newest A.I. model uses nearly five times more text data for training than its predecessor
Sundar Pichai, chief executive officer of Alphabet Inc., during the Google I/O Developers Conference in Mountain View, California, on Wednesday, May 10, 2023.
David Paul Morris | Bloomberg | Getty Images
Google’s new large language model, which the company announced last week, uses almost five times as much training data as its predecessor from 2022, allowing its to perform more advanced coding, math and creative writing tasks, CNBC has learned.
PaLM 2, the company’s new general-use large language model (LLM) that was unveiled at Google I/O, is trained on 3.6 trillion tokens, according to internal documentation viewed by CNBC. Tokens, which are strings of words, are an important building block for training LLMs, because they teach the model to predict the next word that will appear in a sequence.
Google’s previous version of PaLM, which stands for Pathways Language Model, was released in 2022 and trained on 780 billion tokens.
While Google has been eager to showcase the power of its artificial intelligence technology and how it can be embedded into search, emails, word processing and spreadsheets, the company has been unwilling to publish the size or other details of its training data. OpenAI, the Microsoft-backed creator of ChatGPT, has also kept secret the specifics of its latest LLM called GPT-4.
The reason for the lack of disclosure, the companies say, is the competitive nature of the business. Google and OpenAI are rushing to attract users who may want to search for information using conversational chatbots rather than traditional search engines.
But as the AI arms race heats up, the research community is demanding greater transparency.
Since unveiling PaLM 2, Google has said the new model is smaller than prior LLMs, which is significant because it means the company’s technology is becoming more efficient while accomplishing more sophisticated tasks. PaLM 2, according to internal documents, is trained on 340 billion parameters, an indication of the complexity of the model. The initial PaLM was trained on 540 billion parameters.
Google didn’t immediately provide a comment for this story.
Google said in a blog post about PaLM 2 that the model uses a “new technique” called “compute-optimal scaling.” That makes the the LLM “more efficient with overall better performance, including faster inference, fewer parameters to serve, and a lower serving cost.”
In announcing PaLM 2, Google confirmed CNBC’s previous reporting that the model is trained on 100 languages and performs a broad range of tasks. It’s already being used to power 25 features and products, including the company’s experimental chatbot Bard. It’s available in four sizes, from smallest to largest: Gecko, Otter, Bison and Unicorn.
PaLM 2 is more powerful than any existing model, based on public disclosures. Facebook’s LLM called LLaMA, which it announced in February, is trained on 1.4 trillion tokens. The last time OpenAI shared ChatGPT’s training size was with GPT-3, when the company said it was trained on 300 billion tokens at the time. OpenAI released GPT-4 in March, and said it exhibits “human-level performance” on many professional tests.
LaMDA, a conversation LLM that Google introduced two years ago and touted in February alongside Bard, was trained on 1.5 trillion tokens, according to the latest documents viewed by CNBC.
As new AI applications quickly hit the mainstream, controversies surrounding the underlying technology are getting more spirited.
El Mahdi El Mhamdi, a senior Google Research scientist, resigned in February over the company’s lack of transparency. On Tuesday, OpenAI CEO Sam Altman testified at a hearing of the Senate Judiciary subcommittee on privacy and technology, and agreed with lawmakers that a new system to deal with AI is needed.
“For a very new technology we need a new framework,” Altman said. “Certainly companies like ours bear a lot of responsibility for the tools that we put out in the world.”
— CNBC’s Jordan Novet contributed to this report.
Technology
Blip, dip, pullback or the beginning of the end? Global investors weigh in on stock sell-off
Global investor sentiment for artificial intelligence remains buoyant, despite on the ongoing stock sell-off.
European and Asia markets have seen days of consecutive losses, tracking their U.S. counterparts lower as pressures mount on AI-related stocks and their valuations. The pan-European Stoxx 600 on Tuesday notched its lowest level in a month, with major bourses opening mixed on Wednesday, while Asia-Pacific markets fell.
Stateside, stock futures were little changed overnight after major U.S. indexes extended their losses. AI-related stocks such as Nvidia, Palantir, and Microsoft are among those feeling the pressure.
“We do think this is an AI specific pullback. We don’t think this is the beginning of the bear market,” Emma Wall, head of investment analysis at Hargreaves Lansdown, told CNBC’s “Squawk Box Europe.”
When considering whether this is the “beginning of the end” or a moment marking “the big pullback,” Wall argued that while we are overdue a “major global market correction,” the current downturn is yet to bring this shift.
Many markets outside of the U.S. — particularly in Europe and the U.K. — already reflect much of the negative news, she said, adding that she sees the pressure as sector specific.
It is, however, an opportunity to rebalance portfolios, as “even taking into consideration this week, most people have had a really good run, even in AI stocks,” Wall said.
Mike Wilson, chief U.S. equity strategist and chief investment officer at Morgan Stanley, echoed this sentiment. He said markets have been in a correction for the past six weeks but “it’s not the end of the AI cycle.”
All eyes are on Nvidia, considered the bellwether of AI, as it’s due to post third-quarter earnings after the closing bell on Wednesday.
“Whatever happens tonight is, if it is a blip, is a pullback, it’s probably a dip to be bought. But I think we are in the midst of somewhat of a correction right now,” Wilson told CNBC’s “Inside India,” adding that he thinks it’s the middle-inning.
“The credit part of this spending is just beginning, meaning we’re just starting to raise money in the credit markets. It’s not like that money is going to sit there and they’re not going to spend it, which means there’s probably time on the clock with these intermittent kind of pullbacks,” he added.
Companies and investors are engaged in a delicate dance.
On one side, AI labs and their partners are making big promises and aggressive plays, according to Jason Thomas, head of global research and investment strategy at Carlyle. “But it’s not incumbent upon investors to believe them,” he told CNBC’s Julianna Tatelbaum, from the firm’s annual conference.
“Investors, of course, have to ensure that they are getting compensated for the risk that things don’t work out quite as planned, and I think that there’s a sense that perhaps there’s been some assets in the space that have been priced to best case scenarios. So I think that that’s the reassessment that’s going on right now,” he said.
Hyperscalers’ rising capex
The sell-off comes as the pace of debt dealmaking picks up, fueling speculation that it may have unsettled investors, many of whom have remained bullish on AI as long as companies post sound earnings. Google-owner Alphabet and Meta have issued bonds, for example.
“It’s not a problem, as long as the funding markets are there, meaning they’re raising the debt,” Wilson added. “I mean, there’s investors lined up,” he said.
It does however, become a problem when this is no longer the case, but “we haven’t seen that yet,” he said.
AI has fundamentally changed the strategy for many Big Tech firms, particularly when it comes to U.S. hyperscalers, which have morphed into capex-heavy companies from once asset-light businesses. Global investors are now assessing this new dynamic. Bank of America‘s latest Global Fund Managers Survey found that, for the first time in two decades, fund managers are concerned about hyperscalers “overinvesting.“
“[Hyperscalers] traded at very high price-to-book ratios, which made a lot of sense. You don’t value a money-printing machine based on the cost of the paper or based on the cost of the printing press. And that’s essentially what they were, these massive money printing machines where most of their assets were intangible, proprietary technology, the digital platforms,” said Carlyle’s Thomas.
“Now they’ve actually started to invest so much that 70% of their cash flow is being consumed by capital spending and, if you look at their book value now, 70% actually consists of property, plant and equipment, largely data centers. That’s a four-fold increase from a decade ago,” he added.
Technology
Dutch halt state intervention at Chinese-owned chipmaker Nexperia, paving way for exports to resume
-
Sports2 years agoStory injured on diving stop, exits Red Sox game
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports3 years agoButton battles heat exhaustion in NASCAR debut
-
Sports3 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment1 year agoHere are the best electric bikes you can buy at every price level in October 2024