Google’s newest A.I. model uses nearly five times more text data for training than its predecessor
Sundar Pichai, chief executive officer of Alphabet Inc., during the Google I/O Developers Conference in Mountain View, California, on Wednesday, May 10, 2023.
David Paul Morris | Bloomberg | Getty Images
Google’s new large language model, which the company announced last week, uses almost five times as much training data as its predecessor from 2022, allowing its to perform more advanced coding, math and creative writing tasks, CNBC has learned.
PaLM 2, the company’s new general-use large language model (LLM) that was unveiled at Google I/O, is trained on 3.6 trillion tokens, according to internal documentation viewed by CNBC. Tokens, which are strings of words, are an important building block for training LLMs, because they teach the model to predict the next word that will appear in a sequence.
Google’s previous version of PaLM, which stands for Pathways Language Model, was released in 2022 and trained on 780 billion tokens.
While Google has been eager to showcase the power of its artificial intelligence technology and how it can be embedded into search, emails, word processing and spreadsheets, the company has been unwilling to publish the size or other details of its training data. OpenAI, the Microsoft-backed creator of ChatGPT, has also kept secret the specifics of its latest LLM called GPT-4.
The reason for the lack of disclosure, the companies say, is the competitive nature of the business. Google and OpenAI are rushing to attract users who may want to search for information using conversational chatbots rather than traditional search engines.
But as the AI arms race heats up, the research community is demanding greater transparency.
Since unveiling PaLM 2, Google has said the new model is smaller than prior LLMs, which is significant because it means the company’s technology is becoming more efficient while accomplishing more sophisticated tasks. PaLM 2, according to internal documents, is trained on 340 billion parameters, an indication of the complexity of the model. The initial PaLM was trained on 540 billion parameters.
Google didn’t immediately provide a comment for this story.
Google said in a blog post about PaLM 2 that the model uses a “new technique” called “compute-optimal scaling.” That makes the the LLM “more efficient with overall better performance, including faster inference, fewer parameters to serve, and a lower serving cost.”
In announcing PaLM 2, Google confirmed CNBC’s previous reporting that the model is trained on 100 languages and performs a broad range of tasks. It’s already being used to power 25 features and products, including the company’s experimental chatbot Bard. It’s available in four sizes, from smallest to largest: Gecko, Otter, Bison and Unicorn.
PaLM 2 is more powerful than any existing model, based on public disclosures. Facebook’s LLM called LLaMA, which it announced in February, is trained on 1.4 trillion tokens. The last time OpenAI shared ChatGPT’s training size was with GPT-3, when the company said it was trained on 300 billion tokens at the time. OpenAI released GPT-4 in March, and said it exhibits “human-level performance” on many professional tests.
LaMDA, a conversation LLM that Google introduced two years ago and touted in February alongside Bard, was trained on 1.5 trillion tokens, according to the latest documents viewed by CNBC.
As new AI applications quickly hit the mainstream, controversies surrounding the underlying technology are getting more spirited.
El Mahdi El Mhamdi, a senior Google Research scientist, resigned in February over the company’s lack of transparency. On Tuesday, OpenAI CEO Sam Altman testified at a hearing of the Senate Judiciary subcommittee on privacy and technology, and agreed with lawmakers that a new system to deal with AI is needed.
“For a very new technology we need a new framework,” Altman said. “Certainly companies like ours bear a lot of responsibility for the tools that we put out in the world.”
— CNBC’s Jordan Novet contributed to this report.
Technology
Bitcoin rises to a fresh record above $94,000 as investors watch Trump transition, ETF options
Jonathan Raa | Nurphoto | Getty Images
Bitcoin advanced past $94,000 on Wednesday for the first time as traders continued to monitor President-elect Donald Trump’s transition back to the White House and weighed early options trading on bitcoin ETFs.
The price of the cryptocurrency was last higher by more than 1% at $94,461.75, according to Coin Metrics. Earlier, it traded as high as $94,834.33.
Coinbase shares rose 2%. Meanwhile, MicroStrategy jumped 8%, bringing its week-to-date gains to 36%.
Bitcoin has been regularly hitting fresh records since the election, though in smaller increments since the postelection rally faded last week, on hopes that Trump will usher in a crypto-friendly era for the industry that includes a more supportive regulation and a potential national strategic bitcoin reserve or stockpile.
Bitcoin continues its climb toward $95,000
Traders this week are keeping a close eye on Trump’s appointments for Treasury Secretary and the Securities and Exchange Commission chair.
“We’re still very much in a phase of kind of pricing in the Trump trade,” said Joel Kruger, market strategist at LMAX Group.
He also pointed to the “mainstream, institutional adoption that we’re getting by way of the approval of the bitcoin and ETH spot ETFs this year” and options trading on those ETFs going live beginning Tuesday, which he called “another reflection of the maturation of the crypto market.”
Options on BlackRock’s iShares Bitcoin Trust ETF (IBIT) began trading on the Nasdaq Tuesday. The Grayscale Bitcoin Trust (GBTC), the Grayscale Bitcoin Mini Trust (BTC) and the Bitwise Bitcoin ETF (BITB) are expected to have options available Wednesday.
Elsewhere, traders are looking forward to Nvidia earnings after the bell, which could impact bitcoin’s price. The cryptocurrency often benefits from moves in risk assets broadly, more so this year as institutional investors have become more comfortable with it thanks to bitcoin ETFs.
Fabien Pinckaers, CEO of Belgian-based enterprise software startup Odoo.
Odoo
Odoo, a startup taking on SAP in the realm of enterprise software, boosted its valuation to 5 billion euros ($5.3 billion) in a secondary share round led by Alphabet‘s venture fund and Sequoia Capital.
The Belgium-based company develops open-source enterprise resource planning software, with over 80 applications available on its platform offering businesses tools for accounting, customer relationship management, human resources and e-commerce and website building.
Fabien Pinckaers, CEO and co-founder of Odoo, told CNBC in an interview this week that his company didn’t have a need to raise any primary capital as it is “cash profitable” and growing revenue at a rate of 50% year-over-year. Enterprise resource planning, he said, is “still a very fragmented market.”
“The reason everybody [has] failed [in this market] is that it’s quite complex,” Pinckaers told CNBC. “Small companies have complex needs from accounting to inventory, to website, e-commerce, point-of-sale. It’s a lot and they don’t have budget, and they need something that is simple and affordable.”
“Nobody succeeded to get both,” he added. “You have complex products like SAP that run well for large companies. But it’s complex and expensive.”
Andrew Reed, partner at Sequoia Capital, added that the market Odoo is addressing “just requires more gestation time than most startups both because the core system is very complex, and making it simple to use for small businesses and various countries is no small feat.”
Humble beginnings
Odoo “is not your traditional Silicon Valley tech story,” according to Reed.
Pinckaers opened the company’s first-ever office 22 years ago on a farm in Belgium. That was all he could afford at the time. Later, as the company started bringing in revenue, Odoo opened two additional offices in Belgium, home to the firm’s research and development, support and technical teams.
Today, Pinckaers resides in India with his family. He’s lived there for a year now, working to expand the company’s presence there, hiring more people, increasing marketing and broadening Odoo’s overall partner network.
Odoo had billings of 370 million euros last year and is on track to top 650 million of billings in 2025 — after that, the company is hoping to top the 1 billion-euro billings milestone by 2027. Billings — or the total sum of all invoices for a given year — is Odoo’s preferred metric for tracking annual revenue performance.
Around 80% of Odoo’s business today accounts for open-source software, with the remaining 20% coming from software licensed for a fee, Pinckaers said. Open source refers to a type of software that allows users to access the underlying code — most often free of charge — which they can then modify and adjust.
In no rush to IPO
Despite Odoo now being at the scale of an IPO-ready business, Pinckaers said he’s in no rush to take the company public. If anything, remaining private has given Odoo flexibility to stay focused on investing for the long term, he said.
Odoo’s private backers aren’t in a rush for the firm to go public, either. Alex Nichols, partner at Alphabet’s CapitalG, told CNBC that he’s not worried about “IPO timing,” adding that factors like public market conditions are ultimately “out of our control.”
Pinckaers built the business to the size it is today primarily by bootstrapping — that is, growing without raising external funding. Odoo hasn’t had to raise primary capital from investors in a decade, opting instead to let early investors and employees sell shares in secondary sales.
The last time Odoo secured primary funding was in 2014, when it raised $10 million in a Series B round. Prior to the latest secondary round, Odoo was most recently valued by investors at 3.2 billion euros.
Odoo’s other backers include the likes of private equity firms Summit Partners, Noshaq, and Wallonie Entreprendre, which all sold a portion of their shares to CapitalG and Sequoia as part of the 500-million-euro investment announced on Wednesday.
Even after selling a portion of its shares, Summit remains Odoo’s largest institutional shareholder. Pinckaers himself has never sold his own personal shares.
-
Sports2 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports8 months agoStory injured on diving stop, exits Red Sox game
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports1 year agoGame 1 of WS least-watched in recorded history
-
Sports3 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment1 year agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike
-
Business2 years agoBank of England’s extraordinary response to government policy is almost unthinkable | Ed Conway