ChatGPT and Generative AI are booming, but at a very expensive price
OpenAI CEO Sam Altman speaks during a keynote address announcing ChatGPT integration for Bing at Microsoft in Redmond, Washington, on February 7, 2023.
Jason Redmond | AFP | Getty Images
Before OpenAI’s ChatGPT emerged and captured the world’s attention for its ability to create compelling sentences, a small startup called Latitude was wowing consumers with its AI Dungeon game that let them use artifical intelligence to create fantastical tales based on their prompts.
But as AI Dungeon became more popular, Latitude CEO Nick Walton recalled that the cost to maintain the text-based role-playing game began to skyrocket. Powering AI Dungeon’s text-generation software was the GPT language technology offered by the Microsoft-backed artificial intelligence research lab OpenAI. The more people played AI Dungeon, the bigger the bill Latitude had to pay OpenAI.
Compounding the predicament was that Walton also discovered content marketers were using AI Dungeon to generate promotional copy, a use for AI Dungeon that his team never foresaw, but that ended up adding to the company’s AI bill.
At its peak in 2021, Walton estimates Latitude was spending nearly $200,000 a month on OpenAI’s so-called generative AI software and Amazon Web Services in order to keep up with the millions of user queries it needed to process each day.
“We joked that we had human employees and we had AI employees, and we spent about as much on each of them,” Walton said. “We spent hundreds of thousands of dollars a month on AI and we are not a big startup, so it was a very massive cost.”
By the end of 2021, Latitude switched from using OpenAI’s GPT software to a cheaper but still capable language software offered by startup AI21 Labs, Walton said, adding that the startup also incorporated open source and free language models into its service to lower the cost. Latitude’s generative AI bills have dropped to under $100,000 a month, Walton said, and the startup charges players a monthly subscription for more advanced AI features to help reduce the cost.
Latitude’s pricey AI bills underscore an unpleasant truth behind the recent boom in generative AI technologies: The cost to develop and maintain the software can be extraordinarily high, both for the firms that develop the underlying technologies, generally referred to as a large language or foundation models, and those that use the AI to power their own software.
The high cost of machine learning is an uncomfortable reality in the industry as venture capitalists eye companies that could potentially be worth trillions, and big companies such as Microsoft, Meta, and Google use their considerable capital to develop a lead in the technology that smaller challengers can’t catch up to.
But if the margin for AI applications is permanently smaller than previous software-as-a-service margins, because of the high cost of computing, it could put a damper on the current boom.
The high cost of training and “inference” — actually running — large language models is a structural cost that differs from previous computing booms. Even when the software is built, or trained, it still requires a huge amount of computing power to run large language models because they do billions of calculations every time they return a response to a prompt. By comparison, serving web apps or pages requires much less calculation.
These calculations also require specialized hardware. While traditional computer processors can run machine learning models, they’re slow. Most training and inference now takes place on graphics processors, or GPUs, which were initially intended for 3D gaming, but have become the standard for AI applications because they can do many simple calculations simultaneously.
Nvidia makes most of the GPUs for the AI industry, and its primary data center workhorse chip costs $10,000. Scientists that build these models often joke that they “melt GPUs.”
Training models
Nvidia A100 processor
Nvidia
Analysts and technologists estimate that the critical process of training a large language model such as GPT-3 could cost more than $4 million. More advanced language models could cost over “the high-single digit-millions” to train, said Rowan Curran, a Forrester analyst who focuses on AI and machine learning.
Meta’s largest LLaMA model released last month, for example, used 2,048 Nvidia A100 GPUs to train on 1.4 trillion tokens (750 words is about 1,000 tokens), taking about 21 days, the company said when it released the model last month.
It took about 1 million GPU hours to train. With dedicated prices from AWS, that would cost over $2.4 million. And at 65 billion parameters, it’s smaller than the current GPT models at OpenAI, like ChatGPT-3, which has 175 billion parameters.
Clement Delangue, the CEO of AI startup Hugging Face, said the process of training the company’s Bloom large language model took more than two-and-a-half months and required access to a supercomputer that was “something like the equivalent of 500 GPUs.”
Organizations that build large language models must be cautious when they retrain the software, which helps the software improve its abilities, because it costs so much, he said.
“It’s important to realize that these models are not trained all the time, like every day,” Delangue said, noting that’s why some models, like ChatGPT, don’t have knowledge of recent events. ChatGPT’s knowledge stops in 2021, he said.
“We are actually doing a training right now for the version two of Bloom and it’s gonna cost no more than $10 million to retrain,” Delangue said. “So that’s the kind of thing that we don’t want to do every week.”
Inference and who pays for it
Bing with Chat
Jordan Novet | CNBC
To use a trained machine learning model to make predictions or generate text, engineers use the model in a process called “inference,” which can be much more expensive than training because it might need to run millions of times for a popular product.
For a product as popular as ChatGPT — which investment firm UBS estimates to have reached 100 million monthly active users in January — Curran believes that it could have cost OpenAI $40 million to process the millions of prompts people fed into the software that month.
Costs skyrocket when these tools are used billions of times a day. Financial analysts estimate Microsoft’s Bing AI chatbot, which is powered by an OpenAI ChatGPT model, needs at least $4 billion of infrastructure to serve responses to all Bing users.
In the case of Latitude, for instance, while the startup didn’t have to pay to train the underlying OpenAI language model it was accessing, it had to account for the inferencing costs that were something akin to “half-a-cent per call” on “a couple million requests per day,” a Latitude spokesperson said.
“And I was being relatively conservative,” Curran said of his calculations.
In order to sow the seeds of the current AI boom, venture capitalists and tech giants have been investing billions of dollars into startups that specialize in generative AI technologies. Microsoft, for instance, invested as much as $10 billion into GPT’s overseer OpenAI, according to media reports in January. Salesforce‘s venture capital arm, Salesforce Ventures, recently debuted a $250 million fund that caters to generative AI startups.
As investor Semil Shah of the VC firms Haystack and Lightspeed Venture Partners described on Twitter, “VC dollars shifted from subsidizing your taxi ride and burrito delivery to LLMs and generative AI compute.”
Many entrepreneurs see risks in relying on potentially subsidized AI models that they don’t control and merely pay for on a per-use basis.
“When I talk to my AI friends at the startup conferences, this is what I tell them: Do not solely depend on OpenAI, ChatGPT or any other large language models,” said Suman Kanuganti, founder of personal.ai, a chatbot currently in beta mode. “Because businesses shift, they are all owned by big tech companies, right? If they cut access, you’re gone.”
Companies such as enterprise tech firm Conversica are exploring how they can use the tech through Microsoft’s Azure cloud service at its currently discounted price.
While Conversica CEO Jim Kaskade declined to comment about how much the startup is paying, he conceded that the subsidized cost is welcome as it explores how language models can be used effectively.
“If they were truly trying to break even, they’d be charging a hell of a lot more,” Kaskade said.
How it could change
It’s unclear if AI computation will stay expensive as the industry develops. Companies making the foundation models, semiconductor makers and startups all see business opportunities in reducing the price of running AI software.
Nvidia, which has about 95% of the market for AI chips, continues to develop more powerful versions designed specifically for machine learning, but improvements in total chip power across the industry have slowed in recent years.
Still, Nvidia CEO Jensen Huang believes that in 10 years, AI will be “a million times” more efficient because of improvements not only in chips, but also in software and other computer parts.
“Moore’s Law, in its best days, would have delivered 100x in a decade,” Huang said last month on an earnings call. “By coming up with new processors, new systems, new interconnects, new frameworks and algorithms, and working with data scientists, AI researchers on new models, across that entire span, we’ve made large language model processing a million times faster.”
Some startups have focused on the high cost of AI as a business opportunity.
“Nobody was saying ‘You should build something that was purpose-built for inference.’ What would that look like?” said Sid Sheth, founder of D-Matrix, a startup building a system to save money on inference by doing more processing in the computer’s memory, as opposed to on a GPU.
“People are using GPUs today, NVIDIA GPUs, to do most of their inference. They buy the DGX systems that NVIDIA sells that cost a ton of money. The problem with inference is if the workload spikes very rapidly, which is what happened to ChatGPT, it went to like a million users in five days. There is no way your GPU capacity can keep up with that because it was not built for that. It was built for training, for graphics acceleration,” he said.
Delangue, the HuggingFace CEO, believes more companies would be better served focusing on smaller, specific models that are cheaper to train and run, instead of the large language models that are garnering most of the attention.
Meanwhile, OpenAI announced last month that it’s lowering the cost for companies to access its GPT models. It now charges one-fifth of one cent for about 750 words of output.
OpenAI’s lower prices have caught the attention of AI Dungeon-maker Latitude.
“I think it’s fair to say that it’s definitely a huge change we’re excited to see happen in the industry and we’re constantly evaluating how we can deliver the best experience to users,” a Latitude spokesperson said. “Latitude is going to continue to evaluate all AI models to be sure we have the best game out there.”
Watch: AI’s “iPhone Moment” – Separating ChatGPT Hype and Reality
Microsoft CEO Satya Nadella speaks during the Microsoft Build conference at Microsoft headquarters in Redmond, Washington, on May 21, 2024.
Jason Redmond | AFP | Getty Images
A half-century ago, childhood friends Bill Gates and Paul Allen started Microsoft from a strip mall in Albuquerque, New Mexico. Five decades and almost $3 trillion later, the company celebrates its 50th birthday on Friday from its sprawling campus in Redmond, Washington.
Now the second most valuable publicly traded company in the world, Microsoft has only had three CEOs in its history, and all of them are in attendance for the monumental event. One is current CEO Satya Nadella. The other two are Gates and Steve Ballmer, both among the 11 richest people in the world due to their Microsoft fortunes.
While Microsoft has mostly been on the ascent of late, with Nadella turning the company into a major power player in cloud computing and artificial intelligence, the birthday party lands at an awkward moment.
The company’s stock price has dropped for four consecutive months for the first time since 2009 and just suffered its steepest quarterly drop in three years. That was all before President Donald Trump’s announcement this week of sweeping tariffs, which sent the Nasdaq tumbling on Thursday and Microsoft down another 2.4%.
Cloud computing has been Microsoft’s main source of new revenue since Nadella took over from Ballmer as CEO in 2014. But the Azure cloud reported disappointing revenue in the latest quarter, a miss that finance chief Amy Hood attributed in January to power and space shortages and a sales posture that focused too much on AI. Hood said revenue growth in the current quarter will fall to 10% from 17% a year earlier
Nadella said management is refining sales incentives to maximize revenue from traditional workloads, while positioning the company to benefit from the ongoing AI boom.
“You would rather win the new than just protect the past,” Nadella told analysts on a conference call.
The past remains healthy. Microsoft still generates around one-fifth of its roughly $262 billion in annual revenue from productivity software, mostly from commercial clients. Windows makes up around 10% of sales.
Meanwhile, the company has used its massive cash pile to orchestrate its three largest acquisitions on record in a little over eight years, snapping up LinkedIn in late 2016, Nuance Communications in 2022 and Activision Blizzard in 2023, for a combined $121 billion.
“Microsoft has figured out how to stay ahead of the curve, and 50 years later, this is a company that can still be on the forefront of technology innovation,” said Soma Somasegar, a former Microsoft executive who now invests in startups at venture firm Madrona. “That’s a commendable place for the company to be in.”
When Somasegar gave up his corporate vice president position at Microsoft in 2015, the company was fresh off a $7.6 billion write-down from Ballmer’s ill-timed purchase of Nokia’s devices and services business.
Microsoft is now in a historic phase of investment. The company has built a $13.8 billion stake in OpenAI and last year spent almost $76 billion on capital expenditures and finance leases, up 83% from a year prior, partly to enable the use of AI models in the Azure cloud. In January, Nadella said Microsoft has $13 billion in annualized AI revenue, more even than OpenAI, which just closed a financing round valuing the company at $300 billion.
Microsoft’s spending spree has constrained free cash flow growth. Guggenheim analysts wrote in a note after the company’s earnings report in January, “You just have to believe in the future.”
Of the 35 Microsoft analysts tracked by FactSet, 32 recommend buying the stock, which has appreciated tenfold since Nadella became CEO. Azure has become a fearsome threat to Amazon Web Services, which pioneered the cloud market in the 2000s, and startups as well as enterprises are flocking to its cloud technology.
Winston Weinberg, CEO of legal AI startup Harvey, uses OpenAI models through Azure. Weinberg lauded Nadella’s focus on customers of all sizes.
“Satya has literally responded to emails within 15 minutes of us having a technical problem, and he’ll route it to the right person,” Weinberg said.
Still, technology is moving at an increasingly rapid pace and Microsoft’s ability to stay on top is far from guaranteed. Industry experts highlighted four key issues the company has to address as it pushes into its next half-century.
Microsoft didn’t respond to a request for comment.
Regulation
There’s some optimism that the Trump administration and a new head of the Federal Trade Commission will open up the door to the kinds of deal-making that proved very challenging during Joe Biden’s presidency, when Lina Khan headed the FTC.
But regulatory uncertainty remains.
It’s not a new risk for Microsoft. In 1995, the company paid a $46 million breakup fee to tax software maker Intuit after the Justice Department filed suit to block the proposed deal. Years later, the DOJ got Microsoft to revamp some of its practices after a landmark antitrust case.
Microsoft pushed through its largest acquisition ever, the $75 billion purchase of video game publisher Activision, during Biden’s term. But only after a protracted legal battle with the FTC.
At the very end of Biden’s time in office, the FTC opened an antitrust investigation on Microsoft. That probe is ongoing, Bloomberg reported in March.
Nadella has cultivated a relationship with Trump. In January, the two reportedly met for lunch at Trump’s Mar-a-Lago resort in Florida, alongside Tesla CEO Elon Musk.
President Donald Trump shakes hands with Microsoft CEO Satya Nadella during an American Technology Council roundtable at the White House in Washington on June 19, 2017.
Nicholas Kamm | AFP | Getty Images
The U.S. isn’t the only concern. The U.K.’s Competition and Markets Authority said in January that an independent inquiry found that “Microsoft is using its strong position in software to make it harder for AWS and Google to compete effectively for cloud customers that wish to use Microsoft software on the cloud.”
Microsoft last year committed to unbundling Teams from Microsoft 365 productivity software subscriptions globally to address concerns from the European Union’s executive arm, the European Commission.
Noncore markets
Fairly early in Microsoft’s history the company became the world’s largest software maker. And in cloud, Microsoft is the biggest challenger to AWS. Most of the company’s revenue comes from corporations, schools and governments.
But Microsoft is in other markets where its position is weaker. Those include video games, laptops and search advertising.
Mary Jo Foley, editor in chief at advisory group Directions on Microsoft, said the company may be better off focusing on what it does best, rather than continuing to offer Xbox consoles and Surface tablets.
“Microsoft is not good at anything in the consumer space (with the possible exception of gaming),” wrote Foley, who has covered the company on and off since 1984. “You’re wasting time and money on trying to figure it out. Microsoft is an enterprise company — and that is more than OK.”
It’s unlikely Microsoft will back away from games, particularly after the Activision deal. Nearly $12 billion of Microsoft’s $69.6 billion in fourth-quarter revenue came from gaming, search and news advertising, and consumer subscriptions to the Microsoft 365 productivity bundle. That doesn’t include sales of devices, Windows licenses or advertising on LinkedIn.
“As a company, Microsoft’s all-in on gaming,” Nadella said in 2021 in an appearance alongside gaming unit head Phil Spencer. “We believe we can play a leading role in democratizing gaming and defining that future of interactive entertainment, quite frankly, at scale.”
AI pressure
Microsoft has an unquestionably strong position in AI today, thanks in no small part to its early alliance with OpenAI. Microsoft has added the startup’s AI models to Windows, Excel, Bing and other products.
The breakout has been GitHub Copilot, which generates source code and answers developers’ questions. GitHub reached $2 billion in annualized revenue last year, with Copilot accounting for more than 40% of sales growth for the business. Microsoft bought GitHub in 2018 for $7.5 billion.
Microsoft CEO Satya Nadella, right, speaks as OpenAI CEO Sam Altman looks on during the OpenAI DevDay event in San Francisco on Nov. 6, 2023.
Justin Sullivan | Getty Images
But speedy deployment in AI can be worrisome.
The company is “not providing the underpinnings needed to deploy AI properly, in terms of security and governance — all because they care more about being ‘first,'” Foley wrote. Microsoft also hasn’t been great at helping customers understand the return on investment, she wrote.
AI-ready Copilot+ PCs, which Microsoft introduced last year, aren’t gaining much traction. The company had to delay the release of the Recall search feature to prevent data breaches. And the Copilot assistant subscription, at $30 a month for customers of the Microsoft 365 productivity suite, hasn’t become pervasive in the business world.
“Copilot was really their chance to take the lead,” said Jason Wong, an analyst at technology industry researcher Gartner. “But increasingly, what it’s seeming like is Copilot is just an add-on and not like a net-new thing to drive AI.”
Innovation
At 50, the biggest question facing Microsoft is whether it can still build impressive technology on its own. Products like the Surface and HoloLens augmented reality headset generated buzz, but they hit the market years ago.
Teams was a novel addition to its software bundle, though the app’s success came during the Covid pandemic after the explosive growth in products like Zoom and Slack, which Salesforce acquired. And Microsoft is still researching quantum computing.
In AI, Microsoft’s best bet so far was its investment in OpenAI. Somasegar said Microsoft is in prime position to be a big player in the market.
“To me, it’s been 2½ years since ChatGPT showed up, and we are not even at the Uber and Airbnb moment,” Somasegar said. “There is a tremendous amount of value creation that needs to happen in AI. Microsoft as much as everybody else is thinking, ‘What does that mean? How do we get there?'”
Technology
Nvidia positioned to weather Trump tariffs, chip demand ‘off the charts,’ says Altimeter’s Gerstner
Altimeter Capital CEO Brad Gerstner said Thursday that he’s moving out of the “bomb shelter” with Nvidia and into a position of safety, expecting that the chipmaker is positioned to withstand President Donald Trump’s widespread tariffs.
“The growth and the demand for GPUs is off the charts,” he told CNBC’s “Fast Money Halftime Report,” referring to Nvidia’s graphics processing units that are powering the artificial intelligence boom. He said investors just need to listen to commentary from OpenAI, Google and Elon Musk.
President Trump announced an expansive and aggressive “reciprocal tariff” policy in a ceremony at the White House on Wednesday. The plan established a 10% baseline tariff, though many countries like China, Vietnam and Taiwan are subject to steeper rates. The announcement sent stocks tumbling on Thursday, with the tech-heavy Nasdaq down more than 5%, headed for its worst day since 2022.
The big reason Nvidia may be better positioned to withstand Trump’s tariff hikes is because semiconductors are on the list of exceptions, which Gerstner called a “wise exception” due to the importance of AI.
Nvidia’s business has exploded since the release of OpenAI’s ChatGPT in 2022, and annual revenue has more than doubled in each of the past two fiscal years. After a massive rally, Nvidia’s stock price has dropped by more than 20% this year and was down almost 7% on Thursday.
Gerstner is concerned about the potential of a recession due to the tariffs, but is relatively bullish on Nvidia, and said the “negative impact from tariffs will be much less than in other areas.”
He said it’s key for the U.S. to stay competitive in AI. And while the company’s chips are designed domestically, they’re manufactured in Taiwan “because they can’t be fabricated in the U.S.” Higher tariffs would punish companies like Meta and Microsoft, he said.
“We’re in a global race in AI,” Gerstner said. “We can’t hamper our ability to win that race.”
-
Sports2 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports12 months agoStory injured on diving stop, exits Red Sox game
-
Sports1 year agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike
-
Business3 years agoBank of England’s extraordinary response to government policy is almost unthinkable | Ed Conway