ChatGPT and Generative AI are booming, but at a very expensive price
OpenAI CEO Sam Altman speaks during a keynote address announcing ChatGPT integration for Bing at Microsoft in Redmond, Washington, on February 7, 2023.
Jason Redmond | AFP | Getty Images
Before OpenAI’s ChatGPT emerged and captured the world’s attention for its ability to create compelling sentences, a small startup called Latitude was wowing consumers with its AI Dungeon game that let them use artifical intelligence to create fantastical tales based on their prompts.
But as AI Dungeon became more popular, Latitude CEO Nick Walton recalled that the cost to maintain the text-based role-playing game began to skyrocket. Powering AI Dungeon’s text-generation software was the GPT language technology offered by the Microsoft-backed artificial intelligence research lab OpenAI. The more people played AI Dungeon, the bigger the bill Latitude had to pay OpenAI.
Compounding the predicament was that Walton also discovered content marketers were using AI Dungeon to generate promotional copy, a use for AI Dungeon that his team never foresaw, but that ended up adding to the company’s AI bill.
At its peak in 2021, Walton estimates Latitude was spending nearly $200,000 a month on OpenAI’s so-called generative AI software and Amazon Web Services in order to keep up with the millions of user queries it needed to process each day.
“We joked that we had human employees and we had AI employees, and we spent about as much on each of them,” Walton said. “We spent hundreds of thousands of dollars a month on AI and we are not a big startup, so it was a very massive cost.”
By the end of 2021, Latitude switched from using OpenAI’s GPT software to a cheaper but still capable language software offered by startup AI21 Labs, Walton said, adding that the startup also incorporated open source and free language models into its service to lower the cost. Latitude’s generative AI bills have dropped to under $100,000 a month, Walton said, and the startup charges players a monthly subscription for more advanced AI features to help reduce the cost.
Latitude’s pricey AI bills underscore an unpleasant truth behind the recent boom in generative AI technologies: The cost to develop and maintain the software can be extraordinarily high, both for the firms that develop the underlying technologies, generally referred to as a large language or foundation models, and those that use the AI to power their own software.
The high cost of machine learning is an uncomfortable reality in the industry as venture capitalists eye companies that could potentially be worth trillions, and big companies such as Microsoft, Meta, and Google use their considerable capital to develop a lead in the technology that smaller challengers can’t catch up to.
But if the margin for AI applications is permanently smaller than previous software-as-a-service margins, because of the high cost of computing, it could put a damper on the current boom.
The high cost of training and “inference” — actually running — large language models is a structural cost that differs from previous computing booms. Even when the software is built, or trained, it still requires a huge amount of computing power to run large language models because they do billions of calculations every time they return a response to a prompt. By comparison, serving web apps or pages requires much less calculation.
These calculations also require specialized hardware. While traditional computer processors can run machine learning models, they’re slow. Most training and inference now takes place on graphics processors, or GPUs, which were initially intended for 3D gaming, but have become the standard for AI applications because they can do many simple calculations simultaneously.
Nvidia makes most of the GPUs for the AI industry, and its primary data center workhorse chip costs $10,000. Scientists that build these models often joke that they “melt GPUs.”
Training models
Nvidia A100 processor
Nvidia
Analysts and technologists estimate that the critical process of training a large language model such as GPT-3 could cost more than $4 million. More advanced language models could cost over “the high-single digit-millions” to train, said Rowan Curran, a Forrester analyst who focuses on AI and machine learning.
Meta’s largest LLaMA model released last month, for example, used 2,048 Nvidia A100 GPUs to train on 1.4 trillion tokens (750 words is about 1,000 tokens), taking about 21 days, the company said when it released the model last month.
It took about 1 million GPU hours to train. With dedicated prices from AWS, that would cost over $2.4 million. And at 65 billion parameters, it’s smaller than the current GPT models at OpenAI, like ChatGPT-3, which has 175 billion parameters.
Clement Delangue, the CEO of AI startup Hugging Face, said the process of training the company’s Bloom large language model took more than two-and-a-half months and required access to a supercomputer that was “something like the equivalent of 500 GPUs.”
Organizations that build large language models must be cautious when they retrain the software, which helps the software improve its abilities, because it costs so much, he said.
“It’s important to realize that these models are not trained all the time, like every day,” Delangue said, noting that’s why some models, like ChatGPT, don’t have knowledge of recent events. ChatGPT’s knowledge stops in 2021, he said.
“We are actually doing a training right now for the version two of Bloom and it’s gonna cost no more than $10 million to retrain,” Delangue said. “So that’s the kind of thing that we don’t want to do every week.”
Inference and who pays for it
Bing with Chat
Jordan Novet | CNBC
To use a trained machine learning model to make predictions or generate text, engineers use the model in a process called “inference,” which can be much more expensive than training because it might need to run millions of times for a popular product.
For a product as popular as ChatGPT — which investment firm UBS estimates to have reached 100 million monthly active users in January — Curran believes that it could have cost OpenAI $40 million to process the millions of prompts people fed into the software that month.
Costs skyrocket when these tools are used billions of times a day. Financial analysts estimate Microsoft’s Bing AI chatbot, which is powered by an OpenAI ChatGPT model, needs at least $4 billion of infrastructure to serve responses to all Bing users.
In the case of Latitude, for instance, while the startup didn’t have to pay to train the underlying OpenAI language model it was accessing, it had to account for the inferencing costs that were something akin to “half-a-cent per call” on “a couple million requests per day,” a Latitude spokesperson said.
“And I was being relatively conservative,” Curran said of his calculations.
In order to sow the seeds of the current AI boom, venture capitalists and tech giants have been investing billions of dollars into startups that specialize in generative AI technologies. Microsoft, for instance, invested as much as $10 billion into GPT’s overseer OpenAI, according to media reports in January. Salesforce‘s venture capital arm, Salesforce Ventures, recently debuted a $250 million fund that caters to generative AI startups.
As investor Semil Shah of the VC firms Haystack and Lightspeed Venture Partners described on Twitter, “VC dollars shifted from subsidizing your taxi ride and burrito delivery to LLMs and generative AI compute.”
Many entrepreneurs see risks in relying on potentially subsidized AI models that they don’t control and merely pay for on a per-use basis.
“When I talk to my AI friends at the startup conferences, this is what I tell them: Do not solely depend on OpenAI, ChatGPT or any other large language models,” said Suman Kanuganti, founder of personal.ai, a chatbot currently in beta mode. “Because businesses shift, they are all owned by big tech companies, right? If they cut access, you’re gone.”
Companies such as enterprise tech firm Conversica are exploring how they can use the tech through Microsoft’s Azure cloud service at its currently discounted price.
While Conversica CEO Jim Kaskade declined to comment about how much the startup is paying, he conceded that the subsidized cost is welcome as it explores how language models can be used effectively.
“If they were truly trying to break even, they’d be charging a hell of a lot more,” Kaskade said.
How it could change
It’s unclear if AI computation will stay expensive as the industry develops. Companies making the foundation models, semiconductor makers and startups all see business opportunities in reducing the price of running AI software.
Nvidia, which has about 95% of the market for AI chips, continues to develop more powerful versions designed specifically for machine learning, but improvements in total chip power across the industry have slowed in recent years.
Still, Nvidia CEO Jensen Huang believes that in 10 years, AI will be “a million times” more efficient because of improvements not only in chips, but also in software and other computer parts.
“Moore’s Law, in its best days, would have delivered 100x in a decade,” Huang said last month on an earnings call. “By coming up with new processors, new systems, new interconnects, new frameworks and algorithms, and working with data scientists, AI researchers on new models, across that entire span, we’ve made large language model processing a million times faster.”
Some startups have focused on the high cost of AI as a business opportunity.
“Nobody was saying ‘You should build something that was purpose-built for inference.’ What would that look like?” said Sid Sheth, founder of D-Matrix, a startup building a system to save money on inference by doing more processing in the computer’s memory, as opposed to on a GPU.
“People are using GPUs today, NVIDIA GPUs, to do most of their inference. They buy the DGX systems that NVIDIA sells that cost a ton of money. The problem with inference is if the workload spikes very rapidly, which is what happened to ChatGPT, it went to like a million users in five days. There is no way your GPU capacity can keep up with that because it was not built for that. It was built for training, for graphics acceleration,” he said.
Delangue, the HuggingFace CEO, believes more companies would be better served focusing on smaller, specific models that are cheaper to train and run, instead of the large language models that are garnering most of the attention.
Meanwhile, OpenAI announced last month that it’s lowering the cost for companies to access its GPT models. It now charges one-fifth of one cent for about 750 words of output.
OpenAI’s lower prices have caught the attention of AI Dungeon-maker Latitude.
“I think it’s fair to say that it’s definitely a huge change we’re excited to see happen in the industry and we’re constantly evaluating how we can deliver the best experience to users,” a Latitude spokesperson said. “Latitude is going to continue to evaluate all AI models to be sure we have the best game out there.”
Watch: AI’s “iPhone Moment” – Separating ChatGPT Hype and Reality
Technology
Tokenization of the market, from stocks to bonds to real estate is coming, says BlackRock CEO Larry Fink, if we can solve one problem
Bitwise Spot Bitcoin ETF (BITB) signage on the floor of the New York Stock Exchange (NYSE) in New York, US, on Thursday, Jan. 11, 2024, with trading commencing on the first US exchange-traded funds that invest directly in the biggest cryptocurrency.
Bloomberg | Bloomberg | Getty Images
If the vision of Larry Fink — CEO of BlackRock, the world’s biggest money manager — becomes reality, all assets from stocks to bonds to real estate and more would be tradable online, on a blockchain.
“Every asset — can be tokenized,” Fink wrote in his recent annual letter to investors.
Unlike traditional paper certificates signifying financial ownership, tokens live securely on a blockchain, enabling instant buying, selling, and transfers without paperwork or waiting — “much like a digital deed,” he wrote.
Fink says it would be nothing short of a “revolution” for investing. Think 24-hour markets and a trading settlement process that can be compacted down into seconds from a process that today can still take days, with billions of dollars reinvested immediately back into the economy.
But there’s one big problem, one technology challenge that stands in the way: the lack of a coordinated digital identity verification system.
While technology experts say Fink’s idea isn’t improbable, they agree that there are cybersecurity challenges ahead in making it work.
Verifying asset owners in world of AI deep fakes
Today, it’s not easy to verify online that the person you are interacting with is that person because of the prevalence of AI deepfakes and sophisticated cybercriminals, according to Christina Hulka, executive director of the Secure Technology Alliance, an organization focused on identity, access and payments. As a result, having a unified verification system would be useful because there would be cryptographic validation that people are who they say they are.
“The [financial services] industry is focused on how to build a zero-trust framework for identification. You don’t trust anything until it’s verified,” Hulka said. “The challenge is getting everyone together about which technology to use that makes it as simple and as seamless for the consumer as possible,” she added.
It’s hard to say precisely how a broad-based digital verification system would work but to support a fully tokenized financial structure, a system would, at a minimum, need to meet stringent security requirements, particularly those tied to financial regulations like the Know Your Customer rule and anti-money laundering rules, according to Zulfikar Ramzan, chief technology officer at Point Wild, a cybersecurity company.
At the same time, the system would need to be low friction and quick. There’s no shortage of technical tools today, especially from the field of cryptography, that can effectively bind a digital identity to a transaction, Ramzan said. “Fifteen to 20 years ago, this conversation would have been a non-starter,” he added.
There have been some successes with programs like this across the globe, according to Ramzan. India’s Aadhaar system is an example of a digital identity framework at a national scale. It enables most of the population to authenticate transactions via mobile devices, and it’s integrated across both public and private services. Estonia has an e-ID system that allows citizens to do everything from banking to voting online. Singapore and the UAE have also implemented strong national identity programs tied to mobile infrastructure and digital services. “While these systems differ in how they handle issues like privacy, they all share a key trait: centralized government leadership that drove standardization and adoption,” Ramzan said.
Centralized personal data is a big target for cybercriminals
While a centralized system solves one challenge, the storage of personally identifiable information and biometrics data is a security risk, said David Mattei, a strategic advisor in the fraud and AML practice at Datos Insights, which works with financial services, insurance and retail technology companies.
Notably, there have been reports of data stolen from India’s Aadhaar system. And last year, El Salvador’s government had the personal data of 80% of its citizens stolen from a centralized, government-managed citizen identity system. “A lot of security experts do not advocate having a centralized security system because it’s kind of like the pot at the end of the rainbow that every fraudster is trying to get his hands on,” Mattei said.
In the U.S., there’s a long-standing preference for decentralized systems for identity. On mobile devices, Face ID and Fingerprint ID are done not by centralizing all of that data in one spot at Apple or Google, but by storing the data in a secure module on each mobile device. “This makes it much harder, if not impossible, for fraudsters to steal that data en masse,” Mattei said.
Larry Fink, chief executive officer of BlackRock Inc., at the Berlin Global Dialogue in Berlin, Germany, on Tuesday, Oct. 1, 2024.
Bloomberg | Bloomberg | Getty Images
Digital driver’s licenses offer a cautionary tale
It would take a significant coordinated effort to come up with a national identity system used for identity verification.
Identity systems in the U.S. today are fragmented, Ramzan said, giving the example of state departments of motor vehicles. “To move forward, we will either need a cohesive national strategy or a way to better coordinate identity across the state and federal levels,” he said.
That’s not an easy task. Take, for example, the effort many states are making to adopt digital driver’s licenses. About a quarter of states today, including Utah, Maryland, Virginia and New York, issue mobile driver’s licenses, according to mDLConnection, an online resource from the Secure Technology Alliance. Other states have pilot programs in effect, have enacted legislation or are studying the issue. But this undertaking is quite ambitious and has been underway for several years.
To implement a national identity verification system would be a “massive undertaking and would require just about every company that does business online to adopt a government standard for identity verification and authentication,” Mattei said.
Competitive forces are another issue to contend with. “There is an ecosystem of vendors who offer identity verification and authentication solutions that would not want a centralized system for fear of going out of business,” Mattei said.
There are also significant data privacy hurdles to overcome. States and the federal government would need to coordinate to resolve governance issues, and this might prompt “big brother” concerns about the extent to which the federal government could monitor the activities of its citizens.
Many people have “a bit of an allergic reaction” when anything resembling a national ID comes up, Ramzan said.
Fink has been pushing the SEC to look at issue
The idea is not a brand new one for Fink. At Davos earlier this year, he told CNBC that he wanted the SEC “to rapidly expand the tokenization of stocks and bonds.”
There’s BlackRock self-interest at work, and potential cost savings for the firm and many others, which Fink has spoken about. In recent years, BlackRock has been dragged into political battles, and lawsuits, over its voting of a massive amount of shares held in its funds on ESG issues. “We’d never have to vote on a proxy vote anymore,” Fink told CNBC at Davos, referring to “the tax on BlackRock.”
“Every owner would be notified of a vote,” he said, adding that it would bring down the cost of ownership of stocks and bonds.
It is clear from Fink’s decision to give this issue prominent placement in his annual letter — even if it came in third in the order of issues he covered behind both the politics of protectionism and the growing role of private markets — that he isn’t letting up. And what’s needed to make this a reality, he contends, is a new digital identity verification system. The letter is short on details, and BlackRock declined to elaborate, but, at least on the surface, the solution for Fink is clear. “If we’re serious about building an efficient and accessible financial system, championing tokenization alone won’t suffice. We must solve digital verification, too,” he wrote.
Blockchain continues to evolve and people are learning to understand it better. Accordingly, there are initiatives underway to think about how the U.S. can achieve a broad-based identity verification system, Hulka said. There are technical ways to do it, but finding the right way that works for the country is more of a challenge since it has to be interoperable. “The goal is to get to a point where there is one way to verify identity across multiple services,” she said.
Eventually, there will be a tipping point for the financial services industry where it becomes a business imperative, Hulka said. “The question is when, of course.”
Peter Thiel, co-founder of PayPal, Palantir Technologies, and Founders Fund, holds hundred dollar bills as he speaks during the Bitcoin 2022 Conference at Miami Beach Convention Center on April 7, 2022 in Miami, Florida.
Marco Bello | Getty Images
Founders Fund, the venture capital firm run by billionaire Peter Thiel, has closed a $4.6 billion late-stage venture fund, according to a Friday filing with the Securities and Exchange Commission.
The fund, Founders Fund Growth III, includes capital from 270 investors, the filing said. Thiel, Napoleon Ta and Trae Stephens are the three people named as directors. A substantial amount of the capital was provided by the firm’s general partners, according to a person familiar with the matter.
Axios reported in December that Founders Fund was raising about $3 billion for the fund. The firm ended up raising more than that amount from outside investors as part of the total $4.6 billion pool, said the person, who asked not to be named because the details are confidential.
A Founders Fund spokesperson declined to comment.
Thiel, best known for co-founding PayPal before putting the first outside money in Facebook and for funding defense software vendor Palantir, started Founders Fund in 2005. In addition to Palantir, the firm’s top investments include Airbnb, Stripe, Affirm and Elon Musk’s SpaceX.
Founders Fund is also a key investor in Anduril, the defense tech company started by Palmer Luckey. CNBC reported in February that Anduril is in talks to raise funding at a $28 billion valuation.
Hefty amounts of private capital are likely to be needed for the foreseeable future as the IPO market remains virtually dormant. It was also dealt a significant blow last week after President Donald Trump’s announcement of widespread tariffs roiled tech stocks. Companies including Klarna, StubHub and Chime delayed their plans to go public as the Nasdaq sank.
President Trump walked back some of the tariffs this week, announcing a 90-day pause for most new tariffs, excluding those imposed on China, while the administration negotiates with other countries. But the uncertainty of where levies will end up is a troubling recipe for risky bets like tech IPOs.
SpaceX, Stripe and Anduril are among the most high-profile venture-backed companies that are still private. Having access to a large pool of growth capital allows Founders Fund to continue investing in follow-on rounds that are off limits to many traditional venture firms.
Thiel was a major Trump supporter during the 2016 campaign, but later had a falling out with the president and was largely on the sidelines in 2024 even as many of his tech peers rallied behind the Republican leader.
In June, Thiel said that even though he wasn’t providing money to the campaign for Trump, who was the Republican presumptive nominee at the time, he’d vote for him over Joe Biden, who had yet to drop out of the race and endorse Kamala Harris.
“If you hold a gun to my head, I’ll vote for Trump,” Thiel said in an interview on stage at the Aspen Ideas Festival. “I’m not going to give any money to his super PAC.”
WATCH: Anduril founder Palmer Luckey talks $32 billion government contract
From left, U.S. President Donald Trump, Senator Dave McCormick, his wife Dina Powell McCormick and Elon Musk watch the men’s NCAA wrestling competition at the Wells Fargo Center in Philadelphia, Pennsylvania, on March 22, 2025.
Brendan Smialowski | Afp | Getty Images
Meta on Friday announced that it was expanding its board of directors with two new members, including Dina Powell McCormick, a part of President Donald Trump’s first administration.
Powell McCormick served as a deputy national security advisor to Trump from 2017 to 2018. She is also married to Sen. Dave McCormick, a Republican from Pennsylvania who took office in January.
“He’s a good man,” Trump said of McCormick in an endorsement last year, according to the Associated Press. Powell McCormick and her husband were photographed in March beside Trump and Tesla CEO Elon Musk, a current advisor to the president, at a wrestling championship match in Philadelphia, Pennsylvania.
Additionally, Powell McCormick was assistant Secretary of State under Condoleezza Rice in President George W. Bush’s administration.
Besides her political background, Powell McCormick is vice chair, president and head of global client services at BDT & MSD Partners. That company was founded in 2023 when the merchant bank BDT combined with Michael Dell’s investment firm MSD. Powell McCormick arrived at the firm after 16 years at Goldman Sachs, where she had been a partner.
Her appointment represents another sign of Meta’s alignment with Republicans following Trump’s return to the White House.
In January, the company announced a shift away from fact-checking and said it was bringing Trump’s friend Dana White, CEO of Ultimate Fighting Championship, onto the board. The changes follow Trump dubbing the company behind Facebook and Instagram “the enemy of the people” on CNBC last year.
Also on Friday, Meta said Patrick Collison, co-founder and CEO of payments startup Stripe, was also elected to the board. Stripe was valued at $65 billion in a tender offer last year.
“Patrick and Dina bring a lot of experience supporting businesses and entrepreneurs to our board,” Meta co-founder and CEO Mark Zuckerberg said in a statement.
Zuckerberg visited the White House last week, after attending Trump’s inauguration in Washington in January. Politico last week reported that the Meto CEO paid $23 million in cash for a mansion in the nation’s capital.
Powell McCormick and Collison officially become directors on April 15, Meta said.
WATCH: Mark Zuckerberg lobbies Trump to avoid Meta antitrust trial, reports say
-
Sports2 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports1 year agoStory injured on diving stop, exits Red Sox game
-
Sports1 year agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike
-
Business3 years agoBank of England’s extraordinary response to government policy is almost unthinkable | Ed Conway