Meet the $10,000 Nvidia chip powering the race for A.I.
Nvidia CEO Jensen Huang speaks during a press conference at The MGM during CES 2018 in Las Vegas on January 7, 2018.
Mandel Ngan | AFP | Getty Images
Software that can write passages of text or draw pictures that look like a human created them has kicked off a gold rush in the technology industry.
Companies like Microsoft and Google are fighting to integrate cutting-edge AI into their search engines, as billion-dollar competitors such as OpenAI and Stable Diffusion race ahead and release their software to the public.
Powering many of these applications is a roughly $10,000 chip that’s become one of the most critical tools in the artificial intelligence industry: The Nvidia A100.
The A100 has become the “workhorse” for artificial intelligence professionals at the moment, said Nathan Benaich, an investor who publishes a newsletter and report covering the AI industry, including a partial list of supercomputers using A100s. Nvidia takes 95% of the market for graphics processors that can be used for machine learning, according to New Street Research.
The A100 is ideally suited for the kind of machine learning models that power tools like ChatGPT, Bing AI, or Stable Diffusion. It’s able to perform many simple calculations simultaneously, which is important for training and using neural network models.
The technology behind the A100 was initially used to render sophisticated 3D graphics in games. It’s often called a graphics processor, or GPU, but these days Nvidia’s A100 is configured and targeted at machine learning tasks and runs in data centers, not inside glowing gaming PCs.
Big companies or startups working on software like chatbots and image generators require hundreds or thousands of Nvidia’s chips, and either purchase them on their own or secure access to the computers from a cloud provider.
Hundreds of GPUs are required to train artificial intelligence models, like large language models. The chips need to be powerful enough to crunch terabytes of data quickly to recognize patterns. After that, GPUs like the A100 are also needed for “inference,” or using the model to generate text, make predictions, or identify objects inside photos.
This means that AI companies need access to a lot of A100s. Some entrepreneurs in the space even see the number of A100s they have access to as a sign of progress.
“A year ago we had 32 A100s,” Stability AI CEO Emad Mostaque wrote on Twitter in January. “Dream big and stack moar GPUs kids. Brrr.” Stability AI is the company that helped develop Stable Diffusion, an image generator that drew attention last fall, and reportedly has a valuation of over $1 billion.
Now, Stability AI has access to over 5,400 A100 GPUs, according to one estimate from the State of AI report, which charts and tracks which companies and universities have the largest collection of A100 GPUs — although it doesn’t include cloud providers, which don’t publish their numbers publicly.
Nvidia’s riding the A.I. train
Nvidia stands to benefit from the AI hype cycle. During Wednesday’s fiscal fourth-quarter earnings report, although overall sales declined 21%, investors pushed the stock up about 14% on Thursday, mainly because the company’s AI chip business — reported as data centers — rose by 11% to more than $3.6 billion in sales during the quarter, showing continued growth.
Nvidia shares are up 65% so far in 2023, outpacing the S&P 500 and other semiconductor stocks alike.
Nvidia CEO Jensen Huang couldn’t stop talking about AI on a call with analysts on Wednesday, suggesting that the recent boom in artificial intelligence is at the center of the company’s strategy.
“The activity around the AI infrastructure that we built, and the activity around inferencing using Hopper and Ampere to influence large language models has just gone through the roof in the last 60 days,” Huang said. “There’s no question that whatever our views are of this year as we enter the year has been fairly dramatically changed as a result of the last 60, 90 days.”
Ampere is Nvidia’s code name for the A100 generation of chips. Hopper is the code name for the new generation, including H100, which recently started shipping.
More computers needed
Nvidia A100 processor
Nvidia
Compared to other kinds of software, like serving a webpage, which uses processing power occasionally in bursts for microseconds, machine learning tasks can take up the whole computer’s processing power, sometimes for hours or days.
This means companies that find themselves with a hit AI product often need to acquire more GPUs to handle peak periods or improve their models.
These GPUs aren’t cheap. In addition to a single A100 on a card that can be slotted into an existing server, many data centers use a system that includes eight A100 GPUs working together.
This system, Nvidia’s DGX A100, has a suggested price of nearly $200,000, although it comes with the chips needed. On Wednesday, Nvidia said it would sell cloud access to DGX systems directly, which will likely reduce the entry cost for tinkerers and researchers.
It’s easy to see how the cost of A100s can add up.
For example, an estimate from New Street Research found that the OpenAI-based ChatGPT model inside Bing’s search could require 8 GPUs to deliver a response to a question in less than one second.
At that rate, Microsoft would need over 20,000 8-GPU servers just to deploy the model in Bing to everyone, suggesting Microsoft’s feature could cost $4 billion in infrastructure spending.
“If you’re from Microsoft, and you want to scale that, at the scale of Bing, that’s maybe $4 billion. If you want to scale at the scale of Google, which serves 8 or 9 billion queries every day, you actually need to spend $80 billion on DGXs.” said Antoine Chkaiban, a technology analyst at New Street Research. “The numbers we came up with are huge. But they’re simply the reflection of the fact that every single user taking to such a large language model requires a massive supercomputer while they’re using it.”
The latest version of Stable Diffusion, an image generator, was trained on 256 A100 GPUs, or 32 machines with 8 A100s each, according to information online posted by Stability AI, totaling 200,000 compute hours.
At the market price, training the model alone cost $600,000, Stability AI CEO Mostaque said on Twitter, suggesting in a tweet exchange the price was unusually inexpensive compared to rivals. That doesn’t count the cost of “inference,” or deploying the model.
Huang, Nvidia’s CEO, said in an interview with CNBC’s Katie Tarasov that the company’s products are actually inexpensive for the amount of computation that these kinds of models need.
“We took what otherwise would be a $1 billion data center running CPUs, and we shrunk it down into a data center of $100 million,” Huang said. “Now, $100 million, when you put that in the cloud and shared by 100 companies, is almost nothing.”
Huang said that Nvidia’s GPUs allow startups to train models for a much lower cost than if they used a traditional computer processor.
“Now you could build something like a large language model, like a GPT, for something like $10, $20 million,” Huang said. “That’s really, really affordable.”
New competition
Nvidia isn’t the only company making GPUs for artificial intelligence uses. AMD and Intel have competing graphics processors, and big cloud companies like Google and Amazon are developing and deploying their own chips specially designed for AI workloads.
Still, “AI hardware remains strongly consolidated to NVIDIA,” according to the State of AI compute report. As of December, more than 21,000 open-source AI papers said they used Nvidia chips.
Most researchers included in the State of AI Compute Index used the V100, Nvidia’s chip that came out in 2017, but A100 grew fast in 2022 to be the third-most used Nvidia chip, just behind a $1500-or-less consumer graphics chip originally intended for gaming.
The A100 also has the distinction of being one of only a few chips to have export controls placed on it because of national defense reasons. Last fall, Nvidia said in an SEC filing that the U.S. government imposed a license requirement barring the export of the A100 and the H100 to China, Hong Kong, and Russia.
“The USG indicated that the new license requirement will address the risk that the covered products may be used in, or diverted to, a ‘military end use’ or ‘military end user’ in China and Russia,” Nvidia said in its filing. Nvidia previously said it adapted some of its chips for the Chinese market to comply with U.S. export restrictions.
The fiercest competition for the A100 may be its successor. The A100 was first introduced in 2020, an eternity ago in chip cycles. The H100, introduced in 2022, is starting to be produced in volume — in fact, Nvidia recorded more revenue from H100 chips in the quarter ending in January than the A100, it said on Wednesday, although the H100 is more expensive per unit.
The H100, Nvidia says, is the first one of its data center GPUs to be optimized for transformers, an increasingly important technique that many of the latest and top AI applications use. Nvidia said on Wednesday that it wants to make AI training over 1 million percent faster. That could mean that, eventually, AI companies wouldn’t need so many Nvidia chips.
In this photo illustration, the Bluesky Social logo is displayed on a cell phone in Rio de Janeiro, Brazil, on September 4, 2024.
Mauro Pimentel | AFP | Getty Images
Micro-blogging startup Bluesky has gained over 1.25 million new users in the past week, indicating some social media users are changing their habits following the U.S. presidential election.
Bluesky’s influx of users shows that the app has been able to pitch itself as an alternative to X, formerly Twitter, which is owned by Elon Musk, as well as Meta’s Threads. The bulk of the new users are coming from the U.S., Canada and the United Kingdom, the company said Wednesday.
“We’re excited to welcome everyone looking for a better social media experience,” Bluesky CEO Jay Graber told CNBC in a statement.
Despite the surge of users, Bluesky’s total base remains a fraction of its rivals’. The Seattle startup claims 15.2 million total users. Meta CEO Mark Zuckerberg in October said Threads had nearly 275 million monthly users. Musk in May claimed that X had 600 million monthly users, but market intelligence firm Sensor Tower pegged X’s monthly base at 318 million users in October.
Created in 2019 as a project inside Twitter, when Jack Dorsey was still CEO, Bluesky doesn’t show ads and has yet to develop a business model. It became an independent company in 2021. Dorsey said in May of this year that he’s no longer a member of Bluesky’s board.
“Journalists, politicians, and news junkies have also been talking up Bluesky as a better X alternative than Threads,” wrote Similarweb, the internet traffic and monitoring service, in a Tuesday blog.
Some users with new Bluesky accounts posted that they had moved to the service due to Musk and his support for President-elect Donald Trump.
“It’s appalling that Elon Musk has transformed Twitter into a Trump propaganda machine, rife with disinformation and misinformation,” one user posted on Bluesky.
This is Bluesky’s second notable surge in the last couple of months.
Bluesky said it picked up 2 million new users in September after the Brazilian Supreme Court suspended X in the country for failing to comply with regional content moderation policies and not appointing a local representative.
Cisco CEO Chuck Robbins speaks at The Wall Street Journal’s Future of Everything Festival in New York on May 21, 2024.
Dia Dipasupil | Getty Images
Cisco reported a fourth straight quarter of declining revenue even as results topped analysts’ estimates. The stock slipped 2.5% in extended trading.
Here’s how the company did in comparison with LSEG consensus:
- Earnings per share: 91 cents adjusted vs. 87 cents expected
- Revenue: $13.84 billion vs. $13.77 billion expected
Cisco’s revenue dropped 6% in the quarter ended Oct. 26, from $14.7 billion a year earlier, according to a statement. Net income fell to $2.71 billion, or 68 cents per share, from $3.64 billion, or 89 cents per share, in the same quarter a year ago.
Networking revenue plunged 23% to $6.75 billion, slightly below the $6.8 billion consensus of analysts surveyed by StreetAccount.
Security revenue doubled to $2.02 billion, topping the StreetAccount consensus of $1.93 billion. Cisco’s revenue from collaboration was $1.09 billion, a bit below the $1.04 billion consensus estimate.
Cisco CEO Chuck Robbins said on the earnings call on Wednesday that orders from large-scale clients for artificial intelligence infrastructure exceeded $300 million in the quarter. Server makers such as Dell and HPE have also focused on sales of hardware that can help clients implement generative AI.
“We have earned more design wins and remain confident that we will exceed our target of $1 billion of AI orders this fiscal year from web-scale customers,” Robbins said.
Cisco has announced hardware containing Nvidia’s graphics processing units, which are widely used for training AI models, Robbins said.
“Over time, you’ll see us support other GPUs as the market demands,” he said. “But that partnership is still going fine. It’s still early. And I think 2025 is when we’ll start to see enterprise real deployment of some of these technologies.”
For now, enterprises are updating data center infrastructure to prepare for AI and the widespread deployment of AI applications, Robbins said.
U.S. government agencies have delayed deals with Cisco, rather than scrapping them altogether. The Fiscal Responsibility Act of 2023, which became law in June of last year, has limited U.S. government spending, said Scott Herren, Cisco’s finance chief.
Herren said that with Republicans poised to control the White House and both houses of Congress, he expects “to get a budget in place relatively soon.”
During the quarter, Cisco acquired security startups DeepFactor and Robust Intelligence.
Cisco lifted its full-year guidance to $3.60 to $3.66 in adjusted earnings per share on $55.3 billion to $56.3 billion in revenue, up from a prior forecast of $3.52 to $3.58 in EPS and $55 billion to $56.2 billion in revenue. Guidance would indicate projected revenue growth of 3.3% at the middle of the range.
Analysts expected adjusted earnings for the year of $3.58 per share on $55.89 billion in revenue.
As of Wednesday’s close, Cisco’s stock was up 17% year to date, while the S&P 500 index is up around 26% over that stretch.
WATCH: The Earnings Setup: Cisco Systems, Applied Materials and Disney
Republican presidential nominee, former U.S. President Donald Trump, (C) greets attendees during a campaign stop to address Pennsylvanians who are concerned about the threat of Communist China to U.S. agriculture at the Smith Family Farm September 23, 2024 in Smithton, Pennsylvania.
Win Mcnamee | Getty Images
After Donald Trump won the U.S. presidency last week, tech CEOs including Apple‘s Tim Cook, Meta‘s Mark Zuckerberg and Amazon‘s Jeff Bezos publicly praised the president-elect.
One name was conspicuously missing: TikTok CEO Shou Zi Chew.
His absence was notable considering that of all the top tech companies, TikTok faces the most immediate and existential threat from the U.S. government. In April, President Joe Biden signed a law that requires China’s ByteDance to sell TikTok by Jan. 19. If ByteDance fails to comply, internet hosting companies and app store owners such as Apple and Google will be prohibited from supporting TikTok, effectively banning it in the U.S.
Trump’s return to the White House, though, may provide a lifeline for Chew and TikTok.
Although both Republicans and Democrats supported the Biden TikTok ban in April, Trump voiced opposition to the ban during his candidacy. Trump acknowledged the national security and data privacy concerns with TikTok in a March interview with CNBC’s “Squawk Box,” but he also said “there’s a lot of good and there’s a lot of bad” with the app.
Trump also leveraged TikTok’s shaky future in the U.S. as a reason for people to vote against Democrat Vice President Kamala Harris.
“We’re not doing anything with TikTok, but the other side is going to close it up, so if you like TikTok, go out and vote for Trump,” the president-elect said in a September post on his Truth Social service.
Since his election, Trump hasn’t publicly discussed his plans for TikTok, but Trump-Vance transition spokeswoman Karoline Leavitt told CNBC that the president-elect “will deliver.”
“The American people re-elected President Trump by a resounding margin giving him a mandate to implement the promises he made on the campaign trail,” Leavitt said in a statement.
Trump’s rhetoric on TikTok began to turn after the president-elect met in February with billionaire Jeff Yass, a Republican megadonor and a major investor in the Chinese-owned social media app.
Yass’s trading firm Susquehanna International Group owns a 15% stake in ByteDance while Yass maintains a 7% stake in the company, equating to about $21 billion, NBC and CNBC reported in March. That month it was also reported that Yass was a part owner of the business that merged with the parent company of Trump’s Truth Social.
TikTok’s CEO Shou Zi Chew testifies during the Senate Judiciary Committee hearing on online child sexual exploitation, at the U.S. Capitol, in Washington, U.S., January 31, 2024.
Nathan Howard | Reuters
If ByteDance doesn’t sell TikTok by the January deadline, Trump could potentially call on Congress to repeal the law or he can introduce a more “selective enforcement” of the law that would essentially allow TikTok to continue operating in the U.S. without facing penalties, said Sarah Kreps, a Cornell University professor of government. “Selective enforcement” would be akin to police officers not always enforcing every single instance of jaywalking, she said.
At TikTok, meanwhile, Chew has remained quiet since Trump’s victory, just as he had been in the lead-up to Election Day.
The Chinese-owned company may be taking a neutral approach and a wait-and-see strategy for now, said Long Le, a China business expert and Santa Clara University associate teaching professor.
Le said it’s hard to foresee what Trump will do.
“He’s also a contrarian; that’s what makes him unpredictable,” Le said. “He can say one thing, and the next year he’ll change his mind.”
TikTok didn’t respond to requests for comment.
Mark Zuckerberg, CEO of Meta testifies before the Senate Judiciary Committee at the Dirksen Senate Office Building on January 31, 2024 in Washington, DC.
Alex Wong | Getty Images
‘Facebook has been very bad for our country’
When it comes to social media apps, Trump’s campaign comments suggest he’s more concerned with TikTok rival Meta.
In his March interview with “Squawk Box,” Trump said Meta, which owns Facebook and Instagram, posed a much bigger problem than TikTok. He also said a TikTok ban would only benefit Meta, which he labeled “an enemy of the people.”
“Facebook has been very bad for our country, especially when it comes to elections,” Trump said.
But Trump’s negative views on Meta may have changed after comments by CEO Mark Zuckerberg over the past few months, Cornell’s Kreps said.
Zuckerberg described the photo of Trump raising his fist following a failed assassination attempt in July as “one of the most badass things I’ve ever seen in my life.” And after Trump’s win, Zuckerberg congratulated him, saying he was looking forward to working with the president-elect and his administration.
“My sense as an armchair psychologist of Trump is that he really likes people who sing his praises, and so his view on Zuckerberg and Meta, I would imagine, has changed,” Kreps said. “He might then just revert to his American economic nationalism here and say, ‘Let’s protect American industry and continue with the Chinese ban.'”
Meta didn’t respond to a request for comment.
Maintaining support of the TikTok ban could also win Trump political favor with lawmakers concerned about China’s global political and business influence, said Milton Mueller, a professor at Georgia Tech’s School of Public Policy.
“I don’t see him scoring big points politically by standing up for TikTok,” Mueller said, noting that few lawmakers, like Sen. Rand Paul, R-Ky., have opposed the ban.
Even if Trump does provide a lifeline for TikTok, it’s unclear how much damage that would do to his administration since many politicians are reluctant to publicly criticize him, Le said.
“They’re not going to challenge him because he just got so much power,” Le said.
Since launching his TikTok account in June, Trump has amassed over 14 million followers. Given his social media savvy, Trump may not want to make a decision that results in him losing the public attention and influence he’s gained on TikTok, Le said.
WATCH: TikTok is ‘digital nicotine’ for young people, says D.C. Attorney General Brian Schwalb
-
Sports2 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports7 months agoStory injured on diving stop, exits Red Sox game
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports1 year agoGame 1 of WS least-watched in recorded history
-
Environment1 year agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Sports3 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike
-
Business2 years agoBank of England’s extraordinary response to government policy is almost unthinkable | Ed Conway