In an unmarked office building in Austin, Texas, two small rooms contain a handful of Amazon employees designing two types of microchips for training and accelerating generative AI. These custom chips, Inferentia and Trainium, offer AWS customers an alternative to training their large language models on Nvidia GPUs, which have been getting difficult and expensive to procure.
“The entire world would like more chips for doing generative AI, whether that’s GPUs or whether that’s Amazon’s own chips that we’re designing,” Amazon Web Services CEO Adam Selipsky told CNBC in an interview in June. “I think that we’re in a better position than anybody else on Earth to supply the capacity that our customers collectively are going to want.”
Yet others have acted faster, and invested more, to capture business from the generative AI boom. When OpenAI launched ChatGPT in November, Microsoft gained widespread attention for hosting the viral chatbot, and investing a reported $13 billion in OpenAI. It was quick to add the generative AI models to its own products, incorporating them into Bing in February.
That same month, Google launched its own large language model, Bard, followed by a $300 million investment in OpenAI rival Anthropic.
It wasn’t until April that Amazon announced its own family of large language models, called Titan, along with a service called Bedrock to help developers enhance software using generative AI.
“Amazon is not used to chasing markets. Amazon is used to creating markets. And I think for the first time in a long time, they are finding themselves on the back foot and they are working to play catch up,” said Chirag Dekate, VP analyst at Gartner.
In the long run, Dekate said, Amazon’s custom silicon could give it an edge in generative AI.
“I think the true differentiation is the technical capabilities that they’re bringing to bear,” he said. “Because guess what? Microsoft does not have Trainium or Inferentia,” he said.
AWS quietly started production of custom silicon back in 2013 with a piece of specialized hardware called Nitro. It’s now the highest-volume AWS chip. Amazon told CNBC there is at least one in every AWS server, with a total of more than 20 million in use.
AWS started production of custom silicon back in 2013 with this piece of specialized hardware called Nitro. Amazon told CNBC in August that Nitro is now the highest volume AWS chip, with at least one in every AWS server and a total of more than 20 million in use.
Courtesy Amazon
In 2015, Amazon bought Israeli chip startup Annapurna Labs. Then in 2018, Amazon launched its Arm-based server chip, Graviton, a rival to x86 CPUs from giants like AMD and Intel.
“Probably high single-digit to maybe 10% of total server sales are Arm, and a good chunk of those are going to be Amazon. So on the CPU side, they’ve done quite well,” said Stacy Rasgon, senior analyst at Bernstein Research.
Also in 2018, Amazon launched its AI-focused chips. That came two years after Google announced its first Tensor Processor Unit, or TPU. Microsoft has yet to announce the Athena AI chip it’s been working on, reportedly in partnership with AMD.
CNBC got a behind-the-scenes tour of Amazon’s chip lab in Austin, Texas, where Trainium and Inferentia are developed and tested. VP of product Matt Wood explained what both chips are for.
“Machine learning breaks down into these two different stages. So you train the machine learning models and then you run inference against those trained models,” Wood said. “Trainium provides about 50% improvement in terms of price performance relative to any other way of training machine learning models on AWS.”
Trainium first came on the market in 2021, following the 2019 release of Inferentia, which is now on its second generation.
Inferentia allows customers “to deliver very, very low-cost, high-throughput, low-latency, machine learning inference, which is all the predictions of when you type in a prompt into your generative AI model, that’s where all that gets processed to give you the response, ” Wood said.
For now, however, Nvidia’s GPUs are still king when it comes to training models. In July, AWS launched new AI acceleration hardware powered by Nvidia H100s.
“Nvidia chips have a massive software ecosystem that’s been built up around them over the last like 15 years that nobody else has,” Rasgon said. “The big winner from AI right now is Nvidia.”
Amazon’s custom chips, from left to right, Inferentia, Trainium and Graviton are shown at Amazon’s Seattle headquarters on July 13, 2023.
Joseph Huerta
Leveraging cloud dominance
AWS’ cloud dominance, however, is a big differentiator for Amazon.
“Amazon does not need to win headlines. Amazon already has a really strong cloud install base. All they need to do is to figure out how to enable their existing customers to expand into value creation motions using generative AI,” Dekate said.
When choosing between Amazon, Google, and Microsoft for generative AI, there are millions of AWS customers who may be drawn to Amazon because they’re already familiar with it, running other applications and storing their data there.
“It’s a question of velocity. How quickly can these companies move to develop these generative AI applications is driven by starting first on the data they have in AWS and using compute and machine learning tools that we provide,” explained Mai-Lan Tomsen Bukovec, VP of technology at AWS.
AWS is the world’s biggest cloud computing provider, with 40% of the market share in 2022, according to technology industry researcher Gartner. Although operating income has been down year-over-year for three quarters in a row, AWS still accounted for 70% of Amazon’s overall $7.7 billion operating profit in the second quarter. AWS’ operating margins have historically been far wider than those at Google Cloud.
“Let’s rewind the clock even before ChatGPT. It’s not like after that happened, suddenly we hurried and came up with a plan because you can’t engineer a chip in that quick a time, let alone you can’t build a Bedrock service in a matter of 2 to 3 months,” said Swami Sivasubramanian, AWS’ VP of database, analytics and machine learning.
Bedrock gives AWS customers access to large language models made by Anthropic, Stability AI, AI21 Labs and Amazon’s own Titan.
“We don’t believe that one model is going to rule the world, and we want our customers to have the state-of-the-art models from multiple providers because they are going to pick the right tool for the right job,” Sivasubramanian said.
An Amazon employee works on custom AI chips, in a jacket branded with AWS’ chip Inferentia, at the AWS chip lab in Austin, Texas, on July 25, 2023.
Katie Tarasov
One of Amazon’s newest AI offerings is AWS HealthScribe, a service unveiled in July to help doctors draft patient visit summaries using generative AI. Amazon also has SageMaker, a machine learning hub that offers algorithms, models and more.
Another big tool is coding companion CodeWhisperer, which Amazon said has enabled developers to complete tasks 57% faster on average. Last year, Microsoft also reported productivity boosts from its coding companion, GitHub Copilot.
“We have so many customers who are saying, ‘I want to do generative AI,’ but they don’t necessarily know what that means for them in the context of their own businesses. And so we’re going to bring in solutions architects and engineers and strategists and data scientists to work with them one on one,” AWS CEO Selipsky said.
Although so far AWS has focused largely on tools instead of building a competitor to ChatGPT, a recently leaked internal email shows Amazon CEO Andy Jassy is directly overseeing a new central team building out expansive large language models, too.
In the second-quarter earnings call, Jassy said a “very significant amount” of AWS business is now driven by AI and more than 20 machine learning services it offers. Some examples of customers include Philips, 3M, Old Mutual and HSBC.
The explosive growth in AI has come with a flurry of security concerns from companies worried that employees are putting proprietary information into the training data used by public large language models.
“I can’t tell you how many Fortune 500 companies I’ve talked to who have banned ChatGPT. So with our approach to generative AI and our Bedrock service, anything you do, any model you use through Bedrock will be in your own isolated virtual private cloud environment. It’ll be encrypted, it’ll have the same AWS access controls,” Selipsky said.
For now, Amazon is only accelerating its push into generative AI, telling CNBC that “over 100,000” customers are using machine learning on AWS today. Although that’s a small percentage of AWS’s millions of customers, analysts say that could change.
“What we are not seeing is enterprises saying, ‘Oh, wait a minute, Microsoft is so ahead in generative AI, let’s just go out and let’s switch our infrastructure strategies, migrate everything to Microsoft.’ Dekate said. “If you’re already an Amazon customer, chances are you’re likely going to explore Amazon ecosystems quite extensively.”
— CNBC’s Jordan Novet contributed to this report.
CORRECTION: This article has been updated to reflect Inferentia as the chip used for machine learning inference.
Meta CEO Mark Zuckerberg appears at the Meta Connect event in Menlo Park, California, Sept. 25, 2024.
David Paul Morris | Bloomberg | Getty Images
Meta CEO Mark Zuckerberg slammed rival tech giant Apple for lackluster innovation efforts and “random rules” in a lengthy podcast interview on Friday.
“On the one hand, [the iPhone has] been great, because now pretty much everyone in the world has a phone, and that’s kind of what enables pretty amazing things,” Zuckerberg said in an episode of the “Joe Rogan Experience.” “But on the other hand … they have used that platform to put in place a lot of rules that I think feel arbitrary and [I] feel like they haven’t really invented anything great in a while. It’s like Steve Jobs invented the iPhone, and now they’re just kind of sitting on it 20 years later.”
Zuckerberg added that he thought iPhone sales were struggling because consumers are taking longer to upgrade their phones because new models aren’t big improvements from prior iterations.
“So how are they making more money as a company? Well, they do it by basically, like, squeezing people, and, like you’re saying, having this 30% tax on developers by getting you to buy more peripherals and things that plug into it,” Zuckerberg said. “You know, they build stuff like Air Pods, which are cool, but they’ve just thoroughly hamstrung the ability for anyone else to build something that can connect to the iPhone in the same way.”
Apple defends itself from pushback from other companies by saying that it doesn’t want to violate consumers’ privacy and security, according to Zuckerberg. But he said that the problem would be solved if Apple fixed its protocol, like building better security and using encryption.
“It’s insecure because you didn’t build any security into it. And then now you’re using that as a justification for why only your product can connect in an easy way,” Zuckerberg said.
Zuckerberg said that if Apple stopped applying its “random rules,” Meta’s profit would double.
He also took shots at Apple’s Vision Pro headset, which had disappointing U.S. sales. Meta sells its own virtual headsets called the Meta Quest.
“I think the Vision Pro is, I think, one of the bigger swings at doing a new thing that they tried in a while,” Zuckerberg said. “And I don’t want to give them too hard of a time on it, because we do a lot of things where the first version isn’t that good, and you want to kind of judge the third version of it. But I mean, the V1, it definitely did not hit it out of the park.”
“I heard it’s really good for watching movies,” he added.
Apple did not immediately respond to a request for comment from CNBC.
Mark Zuckerberg’s announcement this week that Meta would pivot its moderation policies to allow more “free expression” was widely viewed as the company’s latest effort to appease President-elect Donald Trump.
More than any of its Silicon Valley peers, Meta has taken numerous public steps to make amends with Trump since his election victory in November.
That follows a highly contentious four years between the two during Trump’s first term in office, which ended with Facebook — similar to other social media companies — banning Trump from its platform.
As recently as March, Trump was using his preferred nickname of “Zuckerschmuck” when talking about Meta’s CEO and declaring that Facebook was an “enemy of the people.”
With Meta now positioning itself to be a key player in artificial intelligence, Zuckerberg recognizes the need for White House support as his company builds data centers and pursues policies that will allow it to fulfill its lofty ambitions, according to people familiar with the company’s plans who asked not to be named because they weren’t authorized to speak on the matter.
“Even though Facebook is as powerful as it is, it still had to bend the knee to Trump,” said Brian Boland, a former Facebook vice president, who left the company in 2020.
Meta declined to comment for this article.
In Tuesday’s announcement, Zuckerberg said Meta will end third-party fact-checking, remove restrictions on topics such as immigration and gender identity and bring political content back to users’ feeds. Zuckerberg pitched the sweeping policy changes as key to stabilizing Meta’s content-moderation apparatus, which he said had “reached a point where it’s just too many mistakes and too much censorship.”
The policy change was the latest strategic shift Meta has taken to buddy up with Trump and Republicans since Election Day.
A day earlier, Meta announced that UFC CEO Dana White, a longtime Trump friend, is joining the company’s board.
And last week, Meta announced that it was replacing Nick Clegg, its president of global affairs, with Joel Kaplan, who had been the company’s policy vice president. Clegg previously had a career in British politics with the Liberal Democrats party, including as a deputy prime minister, while Kaplan was a White House deputy chief of staff under former President George W. Bush.
Kaplan, who joined Meta in 2011 when it was still known as Facebook, has longstanding ties to the Republican Party and once worked as a law clerk for the late conservative Supreme Court Justice Antonin Scalia. In December, Kaplan posted photos on Facebook of himself with Vice President-elect JD Vance and Trump during their visit to the New York Stock Exchange.
Joel Kaplan, Facebook’s vice president of global policy, on April 17, 2018.
Niall Carson | PA Images | Getty Images
Many Meta employees criticized the policy change internally, with some saying the company is absolving itself of its responsibility to create a safe platform. Current and former employees also expressed concern that marginalized communities could face more online abuse due to the new policy, which is set to take effect over the coming weeks.
Despite the backlash from employees, people familiar with the company’s thinking said Meta is more willing to make these kinds of moves after laying off 21,000 employees, or nearly a quarter of its workforce, in 2022 and 2023.
Those cuts affected much of Meta’s civic integrity and trust and safety teams. The civic integrity group was the closest thing the company had to a white-collar union, with members willing to push back against certain policy decisions, former employees said. Since the job cuts, Zuckerberg faces less friction when making broad policy changes, the people said.
Zuckerberg’s overtures to Trump began in the months leading up to the election.
Following the first assassination attempt on Trump in July, Zuckerberg called the photo of Trump raising his fist with blood running down his face “one of the most badass things I’ve ever seen in my life.”
A month later, Zuckerberg penned a letter to the House Judiciary Committee alleging that the Biden administration had pressured Meta’s teams to censor certain Covid-19 content.
“I believe the government pressure was wrong, and I regret that we were not more outspoken about it,” he wrote.
After Trump’s presidential victory, Zuckerberg joined several other technology executives who visited the president-elect’s Mar-a-Lago resort in Florida. Meta also donated $1 million to Trump’s inaugural fund.
On Friday, Meta revealed to its workforce in a memo obtained by CNBC that it intends to shutter several internal programs related to diversity and inclusion in its hiring process, representing another Trump-friendly move.
The previous day, some details of the company’s new relaxed content-moderation guidelines were published by the news site The Intercept, showing the kind of offensive rhetoric that Meta’s new policy would now allow, including statements such as “Migrants are no better than vomit” and “I bet Jorge’s the one who stole my backpack after track practice today. Immigrants are all thieves.”
Recalibrating for Trump
Zuckerberg, who has been dragged to Washington eight times to testify before congressional committees during the last two administrations, wants to be perceived as someone who can work with Trump and the Republican Party, people familiar with the matter said.
Though Meta’s content-policy updates caught many of its employees and fact-checking partners by surprise, a small group of executives were formulating the plans in the aftermath of the U.S. election results. By New Year’s Day, leadership began planning the public announcements of its policy change, the people said.
Meta typically undergoes major “recalibrations” after prominent U.S. elections, said Katie Harbath, a former Facebook policy director and CEO of tech consulting firm Anchor Change. When the country undergoes a change in power, Meta adjusts its policies to best suit its business and reputational needs based on the political landscape, Harbath said.
“In 2028, they’ll recalibrate again,” she said.
After the 2016 election and Trump’s first victory, for example, Zuckerberg toured the U.S. to meet people in states he hadn’t previously visited. He published a 6,000-word manifesto emphasizing the need for Facebook to build more community.
The social media company faced harsh criticism about fake news and Russian election interference on its platforms after the 2016 election.
Following the 2020 election, during the heart of the pandemic, Meta took a harder stand on Covid-19 content, with a policy executive saying in 2021 that the “amount of COVID-19 vaccine misinformation that violates our policies is too much by our standards.” Those efforts may have appeased the Biden administration, but it drew the ire of Republicans.
Meta is once again reacting to the moment, Harbath said.
“There wasn’t a business risk here in Silicon Valley to be more right-leaning,” Harbath said.
While Trump has offered few specific policy proposals for his second administration, Meta has plenty at stake.
The White House could create more relaxed AI regulations compared with those in the European Union, where Meta says harsh restrictions have resulted in the company not releasing some of its more advanced AI technologies. Meta, like other tech giants, also needs more massive data centers and cutting-edge computer chips to help train and run their advanced AI models.
“There’s a business benefit to having Republicans win, because they are traditionally less regulatory,” Harbath said.
Meta’s CEO Mark Zuckerberg reacts as he testifies during the Senate Judiciary Committee hearing on online child sexual exploitation at the U.S. Capitol in Washington, U.S., January 31, 2024.
Evelyn Hockstein | Reuters
Meta isn’t alone in trying to cozy up to Trump. But the extreme measures the company is taking reflects a particular level of animus expressed by Trump over the years.
Trump has accused Meta of censorship and has expressed resentment over the company’s two-year suspension of his Facebook and Instagram accounts following the Jan. 6 attack on the Capitol.
In July 2024, Trump posted on Truth Social that he intended to “pursue Election Fraudsters at levels never seen before, and they will be sent to prison for long periods of time,” adding “ZUCKERBUCKS, be careful!” Trump reiterated that statement in his book, “Save America,” writing that Zuckerberg plotted against him during the 2020 election and that the Meta CEO would “spend the rest of his life in prison” if it happened again.
Meta spends $14 million annually on providing personal security for Zuckerberg and his family, according to the company’s 2024 proxy statement. As part of that security, the company analyzes any threats or perceived threats against its CEO, according to a person familiar with the matter. Those threats are cataloged, analyzed and dissected by Meta’s multitude of security teams.
After Trump’s comments, Meta’s security teams analyzed how Trump could weaponize the Justice Department and the country’s intelligence agencies against Zuckerberg and what it would cost the company to defend its CEO against a sitting president, said the person, who asked not to be named because of confidentiality.
Meta’s efforts to appease the incoming president bring their own risks.
After Zuckerberg announced the new speech policy Tuesday, Boland, the former executive, was among a number of users who took to Meta’s Threads service to tell their followers that they were quitting Facebook.
“Last post before deleting,” Boland wrote in his post.
Before the post could be seen by any of his Threads followers, Meta’s content moderation system had taken it down, citing cybersecurity reasons.
Boland told CNBC in an interview that he couldn’t help but chuckle at the situation.
“It’s deeply ironic,” Boland said.
— CNBC’s Salvador Rodriguez contributed to this report.
Apple is losing market share in China due to declining iPhone shipments, supply chain analyst Ming-Chi Kuo wrote in a report on Friday. The stock slid 2.4%.
“Apple has adopted a cautious stance when discussing 2025 iPhone production plans with key suppliers,” Kuo, an analyst at TF Securities, wrote in the post. He added that despite the expected launch of the new iPhone SE 4, shipments are expected to decline 6% year over year for the first half of 2025.
Kuo expects Apple’s market share to continue to slide, as two of the coming iPhones are so thin that they likely will only support eSIM, which the Chinese market currently does not promote.
“These two models could face shipping momentum challenges unless their design is modified,” he wrote.
Kuo wrote that in December, overall smartphone shipments in China were flat from a year earlier, but iPhone shipments dropped 10% to 12%.
There is also “no evidence” that Apple Intelligence, the company’s on-device artificial intelligence offering, is driving hardware upgrades or services revenue, according to Kuo. He wrote that the feature “has not boosted iPhone replacement demand,” according to a supply chain survey he conducted, and added that in his view, the feature’s appeal “has significantly declined compared to cloud-based AI services, which have advanced rapidly in subsequent months.”
Apple’s estimated iPhone shipments total about 220 million units for 2024 and between about 220 million and 225 million for this year, Kuo wrote. That is “below the market consensus of 240 million or more,” he wrote.
Apple did not immediately respond to CNBC’s request for comment.