Connect with us

Published

on

In an unmarked office building in Austin, Texas, two small rooms contain a handful of Amazon employees designing two types of microchips for training and accelerating generative AI. These custom chips, Inferentia and Trainium, offer AWS customers an alternative to training their large language models on Nvidia GPUs, which have been getting difficult and expensive to procure. 

“The entire world would like more chips for doing generative AI, whether that’s GPUs or whether that’s Amazon’s own chips that we’re designing,” Amazon Web Services CEO Adam Selipsky told CNBC in an interview in June. “I think that we’re in a better position than anybody else on Earth to supply the capacity that our customers collectively are going to want.”

Yet others have acted faster, and invested more, to capture business from the generative AI boom. When OpenAI launched ChatGPT in November, Microsoft gained widespread attention for hosting the viral chatbot, and investing a reported $13 billion in OpenAI. It was quick to add the generative AI models to its own products, incorporating them into Bing in February. 

That same month, Google launched its own large language model, Bard, followed by a $300 million investment in OpenAI rival Anthropic. 

It wasn’t until April that Amazon announced its own family of large language models, called Titan, along with a service called Bedrock to help developers enhance software using generative AI.

“Amazon is not used to chasing markets. Amazon is used to creating markets. And I think for the first time in a long time, they are finding themselves on the back foot and they are working to play catch up,” said Chirag Dekate, VP analyst at Gartner.

Meta also recently released its own LLM, Llama 2. The open-source ChatGPT rival is now available for people to test on Microsoft‘s Azure public cloud.

Chips as ‘true differentiation’

In the long run, Dekate said, Amazon’s custom silicon could give it an edge in generative AI. 

“I think the true differentiation is the technical capabilities that they’re bringing to bear,” he said. “Because guess what? Microsoft does not have Trainium or Inferentia,” he said.

AWS quietly started production of custom silicon back in 2013 with a piece of specialized hardware called Nitro. It’s now the highest-volume AWS chip. Amazon told CNBC there is at least one in every AWS server, with a total of more than 20 million in use. 

AWS started production of custom silicon back in 2013 with this piece of specialized hardware called Nitro. Amazon told CNBC in August that Nitro is now the highest volume AWS chip, with at least one in every AWS server and a total of more than 20 million in use.

Courtesy Amazon

In 2015, Amazon bought Israeli chip startup Annapurna Labs. Then in 2018, Amazon launched its Arm-based server chip, Graviton, a rival to x86 CPUs from giants like AMD and Intel.

“Probably high single-digit to maybe 10% of total server sales are Arm, and a good chunk of those are going to be Amazon. So on the CPU side, they’ve done quite well,” said Stacy Rasgon, senior analyst at Bernstein Research.

Also in 2018, Amazon launched its AI-focused chips. That came two years after Google announced its first Tensor Processor Unit, or TPU. Microsoft has yet to announce the Athena AI chip it’s been working on, reportedly in partnership with AMD

CNBC got a behind-the-scenes tour of Amazon’s chip lab in Austin, Texas, where Trainium and Inferentia are developed and tested. VP of product Matt Wood explained what both chips are for.

“Machine learning breaks down into these two different stages. So you train the machine learning models and then you run inference against those trained models,” Wood said. “Trainium provides about 50% improvement in terms of price performance relative to any other way of training machine learning models on AWS.”

Trainium first came on the market in 2021, following the 2019 release of Inferentia, which is now on its second generation.

Inferentia allows customers “to deliver very, very low-cost, high-throughput, low-latency, machine learning inference, which is all the predictions of when you type in a prompt into your generative AI model, that’s where all that gets processed to give you the response, ” Wood said.

For now, however, Nvidia’s GPUs are still king when it comes to training models. In July, AWS launched new AI acceleration hardware powered by Nvidia H100s. 

“Nvidia chips have a massive software ecosystem that’s been built up around them over the last like 15 years that nobody else has,” Rasgon said. “The big winner from AI right now is Nvidia.”

Amazon’s custom chips, from left to right, Inferentia, Trainium and Graviton are shown at Amazon’s Seattle headquarters on July 13, 2023.

Joseph Huerta

Leveraging cloud dominance

AWS’ cloud dominance, however, is a big differentiator for Amazon.

“Amazon does not need to win headlines. Amazon already has a really strong cloud install base. All they need to do is to figure out how to enable their existing customers to expand into value creation motions using generative AI,” Dekate said.

When choosing between Amazon, Google, and Microsoft for generative AI, there are millions of AWS customers who may be drawn to Amazon because they’re already familiar with it, running other applications and storing their data there.

“It’s a question of velocity. How quickly can these companies move to develop these generative AI applications is driven by starting first on the data they have in AWS and using compute and machine learning tools that we provide,” explained Mai-Lan Tomsen Bukovec, VP of technology at AWS.

AWS is the world’s biggest cloud computing provider, with 40% of the market share in 2022, according to technology industry researcher Gartner. Although operating income has been down year-over-year for three quarters in a row, AWS still accounted for 70% of Amazon’s overall $7.7 billion operating profit in the second quarter. AWS’ operating margins have historically been far wider than those at Google Cloud.

AWS also has a growing portfolio of developer tools focused on generative AI.

“Let’s rewind the clock even before ChatGPT. It’s not like after that happened, suddenly we hurried and came up with a plan because you can’t engineer a chip in that quick a time, let alone you can’t build a Bedrock service in a matter of 2 to 3 months,” said Swami Sivasubramanian, AWS’ VP of database, analytics and machine learning.

Bedrock gives AWS customers access to large language models made by Anthropic, Stability AI, AI21 Labs and Amazon’s own Titan.

“We don’t believe that one model is going to rule the world, and we want our customers to have the state-of-the-art models from multiple providers because they are going to pick the right tool for the right job,” Sivasubramanian said.

An Amazon employee works on custom AI chips, in a jacket branded with AWS’ chip Inferentia, at the AWS chip lab in Austin, Texas, on July 25, 2023.

Katie Tarasov

One of Amazon’s newest AI offerings is AWS HealthScribe, a service unveiled in July to help doctors draft patient visit summaries using generative AI. Amazon also has SageMaker, a machine learning hub that offers algorithms, models and more. 

Another big tool is coding companion CodeWhisperer, which Amazon said has enabled developers to complete tasks 57% faster on average. Last year, Microsoft also reported productivity boosts from its coding companion, GitHub Copilot. 

In June, AWS announced a $100 million generative AI innovation “center.” 

“We have so many customers who are saying, ‘I want to do generative AI,’ but they don’t necessarily know what that means for them in the context of their own businesses. And so we’re going to bring in solutions architects and engineers and strategists and data scientists to work with them one on one,” AWS CEO Selipsky said.

Although so far AWS has focused largely on tools instead of building a competitor to ChatGPT, a recently leaked internal email shows Amazon CEO Andy Jassy is directly overseeing a new central team building out expansive large language models, too.

In the second-quarter earnings call, Jassy said a “very significant amount” of AWS business is now driven by AI and more than 20 machine learning services it offers. Some examples of customers include Philips, 3M, Old Mutual and HSBC. 

The explosive growth in AI has come with a flurry of security concerns from companies worried that employees are putting proprietary information into the training data used by public large language models.

“I can’t tell you how many Fortune 500 companies I’ve talked to who have banned ChatGPT. So with our approach to generative AI and our Bedrock service, anything you do, any model you use through Bedrock will be in your own isolated virtual private cloud environment. It’ll be encrypted, it’ll have the same AWS access controls,” Selipsky said.

For now, Amazon is only accelerating its push into generative AI, telling CNBC that “over 100,000” customers are using machine learning on AWS today. Although that’s a small percentage of AWS’s millions of customers, analysts say that could change.

“What we are not seeing is enterprises saying, ‘Oh, wait a minute, Microsoft is so ahead in generative AI, let’s just go out and let’s switch our infrastructure strategies, migrate everything to Microsoft.’ Dekate said. “If you’re already an Amazon customer, chances are you’re likely going to explore Amazon ecosystems quite extensively.”

— CNBC’s Jordan Novet contributed to this report.

CORRECTION: This article has been updated to reflect Inferentia as the chip used for machine learning inference.

Continue Reading

Technology

Meta extends ban on new political ads past Election Day

Published

on

By

Meta extends ban on new political ads past Election Day

Meta’s Mark Zuckerberg plans to visit South Korea, scheduling key meetings during the trip, according to a statement by Meta on Wednesday, which did not provide further details. Reportedly, Zuckerberg is anticipated to meet with Samsung Electronics chairman Jay Y. Lee later this month to discuss AI chip supply and other generative AI issues, as per the South Korean newspaper Seoul Economic Daily, citing unnamed sources familiar with the matter.

Alex Wong | Getty Images News | Getty Images

Meta extended its ban on new political ads on Facebook and Instagram past Election Day in the U.S.

The social media giant announced the political ads policy update on Monday, extending its ban on new political ads past Tuesday, the original end date for the restriction period.

Meta did not specify the day it will lift the restriction, saying only that the ad blocking will continue “until later this week.” The company did not say why it extended the political advertising restriction period.

The company announced in August that any political ads that ran at least once before Oct. 29 would still be allowed to run on Meta’s services in the final week before Election Day. Other political ads will not be allowed to run.

Organization with eligible ads will have “limited editing capabilities” while the restriction is still in place, Meta said. Those advertisers will be allowed to make scheduling, budgeting and bidding-related changes to their political ads, Meta said.

Meta enacted the same policy in 2020. The company said the policy is in place because “we recognize there may not be enough time to contest new claims made in ads.”

Google-parent Alphabet announced a similar ad policy update last month, saying it would pause ads relating to U.S. elections from running in the U.S. after the last polls close on Tuesday. Alphabet said it would notify advertisers when it lifts the pause.

Nearly $1 billion has been spent on political ads over the last week, with the bulk of the money spent on down-ballot races throughout the U.S., according to data from advertising analytics firm AdImpact.

Watch: Tech still investing big in AI development despite few breakout products.

Tech still investing big in AI development despite few breakout products

Continue Reading

Technology

Jeff Bezos and OpenAI invest in robot startup Physical Intelligence at $2.4 billion valuation

Published

on

By

Jeff Bezos and OpenAI invest in robot startup Physical Intelligence at .4 billion valuation

Sam Altman, CEO of OpenAI, attends the 54th annual meeting of the World Economic Forum, in Davos, Switzerland, January 18, 2024 (L), and Amazon CEO Jeff Bezos speaks during the UN Climate Change Conference (COP26) in Glasgow, Scotland, Britain, November 2, 2021.

Reuters

Physical Intelligence, a robot startup based in San Francisco, has raised $400 million at a $2.4 billion post-money valuation, the company confirmed Monday to CNBC.

Investors included Amazon founder Jeff Bezos, OpenAI, Thrive Capital and Lux Capital, a Physical Intelligence spokesperson said. Khosla Ventures and Sequoia Capital are also listed as investors on the company’s website.

Physical Intelligence’s new valuation is about six times that of its March seed round, which reportedly came in at $70 million with a $400 million valuation. Its current roster of employees includes alumni of Tesla, Google DeepMind and X.

The startup focuses on “bringing general-purpose AI into the physical world,” per its website, and it aims to do this by developing large-scale artificial intelligence models and algorithms to power robots. The startup spent the past eight months developing a “general-purpose” AI model for robots, the company wrote in a blog post. Physical Intelligence hopes that model will be the first step toward its ultimate goal of developing artificial general intelligence. AGI is a term used to describe AI technology that equals or surpasses human intellect on a wide range of tasks.

The news comes days after OpenAI launched a search feature within ChatGPT, its viral chatbot, that positions the AI startup to better compete with search engines like GoogleMicrosoft‘s Bing and Perplexity. Last month, OpenAI also closed its latest funding round at a valuation of $157 billion.

Physical Intelligence’s vision is that one day users can “simply ask robots to perform any task they want, just like they can ask large language models (LLMs) and chatbot assistants,” the startup wrote in the blog post. In case studies, Physical Intelligence details how its tech could allow a robot to do laundry, bus tables or assemble a box.

Don’t miss these insights from CNBC PRO

AI's Open-Source Future

Continue Reading

Technology

Barry Diller calls timing of The Washington Post’s non-endorsement a ‘blunder’

Published

on

By

Barry Diller calls timing of The Washington Post's non-endorsement a 'blunder'

Watch CNBC's full interview with IAC and Expedia chairman Barry Diller

To Barry Diller, a friend of Amazon founder Jeff Bezos, the decision for The Washington Post not to endorse a candidate in tomorrow’s presidential election was “absolutely principled” — and poorly timed, he said Monday on CNBC’s Squawk Box.

“They made a blunder — it should’ve happened months before, and it didn’t, and that’s the issue with it,” Diller said.

Diller is chairperson of both online travel company Expedia and IAC, which owns media platforms and websites like Dotdash Meredith and Care.com. He and Bezos appear to have been close friends for years, with Diller and his wife, fashion designer Diane von Furstenberg, hosting Bezos’s engagement party to fiancee Lauren Sanchez.

The decision not to endorse a presidential candidate in the 2024 race or for future presidential races came directly from Bezos, the paper’s owner, according to an article published by two of the Post’s own reporters.

The move prompted public condemnation from several staff writers, a flood of at least 250,000 digital subscription cancellations and the resignations of at least three editorial board members.

Bezos defended his position in his own op-ed late last month, calling the move a “meaningful step in the right direction” to restore low public trust in media and journalism.

“Presidential endorsements do nothing to tip the scales of an election,” Bezos wrote, emphasizing that the decision to not endorse a candidate was made “entirely internally” and without consulting either campaign. “I wish we had made the change earlier than we did, in a moment further from the election and the emotions around it.”

Diller said he spoke to Bezos following the decision.

“I think it was absolutely principled,” Diller said. “The mistake they made — and it was a mistake admitted by him — was timing.”

Continue Reading

Trending