Yoshua Bengio (L) and Max Tegmark (R) discuss the development of artificial general intelligence during a live podcast recording of CNBC’s “Beyond The Valley” in Davos, Switzerland in January 2025.
CNBC
Artificial general intelligence built like “agents” could prove dangerous as its creators might lose control of the system, two of of the world’s most prominent AI scientists told CNBC.
In the latest episode of CNBC’s “Beyond The Valley” podcast released on Tuesday, Max Tegmark, a professor at the Massachusetts Institute of Technology and the President of the Future of Life Institute, and Yoshua Bengio, dubbed one of the “godfathers of AI” and a professor at the Université de Montréal, spoke about their concerns about artificial general intelligence, or AGI. The term broadly refers to AI systems that are smarter than humans.
Their fears stem from the world’s biggest firms now talking about “AI agents” or “agentic AI” — which companies claim will allow AI chatbots to act like assistants or agents and assist in work and everyday life. Industry estimates vary on when AGI will come into existence.
With that concept comes the idea that AI systems could have some “agency” and thoughts of their own, according to Bengio.
“Researchers in AI have been inspired by human intelligence to build machine intelligence, and, in humans, there’s a mix of both the ability to understand the world like pure intelligence and the agentic behavior, meaning … to use your knowledge to achieve goals,” Bengio told CNBC’s “Beyond The Valley.”
“Right now, this is how we’re building AGI: we are trying to make them agents that understand a lot about the world, and then can act accordingly. But this is actually a very dangerous proposition.”
Bengio added that pursuing this approach would be like “creating a new species or a new intelligent entity on this planet” and “not knowing if they’re going to behave in ways that agree with our needs.”
“So instead, we can consider, what are the scenarios in which things go badly and they all rely on agency? In other words, it is because the AI has its own goals that we could be in trouble.”
The idea of self-preservation could also kick in, as AI gets even smarter, Bengio said.
“Do we want to be in competition with entities that are smarter than us? It’s not a very reassuring gamble, right? So we have to understand how self-preservation can emerge as a goal in AI.”
AI tools the key
For MIT’s Tegmark, the key lies in so-called “tool AI” — systems that are created for a specific, narrowly-defined purpose, but that don’t have to be agents.
Tegmark said a tool AI could be a system that tells you how to cure cancer, or something that possesses “some agency” like a self-driving car “where you can prove or get some really high, really reliable guarantees that you’re still going to be able to control it.”
“I think, on an optimistic note here, we can have almost everything that we’re excited about with AI … if we simply insist on having some basic safety standards before people can sell powerful AI systems,” Tegmark said.
“They have to demonstrate that we can keep them under control. Then the industry will innovate rapidly to figure out how to do that better.”
Tegmark’s Future of Life Institute in 2023 called for a pause to the development of AI systems that can compete with human-level intelligence. While that has not happened, Tegmark said people are talking about the topic, and now it is time to take action to figure out how to put guardrails in place to control AGI.
“So at least now a lot of people are talking the talk. We have to see if we can get them to walk the walk,” Tegmark told CNBC’s “Beyond The Valley.”
“It’s clearly insane for us humans to build something way smarter than us before we figured out how to control it.”
There are several views on when AGI will arrive, partly driven by varying definitions.
OpenAI CEO Sam Altman said his company knows how to build AGI and said it will arrive sooner than people think, though he downplayed the impact of the technology.
“My guess is we will hit AGI sooner than most people in the world think and it will matter much less,” Altman said in December.
Amazon CEO Andy Jassy speaks during a keynote address at AWS re:Invent 2024, a conference hosted by Amazon Web Services, at The Venetian Las Vegas on December 3, 2024 in Las Vegas, Nevada.
Noah Berger | Getty Images Entertainment | Getty Images
Amazon said Thursday it plans to boost its capital expenditures to $100 billion in 2025, as it continues its investments in artificial intelligence.
The capex figure exceeds last year’s spending of roughly $83 billion. Amazon CEO Andy Jassy had predicted in October that the company’s 2025 capex would surpass last year’s figure, primarily driven by growth in generative AI.
“We spent $26.3 billion in capex in Q4, and I think that is reasonably representative of what you expect an annualized capex rate in 2025,” Jassy said on call with investors after the company released its fourth-quarter earnings report. “The vast majority of that capex spend is on AI for AWS.”
Amazon has been rushing to invest in data centers, networking gear and hardware to meet vast demand for generative AI, which has exploded in popularity since OpenAI released its ChatGPT assistant in late 2022. Amazon has introduced a flurry of AI products, including its own set of Nova models, Trainium chips, a shopping chatbot, and a marketplace for third-party models called Bedrock.
Other tech companies are also spending big on AI. Google parent Alphabetsaid Tuesday it expects to invest about $75 billion in capital expenditures this year. Last month, Microsoftsaid it planned to spend $80 billion in fiscal 2025 on the buildout of data centers to support AI workloads. Metasaid it will spend as much as $65 billion on capital expenditures as it works to construct more data center and computing infrastructure.
Amazon gave an update on its spending plans after reporting mixed results for the fourth quarter. The company projected weaker-than-expected sales for the current period, which overshadowed a beat on the top and bottom lines in the fourth quarter. Shares fell more than 4% in extended trading.
Jassy tried to reassure investors on the call that the jump in spending will be worthwhile, calling it a “once-in-a-lifetime type of business opportunity.”
“I think that both our business, our customers and shareholders will be happy, medium to long-term, that we’re pursuing the capital opportunity and the business opportunity in AI,” Jassy said. “We also have capex that we’re spending this year in our stores business, really with an aim towards trying to continue to improve the delivery speed and our cost to serve.”
Tech companies are facing fresh skepticism of their AI spending plans after the early success of Chinese AI startup DeepSeek. The lab claims it only took two months and less than $6 million to develop its R1 model, which it says rivals OpenAI’s o1. Markets were roiled by the launch last week, with chipmakers Nvidia and Broadcom losing a combined $800 billion in market cap.
Verily is selling its stop-loss insurance subsidiary, Granular Insurance Company, to the insurance provider Elevance Health, the Alphabet health tech company confirmed to CNBC on Thursday.
Verily is one of Google’s sister companies and operates within Alphabet’s “Other Bets” category. The Granular sale is the latest in a series of sweeping changes at the precision health company, which has slashed its workforce, restructured its business and overhauled its executive leadership in recent years.
The terms of the deal were not disclosed.
Verily launched Granular, initially called Coefficient Insurance Company, in 2020 with financial backing from the commercial insurance unit of the Swiss Re Group. The business offered self-funded employers and captives medical stop-loss, fronting reinsurance and fronting solutions that used “proprietary technology,” Verily said.
Alphabet’s health company has raised more than $1 billion, and it has attracted big-name talent.Apple’s former head of health strategic initiatives, Myoung Cha, joined Verily as chief product officer last year, and Andrew Trister, Verily’s chief medical and scientific officer, was a founding member of Apple’s health team. Amy Abernethy, who served as principal deputy commissioner at the U.S. Food and Drug administration, joined the company in 2021 before departing in late 2023.
But Verily has struggled to find and stick to a winning niche in health care.
The company started as a moonshot in 2015 within Alphabet’s innovation lab X, formerly Google X, where it developed hardware like continuous glucose monitors. Verily pivoted to pandemic response when Covid-19 broke out in 2020, and it switched directions again to focus on precision medicine in 2022.
Verily introduced a new artificial intelligence-powered chronic care solution in June called Lightpath. The first offering is metabolic health focused, and it will support patients taking the blockbuster weight loss medications called GLP-1s, using continuous glucose monitors or other interventions, according to a release.
And now, the company is getting out of the insurance business.
Elevance Health did not immediately respond to CNBC’s request for comment. The deal was first reported by Business Insider.
CEO of Meta and Facebook Mark Zuckerberg, Lauren Sanchez, Amazon founder Jeff Bezos, Google CEO Sundar Pichai and Tesla and SpaceX CEO Elon Musk attend the inauguration ceremony before Donald Trump is sworn in as the 47th US President in the US Capitol Rotunda in Washington, DC, on Jan. 20, 2025.
Saul Loeb | Via Reuters
The strengthening dollar is posing challenges for the biggest U.S. tech companies, which have become increasingly reliant on overseas revenue. With other currencies weakening, money made elsewhere is worth less when converted into dollars.
Amazon should suffer less than its megacap peers as the e-commerce giant generates a higher percentage of sales in the U.S. However, in its fourth-quarter earnings report on Thursday, Amazon said foreign exchange rates are to blame for the company’s weaker-than-expected first-quarter forecast and the possibility of its slowest revenue growth on record.
Revenue in the current quarter will land between $151 billion and $155.5 billion, suggesting annual growth of just 5% to 9%. Amazon’s slowest quarter for growth came in mid-2022, when revenue increased by 7.2%.
“This guidance anticipates an unusually large, unfavorable impact of approximately $2.1 billion, or 150 basis points, from foreign exchange rates,” Amazon said in the earnings release.
On its earnings call that followed, Amazon said it saw $700 million “more of foreign exchange headwind than we anticipated” in the fourth quarter. During the period, international revenue totaled $43.4 billion, or 23% of overall sales.
At Apple, roughly 58% of revenue came from overseas in the latest period. For Meta, it was 55%, Alphabet reported about 52%, Microsoft slightly under 50% and Tesla just over 50% for all of 2024.
The U.S. dollar index — which measures the greenback against a basket of rivals — hit its highest level in more than two years last month, ahead of President Donald Trump’s inauguration. The dollar climbed steadily from late November through mid-January and has since fallen slightly.
The dollar may be particularly volatile in the coming weeks and months due to uncertainties surrounding Trump’s tariff policies and the threat of a trade war, most notably China, along with a lack of clarity about U.S. foreign policy, given comments Trump has made about potentially trying to take over Greenland and Gaza.
Here’s what other companies had to say on the topic of foreign exchange in issuing their financial results.
Microsoft CFO Amy Hood said foreign exchange did “not have a significant impact on our results and was roughly in line with expectation,” though for the current quarter it would bring down revenue growth by “more than 1 point.”
Susan Li, Meta’s finance chief, said the company expects “a three-point headwind in Q1” after foreign exchange “approximately neutral to revenue in Q4, just with the dollar strengthening, in particular against the euro.”
Alphabet CFO Anat Ashkenazi said investors can “expect a larger headwind to our revenues from the strengthening of the U.S. dollar relative to key currencies in Q1 versus Q4 2024.”
Apple finance chief Kevan Parekh warned last week that, “As the dollar strengthens significantly, we expect foreign exchange to be a headwind and to have a negative impact on revenue of about 2.5 percentage points on a year-over-year basis.”
The rise of the dollar will lead investors to pay close attention to job numbers out on Friday. When the Bureau of Labor Statistics releases its nonfarm payrolls count for January, it’s projected to show growth of 169,000, down from 256,000 in December, but nearly in line with the past three-month average. The unemployment rate is projected to stay at 4.1%, according to the Dow Jones consensus for the report.
After that, the the tech industry will wait to see what Nvidia has to say about foreign exchange when the chipmaker reports earnings later in February. In the period ending in October, Nvidia generated about 58% of its revenue from outside the U.S.