Connect with us

Published

on

Yoshua Bengio (L) and Max Tegmark (R) discuss the development of artificial general intelligence during a live podcast recording of CNBC’s “Beyond The Valley” in Davos, Switzerland in January 2025.

CNBC

Artificial general intelligence built like “agents” could prove dangerous as its creators might lose control of the system, two of of the world’s most prominent AI scientists told CNBC.

In the latest episode of CNBC’s “Beyond The Valley” podcast released on Tuesday, Max Tegmark, a professor at the Massachusetts Institute of Technology and the President of the Future of Life Institute, and Yoshua Bengio, dubbed one of the “godfathers of AI” and a professor at the Université de Montréal, spoke about their concerns about artificial general intelligence, or AGI. The term broadly refers to AI systems that are smarter than humans.

Their fears stem from the world’s biggest firms now talking about “AI agents” or “agentic AI” — which companies claim will allow AI chatbots to act like assistants or agents and assist in work and everyday life. Industry estimates vary on when AGI will come into existence.

With that concept comes the idea that AI systems could have some “agency” and thoughts of their own, according to Bengio.

“Researchers in AI have been inspired by human intelligence to build machine intelligence, and, in humans, there’s a mix of both the ability to understand the world like pure intelligence and the agentic behavior, meaning … to use your knowledge to achieve goals,” Bengio told CNBC’s “Beyond The Valley.”

“Right now, this is how we’re building AGI: we are trying to make them agents that understand a lot about the world, and then can act accordingly. But this is actually a very dangerous proposition.”

Bengio added that pursuing this approach would be like “creating a new species or a new intelligent entity on this planet” and “not knowing if they’re going to behave in ways that agree with our needs.”

“So instead, we can consider, what are the scenarios in which things go badly and they all rely on agency? In other words, it is because the AI has its own goals that we could be in trouble.”

The idea of self-preservation could also kick in, as AI gets even smarter, Bengio said.

“Do we want to be in competition with entities that are smarter than us? It’s not a very reassuring gamble, right? So we have to understand how self-preservation can emerge as a goal in AI.”

AI tools the key

For MIT’s Tegmark, the key lies in so-called “tool AI” — systems that are created for a specific, narrowly-defined purpose, but that don’t have to be agents.

Tegmark said a tool AI could be a system that tells you how to cure cancer, or something that possesses “some agency” like a self-driving car “where you can prove or get some really high, really reliable guarantees that you’re still going to be able to control it.”

“I think, on an optimistic note here, we can have almost everything that we’re excited about with AI … if we simply insist on having some basic safety standards before people can sell powerful AI systems,” Tegmark said.

“They have to demonstrate that we can keep them under control. Then the industry will innovate rapidly to figure out how to do that better.”

Tegmark’s Future of Life Institute in 2023 called for a pause to the development of AI systems that can compete with human-level intelligence. While that has not happened, Tegmark said people are talking about the topic, and now it is time to take action to figure out how to put guardrails in place to control AGI.

“So at least now a lot of people are talking the talk. We have to see if we can get them to walk the walk,” Tegmark told CNBC’s “Beyond The Valley.”

“It’s clearly insane for us humans to build something way smarter than us before we figured out how to control it.”

There are several views on when AGI will arrive, partly driven by varying definitions.

OpenAI CEO Sam Altman said his company knows how to build AGI and said it will arrive sooner than people think, though he downplayed the impact of the technology.

“My guess is we will hit AGI sooner than most people in the world think and it will matter much less,” Altman said in December.

Continue Reading

Technology

Waymo begins offering freeway robotaxi rides in San Francisco, LA and Phoenix

Published

on

By

Waymo begins offering freeway robotaxi rides in San Francisco, LA and Phoenix

Watch: Waymo launches paid robotaxi rides on freeways

Waymo robotaxis will now take passengers on freeways in three major U.S. cities, marking a major milestone for the driverless, ride-hailing company.

Alphabet-owned Waymo on Wednesday said it will begin offering those types of trips in the San Francisco, Phoenix and Los Angeles markets “when a freeway route is meaningfully faster.” The Google sister company will gradually extend freeway trips to more riders and locations over time.

Although Waymo’s driverless cars have previously taken passengers on smaller highways and side streets, Wednesday’s expansion marks the first time the company will take payment from public riders to go on freeways with higher speed limits.

“Freeway driving is one of those things that’s very easy to learn, but very hard to master when we’re talking about full autonomy without a human driver as a backup, and at scale,” Waymo co-CEO Dmitri Dolgov said at a press event ahead of the announcement. “It took time to do it properly.”

Waymo vehicles will generally travel up to a freeway’s maximum posted speed limit, which is 65 mph in many cases, the company said. However, a spokesperson confirmed, the robotaxis may sometimes go a few miles over the limit for safety purposes in extraordinary circumstances.

Freeway operations required expanded operational protocols, including coordination with safety officials at the California Highway Patrol and the Arizona Department of Public Safety, Waymo said. The company also installed additional infrastructure needed to charge its fleet of electric robotaxis given the freeway expansion.

Over the last year, Waymo has offered select Alphabet employees robotaxi freeway rides around San Francisco, Los Angeles and Phoenix in preparation for Wednesday’s launch, said Waymo Product Manager Jacopo Sannazzaro.

The company has been testing on freeways for more than a decade in total, he added. Besides testing on public roads and closed courses, Waymo also conducts testing in simulation to determine how its vehicles will respond to both typical and hard-to-replicate events, like merging onto freeways, lane-splitting motorcyclists or another car flipping over.

CNBC took a freeway test ride in a Waymo in the San Francisco Bay Area, from YouTube’s offices in San Bruno to San Mateo and back. The ride went on and off ramps along the California 101 seamlessly, with no incidents.

Waymo’s continued expansion

After already launching its robotaxi service in Austin, Texas, San Francisco, Phoenix and Los Angeles, Waymo has also announced plans to expand to Miami, San Diego and Washington, D.C., in 2026. The company is also testing its vehicles in New York City, Tokyo and plans to begin offering rides to the public in London next year.

Waymo on Wednesday also announced that it’s expanding its service footprint in the San Francisco Bay Area to San Jose. That includes rides to and from San Jose Mineta International Airport, marking the company’s second international airport destination. The SJC airport plans were first announced in September.

In 2023, Waymo launched at Phoenix Sky Harbor International Airport, which has become its most popular destination in the Phoenix metropolitan service area.

The company expanded its service in March to include an additional 27 square miles of coverage in the region, including cities like Mountain View and Palo Alto. After the Wednesday expansion, Waymo now offers service in about 260 square miles of Silicon Valley.

Would-be Waymo competitor Tesla also takes passengers to and from SJC. Customers can hail a ride via Tesla’s “Robotaxi” app, but that name is not precisely descriptive. Tesla only operates a car service with human drivers on board, not a commercial robotaxi service like Waymo’s, due to a mix of technical limits and permit requirements in California.

Continue Reading

Technology

AMD’s Lisa Su dismisses AI spending fears as stock rallies on growth projections: ‘It’s the right gamble’

Published

on

By

AMD's Lisa Su dismisses AI spending fears as stock rallies on growth projections: 'It's the right gamble'

AMD CEO Lisa Su dismisses AI spending fears: 'It's the right gamble'

Advanced Micro Devices‘ CEO Lisa Su shut down concerns over Big Tech’s elevated spending during an interview with CNBC’s “Squawk Box” on Wednesday and said investing in more computing will accelerate the pace of innovation.

“I don’t think it’s a big gamble,” she said. “I think it’s the right gamble.”

Many of AMD’s hyperscaler customers over the last 12 months have beefed up spending as the technology reaches an “inflection point” and companies can see the return on that spending, Su added.

Su’s comments come as tech’s megacaps announced more than $380 billion in AI spending in their latest earnings reports as the firms race to build out infrastructure to support soaring demand.

Read more CNBC tech news

On Tuesday, Su told analysts that AMD expects revenues to grow 35% per year over the next three to five years due to “insatiable” AI chip demand.

Shares were last up more than 7%.

Concerns of a potential AI bubble have jolted markets in recent sessions as Wall Street raises concerns that valuations have gotten too high.

Stock Chart IconStock chart icon

hide content

AMD 5-year stock chart.

Continue Reading

Technology

Coinbase moves incorporation to Texas from Delaware, following Musk’s lead

Published

on

By

Coinbase moves incorporation to Texas from Delaware, following Musk's lead

Brian Armstrong, chief executive officer of Coinbase Global Inc., speaks during the Messari Mainnet summit in New York, on Thursday, Sept. 21, 2023.

Michael Nagle | Bloomberg | Getty Images

Coinbase is following Tesla out of Delaware and into Texas.

Paul Grewal, Coinbase’s chief legal officer, wrote in a Wall Street Journal op-ed on Wednesday that the crypto exchange is moving its state of incorporation, a year after Elon Musk did the same with his electric vehicle maker. Musk also reincorporated his rocket maker SpaceX from Delaware to Texas.

“Delaware’s legal framework once provided companies with consistency. But no more,” Grawal wrote, pointing to recent “unpredictable outcomes” in the Delaware Chancery Court.

A handful of notable names, including Dropbox, TripAdvisor and venture firm Andreessen Horowitz have announced departures from Delaware. It’s a move that was championed by Musk following a Delaware Chancery Court ruling that ordered Tesla to rescind the CEO’s 2018 pay package, worth about $56 billion in options.

“If your company is still incorporated in Delaware, I recommend moving to another state as soon as possible,” Musk wrote in a post on X in February 2024, when he filed to change SpaceX’s incorporation state.

Read more CNBC tech news

Last week, Tesla shareholders voted to approve Musk’s more recent pay package, which could be worth up to $1 trillion.

Delaware has long been the dominant state for U.S. companies to incorporate due to its flexible corporate code and expert judiciary, and is seen as balancing the rights of executives and shareholders. A Texas state law allows corporations to limit shareholder lawsuits against insiders for breach of fiduciary duty. 

Coinbase and Andreessen Horowitz, an early backer, currently face a lawsuit in Delaware concerning the sale of shares in the crypto company tied to its public listing in 2021.

Like Musk, Coinbase CEO Brian Armstrong was a major contributor to President Donald Trump’s 2024 campaign for the White House.

— CNBC’s Lora Kolodny contributed to this report.

WATCH: Delaware Gov. Matt Meyer says laws didn’t change as a result of Musk

Delaware Gov. Matt Meyer: Our corporate laws did not change as a result of Elon Musk's Tesla case

Continue Reading

Trending