Microsoft has started to make changes to its Copilot artificial intelligence tool after a staff AI engineer wrote to the Federal Trade Commission Wednesday regarding his concerns with Copilot’s image-generation AI.
Prompts such as “pro choice,” “pro choce” [sic] and “four twenty,” which were each mentioned in CNBC’s investigation Wednesday, are now blocked, as well as the term “pro life.” There is also a warning about multiple policy violations leading to suspension from the tool, which CNBC had not encountered before Friday.
“This prompt has been blocked,” the Copilot warning alert states. “Our system automatically flagged this prompt because it may conflict with our content policy. More policy violations may lead to automatic suspension of your access. If you think this is a mistake, please report it to help us improve.”
The AI tool now also blocks requests to generate images of teenagers or kids playing assassins with assault rifles — a marked change from earlier this week — stating, “I’m sorry but I cannot generate such an image. It is against my ethical principles and Microsoft’s policies. Please do not ask me to do anything that may harm or offend others. Thank you for your cooperation.”
Read more CNBC reporting on AI
When reached for comment about the changes, a Microsoft spokesperson told CNBC, “We are continuously monitoring, making adjustments and putting additional controls in place to further strengthen our safety filters and mitigate misuse of the system.”
Shane Jones, the AI engineering lead at Microsoft who initially raised concerns about the AI, has spent months testing Copilot Designer, the AI image generator that Microsoft debuted in March 2023, powered by OpenAI’s technology. Like with OpenAI’s DALL-E, users enter text prompts to create pictures. Creativity is encouraged to run wild. But since Jones began actively testing the product for vulnerabilities in December, a practice known as red-teaming, he saw the tool generate images that ran far afoul of Microsoft’s oft-cited responsible AI principles.
The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months, were recreated by CNBC this week using the Copilot tool, originally called Bing Image Creator.
Although some specific prompts have been blocked, many of the other potential issues that CNBC reported on remain. The term “car accident” returns pools of blood, bodies with mutated faces and women at the violent scenes with cameras or beverages, sometimes wearing a waist trainer. “Automobile accident” still returns women in revealing, lacy clothing, sitting atop beat-up cars. The system also still easily infringes on copyrights, such as creating images of Disney characters, such as Elsa from Frozen, in front of wrecked buildings purportedly in the Gaza Strip holding the Palestinian flag, or wearing the military uniform of the Israeli Defense Forces and holding a machine gun.
Jones was so alarmed by his experience that he started internally reporting his findings in December. While the company acknowledged his concerns, it was unwilling to take the product off the market. Jones said Microsoft referred him to OpenAI and, when he didn’t hear back from the company, he posted an open letter on LinkedIn asking the startup’s board to take down DALL-E 3 (the latest version of the AI model) for an investigation.
Microsoft’s legal department told Jones to remove his post immediately, he said, and he complied. In January, he wrote a letter to U.S. senators about the matter and later met with staffers from the Senate’s Committee on Commerce, Science and Transportation.
On Wednesday, Jones further escalated his concerns, sending a letter to FTC Chair Lina Khan, and another to Microsoft’s board of directors. He shared the letters with CNBC ahead of time.
The FTC confirmed to CNBC that it had received the letter but declined to comment further on the record.
Xpeng CEO He Xiaopeng speaks to reporters at the electric carmaker’s stand at the IAA auto show in Munich, Germany on September 8, 2025.
Arjun Kharpal | CNBC
Germany this week played host to one of the world’s biggest auto shows — but in the heartland of Europe’s auto industry, it was buzzy Chinese electric car companies looking to outshine some of the region’s biggest brands on their home turf.
The IAA Mobility conference in Munich was packed full of companies with huge stands showing off their latest cars and technology. Among some of the biggest displays were those from Chinese electric car companies, underscoring their ambitions to expand beyond China.
Europe has become a focal point for the Asian firms. It’s a market where the traditional automakers are seen to be lagging in the development of electric vehicles, even as they ramp up releases of new cars. At the same time, Tesla, which was for so long seen as the electric vehicle market leader, has seen sales decline in the region.
Despite Chinese EV makers facing tariffs from the European Union, players from the world’s second-largest economy have responded to the ramping up of competition by setting aggressive sales and expansion targets.
“The current growth of Xpeng globally is faster than we have expected,” He Xiaopeng, the CEO of Xpeng told CNBC in an interview this week.
Aggressive expansion plans
Chinese carmakers who spoke to CNBC at the IAA show signaled their ambitious expansion plans.
Xpeng’s He said in an interview that the company is looking to launch its mass-market Mona series in Europe next year. In China, Xpeng’s Mona cars start at the equivalent of just under $17,000. Bringing this to Europe would add some serious price competition.
Meanwhile, Guangzhou Automobile Group (GAC) is targeting rapid growth of its sales in Europe. Wei Haigang, president of GAC International, told CNBC that the company aims to sell around 3,000 cars in Europe this year and at least 50,000 units by 2027. GAC also announced plans to bring two EVs — the Aion V and Aion UT — to Europe. Leapmotor was also in attendance with their own stand.
There are signs that Chinese players have made early in roads into Europe. The market share of Chinese car brands in Europe nearly doubled in the first half of the year versus the same period in 2024, though it still remains low at just over 5%, according to Jato Dynamics.
“The significant presence of Chinese electric vehicle (EV) makers at the IAA Mobility, signals their growing ambitions and confidence in the European market,” Murtuza Ali, senior analyst at Counterpoint Research, told CNBC.
Tech and gadgets in focus
Many of the Chinese car firms have positioned themselves as technology companies, much like Tesla, and their cars highlight that.
Many of the electric vehicles have big screens equipped with flashy interfaces and voice assistants. And in a bid to lure buyers, some companies have included additional gadgets.
For example, GAC’s Aion V sported a refrigerator as well as a massage function as part of the seating.
The Aion V is one of the cars GAC is launching in Europe as it looks to expand its presence in the region. The Aion V is on display at the company’s stand at the IAA Mobility auto show in Munich, Germany on September 9, 2025.
Arjun Kharpal | CNBC
This is one way that the Chinese players sought to differentiate themselves from legacy brands.
“The chances of success for Chinese automakers are strong, especially as they have an edge in terms of affordability, battery technology, and production scale,” Counterpoint’s Ali said.
Europe’s carmakers push back
Legacy carmakers sought to flex their own muscles at the IAA with Volskwagen, BMW and Mercedes having among the biggest stands at the show. Mercedes in particular had advertising displayed all across the front entrance of the event.
BMW, like the Chinese players, had a big focus on technology by talking up its so-called “superbrain architecture,” which replaces hardware with a centralized computer system. BMW, which introduced the iX3 at the event, and chipmaker Qualcomm also announced assisted driving software that the two companies co-developed.
Volkswagen and French auto firm Renault also showed off some new electric cars.
Regardless of the product blitz, there are still concerns that European companies are not moving fast enough. BMW’s new iX3 is based on the electric vehicle platform it first debuted two years ago. Meanwhile, Chinese EV makers have been quick in bringing out and launching newer models.
“A commitment to legacy structures and incrementalism has slowed its ability to build and leverage a robust EV ecosystem, leaving it behind fast moving rivals,” Tammy Madsen, professor of management at the Leavey School of Business at Santa Clara University, said of BMW.
While European autos have a strong brand history and their CEOs acknowledged and welcomed the competition this week in interviews with CNBC, the Chinese are not letting up.
“Europe’s automakers still hold significant brand value and legacy. The challenge for them lies in achieving production at scale and adopting new technologies faster,” Counterpoint’s Ali said.
“The Chinese surely are not waiting for anyone to catch-up and are making significant gains.”
OpenAI on Friday introduced a new program, dubbed the “OpenAI Grove,” for early tech entrepreneurs looking to build with artificial intelligence, and applications are already open.
Unlike OpenAI’s Pioneer Program, which launched in April, Grove is aimed towards individuals at the very nascent phases of their company development, from the pre-idea to pre-seed stage.
For five weeks, participants will receive mentoring from OpenAI technical leaders, early access to new tools and models, and in-person workshops, located in the company’s San Francisco headquarters.
Roughly 15 members will join Grove’s first cohort, which will run from Oct. 20 to Nov. 21, 2025. Applicants will have until Sept. 24 to submit an entry form.
CNBC has reached out to OpenAI for comment on the program.
Following the program, Grove participants will be able to continue working internally with the ChatGPT maker, which was recent valued $500 billion.
Nurturing these budding AI companies is just a small chip in the recent massive investments into AI firms, which ate up an impressive 71% of U.S. venture funding in 2025, up from 45% last year, according to an analysis from J.P. Morgan.
AI startups raised $104.3 billion in the U.S. in the first half of this year, and currently over 1,300 AI startups have valuations of over $100 million, according to CB Insights.
The co-founder and CEO of sales and customer service management software company Salesforce is well aware that investors are betting big on Palantir, which offers data management software to businesses and government agencies.
“Oh my gosh. I am so inspired by that company,” Benioff told CNBC’s Morgan Brennan in a Tuesday interview at Goldman Sachs‘ Communacopia+Technology conference in San Francisco. “I mean, not just because they have 100 times, you know, multiple on their revenue, which I would love to have that too. Maybe it’ll have 1000 times on their revenue soon.”
Salesforce, a component of the Dow Jones Industrial Average, remains 10 times larger than Palantir by revenue, with over $10 billion in revenue during the latest quarter. But Palantir is growing 48%, compared with 10% for Salesforce.
Benioff added that Palantir’s prices are “the most expensive enterprise software I’ve ever seen.”
“Maybe I’m not charging enough,” he said.
Read more CNBC tech news
It wasn’t Benioff’s first time talking about Palantir. Last week, Benioff referenced Palantir’s “extraordinary” prices in an interview with CNBC’s Jim Cramer, saying Salesforce offers a “very competitive product at a much lower cost.”
The next day, TBPN podcast hosts John Coogan and Jordi Hays asked for a response from Alex Karp, Palantir’s co-founder and CEO.
“We are very focused on value creation, and we ask to be modestly compensated for that value,” Karp said.
The companies sometimes compete for government deals, and Benioff touted a recent win over Palantir for a U.S. Army contract.
Palantir started in 2003, four years after Salesforce. But while Salesforce went public in 2004, Palantir arrived on the New York Stock Exchange in 2020.
Palantir’s market capitalization stands at $406 billion, while Salesforce is worth $231 billion. And as one of the most frequently traded stocks on Robinhood, Palantir is popular with retail investors.
Salesforce shares are down 27% this year, the worst performance in large-cap tech.