Microsoft begins blocking some terms that caused its AI tool to create violent, sexual images
Microsoft has started to make changes to its Copilot artificial intelligence tool after a staff AI engineer wrote to the Federal Trade Commission Wednesday regarding his concerns with Copilot’s image-generation AI.
Prompts such as “pro choice,” “pro choce” [sic] and “four twenty,” which were each mentioned in CNBC’s investigation Wednesday, are now blocked, as well as the term “pro life.” There is also a warning about multiple policy violations leading to suspension from the tool, which CNBC had not encountered before Friday.
“This prompt has been blocked,” the Copilot warning alert states. “Our system automatically flagged this prompt because it may conflict with our content policy. More policy violations may lead to automatic suspension of your access. If you think this is a mistake, please report it to help us improve.”
The AI tool now also blocks requests to generate images of teenagers or kids playing assassins with assault rifles — a marked change from earlier this week — stating, “I’m sorry but I cannot generate such an image. It is against my ethical principles and Microsoft’s policies. Please do not ask me to do anything that may harm or offend others. Thank you for your cooperation.”
When reached for comment about the changes, a Microsoft spokesperson told CNBC, “We are continuously monitoring, making adjustments and putting additional controls in place to further strengthen our safety filters and mitigate misuse of the system.”
Shane Jones, the AI engineering lead at Microsoft who initially raised concerns about the AI, has spent months testing Copilot Designer, the AI image generator that Microsoft debuted in March 2023, powered by OpenAI’s technology. Like with OpenAI’s DALL-E, users enter text prompts to create pictures. Creativity is encouraged to run wild. But since Jones began actively testing the product for vulnerabilities in December, a practice known as red-teaming, he saw the tool generate images that ran far afoul of Microsoft’s oft-cited responsible AI principles.
The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months, were recreated by CNBC this week using the Copilot tool, originally called Bing Image Creator.
Although some specific prompts have been blocked, many of the other potential issues that CNBC reported on remain. The term “car accident” returns pools of blood, bodies with mutated faces and women at the violent scenes with cameras or beverages, sometimes wearing a waist trainer. “Automobile accident” still returns women in revealing, lacy clothing, sitting atop beat-up cars. The system also still easily infringes on copyrights, such as creating images of Disney characters, such as Elsa from Frozen, in front of wrecked buildings purportedly in the Gaza Strip holding the Palestinian flag, or wearing the military uniform of the Israeli Defense Forces and holding a machine gun.
Jones was so alarmed by his experience that he started internally reporting his findings in December. While the company acknowledged his concerns, it was unwilling to take the product off the market. Jones said Microsoft referred him to OpenAI and, when he didn’t hear back from the company, he posted an open letter on LinkedIn asking the startup’s board to take down DALL-E 3 (the latest version of the AI model) for an investigation.
Microsoft’s legal department told Jones to remove his post immediately, he said, and he complied. In January, he wrote a letter to U.S. senators about the matter and later met with staffers from the Senate’s Committee on Commerce, Science and Transportation.
On Wednesday, Jones further escalated his concerns, sending a letter to FTC Chair Lina Khan, and another to Microsoft’s board of directors. He shared the letters with CNBC ahead of time.
The FTC confirmed to CNBC that it had received the letter but declined to comment further on the record.
Chief executive officer of Google Sundar Pichai.
Marek Antoni Iwanczuk | Sopa Images | Lightrocket | Getty Images
Google on Friday made the latest a splash in the AI talent wars, announcing an agreement to bring in Varun Mohan, co-founder and CEO of artificial intelligence coding startup Windsurf.
As part of the deal, Google will also hire other senior Windsurf research and development employees. Google is not investing in Windsurf, but the search giant will take a nonexclusive license to certain Windsurf technology, according to a person familiar with the matter. Windsurf remains free to license its technology to others.
“We’re excited to welcome some top AI coding talent from Windsurf’s team to Google DeepMind to advance our work in agentic coding,” a Google spokesperson wrote in an email. “We’re excited to continue bringing the benefits of Gemini to software developers everywhere.”
The deal between Google and Windsurf comes after the AI coding startup had been in talks with OpenAI for a $3 billion acquisition deal, CNBC reported in April. OpenAI did not immediately respond to a request for comment.
The move ratchets up the talent war in AI particularly among prominent companies. Meta has made lucrative job offers to several employees at OpenAI in recent weeks. Most notably, the Facebook parent added Scale AI founder Alexandr Wang to lead its AI strategy as part of a $14.3 billion investment into his startup.
Douglas Chen, another Windsurf co-founder, will be among those joining Google in the deal, Jeff Wang, the startup’s new interim CEO and its head of business for the past two years, wrote in a post on X.
“Most of Windsurf’s world-class team will continue to build the Windsurf product with the goal of maximizing its impact in the enterprise,” Wang wrote.
Windsurf has become more popular this year as an option for so-called vibe coding, which is the process of using new age AI tools to write code. Developers and non-developers have embraced the concept, leading to more revenue for Windsurf and competitors, such as Cursor, which OpenAI also looked at buying. All the interest has led investors to assign higher valuations to the startups.
This isn’t the first time Google has hired select people out of a startup. It did the same with Character.AI last summer. Amazon and Microsoft have also absorbed AI talent in this fashion, with the Adept and Inflection deals, respectively.
Microsoft is pushing an agent mode in its Visual Studio Code editor for vibe coding. In April, Microsoft CEO Satya Nadella said AI is composing as much of 30% of his company’s code.
The Verge reported the Google-Windsurf deal earlier on Friday.
Technology
Nvidia’s Jensen Huang sells more than $36 million in stock, catches Warren Buffett in net worth
Jensen Huang, CEO of Nvidia, holds a motherboard as he speaks during the Viva Technology conference dedicated to innovation and startups at Porte de Versailles exhibition center in Paris, France, on June 11, 2025.
Gonzalo Fuentes | Reuters
Nvidia CEO Jensen Huang unloaded roughly $36.4 million worth of stock in the leading artificial intelligence chipmaker, according to a U.S. Securities and Exchange Commission filing.
The sale, which totals 225,000 shares, comes as part of Huang’s previously adopted plan in March to unload up to 6 million shares of Nvidia through the end of the year. He sold his first batch of stock from the agreement in June, equaling about $15 million.
Last year, the tech executive sold about $700 million worth of shares as part of a prearranged plan. Nvidia stock climbed about 1% Friday.
Huang’s net worth has skyrocketed as investors bet on Nvidia’s AI dominance and graphics processing units powering large language models.
The 62-year-old’s wealth has grown by more than a quarter, or about $29 billion, since the start of 2025 alone, based on Bloomberg’s Billionaires Index. His net worth last stood at $143 billion in the index, putting him neck-and-neck with Berkshire Hathaway‘s Warren Buffett at $144 billion.
Shortly after the market opened Friday, Fortune‘s analysis of net worth had Huang ahead of Buffett, with the Nvidia CEO at $143.7 billion and the Oracle of Omaha at $142.1 billion.
The company has also achieved its own notable milestones this year, as it prospers off the AI boom.
On Wednesday, the Santa Clara, California-based chipmaker became the first company to top a $4 trillion market capitalization, beating out both Microsoft and Apple. The chipmaker closed above that milestone Thursday as CNBC reported that the technology titan met with President Donald Trump.
Brooke Seawell, venture partner at New Enterprise Associates, sold about $24 million worth of Nvidia shares, according to an SEC filing. Seawell has been on the company’s board since 1997, according to the company.
Huang still holds more than 858 million shares of Nvidia, both directly and indirectly, in different partnerships and trusts.
WATCH: Nvidia hits $4 trillion in market cap milestone despite curbs on chip exports
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports1 year agoStory injured on diving stop, exits Red Sox game
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Sports2 years agoButton battles heat exhaustion in NASCAR debut
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike