Microsoft begins blocking some terms that caused its AI tool to create violent, sexual images
Microsoft has started to make changes to its Copilot artificial intelligence tool after a staff AI engineer wrote to the Federal Trade Commission Wednesday regarding his concerns with Copilot’s image-generation AI.
Prompts such as “pro choice,” “pro choce” [sic] and “four twenty,” which were each mentioned in CNBC’s investigation Wednesday, are now blocked, as well as the term “pro life.” There is also a warning about multiple policy violations leading to suspension from the tool, which CNBC had not encountered before Friday.
“This prompt has been blocked,” the Copilot warning alert states. “Our system automatically flagged this prompt because it may conflict with our content policy. More policy violations may lead to automatic suspension of your access. If you think this is a mistake, please report it to help us improve.”
The AI tool now also blocks requests to generate images of teenagers or kids playing assassins with assault rifles — a marked change from earlier this week — stating, “I’m sorry but I cannot generate such an image. It is against my ethical principles and Microsoft’s policies. Please do not ask me to do anything that may harm or offend others. Thank you for your cooperation.”
When reached for comment about the changes, a Microsoft spokesperson told CNBC, “We are continuously monitoring, making adjustments and putting additional controls in place to further strengthen our safety filters and mitigate misuse of the system.”
Shane Jones, the AI engineering lead at Microsoft who initially raised concerns about the AI, has spent months testing Copilot Designer, the AI image generator that Microsoft debuted in March 2023, powered by OpenAI’s technology. Like with OpenAI’s DALL-E, users enter text prompts to create pictures. Creativity is encouraged to run wild. But since Jones began actively testing the product for vulnerabilities in December, a practice known as red-teaming, he saw the tool generate images that ran far afoul of Microsoft’s oft-cited responsible AI principles.
The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months, were recreated by CNBC this week using the Copilot tool, originally called Bing Image Creator.
Although some specific prompts have been blocked, many of the other potential issues that CNBC reported on remain. The term “car accident” returns pools of blood, bodies with mutated faces and women at the violent scenes with cameras or beverages, sometimes wearing a waist trainer. “Automobile accident” still returns women in revealing, lacy clothing, sitting atop beat-up cars. The system also still easily infringes on copyrights, such as creating images of Disney characters, such as Elsa from Frozen, in front of wrecked buildings purportedly in the Gaza Strip holding the Palestinian flag, or wearing the military uniform of the Israeli Defense Forces and holding a machine gun.
Jones was so alarmed by his experience that he started internally reporting his findings in December. While the company acknowledged his concerns, it was unwilling to take the product off the market. Jones said Microsoft referred him to OpenAI and, when he didn’t hear back from the company, he posted an open letter on LinkedIn asking the startup’s board to take down DALL-E 3 (the latest version of the AI model) for an investigation.
Microsoft’s legal department told Jones to remove his post immediately, he said, and he complied. In January, he wrote a letter to U.S. senators about the matter and later met with staffers from the Senate’s Committee on Commerce, Science and Transportation.
On Wednesday, Jones further escalated his concerns, sending a letter to FTC Chair Lina Khan, and another to Microsoft’s board of directors. He shared the letters with CNBC ahead of time.
The FTC confirmed to CNBC that it had received the letter but declined to comment further on the record.
Technology
How Broadcom’s big OpenAI deal fits into the data center boom and what it means for the AI trade
Oracle CEO, Clay Magouyrk, sits down with CNBC’s David Faber on Oct. 13, 2025.
CNBC
Oracle CEO Clay Magouyrk, one of the two people tapped last month to lead the software company, is confident that OpenAI will be able to cover the costs of the massive amount of cloud infrastructure services it consumes.
In an interview with CNBC’s David Faber at Oracle’s AI World conference on Monday, Magouyrk said “of course” OpenAI can pay $60 billion for a year’s worth of cloud resources. In July, OpenAI agreed to a five-year deal with Oracle that’s worth over $300 billion.
“Just look at the rate at which they’ve grown to, you know, almost a billion users. That’s just unheard of,” said Magouyrk, who sat alongside fellow Oracle CEO Mike Sicilia for the interview in Las Vegas.
OpenAI said last week that its flagship ChatGPT chatbot, which was publicly launched less than three years ago, now has 800 million weekly active users. In 2024, OpenAI recorded a $5 billion net loss.
Sicilia said Oracle has started integrating OpenAI artificial intelligence models into a patient portal for viewing electronic health records. Oracle acquired EHR vendor Cerner for about $28 billion in 2022.
“I’ve seen the results, and I really do think that they’re going to have a dramatic impact on industries, on enterprises of all types,” Sicilia said of OpenAI.
OpenAI rents out Nvidia graphics chips to run models through Oracle, as well as CoreWeave, Google and Microsoft. At the same time, the company is designing a custom AI processor that Broadcom will build. Earlier on Monday, Broadcom and OpenAI said they will jointly deploy 10 gigawatts worth of the new OpenAI chips.
Building out that much infrastructure requires a hefty amount of new energy.
“I think it’s a factor of time, not a factor of if we’ll have enough power,” Sicilia said.
Oracle shares rose almost 6% on Monday. The stock has gained 86% this year, lifting Oracle’s market cap close to $900 billion.
WATCH: Oracle CEO Magouyrk: ‘Of course’ OpenAI can pay $60 billion per year
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports2 years agoStory injured on diving stop, exits Red Sox game
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports3 years agoButton battles heat exhaustion in NASCAR debut
-
Sports3 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment1 year agoHere are the best electric bikes you can buy at every price level in October 2024