Microsoft begins blocking some terms that caused its AI tool to create violent, sexual images
Microsoft has started to make changes to its Copilot artificial intelligence tool after a staff AI engineer wrote to the Federal Trade Commission Wednesday regarding his concerns with Copilot’s image-generation AI.
Prompts such as “pro choice,” “pro choce” [sic] and “four twenty,” which were each mentioned in CNBC’s investigation Wednesday, are now blocked, as well as the term “pro life.” There is also a warning about multiple policy violations leading to suspension from the tool, which CNBC had not encountered before Friday.
“This prompt has been blocked,” the Copilot warning alert states. “Our system automatically flagged this prompt because it may conflict with our content policy. More policy violations may lead to automatic suspension of your access. If you think this is a mistake, please report it to help us improve.”
The AI tool now also blocks requests to generate images of teenagers or kids playing assassins with assault rifles — a marked change from earlier this week — stating, “I’m sorry but I cannot generate such an image. It is against my ethical principles and Microsoft’s policies. Please do not ask me to do anything that may harm or offend others. Thank you for your cooperation.”
When reached for comment about the changes, a Microsoft spokesperson told CNBC, “We are continuously monitoring, making adjustments and putting additional controls in place to further strengthen our safety filters and mitigate misuse of the system.”
Shane Jones, the AI engineering lead at Microsoft who initially raised concerns about the AI, has spent months testing Copilot Designer, the AI image generator that Microsoft debuted in March 2023, powered by OpenAI’s technology. Like with OpenAI’s DALL-E, users enter text prompts to create pictures. Creativity is encouraged to run wild. But since Jones began actively testing the product for vulnerabilities in December, a practice known as red-teaming, he saw the tool generate images that ran far afoul of Microsoft’s oft-cited responsible AI principles.
The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months, were recreated by CNBC this week using the Copilot tool, originally called Bing Image Creator.
Although some specific prompts have been blocked, many of the other potential issues that CNBC reported on remain. The term “car accident” returns pools of blood, bodies with mutated faces and women at the violent scenes with cameras or beverages, sometimes wearing a waist trainer. “Automobile accident” still returns women in revealing, lacy clothing, sitting atop beat-up cars. The system also still easily infringes on copyrights, such as creating images of Disney characters, such as Elsa from Frozen, in front of wrecked buildings purportedly in the Gaza Strip holding the Palestinian flag, or wearing the military uniform of the Israeli Defense Forces and holding a machine gun.
Jones was so alarmed by his experience that he started internally reporting his findings in December. While the company acknowledged his concerns, it was unwilling to take the product off the market. Jones said Microsoft referred him to OpenAI and, when he didn’t hear back from the company, he posted an open letter on LinkedIn asking the startup’s board to take down DALL-E 3 (the latest version of the AI model) for an investigation.
Microsoft’s legal department told Jones to remove his post immediately, he said, and he complied. In January, he wrote a letter to U.S. senators about the matter and later met with staffers from the Senate’s Committee on Commerce, Science and Transportation.
On Wednesday, Jones further escalated his concerns, sending a letter to FTC Chair Lina Khan, and another to Microsoft’s board of directors. He shared the letters with CNBC ahead of time.
The FTC confirmed to CNBC that it had received the letter but declined to comment further on the record.
Salesforce CEO Marc Benioff speaks at the Dreamforce conference in San Francisco on Sept. 17, 2024.
David Paul Morris | Bloomberg | Getty Images
Salesforce is adding voice to its Agentforce software, letting clients go beyond text when using artificial intelligence agents to respond to customer questions.
With Agentforce Voice, companies can customize the tone and speed of voices and adjust the pronunciation of specific terms, Salesforce said Monday, ahead of its Dreamforce conference in San Francisco this week. The feature also allows people to interrupt the AI agent during phone calls.
Voice is becoming a bigger part of the generative AI boom, which started with text-based prompts in late 2022, when OpenAI launched ChatGPT. In the past year, OpenAI and Anthropic have enabled their chatbots to conduct spoken conversations without sounding overly robotic. Now that capability is taking hold inside business software.
Agentforce Voice will integrate with corporate phone systems from Amazon, Five9, Genesys, Nice and Ericsson’s Vonage.
Former Salesforce co-CEO Bret Taylor is also trying his hand in the market. Taylor helped start Sierra in 2023, and last year the startup announced that its AI agents “can now pick up the phone.” Sierra has been valued at $10 billion, and has a client list that includes ADT, SiriusXM and SoFi.
Salesforce has been under pressure this year in part due to investor concern that software companies could lose business as AI moves deeper into coding. The stock is down about 28% so far in 2025, while the Nasdaq has gained around 15% over that stretch.
Anthropic told reporters in September that its Claude Sonnet 4.5 model built a chat app similar to Salesforce’s Slack in 30 hours. In Salesforce’s latest earnings report, the company warned that new AI products “may disrupt workforce needs and negatively impact demand for our offerings.”
Salesforce CEO Marc Benioff has downplayed the risk to this company.
“When we get into this kind of zero-sum game, well, all this is going to get wiped out, or all this is going to change, then, you know, you’re not dealing with somebody who actually runs a company, because that’s not the way business works,” Benioff told CNBC’s Morgan Brennan last month. “Business is incremental, it’s evolutionary, it’s growing, it’s evolving, and we don’t see that kind of change.”
Salesforce launched Agentforce last year as a service that could respond to customer requests over text chats with help from generative AI models. Agentforce now has more than 12,000 implementations, according to a statement. But there’s some skepticism about its popularity.
“Investor enthusiasm around Agentforce has moderated as adoption has lagged expectations,” RBC Capital Markets analysts, who recommend holding the stock, wrote in a note to clients last week.
In November, Salesforce will provide early access to Agent Script software, which organizations can use to customize what agents say and do.
WATCH: Salesforce CEO Marc Benioff on what the market is getting wrong about AI
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports2 years agoStory injured on diving stop, exits Red Sox game
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports3 years agoButton battles heat exhaustion in NASCAR debut
-
Sports3 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment1 year agoHere are the best electric bikes you can buy at every price level in October 2024