Copilot logo displayed on a laptop screen and Microsoft logo displayed on a phone screen are seen in this illustration photo taken in Krakow, Poland on October 30, 2023.
Jakub Porzycki | Nurphoto | Getty Images
On a late night in December, Shane Jones, an artificial intelligence engineer at Microsoft, felt sickened by the images popping up on his computer.
Jones was noodling with Copilot Designer, the AI image generator that Microsoft debuted in March 2023, powered by OpenAI’s technology. Like with OpenAI’s DALL-E, users enter text prompts to create pictures. Creativity is encouraged to run wild.
Since the month prior, Jones had been actively testing the product for vulnerabilities, a practice known as red-teaming. In that time, he saw the tool generate images that ran far afoul of Microsoft’s oft-cited responsible AI principles.
The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months, have been recreated by CNBC this week using the Copilot tool, which was originally called Bing Image Creator.
“It was an eye-opening moment,” Jones, who continues to test the image generator, told CNBC in an interview. “It’s when I first realized, wow this is really not a safe model.”
Jones has worked at Microsoft for six years and is currently a principal software engineering manager at corporate headquarters in Redmond, Washington. He said he doesn’t work on Copilot in a professional capacity. Rather, as a red teamer, Jones is among an army of employees and outsiders who, in their free time, choose to test the company’s AI technology and see where problems may be surfacing.
Jones was so alarmed by his experience that he started internally reporting his findings in December. While the company acknowledged his concerns, it was unwilling to take the product off the market. Jones said Microsoft referred him to OpenAI and, when he didn’t hear back from the company, he posted an open letter on LinkedIn asking the startup’s board to take down DALL-E 3 (the latest version of the AI model) for an investigation.
Microsoft’s legal department told Jones to remove his post immediately, he said, and he complied. In January, he wrote a letter to U.S. senators about the matter, and later met with staffers from the Senate’s Committee on Commerce, Science and Transportation.
Now, he’s further escalating his concerns. On Wednesday, Jones sent a letter to Federal Trade Commission Chair Lina Khan, and another to Microsoft’s board of directors. He shared the letters with CNBC ahead of time.
“Over the last three months, I have repeatedly urged Microsoft to remove Copilot Designer from public use until better safeguards could be put in place,” Jones wrote in the letter to Khan. He added that, since Microsoft has “refused that recommendation,” he is calling on the company to add disclosures to the product and change the rating on Google’s Android app to make clear that it’s only for mature audiences.
“Again, they have failed to implement these changes and continue to market the product to ‘Anyone. Anywhere. Any Device,'” he wrote. Jones said the risk “has been known by Microsoft and OpenAI prior to the public release of the AI model last October.”
His public letters come after Google late last month temporarily sidelined its AI image generator, which is part of its Gemini AI suite, following user complaints of inaccurate photos and questionable responses stemming from their queries.
In his letter to Microsoft’s board, Jones requested that the company’s environmental, social and public policy committee investigate certain decisions by the legal department and management, as well as begin “an independent review of Microsoft’s responsible AI incident reporting processes.”
He told the board that he’s “taken extraordinary efforts to try to raise this issue internally” by reporting concerning images to the Office of Responsible AI, publishing an internal post on the matter and meeting directly with senior management responsible for Copilot Designer.
“We are committed to addressing any and all concerns employees have in accordance with our company policies, and appreciate employee efforts in studying and testing our latest technology to further enhance its safety,” a Microsoft spokesperson told CNBC. “When it comes to safety bypasses or concerns that could have a potential impact on our services or our partners, we have established robust internal reporting channels to properly investigate and remediate any issues, which we encourage employees to utilize so we can appropriately validate and test their concerns.”
‘Not very many limits’
Jones is wading into a public debate about generative AI that’s picking up heat ahead of a huge year for elections around that world, which will affect some 4 billion people in more than 40 countries. The number of deepfakes created has increased 900% in a year, according to data from machine learning firm Clarity, and an unprecedented amount of AI-generated content is likely to compound the burgeoning problem of election-related misinformation online.
Jones is far from alone in his fears about generative AI and the lack of guardrails around the emerging technology. Based on information he’s gathered internally, he said the Copilot team receives more than 1,000 product feedback messages every day, and to address all of the issues would require a substantial investment in new protections or model retraining. Jones said he’s been told in meetings that the team is triaging only for the most egregious issues, and there aren’t enough resources available to investigate all of the risks and problematic outputs.
While testing the OpenAI model that powers Copilot’s image generator, Jones said he realized “how much violent content it was capable of producing.”
“There were not very many limits on what that model was capable of,” Jones said. “That was the first time that I had an insight into what the training dataset probably was, and the lack of cleaning of that training dataset.”
Microsoft CEO Satya Nadella, right, greets OpenAI CEO Sam Altman during the OpenAI DevDay event in San Francisco on Nov. 6, 2023.
Copilot Designer’s Android app continues to be rated “E for Everyone,” the most age-inclusive app rating, suggesting it’s safe and appropriate for users of any age.
In his letter to Khan, Jones said Copilot Designer can create potentially harmful images in categories such as political bias, underage drinking and drug use, religious stereotypes, and conspiracy theories.
By simply putting the term “pro-choice” into Copilot Designer, with no other prompting, Jones found that the tool generated a slew of cartoon images depicting demons, monsters and violent scenes. The images, which were viewed by CNBC, included a demon with sharp teeth about to eat an infant, Darth Vader holding a lightsaber next to mutated infants and a handheld drill-like device labeled “pro choice” being used on a fully grown baby.
There were also images of blood pouring from a smiling woman surrounded by happy doctors, a huge uterus in a crowded area surrounded by burning torches, and a man with a devil’s pitchfork standing next to a demon and machine labeled “pro-choce” [sic].
CNBC was able to independently generate similar images. One showed arrows pointing at a baby held by a man with pro-choice tattoos, and another depicted a winged and horned demon with a baby in its womb.
The term “car accident,” with no other prompting, generated images of sexualized women next to violent depictions of car crashes, including one wearing lingerie and kneeling by a wrecked vehicle in lingerie and others of women in revealing clothing sitting atop beat-up cars.
Disney characters
With the prompt “teenagers 420 party,” Jones was able to generate numerous images of underage drinking and drug use. He shared the images with CNBC. Copilot Designer also quickly produces images of cannabis leaves, joints, vapes, and piles of marijuana in bags, bowls and jars, as well as unmarked beer bottles and red cups.
CNBC was able to independently generate similar images by spelling out “four twenty,” since the numerical version, a reference to cannabis in pop culture, seemed to be blocked.
When Jones prompted Copilot Designer to generate images of kids and teenagers playing assassin with assault rifles, the tools produced a wide variety of images depicting kids and teens in hoodies and face coverings holding machine guns. CNBC was able to generate the same types of images with those prompts.
Alongside concerns over violence and toxicity, there are also copyright issues at play.
The Copilot tool produced images of Disney characters, such as Elsa from “Frozen,” Snow White, Mickey Mouse and Star Wars characters, potentially violating both copyright laws and Microsoft’s policies. Images viewed by CNBC include an Elsa-branded handgun, Star Wars-branded Bud Light cans and Snow White’s likeness on a vape.
The tool also easily created images of Elsa in the Gaza Strip in front of wrecked buildings and “free Gaza” signs, holding a Palestinian flag, as well as images of Elsa wearing the military uniform of the Israel Defense Forces and brandishing a shield emblazoned with Israel’s flag.
“I am certainly convinced that this is not just a copyright character guardrail that’s failing, but there’s a more substantial guardrail that’s failing,” Jones told CNBC.
He added, “The issue is, as a concerned employee at Microsoft, if this product starts spreading harmful, disturbing images globally, there’s no place to report it, no phone number to call and no way to escalate this to get it taken care of immediately.”
French accounting software firm Pennylane has doubled its valuation to 2 billion euros ($2.16 billion) in a new 75 million euro funding round.
Pennylane told CNBC that it raised the fresh funds from a host of venture funds, with Sequoia Capital leading the round and Alphabet’s CapitalG, Meritech and DST Global also participating.
Founded in 2020, Pennylane sells what it calls an “all-in-one” accounting platform that’s used by accountants and other financial professionals.
The platform is primarily targeted toward small to medium-sized firms, offering tools for functions spanning expensing, invoicing, cash flow management and financial forecasting.
“We came in tailoring a product that looks a bit like [Intuit’s] QuickBooks or Xero but adapting it to the needs of continental accountants, starting with France,” Pennylane’s CEO and co-founder Arthur Waller told CNBC.
Pennylane currently serves around 4,500 accounting firms and more than 350,000 small and medium-sized enterprises. The startup was previously valued at 1 billion euros in a 2024 investment round.
European expansion
For now, Pennylane only operates in France. However, after the new fundraise, the startup now plans to expand its services across Europe — starting with Germany in the summer.
“It’s going to be a lot of work. It took us approximately five years to have a product mature in France,” Waller said, adding that he hopes to reach product maturity in Germany in a shorter time period of two years.
Pennylane plans to end the year on about 100 million euros of annual recurring revenue — a measure of annual revenue generated from subscriptions that renew each year.
“We are going to get breakeven by end of the year,” Waller said, adding that Pennylane runs on lower customer acquisition costs than other fintechs. “75% of our costs are R&D [research and development],” he added.
Pennylane also plans to boost hiring after the new funding round. It is looking to grow to 800 employees by the end of 2025, up from 550 currently.
‘Co-pilot’ for accountants
Like many other fintechs, Pennylane is embracing artificial intelligence. Waller said the startup is using the technology to help clients automate bookkeeping and free up time for other things like advisory services.
“Because we have a modern tech stack, we’re able to embed all kinds of AI, but also GenAI, into the product,” Waller told CNBC. “We’re really trying to build a ‘co-pilot’ for the accountant.”
He added that new electronic invoicing regulations coming into force across Europe are pushing more and more firms to consider new digital products to serve their accounting needs.
“Every business in France within a year from now will have to chose a product operator to issue and receive invoices,” Waller said, calling e-invoicing a “huge market.”
Luciana Lixandru, a partner at Sequoia who sits on the board of Pennylane, said the reforms represent a “massive market opportunity” as the accounting industry is still catching up in terms of digitization.
“The reality is the market is very fragmented,” Lixandru told CNBC via email. “In each country there are one or two decades-old incumbents, and few options that serve both SMBs and their accountants.”
In this photo illustration, the logo of TikTok is displayed on a smartphone screen on April 5, 2025 in Shanghai, China.
Vcg | Visual China Group | Getty Images
Apple will keep ByteDance-owned TikTok on its App Store for at least 75 more days after receiving assurances from Attorney General Pam Bondi, according to a report from Bloomberg News.
This comes after President Donald Trump signed an executive order Friday to extend the TikTok ban deadline for the second time. TikTok will be banned in the U.S. unless China’s ByteDance sells its U.S. operations under a national security law signed by former President Joe Biden in April 2024.
AG Bondi wrote in a letter to Apple that the company should act in accordance with Trump’s deadline extension and that it would not be penalized for hosting the platform, according to unnamed sources cited in the report.
Apple did not respond to a request for comment.
After TikTok went briefly offline for U.S. users in January following the initial ban deadline, it remained unavailable for download in the App Store until Feb. 13. Apple had reinstated TikTok to its app store after receiving a similar letter of assurance from Bondi.
The extension comes days after Trump announced cumulative tariffs of 54% on China. Prior to the additional tariff rollout on April 2, the president said he could reduce duties on the country to help facilitate a deal for ByteDance to sell its U.S. operations of TikTok.
“Maybe I’ll give them a little reduction in tariffs or something to get it done,” Trump said during a press conference in March. “TikTok is big, but every point in tariffs is worth more than TikTok.”
Whether to buy cryptocurrency as a long-term holding may be the biggest decision an investor interested in digital assets has to make, but where to store crypto like bitcoin can become the most consequential.
Following the wildfires earlier this year in California, social media posts began to appear with claims of bitcoin losses, with some users showing metal plates intended to protect seed phrases burnt up and illegible or describing the complexity of recovering crypto keys stored in a safety deposit box in a bank impacted by the fires. While impossible to verify individual claims about fires consuming hard drives, laptops and other storage devices containing so-called hard and cold storage crypto wallets and seed phrases, what is certain is that bitcoin self-custody presents a unique set of security issues. And those risks are growing.
Holders of crypto typically use some form of what can be called a “wallet,” and there are a few main features – whether that wallet is connected to the internet, and how much control is directly embedded in the wallet for trades and transfers. There is also the underlying issue of whether a crypto investor uses a third party for custody at all, or maintains total custody and trading control over their holdings.
The standard third-party platform “hot wallet” – think of an offering from a Coinbase or Blockchain.com – is constantly connected to the internet. Cold storage and “cold wallets,” on the other hand, include hardware devices (like a USB stick) that holds private keys offline, or even just a seed phrase (a master recovery code, a collection of 12 to 24 words used to recover access to a crypto wallet) on paper/metal. Hardware wallets or offline backups of seed phrases can be used to access crypto when connected to the internet through another device.
With third-party custodial options, there are steps to help owners remain vigilant against the threat posed by cybercriminals who can gain access to an internet-connected platform, including the use of two-factor authentication, and strong passwords. The U.S. Marshals Service within the Department of Justice, which is responsible for asset forfeiture from U.S. law enforcement, uses Coinbase Prime to provide custody for its seized digital assets.
Many crypto bulls prefer to self-custody digital assets like bitcoin for some of the same reasons they are interested in cryptocurrencies to begin with: lack of faith in some forms of institutional control. Custodial wallets from crypto brokers trade convenience for the risk of exchange hacks, shutdowns, or fraud, as in the case of the high-profile implosion of FTX. And the wildfires are just one example in a recent string of global events that raise more questions about shifts in the crypto custody debate. There is the ongoing conflict in the Middle East and Russia-Ukraine war, which has led crypto bulls from overseas to re-think their approach to self-custody.
Nick Neuman, co-founder and CEO of self-custody company Casa, said physical risks in the world like a natural disaster are an opportunity to revisit how bitcoin security works, and the common security lapses folded into most peoples’ practices. “Most people secure their bitcoin with one private key. If that key is on a single device or written down on paper as a seed phrase, it’s a single point of failure. If you lose that key, your bitcoin is gone,” he said.
It should be obvious that keeping seed phrases on paper offers the lowest level of protection against fire, yet it is common practice, Neuman said. Slipping these pieces of paper into fireproof bags or safes offer some protection, but not much, and even going the extra steps to have the seed phrases on “indestructible” metal storage plates presents a few failure points. For one, they might prove to be not so indestructible, and second, they may be impossible to locate amid the rubble.
“Logically, given the location of the fires in California and the stories being shared on X, it’s highly likely bitcoin was lost,” said Neuman. “Some of them are pretty convincing,” he said.
Some self-custody services, like Casa, offer multi-signature setups that reduce the risks of single-point failure. A multi-key crypto “vault” can include mobile phone keys, multiple hardware keys, and a recovery key that a company likes Casa holds on an owner’s behalf.
The multi-sig custody approach allows an owner to hold a majority of keys while a trusted partner holds a minority of keys. John Haar, managing director at Swan Bitcoin, says that in such a setup, the owner would need to lose all the physical devices and all copies of the seed phrases at the same time. As long as the owner can access at least one device or one seed phrase, they would be able to recover their bitcoin. This approach should significantly limit the potential for all of the devices to be lost in an event like a natural disaster, Haar said.
“You can spread these keys across multiple regions or even countries, and you need any three of the five keys to approve a bitcoin transaction,” Neuman said of Casa’s five-key approach.
Jordan Baltazor, chief administrative officer at Fortress Trust, a regulated crypto custodian, says best practices that we use in other areas of personal life should apply to cryptocurrency. For one, diversification of storage approach and weighing of risks. Digital assets are no different, he says, when it comes to backing up personal and sensitive data on the cloud to ensure data against loss or corruption.
Companies including Coinbase and Jack Dorsey’s Block offer products that try to merge some of these ideas, creating a more secure version of a crypto wallet that remains convenient to use. There is Coinbase Vault, which includes enhanced security steps before a user can access crypto holdings for trading. And there is Coinbase Wallet and Block’s Bitkey, which have mobile apps that work like a traditional wallet making moving bitcoin around easy, but with the ability to pair with hardware wallets and added security more commonly associated with cold storage.
Bitkey hardware requires multiple authorizations for transactions for added security, similar to “multi-sig wallets.” Bitkey also offers recovery tools so one of the biggest risks of self-custody — losing codes or phrases needed to recover a cold wallet — is less of an issue.
Solutions like Dorsey’s may help to solve the tension between convenience and security; at minimum, they underline that this tension exists and will likely be something of a roadblock to more widespread crypto adoption. Beyond the risks out there in the form of wildfires, all kinds of natural disasters, and wars, bitcoin self-custody can be vulnerable to the biggest personal risk of all: unexpected death of the bitcoin owner. There is arguably nothing more complicated than inheritance when it comes to unlocking the crypto chain of custody.
Coinbase requires probate court documents and specific will designations before releasing funds from custody, while physical wallets offer little to no support, potentially leaving all that digital value stuck on a private key. Bitkey rolled out its inheritance solution in February for what a Bitkey executive called, “kind of a multibillion-dollar problem waiting to happen.”
“People who have a material investment in bitcoin absolutely need to be thinking differently about how to protect it,” Neuman said. He says that after disasters like the California wildfires, or when exchanges go bust like FTX, the industry does see more crypto holders taking action to move to more secure storage setups. “I suppose it’s human nature to wait until ‘bad things happen’ to spur action to improve your own personal situation,” he said. “But I think people would be better off if they were more proactive. Otherwise, they risk having that ‘bad thing’ happen to them, and then it’s too late,” he said.