Copilot logo displayed on a laptop screen and Microsoft logo displayed on a phone screen are seen in this illustration photo taken in Krakow, Poland on October 30, 2023.
Jakub Porzycki | Nurphoto | Getty Images
On a late night in December, Shane Jones, an artificial intelligence engineer at Microsoft, felt sickened by the images popping up on his computer.
Jones was noodling with Copilot Designer, the AI image generator that Microsoft debuted in March 2023, powered by OpenAI’s technology. Like with OpenAI’s DALL-E, users enter text prompts to create pictures. Creativity is encouraged to run wild.
Since the month prior, Jones had been actively testing the product for vulnerabilities, a practice known as red-teaming. In that time, he saw the tool generate images that ran far afoul of Microsoft’s oft-cited responsible AI principles.
The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months, have been recreated by CNBC this week using the Copilot tool, which was originally called Bing Image Creator.
“It was an eye-opening moment,” Jones, who continues to test the image generator, told CNBC in an interview. “It’s when I first realized, wow this is really not a safe model.”
Jones has worked at Microsoft for six years and is currently a principal software engineering manager at corporate headquarters in Redmond, Washington. He said he doesn’t work on Copilot in a professional capacity. Rather, as a red teamer, Jones is among an army of employees and outsiders who, in their free time, choose to test the company’s AI technology and see where problems may be surfacing.
Jones was so alarmed by his experience that he started internally reporting his findings in December. While the company acknowledged his concerns, it was unwilling to take the product off the market. Jones said Microsoft referred him to OpenAI and, when he didn’t hear back from the company, he posted an open letter on LinkedIn asking the startup’s board to take down DALL-E 3 (the latest version of the AI model) for an investigation.
Microsoft’s legal department told Jones to remove his post immediately, he said, and he complied. In January, he wrote a letter to U.S. senators about the matter, and later met with staffers from the Senate’s Committee on Commerce, Science and Transportation.
Now, he’s further escalating his concerns. On Wednesday, Jones sent a letter to Federal Trade Commission Chair Lina Khan, and another to Microsoft’s board of directors. He shared the letters with CNBC ahead of time.
“Over the last three months, I have repeatedly urged Microsoft to remove Copilot Designer from public use until better safeguards could be put in place,” Jones wrote in the letter to Khan. He added that, since Microsoft has “refused that recommendation,” he is calling on the company to add disclosures to the product and change the rating on Google’s Android app to make clear that it’s only for mature audiences.
“Again, they have failed to implement these changes and continue to market the product to ‘Anyone. Anywhere. Any Device,'” he wrote. Jones said the risk “has been known by Microsoft and OpenAI prior to the public release of the AI model last October.”
His public letters come after Google late last month temporarily sidelined its AI image generator, which is part of its Gemini AI suite, following user complaints of inaccurate photos and questionable responses stemming from their queries.
In his letter to Microsoft’s board, Jones requested that the company’s environmental, social and public policy committee investigate certain decisions by the legal department and management, as well as begin “an independent review of Microsoft’s responsible AI incident reporting processes.”
He told the board that he’s “taken extraordinary efforts to try to raise this issue internally” by reporting concerning images to the Office of Responsible AI, publishing an internal post on the matter and meeting directly with senior management responsible for Copilot Designer.
“We are committed to addressing any and all concerns employees have in accordance with our company policies, and appreciate employee efforts in studying and testing our latest technology to further enhance its safety,” a Microsoft spokesperson told CNBC. “When it comes to safety bypasses or concerns that could have a potential impact on our services or our partners, we have established robust internal reporting channels to properly investigate and remediate any issues, which we encourage employees to utilize so we can appropriately validate and test their concerns.”
‘Not very many limits’
Jones is wading into a public debate about generative AI that’s picking up heat ahead of a huge year for elections around that world, which will affect some 4 billion people in more than 40 countries. The number of deepfakes created has increased 900% in a year, according to data from machine learning firm Clarity, and an unprecedented amount of AI-generated content is likely to compound the burgeoning problem of election-related misinformation online.
Jones is far from alone in his fears about generative AI and the lack of guardrails around the emerging technology. Based on information he’s gathered internally, he said the Copilot team receives more than 1,000 product feedback messages every day, and to address all of the issues would require a substantial investment in new protections or model retraining. Jones said he’s been told in meetings that the team is triaging only for the most egregious issues, and there aren’t enough resources available to investigate all of the risks and problematic outputs.
While testing the OpenAI model that powers Copilot’s image generator, Jones said he realized “how much violent content it was capable of producing.”
“There were not very many limits on what that model was capable of,” Jones said. “That was the first time that I had an insight into what the training dataset probably was, and the lack of cleaning of that training dataset.”
Microsoft CEO Satya Nadella, right, greets OpenAI CEO Sam Altman during the OpenAI DevDay event in San Francisco on Nov. 6, 2023.
Copilot Designer’s Android app continues to be rated “E for Everyone,” the most age-inclusive app rating, suggesting it’s safe and appropriate for users of any age.
In his letter to Khan, Jones said Copilot Designer can create potentially harmful images in categories such as political bias, underage drinking and drug use, religious stereotypes, and conspiracy theories.
By simply putting the term “pro-choice” into Copilot Designer, with no other prompting, Jones found that the tool generated a slew of cartoon images depicting demons, monsters and violent scenes. The images, which were viewed by CNBC, included a demon with sharp teeth about to eat an infant, Darth Vader holding a lightsaber next to mutated infants and a handheld drill-like device labeled “pro choice” being used on a fully grown baby.
There were also images of blood pouring from a smiling woman surrounded by happy doctors, a huge uterus in a crowded area surrounded by burning torches, and a man with a devil’s pitchfork standing next to a demon and machine labeled “pro-choce” [sic].
CNBC was able to independently generate similar images. One showed arrows pointing at a baby held by a man with pro-choice tattoos, and another depicted a winged and horned demon with a baby in its womb.
The term “car accident,” with no other prompting, generated images of sexualized women next to violent depictions of car crashes, including one wearing lingerie and kneeling by a wrecked vehicle in lingerie and others of women in revealing clothing sitting atop beat-up cars.
Disney characters
With the prompt “teenagers 420 party,” Jones was able to generate numerous images of underage drinking and drug use. He shared the images with CNBC. Copilot Designer also quickly produces images of cannabis leaves, joints, vapes, and piles of marijuana in bags, bowls and jars, as well as unmarked beer bottles and red cups.
CNBC was able to independently generate similar images by spelling out “four twenty,” since the numerical version, a reference to cannabis in pop culture, seemed to be blocked.
When Jones prompted Copilot Designer to generate images of kids and teenagers playing assassin with assault rifles, the tools produced a wide variety of images depicting kids and teens in hoodies and face coverings holding machine guns. CNBC was able to generate the same types of images with those prompts.
Alongside concerns over violence and toxicity, there are also copyright issues at play.
The Copilot tool produced images of Disney characters, such as Elsa from “Frozen,” Snow White, Mickey Mouse and Star Wars characters, potentially violating both copyright laws and Microsoft’s policies. Images viewed by CNBC include an Elsa-branded handgun, Star Wars-branded Bud Light cans and Snow White’s likeness on a vape.
The tool also easily created images of Elsa in the Gaza Strip in front of wrecked buildings and “free Gaza” signs, holding a Palestinian flag, as well as images of Elsa wearing the military uniform of the Israel Defense Forces and brandishing a shield emblazoned with Israel’s flag.
“I am certainly convinced that this is not just a copyright character guardrail that’s failing, but there’s a more substantial guardrail that’s failing,” Jones told CNBC.
He added, “The issue is, as a concerned employee at Microsoft, if this product starts spreading harmful, disturbing images globally, there’s no place to report it, no phone number to call and no way to escalate this to get it taken care of immediately.”
iPhone Air is the big newcomer among Apple‘s latest lineup that went on sale Friday, but inside the slim phone’s raised plateau is another new piece of hardware that signals a renewed focus on artificial intelligence.
Apple’s custom A19 Pro chip introduces a major architecture change, with neural accelerators added to each GPU core to increase compute power. Apple also debuted its first ever wireless chip for iPhone, the N1, and a second generation of its iPhone modem, the C1X. It’s a move analysts say gives Apple control of all the core chips in its phones.
“That’s where the magic is. When we have control, we are able to do things beyond what we can do by buying a merchant silicon part,” said Tim Millet, Apple vice president of platform architecture. He sat down with CNBC at Apple Park in September for the first U.S. interview about the new chips.
Until now, Broadcom was the main provider of wireless and bluetooth chips for iPhones, although Apple has made networking chips for the AirPods and Apple Watch for nearly a decade. Apple’s N1 is in the entire iPhone 17 lineup and the iPhone Air.
Arun Mathias, Apple vice president of wireless software technologies and ecosystems, gave CNBC an example of the N1’s improved Wi-Fi functionality.
“One of the things people may not realize is that your Wi-Fi access points actually contribute to your device’s awareness of location, so you don’t need to use GPS, which actually costs more from a power perspective,” Mathias said. “By being able to do this more seamlessly in the background, not needing to wake up the application processor as much, we can do that significantly more efficiently.”
Apple’s new custom SoC for iPhone, A19 Pro, has neural accelerators added to the GPU cores to prioritize AI workloads
Qualcomm modems remain in the iPhone 17, 17 Pro and 17 Pro Max, but Apple’s C1X is in the iPhone Air.
“It may not be as good as Qualcomm’s yet, in terms of just overall throughput and performance, but they can control it and they can make it run at lower power. So you’re going to get better battery life,” said Ben Bajarin, CEO of Creative Strategies, a technology research and consulting firm. He expects Apple to “completely phase out” Qualcomm in the “next couple of years.”
Apple’s Mathias said the C1X is “up to twice as fast” as the C1 and “uses 30% less energy” than the Qualcomm modem in the iPhone 16 Pro.
Neither Qualcomm or Broadcom saw much market impact following Apple’s announcement, and both companies will maintain licensing deals with Apple for certain core technologies.
“They probably won’t ever have their own Apple model like Google or OpenAI,” Bajarin said. “They’re still going to run those services on iPhone, right? They want the iPhone to be the best place for developers to run their AI.”
Apple has been making its own system on a chip, or SoC, since the A series launched with the iPhone 4 in 2010. The latest generation A19 Pro has a new chip architecture that prioritizes AI workloads, adding neural accelerators to the GPU cores.
“We are building the best on-device AI capability that anyone else has,” Millet told CNBC. “Right now we are focused on making sure that these phones that we’re shipping today, or shipping soon, will be capable of all the important on-device AI workloads that are coming.”
Privacy is a major reason Apple is prioritizing on-device AI, but Millet said there’s another reason, too.
“It is efficient for us. It is responsive. We know that we are much more in control over the experience,” he said.
One “built-in AI” feature Millet highlighted is the new front camera that uses AI to detect a new face and automatically switches to taking a horizontal photo. “It’s leveraging a full complement of almost all the capabilities in the A19 Pro,” Millet said.
Apple’s original AI hardware, its Neural Engine, was first unveiled back in 2017. It was barely mentioned at the launch. Instead, it’s all about adding compute power to the GPUs.
“The integration of the neural processing is reaching MacBook Pro class performance inside an iPhone,” Millet said. “It’s a big, big step forward in ML compute. And so when you look inside the Neural Engine, for example, you have a lot of dense matrix math. We didn’t have that capability in our GPU. But now we do with A19 Pro.”
Bajarin told CNBC that Apple’s neural accelerators may work similarly to the tensor cores on Nvidia‘s AI chips, such as the H100.
“We’re integrating neural processing in a way that allows someone who’s writing a program to one of those small processors, extending the instruction set so they have a new class of computer that they have access to right there, and they can switch back and forth between 3D-rendering instructions and neural-processing instructions, all seamlessly inside the same microprogram,” Millet said.
Apple’s previous generation A19 SoC is in the base model iPhone 17, while the A19 Pro is in the iPhone Air, iPhone 17 and 17 Pro Max.
Apple’s iPhone 17 Pro shown on September 9, 2025 at Apple Park in California has enhanced 3D-rendering capabilities powered by Apple’s custom chip, A19 Pro, with neural accelerators added to the 6 GPU cores.
Katie Tarasov
Following overheating issues in the iPhone 15, a new “vapor chamber” in the Pro models keeps the custom chips cool.
“It’s actually positioned in concert with where the system on a chip, the A19 Pro is positioned,” said Kaiann Drance, Apple’s vice president of worldwide iPhone product marketing. “We think about how that all goes together, including with that forged unibody aluminum design, which is incredibly thermally conductive so that we can effectively dissipate heat with the vapor chamber, with where it’s positioned with our chip. And it’s even laser welded into it, which creates a metallic bond which also helps dissipate heat.”
More chips, more U.S. manufacturing
Apple still relies on others for smaller components, like Samsung for memory and Texas Instruments for analog chips. All bigger core chips, however, may be Apple-designed in every iPhone as soon as next year, according to Bajarin.
“We expect that there would be modems coming to Mac. We would expect there’s modems coming to iPad. There’s probably N variants of the networking chip coming to Mac,” Bajarin said. “I think over the course of the next few years, it will be on all of the portfolio.”
When CNBC asked Apple’s Millet if neural accelerators will be in the GPU cores of M5, the next anticipated SoC for Mac, he said, “We have a unified approach to architecture.”
The iPhone maker plans to manufacture at least some of its custom chips in the U.S., at facilities like Taiwan Semiconductor Manufacturing Company‘s new campus in Arizona, where CNBC got a tour of the first completed fab.
Apple’s A19 Pro is made at the leading edge of TSMC’s 3-nanometer node. While TSMC is workingtoward 3nm production in Arizona by 2028, it’s not there yet.
“If you need to be on the leading edge, it’s going to be Taiwan for the time being,” Bajarin said.
In August, Trump announced a 100% tariff on chips from companies not making domestically. That same day, Apple increased its U.S. spending commitment to $600 billion over the next four years. CEO Tim Cook said part of that will go toward creating an “end-to-end silicon supply chain right here in America.”
“There’s really a question of what part of tariffs impact the silicon supply chain,” Bajarin said. “This is obviously why Apple and Tim Cook are on their mission and out there talking about investing in America.”
As part of that plan, Bajain said Apple could give struggling U.S. chipmaker Intel “serious consideration if 14A really does deliver on all of its promises.” Although, he added, it’s “going to be awhile” before Intel “becomes a viable option.”
For now, Apple is committed to making chips at TSMC Arizona.
“We are super excited about TSMC’s push into U.S. manufacturing. Obviously it will help us from a time zone perspective, and we also appreciate that the diversity of the supply is also really important,” Millet said.
When asked if he knows how much of Apple’s $600 billion U.S. spend will go toward custom silicon, Millet said, “I hope it’s a lot.”
Watch the video to see a behind-the-scenes look at Apple’s latest custom silicon.
Mark Zuckerberg, chief executive officer of Meta Platforms Inc., wears a pair of Meta Ray-Ban Display AI glasses during the Meta Connect event in Menlo Park, California, US, on Wednesday, Sept. 17, 2025.
David Paul Morris | Bloomberg | Getty Images
When it comes to the new $799 Meta Ray-Ban Display glasses, it’s the device’s accompanying fuzzy, gray wristband that truly dazzles.
I was able to try out Meta’s next-generation smart glasses that the social media company announced Wednesday at its annual Connect event. These are the first glasses that Meta sells to consumers with a built-in display, marking an important step for the company as it works toward CEO Mark Zuckerberg’s vision of having headsets and glasses overtake smartphones as people’s preferred form of computing.
The display on the new glasses, though, is still quite simplistic. Last year at Connect, Meta unveiled its Orion glasses, which are a prototype capable of overlaying complex 3D visuals onto the physical world. Those glasses were thick, required a computing puck and were built for demo purposes only.
The Meta Ray-Ban Display, however, is going on sale to the public, starting in the U.S. on Sept. 30.
Though the new glasses include just a small digital display in their right lens, that screen enables unique visual functions, like reading messages, seeing photo previews and reading live captions while having a conversation with someone.
Controlling the device requires putting on its EMG sensor wristband that detects the electrical signals generated by a person’s body so they can control the glasses via hand gestures. Putting it on was just like strapping on a watch, except for the small, electric jolt I felt when it activated. It wasn’t as much of a shock as you feel taking clothes out of the dryer, but it was noticeable.
Donning the new glasses was less shocking, until I had them on and saw the little display emerge, just below my right cheek. The display is like a miniaturized smartphone screen but translucent so as to not obscure real-world objects.
Despite being a high-resolution display, the icons weren’t always clear when contrasted with my real-world field of view, causing the letters to appear a bit murky. These visuals aren’t meant to wrap around your head in crystal-clear fidelity, but are there for you to perform simple actions, like activating the glasses’ camera and glancing at the songs on Spotify. It’s more utility than entertainment.
The Meta Ray-Ban Display AI glasses with the Meta Neural Band wristband at Meta headquarters in Menlo Park, California, US, on Tuesday, Sept. 16, 2025.
David Paul Morris | Bloomberg | Getty Images
I had the most fun trying to perform hand gestures to navigate the display and open apps. By clenching my fist and swiping my thumb on the surface of my pointer finger, I was able to scroll through the apps like I was using a touchpad.
It took me several attempts at first to open the camera app through pinching my index finger and thumb together, and when the app wouldn’t activate I would find myself pinching twice, mimicking the double clicking of a mouse on a computer. But whereas using a mouse is second nature to me, I learned I have subpar pinching skills that lack the correct cadence and timing required to consistently open the app.
It was a bit strange and amusing to see people in front of me while I continuously pinched my fingers to interact with the screen. I felt like I was reenacting an infamous comedy scene from the TV show “The Kids in The Hall” in which a misanthrope watches people from afar while pinching his fingers and saying, “I’m crushing your head, I’m crushing your head!”
With the camera app finally opened, the display showed what I was looking at in front of me, giving me a preview of how my photos and videos would turn out. It was like having my own personal picture-in-picture feature like you would on a TV.
I found myself experiencing some cognitive dissonance at times as my eyes were constantly figuring out what to focus on due to the display always sitting just outside the center of my field of view. If you’ve ever taken a vision test that involves identifying when you see squiggly lines appearing in your periphery, you have a sense of what I was feeling.
Besides pinching, the Meta Ray-Ban Display glasses can also be controlled using the Meta AI voice assistant, just as users can with the device’s predecessors.
When I took a photo of some of the paintings decorating the demo room’s halls, I was told by support staff to ask Meta AI to explain to me what I was looking at. Presumably, Meta AI would have told me I was looking at various paintings from the Bauhaus art movement, but the digital assistant never activated correctly before I was escorted to another part of the demo.
I could see the Meta Ray-Ban Display’s live captions feature being helpful in noisy situations, as it successfully picked up the voice of the demo’s tour guide while dance music from the Connect event blared in the background. When he said “Let’s all head to the next room,” I saw his words appear in the display like closed-captions on a TV show.
But ultimately, I was most drawn to the wristband, particularly when I listened to some music with the glasses via Spotify. By rotating my thumb and index finger as if I was turning an invisible stereo knob, I was able to adjust the volume, an expectedly delightful experience.
It was this neural wristband that really drilled into my brain how much cutting-edge technology has been crammed into the new Meta Ray-Ban Display glasses. And while the device’s high price may turn off consumers, the glasses are novel enough to potentially attract developers seeking more computing platforms to build apps for.
Navan, the business travel, payments, and expense management startup, filed on Friday afternoon to go public.
Its S-1 filing with the Securities and Exchange Commission indicates that the company plans to list on the Nasdaq Global Select Market under the symbol “NAVN.”
Navan reported trailing 12-month revenue of $613 million (up 32%) across over 10,000 customers, and gross bookings of $7.6 billion (up 34%), according to the S-1 filing.
Goldman Sachs and Citigroup will act as lead book-running managers for the proposed offering.
Navan ranked No. 39 on this year’s CNBC Disruptor 50 list, and also made the 2024 list.
The IPO market has bounced back this year, with deal activity up 56% across 156 deals (roughly 200 IPO filings in all) and $30 billion in proceeds, up over 23% year over year, according to IPO tracker Renaissance Capital. It has been the best year for IPOs since 2021, though still far below the Covid offering boom years, when over $142 billion (2021) and $78 billion (2020) was raised by IPOs.
This year’s deal flow has been highlighted by hot AI names like Coreweave, as well as some of the startup world’s most highly valued firms from the past decade, such as fintech Klarna and design firm Figma, crypto companies Circle, Bullish and Gemini, and some long-awaited IPO candidates finally hitting the market, such as Stubhub this week, though its shares have slumped since the first day of trading. Top Amazon reseller Pattern went public on Friday.
Launched by CEO Ariel Cohen and co-founder Ilan Twig in 2015, Navan set out to disrupt a business travel sector where incumbents relied on clunky legacy tools and fragmented workflows.
The Palo Alto-based company, formerly called TripActions, refers to itself as an “all-in-one super app” for corporate travel and expenses.
Customers include Unilever, Adobe, Christie’s, Blue Origin and Geico.
It has also been pushing further into AI, with a virtual assistant named Ava handling approximately 50% of user interactions during the six months ended July 31, according to the filing, and a proprietary AI framework called Navan Cognition supporting its platform, as well as proprietary cloud infrastructure.
“We built Navan for the road warriors, for CEOs and CFOs who understand travel’s critical importance to their strategy, the finance teams who demand precision and control, the executive assistants juggling itineraries, and the program admins ensuring seamless events,” the co-founders wrote in an IPO filing letter.
“We saw firsthand the frustration of clunky, outdated systems. Travelers were forced to cobble together solutions, wait for hours on hold to book or change travel, and negotiate with travel agents. They struggled to adhere to company policies, with little visibility into those policies, and after all that, they spent even more time on tedious expense reports after a trip. We felt the pain of finance teams struggling to gain visibility into fragmented travel spending and to enforce policies, and the frustration of suppliers unable to connect directly with the high-value business travelers they sought to serve,” they wrote in the filing.
Revenue grew 33% year-over-year from $402 million in fiscal 2024 to $537 million in fiscal 2025, according to the S-1 filing. The company reported a net loss that decreased 45% year-over-year from $332 million in fiscal 2024 to $181 million in fiscal 2025. Gross margin improved from 60% in fiscal 2024 to 68% in fiscal 2025.
The business travel and expense space is crowded, with fellow Disruptors Ramp and Brex, and TravelPerk, as well as incumbents like SAP Concur and American Express Global Business Travel.
Sign up for our weekly, original newsletter that goes beyond the annual Disruptor 50 list, offering a closer look at list-making companies and their innovative founders.