Ex-Meta staffers see booming business targeting online disinformation after two wars break out
People using their mobile phones outside the offices of Meta, the parent company of Facebook and Instagram, in King’s Cross, London.
Joshua Bratt | Pa Images | Getty Images
Lauren Wagner knows a lot about disinformation. Heading into the 2020 U.S. presidential election, she worked at Facebook, focusing on information integrity and overseeing products designed to make sure content was moderated and fact-checked.
She can’t believe what’s she’s seeing now. Since war erupted last month between Israel and Hamas, the constant deluge of misinformation and violent content spreading across the internet is hard for her to comprehend. Wagner left Facebook parent Meta last year, and her work in trust and safety feels like it was from a prior era.
“When you’re in a situation where there’s such a large volume of visual content, how do you even start managing that when it’s like long video clips and there’s multiple points of view?” Wagner said. “This idea of live-streaming terrorism, essentially at such a deep and in-depth scale, I don’t know how you manage that.”
The problem is even more pronounced because Meta, Google parent Alphabet, and X, formerly Twitter, have all eliminated jobs tied to content moderation and trust and safety as part of broader cost-cutting measures that began late last year and continued through 2023. Now, as people post and share out-of-context videos of previous wars, fabricated audio in news clips, and graphic videos of terrorist acts, the world’s most trafficked websites are struggling to keep up, experts have noted.
As the founder of a new venture capital firm, Radium Ventures, Wagner is in the midst of raising her first fund dedicated solely to startup founders working on trust and safety technologies. She said many more platforms that think they are “fairly innocuous” are seeing the need to act.
“Hopefully this is shining a light on the fact that if you house user-generated content, there’s an opportunity for misinformation, for charged information or potentially damaging information to spread,” Wagner said.
In addition to the traditional social networks, the highly polarized nature of the Israel-Hamas war affects internet platforms that weren’t typically known for hosting political discussions but now have to take precautionary measures. Popular online messaging and discussion channels such as Discord and Telegram could be exploited by terrorist groups and other bad actors who are increasingly using multiple communication services to create and conduct their propaganda campaigns.
A Discord spokesperson declined to comment. Telegram didn’t respond to a request for comment.
A demonstrator places flowers on white-shrouded body bags representing victims in the Israel-Hamas conflict, in front of the White House in Washington, DC, on November 15, 2023.
Mandel Ngan | AFP | Getty Images
On kids gaming site Roblox, thousands of users recently attended pro-Palestinian protests held within the virtual world. That has required the company to closely monitor for posts that violate its community standards, a Roblox spokesperson told CNBC in a statement.
Roblox has thousands of moderators and “automated detection tools in place to monitor,” the spokesperson said, adding that the site “allows for expressions of solidarity,” but does “not allow for content that endorses or condones violence, promotes terrorism or hatred against individuals or groups, or calls for supporting a specific political party.”
When it comes to looking for talent in the trust and safety space, there’s no shortage. Many of Wagner’s former colleagues at Meta lost their jobs and remain dedicated to the cause.
One of her first investments was in a startup called Cove, which was founded by former Meta trust and safety staffers. Cove is among a handful of emerging companies developing technology that they can sell to organizations, following an established enterprise software model. Other Meta veterans have recently started Cinder and Sero AI to go after the same general market.
“It adds some more coherence to the information ecosystem,” Wagner, who is also a senior advisor at the Responsible Innovation Labs nonprofit, said regarding the new crop of trust and safety tools. “They provide some level of standardized processes across companies where they can access tools and guidelines to be able to manage user-generated content effectively.”
‘Brilliant people out there’
It’s not just ex-Meta staffers who recognize the opportunity.
The founding team of startup TrustLab came from companies including Google, Reddit and TikTok parent ByteDance. And the founders of Intrinsic previously worked on trust and safety-related issues at Apple and Discord.
For the TrustCon conference in July, tech policy wonks and other industry experts headed to San Francisco to discuss the latest hot topics in online trust and safety, including their concerns about the potential societal effects of layoffs across the industry.
Several startups showcased their products in the exhibition hall, promoting their services, talking to potential clients and recruiting talent. ActiveFence, which describes itself as a “leader in providing Trust & Safety solutions to protect online platforms and their users from malicious behavior and content,” had a booth at the conference. So did Checkstep, a content moderation platform.
Cove also had an exhibit at the event.
“I think the cost-cutting has definitely obviously affected the labor markets and the hiring market,” said Cove CEO Michael Dworsky, who co-founded the company in 2021 after more than three years at Facebook. “There are a bunch of brilliant people out there that we can now hire.”
Cove has developed software to help manage a company’s content policy and review process. The management platform works alongside various content moderation systems, or classifiers, to detect issues such as harassment, so businesses can protect their users without needing expensive engineers to develop the code. The company, which counts anonymous social media apps YikYak and Sidechat as customers, says on its website that Cove is “the solution we wish we had at Meta.”
“When Facebook started really investing in trust and safety, it’s not like there were tools on the market that they could have bought,” said Cove technology chief Mason Silber, who previously spent seven years at Facebook. “They didn’t want to build, they didn’t want to become the experts. They did it more out of necessity than desire, and they built some of the most robust, trusted safety solutions in the world.”
A Meta spokesperson declined to comment for this story.
Wagner, who left Meta in mid-2022 after about two and a half years at the company, said that earlier content moderation was more manageable than it is today, particularly with the current Middle East crisis. In the past, for instance, a trust and safety team member could analyze a picture and determine whether it contained false information through a fairly routine scan, she said.
But the quantity and speed of photos and videos being uploaded and the ability of people to manipulate details, especially as generative AI tools become more mainstream, has created a whole new hassle.
Social media sites are now dealing with a swarm of content related to two simultaneous wars, one in the Middle East and another between Russia and Ukraine. On top of that, they have to get ready for the 2024 presidential election in less than a year. Former President Donald Trump, who is under criminal indictment in Georgia for alleged interference in the 2020 election, is the front-runner to become the Republican nominee.
Manu Aggarwal, a partner at research firm Everest Group, said trust and safety is among the fastest-growing segments of a part of the market called business process services, which includes the outsourcing of various IT-related tasks and call centers.
By 2024, Everest Group projects the overall business process services market to be about $300 billion, with trust and safety representing about $11 billion of that figure. Companies such as Accenture and Genpact, which offer outsourced trust and safety services and contract workers, currently capture the bulk of spending, primarily because Big Tech companies have been “building their own” tools, Aggarwal said.
As startups focus on selling packaged and easy-to-use technology to a wider swath of clients, Everest Group practice director Abhijnan Dasgupta estimates that spending on trust and safety tools could be between $750 million and $1 billion by the end of 2024, up from $500 million in 2023. This figure is partly dependent on whether companies adopt more AI services, thus requiring them to potentially abide by emerging AI regulations, he added.
Tech investors are circling the opportunity. Venture capital firm Accel is the lead investor in Cinder, a two-year-old startup whose founders helped build much of Meta’s internal trust and safety systems and also worked on counterterrorism efforts.
“What better team to solve this challenge than the one that played a major role in defining Facebook’s Trust and Safety operations?” Accel’s Sara Ittelson said in a press release announcing the financing in December.
Ittelson told CNBC that she expects the trust and safety technology market to grow as more platforms see the need for greater protection and as the social media market continues to fragment.
New content policy regulations have also spurred investment in the area.
The European Commission is now requiring large online platforms with big audiences in the EU to document and detail how they moderate and remove illegal and violent content on their services or face fines of up to 6% of their annual revenue.
Cinder and Cove are promoting their technologies as ways that online businesses can streamline and document their content moderation procedures to comply with the EU’s new regulations, called the Digital Services Act.
‘Frankenstein’s monster’
In the absence of specialized tech tools, Cove’s Dworsky said, many companies have tried to customize Zendesk, which sells customer support software, and Google Sheets to capture their trust and safety policies. That can result in a “very manual, unscalable approach,” he said, describing the process for some companies as “rebuilding and building a Frankenstein’s monster.”
Still, industry experts know that even the most effective trust and safety technologies aren’t a panacea for a problem as big and seemingly uncontrollable as the spread of violent content and disinformation. According to a survey published last week by the Anti-Defamation League, 70% of respondents said that on social media, they’d been exposed to at least one of several types of misinformation or hate related to the Israel-Hamas conflict.
As the problem expands, companies are dealing with the constant struggle over determining what constitutes free speech and what crosses the line into unlawful, or at least unacceptable, content.
Alex Goldenberg, the lead intelligence analyst at the Network Contagion Research Institute, said that in addition to doing their best to maintain integrity on their sites, companies should be honest with their users about their content moderation efforts.
“There’s a balance that is tough to strike, but it is strikable,” he said. “One thing I would recommend is transparency at a time where third-party access and understanding to what is going on at scale on social platforms is what is needed.”
Noam Bardin, the former CEO of navigation firm Waze, now owned by Google, founded the social news-sharing and real-time messaging service Post last year. Bardin, who’s from Israel, said he’s been frustrated with the spread of misinformation and disinformation since the war began in October.
“The whole perception of what’s going on is fashioned and managed through social media, and this means there’s a tremendous influx of propaganda, disinformation, AI-generated content, bringing content from other conflicts into this conflict,” Bardin said.
Bardin said that Meta and X have struggled to manage and remove questionable posts, a challenge that’s become even greater with the influx of videos.
At Post, which is most similar to Twitter, Bardin said he’s been incorporating “all these moderation tools, automated tools and processes” since his company’s inception. He uses services from ActiveFence and OpenWeb, which are both based in Israel.
“Basically, anytime you comment or you post on our platform, it goes through it,” Bardin said regarding the trust and safety software. “It looks at it from an AI perspective to understand what it is and to rank it in terms of harm, pornography, violence, etc.”
Post is an example of the kinds of companies that trust and safety startups are focused on. Active online communities with live-chatting services have also emerged on video game sites, online marketplaces, dating apps and music streaming sites, opening them up to potentially harmful content from users.
Brian Fishman, co-founder of Cinder, said “militant organizations” rely on a network of services to spread propaganda, including platforms like Telegram, and sites such as Rumble and Vimeo, which have less advanced technology than Facebook.
Representatives from Rumble and Vimeo didn’t respond to requests for comment.
Fishman said customers are starting to see trust and safety tools as almost an extension of their cybersecurity budgets. In both cases, companies have to spend money to prevent possible disasters.
“Some of it is you’re paying for insurance, which means that you’re not getting full return on that investment every day,” Fishman said. “You’re investing a little bit more during black times, so that you got capability when you really, really need it, and this is one of those moments where companies really need it.”
WATCH: Lawmakers ask social media and AI companies to crack down on misinformation
Meta CEO Mark Zuckerberg makes a keynote speech at the Meta Connect annual event, at the company’s headquarters in Menlo Park, California, U.S. September 25, 2024.
Manuel Orbegozo | Reuters
Meta CEO Mark Zuckerberg plans to continue his company’s artificial intelligence spending blitz well into the next year as rival tech giants do the same.
Zuckerberg told analysts Wednesday during a second-quarter earnings call that AI’s rapid pace of progress has informed much of Meta’s recent business decisions, including the company’s $14.3 billion June investment into the data-annotating startup Scale AI as part of a revamped AI strategy involving a wave of high-profile hires.
AI’s swift advancement warrants that Meta have “the absolute best and most elite talent-dense team” that can access the resources they need from a “leading compute fleet,” Zuckerberg said about the AI Superintelligence team he assembled for his company this summer. Whatever these top-tier AI researchers build can then be implemented throughout Facebook, Instagram and the rest of the company’s family of apps, he said.
“When we take a technology, we’re good at driving that through all of our apps and our ad systems,” Zuckerberg said. “There’s no other company that is as good as us at kind of taking something and getting it in front of billions of people.”
Those AI endeavors, however, come at a cost.
Meta on Wednesday said it expects its total expenses for 2025 to come in the range of $114 billion and $118 billion, raising the low end of its previous outlook of between $113 billion and $118 billion. And while Meta is still planning out next year, the company said its AI initiatives will “result in a 2026 year-over-year expense growth rate that is above the 2025 expense growth.”
Other tech giants are also spending heavy on AI projects and talent.
Alphabet said last week during its earnings report that it is raising its 2025 capital expenditures forecast to $85 billion, which is $10 billion higher from its prior forecast. Microsoft said Wednesday that its fiscal first-quarter capital expenditures will be $30 billion, ahead of analyst expectations of $24.23 billion.
For now, investors are OK with Meta’s big AI investments, with the company’s shares up nearly 12% in after-hour trading on Wednesday. It helps that Meta reported strong second-quarter earnings that beat on the top and bottom while providing third-quarter sales guidance that topped Wall Street expectations.
It also helps that Zuckerberg said AI drove “greater efficiency and gains across our ad system,” likely reassuring worried investors that Meta’s big AI spending is leading to some immediate results.
And while the company’s Reality Labs unit continues bleeding money, posting an operating loss of $4.53 billion in the second quarter, the surprise hit of the Ray-Ban Meta smart glasses seems to have quelled investor discontent for the time being.
“I continue to think that glasses are basically going to be the ideal form factor for AI, because you can let an AI see what you see throughout the day, hear what you hear, talk to you,” Zuckerberg said. “Once you get a display in there, whether it’s the kind of wide holographic field of view, like we showed with Orion, or just a smaller display that might be good for displaying some information, that’s going to unlock a lot of value, where you can just interact with an AI system throughout the day.”
WATCH: I like Meta as a stock, don’t love it, prefer Google.
Elon Musk has expanded a number of his companies within Texas, including Tesla, SpaceX, the Boring Co. and Neuralink. Tesla broke ground on a lithium refinery in Texas earlier this year with Governor Greg Abbott in attendance.
Christophe Gateau | Picture Alliance | Getty Images
Elon Musk’s tunneling venture, The Boring Company, announced plans earlier this week to build a 10-mile underground loop in Nashville, in coordination with Tennessee Republican Governor Bill Lee, who put out a press release praising the project.
Democratic lawmakers in Nashville are demanding answers on the plans, while the state’s Republican leaders have jumped at the chance to partner with Musk. A state commission is holding an emergency meeting and public hearing Thursday morning to discuss a “no cost/mutual benefit” lease arrangement that’s been proposed to help the company get the tunnels started.
“We are aware of the state’s conversations with the Boring Company, and we have a number of operational questions to understand the potential impacts on Metro and Nashvillians,” Freddie O’Connell, Nashville’s mayor, said in an e-mailed statement.
Based in Pflugerville, Texas, The Boring Co. is poised to take over a chunk of public property about the size of a football field in downtown Nashville. The commission that’s meeting on Thursday includes Tennessee’s governor, speaker of the house, speaker of the senate and secretary of state. Members of the public were invited to give testimony but with less than a week’s notice.
On Monday, The Boring Co. and state officials divulged that Musk’s venture would dig its tunnels under state-owned roadways in order to “connect downtown and the Convention Center to Nashville International Airport with a transit time of approximately 8 minutes.”
It’s called the Music City Loop, and the project marks Musk’s latest effort to bolster his budding business empire in Tennessee. His artificial intelligence startup xAI, the parent of social media platform X, is building data centers and a power plant in Memphis, on the western side of the state.
The governor’s office said on Monday that the Nashville project would come “at zero cost to taxpayers” and would be “entirely privately funded,” though no details were provided about whether or what type of cost-benefit analysis, environment, safety or traffic assessment had been completed by the state before agreeing to the deal.
Musk became a major force in Republican politics last year, when he spent almost $300 million to help reelect President Donald Trump before working for the Trump administration in the first few months of this year. Musk brought The Boring Co. CEO Steve Davis with him to lead Trump’s DOGE initiative, slashing federal agencies, regulations and personnel.
Justin Jones, a Democratic state representative in Nashville, told CNBC on Wednesday that his district had not been able to participate in any public comment period, and hadn’t seen any environmental impact report or health assessment related to the Music City Loop or its construction.
‘Not allowed to be here’
On Wednesday evening, The Boring Co. held a recruiting event, with Davis in attendance, at the parking lot where the company expects the state to grant it a no-cost lease. Jones went to the event hoping to discuss the jobs that Musk’s company is looking to create in his district, the lawmaker told CNBC.
“The CEO is here and the other members of their team, but they sent someone out to tell me that I’m not allowed to be here,” Jones said in a text message, sharing a video of his interaction with The Boring Co. employees at the event.
On Monday, Jones arrived to a separate company event at the Nashville airport only to have authorities claim he lacked proper credentials to attend.
Jones told CNBC that state officials explained to him that only state-level authorizations would be required for The Boring Co. project to begin because the tunnels would go under state roads, and would not require the use of taxpayer funds.
“We’re not even being informed where or what exactly these tunnels are going to run through,” Jones said. “Tomorrow they’re voting to give away state land for no cost. But giving away land obviously has a cost.”
The governor’s office didn’t respond to a request for comment regarding Jones’ concerns. Representatives for The Boring Co. weren’t immediately available to comment.
The Boring Co. has previously built tunnels in Las Vegas, including an initial two miles to carry visitors to different exhibit halls around the Las Vegas Convention Center. Tesla drivers travel through the tunnels to pick up and drop off passengers, who book their rides using an app.
The initial loop cost Nevada taxpayers about $50 million and has been criticized for a lack of pedestrian entrances, walkways and platforms, and its limitations relative to a subway system. The Boring Co. was previously fined by the Nevada Occupational Safety and Health Administration for repeated violations and worker injuries in Las Vegas.
The Musk-owned company also abandoned plans to build tunnels in other locations, including Chicago.
One particular concern in Nashville is that the city is prone to flooding with an average annual rainfall of around 50 inches, according to the National Weather Service, which compares to around 4 inches in Las Vegas. The city’s Metro Water Services previously arranged, with federal support, to purchase homes from residents in vulnerable areas at reduced prices, and convert the land there to green spaces.
The Boring Co. has no experience building in areas with that kind of rainfall and flooding concern.
The public hearing to discuss whether the state will give the parking lots to The Boring Co. in a no-cost, mutual benefit lease agreement starts at 8 a.m. local time on Thursday at Cordell Hull State Office Building, according to a copy of the agenda on the state government’s website.
In Memphis, xAI has faced a community backlash over its use of natural gas-burning turbines which power its data center and supercomputer there. The facility, housed in a former home appliance factory, is responsible for training xAI’s controversial chatbot Grok.
The NAACP and other environmental and public health advocates are suing xAI, saying the company exacerbated air pollution in the area, harmed majority-Black communities who live near their facilities, and violated the Clean Air Act. An xAI spokesperson said at the time the groups announced their intent to sue that the company takes “our commitment to the community and environment seriously.”
WATCH: Tesla CEO Elon Musk says he has no plans to merge the automaker with xAI
-
Sports3 years ago‘Storybook stuff’: Inside the night Bryce Harper sent the Phillies to the World Series
-
Sports1 year agoStory injured on diving stop, exits Red Sox game
-
Sports2 years agoGame 1 of WS least-watched in recorded history
-
Sports2 years agoMLB Rank 2023: Ranking baseball’s top 100 players
-
Sports4 years ago
Team Europe easily wins 4th straight Laver Cup
-
Sports2 years agoButton battles heat exhaustion in NASCAR debut
-
Environment2 years agoJapan and South Korea have a lot at stake in a free and open South China Sea
-
Environment2 years agoGame-changing Lectric XPedition launched as affordable electric cargo bike