Homepage / Technology / A former YouTube product manager explains how big tech companies really police content
Amazon says this Prime Day was its biggest shopping event ever Kudlow says President Trump is 'so dissatisfied' with China trade talks that he is keeping the pressure on As stocks regain their footing, an ominous warning looms Goldman Sachs downgrades Clorox to sell, says valuation is 'unsustainably high' How Satya Nadella has spurred a tripling of Microsoft's stock price in just over four years Kudlow says economic growth could top 4% for 'a quarter or two,' more tax cuts could be coming The one chart that explains Netflix’s stunning comeback US housing starts plunge 12% in June to a nine-month low Aerospace titans Boeing and Airbus top $110 billion in orders at Farnborough Target uses Prime Day to its advantage, logging its 'biggest online shopping day' so far this year Billionaire Marc Lasry sees bitcoin reaching up to $40,000 as it becomes more mainstream and easier to trade These are the 10 US airports where you're most likely to be hacked Amazon shares slightly higher as investors await Prime Day results Wreck of Russian warship found, believed to hold gold worth $130 billion A bullish ‘phenomenon’ in bond market is weeks away from fading, top credit strategist says Stocks making the biggest moves premarket: MS, GOOGL, TXN, UAL, NFLX & more Twitter shares up 50% since late April means most upside priced in, analyst says in downgrade EU fines Google $5 billion over Android antitrust abuse Mortgage applications fall 2.5% as buyers struggle to find affordable homes America may not have the tools to counter the next financial crisis, warn Bernanke, Geithner and Paulson Investors are getting spooked as the risk of a no-deal Brexit rises EU expected to fine Google $5 billion over Android antitrust abuse Ex-FBI chief James Comey urges Americans to vote for Democrats in midterm elections Elon Musk apologizes to British cave diver following baseless 'pedo guy' claim Disney, Comcast and Fox: All you need to know about one of the biggest media battles ever Xiaomi shares notch new high after Hong Kong, mainland China stock exchanges reach agreement The trade war is complicating China's efforts to fix its economy European markets set for a strong open amid earnings; Google in focus Hedge fund billionaire Einhorn places sixth in major poker tournament The biggest spender of political ads on Facebook? President Trump Asian stocks poised to gain after Fed's Powell gives upbeat comments; dollar firmer Stocks are setting up to break to new highs Not all FAANG stocks are created equal EU ruling may be too little, too late to stop Google's mobile dominance Cramer explains how Netflix's stock managed to taper its drop after disappointing on earnings Airbnb condemns New York City's 'bellhop politics,' threatens legal retaliation Amazon sellers say they were unfairly suspended right before Prime Day, and now have two bad choices Investor explains why 'duller' tech stocks can have better returns than 'high-flying' tech names Elon Musk is 'thin-skinned and short-tempered,' says tech VC Texas Instruments CEO Brian Crutcher resigns for violating code of conduct Google Cloud Platform fixes issues that took down Spotify, Snapchat and other popular sites Uber exec: We want to become the 'one stop' transportation app 'What a dumb hearing,' says Democrat as Congress grills tech companies on conservative bias Amazon shares rebound, report says Prime Day sales jumped 89 percent in first 12 hours of the event How to put your medical history on your iPhone in less than 5 minutes Investment chief: Watch these two big events in 2018 Even with Netflix slowing, the market rally is likely not over Cramer: Netflix subscriber weakness debunks the 'sky's the limit' theory on the stock Netflix is looking at watch time as a new area of growth, but the competition is stiff Why Nobel laureate Richard Thaler follows Warren Buffett's advice to avoid bitcoin Rolls-Royce is developing tiny 'cockroach' robots to crawl in and fix airplane engines After Netflix plunge, Wall Street analysts forecast just tame returns ahead for the once high-flying FANG group Roku shares rise after analyst raises streaming video company's price target due to customer growth China is investing 9 times more into Europe than into North America, report reveals Amazon says US Prime Day sales 'so far bigger than ever' as glitch is resolved Netflix is on pace for its worst day in two years US lumber producers see huge opportunity, rush to expand San Francisco to consider tax on companies to help homeless Homebuilder sentiment, still high, stalls as tariffs, labor and land drive up costs Powell backs more rate hikes as economy growing 'considerably stronger' Netflix history is filled with big stock declines – like today – followed by bigger rebounds Intel shares get downgraded by Evercore ISI due to rising competition from Nvidia, AMD Petco aims to reinvent the pet store with something you can't buy online Genetic testing is coming of age, but for consumers it's buyer beware Tech 'FAANG' was the most-crowded trade in the world heading into the Netflix implosion, survey shows Netflix weak subscriber growth may indicate a 'maturity wall' that could whack the stock even more: Analyst This chart may be predicting the bull market's demise Wall Street says Netflix's stock plunge is a ‘compelling’ buying opportunity because the streaming giant ‘never misses twice’ Tesla sinks after Musk tweets, again Boeing announces new division devoted to flying taxis Stocks making the biggest move premarket: NFLX, UNH, GS, AMZN, WMT & more Deutsche Bank downgrades Netflix, but says big subscriber miss is not 'thesis changing' IBM is experimenting with a cryptocurrency that’s pegged to the US dollar North Korea and Zimbabwe: A friendship explained Virgin Galactic spinoff Orbit to launch rockets from the UK with space deal Artificial intelligence will create more jobs than it destroys? That’s what PwC says ‘Treasonous’ Trump and ‘Putin’s poodle:' Scathing headlines follow the Trump-Putin summit China’s fintech companies offer ‘enormous’ opportunity, investment manager says Trump's performance at summit with Putin was 'unprecedented,' experts say Walmart and Microsoft link up on cloud technology as they both battle Amazon European stocks seen mixed amid earnings; Fed’s Powell to address Congress How I knew I should quit my day job and run my start-up full-time: Viral website founder China's stocks have been trounced, but the trade war may ultimately be good news for those shares Billionaire tech investor Peter Thiel bets on crypto start-up Block.one Asian shares subdued open after mixed close on Wall Street; energy stocks under pressure Amazon cloud hits snags after Amazon Prime Day downtime Netflix isn't doomed by one quarter unless people start questioning the long-term investor thesis Tech stocks set to sink on Tuesday after rough evening for ‘FANG’ Netflix plummets after missing big on subscriber growth This wristband lets humans control machines with their minds The U.S. has a rocky history convincing Russia to extradite computer criminals Amazon suffers glitches at the start of Prime Day Jeff Bezos is now the richest man in modern history 'The United States has been foolish': Read Trump and Putin's full exchange Goldman Sachs recommends these 5 highly profitable companies — including Nvidia — to combat rising inflation Goldman Sachs releases 'tactical' stock picks for this earnings season Three red flags for Netflix ahead of its earnings report The bond market may be raising recession fears, but don't expect one anytime soon Cramer: Banks are 'making fortunes' but are still as hated as they were during the financial crisis Putin told Trump at summit: Russia never meddled in US election


A former YouTube product manager explains how big tech companies really police content

This post originally appeared on Hunter Walk’s blog and is republished here by permission.

Since Facebook, Twitter and Alphabet‘s YouTube have all been vocal (to various degrees) about staffing up the human element of their content moderation teams, here are a few things to understand about how these systems typically work. Most of this is based on my time at YouTube (which ended almost five years ago, so nothing here should be considered a definitive statement of current operations), but I found our peer companies approached it similarly. Note, I’m going to focus on user generated/shared content, not advertising policies. It’s typical that ads have their own, separate criteria. This is more about text, images & video/audio that a regular user would create, upload and publish.

Content Moderation or Content Review is a term applied to content (text, images, audio, video) that a user has uploaded, published or shared on a social platform. It’s distinct from Ads or Editorial (eg finding content on the site to feature/promote if such a function exists within an org), which typically have separate teams and guidelines for when they review content.

The goal of most Content Moderation teams is to enforce the product’s Community Standards or Terms of Service, which state what can and cannot be shared on the platform. As you might guess, there’s black and white and gray areas in all of this, which mean there are guidelines, training and escalation policies for human reviewers.

It would be very rare (and undesirable) for humans to (a) review all the content shared on a site and (b) review content pre-publish – that is, when a user tries to share something, having it “approved” by a human before it goes live on the site/app.

Instead, companies rely upon content review algorithms which do a lot of the heavy lifting. The algorithms attempt to “understand” the content being created and shared. At point of creation there are limited signals – who uploaded it (account history or lack thereof), where it was uploaded from, the content itself and other metadata. As the content exists within the product more data is gained – who is consuming it, is it being flagged by users, is it being shared by users and so on.

These richer signals factor into the algorithm continuing to tune its conclusion about whether a piece of content is appropriate for the site or not. Most of these systems have user flagging tools which factor heavily into the algorithmic scoring of whether content should be elevated for review.

Most broadly, you can think about a piece of content as being Green, Yellow or Red at any given time. Green means the algorithm thinks it’s fine to exist on the site. Yellow means it’s questionable. And Red, well, red means it shouldn’t be on the site. Each of these designations are fluid and not perfect. There are false positives and false negatives all the time.

To think about the effectiveness of a Content Policy as *just* the quality of the technology would be incomplete. It’s really a policy question decided by people and enforced at the code level. Management needs to set thresholds for the divisions between Green, Yellow and Red. They determine whether an unknown new user should default to be trusted or not. They conclude how to prioritize human review of items in the Green, Yellow or Red buckets. And that’s where humans mostly come into play…

Human reviewers help create training sets for the algorithms but their main function is continually staffing the review queues of content that the algorithm has spit out for them. Queues are typically broken into different buckets based on priority of review (eg THIS IS URGENT, REVIEW IN REAL TIME 24-7) as well as characteristics of the reviewers – trained in different types of content review, speak different languages, etc. It’s a complex factory-like system with lots of logic built in.

Amount of content coming on to the platform and the algorithmic thresholds needed to trigger a human review are what influence the amount of content that goes into a review queue. The number of human reviewers, their training/quality, and the effectiveness of the tools they work in are what impact the speed with which content gets reviewed.

So basically when you hear about “10,000 human reviewers being added” it can be (a) MORE content is going to be reviewed [thresholds are being changed to put more content into review queues] and/or (b) review queue content will be reviewed FASTER [same content but more humans to review].

The honest answer is “Yes But….”

Yes but Content Operations is typically a cost center, not a revenue center, so it gets managed to a cost exposure and can be starved for resources.

Yes but Content Operations can sometimes be thought of as a “beginner” job for product managers, designers, engineers so it gets younger, less influential staffing which habitually rotates off after 1-2 years to a new project.

Yes but lack of diversity and misaligned incentives in senior leadership and teams can lead to an under-assessing of the true cost (to brand, to user experience) of “bad” content on the platform.

Because there are much better places to share porn than Twitter, Facebook and YouTube. And because algorithms are actually really good at detecting nudity. However, content created for sexual gratification that doesn’t expressly have nudity involved is much tougher for platforms. Did I ever write about creating YouTube’s fetish video policy? That was an interesting discussion…

  1. Make it a dashboard level metric – If the CEO and her team is looking at content safety metrics alongside usage, revenue and so on, it’ll prove that it matters and it’ll be staffed more appropriately.
  2. Talk in #s not percentages – These HUGE platforms always say “well, 99% of our content is safe” but what they’re actually saying is “1% of a gazillion is still a really large number.” The minimization framework – which is really a PR thing – betrays the true goals of taking this stuff seriously.
  3. Focus on preventing repeat infringement and recovering quickly from initial infringement – No one expects these systems to be perfect and I think it’s generally good to trust a user until they prove themselves to be non-trustworthy. And then hit them hard. Twitter feels especially poor at this – there are so many gray-area users on the system at any given time.
  4. Management should spend time in the review queues – When I was leading product at YouTube I tried to habitually spend time in the content review queues because I didn’t want to insulate myself from the on-the-ground realities. I saw lots of nasty stuff but also maintained an appreciation for what our review teams and users had to go through.
  5. Response times are the new regulatory framework – I wonder if there’s role for our government to not regulate content but to regulate response time to content flagging. There’s a ton of complexity here and regulations can create incentives to *not* flag content, but it’s an area I’m noodling about.

Hope that helps folks understand these systems a bit more. If you have any questions, reach out to me on Twitter.

Hunter Walk is an investor at Homebrew, a seed-stage venture fund. He previously led consumer product management for YouTube, oversaw product and sales for Google’s contextual ad business, and was a founding member of the product team that created virtual world Second Life.

Source: Tech CNBC
A former YouTube product manager explains how big tech companies really police content

Comments are closed.