Tech Capture: The Builders

Tech capture is another phrase for how social media platforms and AI products are manipulating human attention, through a combination of psychology, algorithmic design, and our oldest survival instincts, used against our best interests.

It isn’t an accident

The most common assumption about social media and the large language models (LLMs) behind our friendly chatbot is that the addictive quality is a side effect. Something that emerged from a product built for connection, and that the companies have been scrambling to fix ever since.

That’s not what happened.

Sean Parker was Facebook’s founding president. In 2017 he shared the team’s central question: "How do we consume as much of your time and conscious attention as possible?"

The answer was the like button: "a little dopamine hit every once in a while, because someone liked or commented on a photo," which prompts us to post more, which produces more likes. He called it a "social-validation feedback loop" that was "exploiting a vulnerability in human psychology."

He then said the people who built it "understood this consciously. And we did it anyway."

He also remembered people telling him they weren’t on social media, that they valued real-life interactions, presence, intimacy. His answer: "We’ll get you eventually." Looking back, he admitted it "literally changes your relationship with society, with each other … God only knows what it’s doing to our children’s brains."

He wasn’t alone. Chamath Palihapitiya ran user growth at Facebook from 2007 to 2011. Speaking at Stanford, he said, "The short-term, dopamine-driven feedback loops we’ve created are destroying how society works," and that he felt "tremendous guilt" for his part in building them.

Then there is Guillaume Chaslot, who helped build YouTube’s recommendation engine. Their priority was how much time we spend watching, how much of our attention they could grab. That was what mattered, and everything else was treated as a distraction. When he worked on a project to introduce more diversity into recommendations, they shut it down, because it didn’t keep us watching. According to Chaslot, YouTube’s "machine learning" system figured out that the best way to get people to spend more time on the platform was to show them videos "light on facts but rife with wild speculation." In other words, conspiracy theories. So the system pushed people toward them.

There is also Tristan Harris. He was Google’s design ethicist before he left in 2016 to found the Center for Humane Technology. He said their goal was to turn the phone into a slot machine. Turn dating into a slot machine. Turn email into a slot machine.

Roger McNamee, an early investor in both Facebook and Google and once a mentor to Mark Zuckerberg, wrote Zucked: Waking Up to the Facebook Catastrophe (Penguin Press, 2019), arguing the two companies now threaten public health and democracy.

These accounts come from the founding president, the head of growth, the engineer who built the recommendation system, the design ethicist, and the money.

And then they ran an experiment on us without asking our permission

In January 2012, Facebook ran a week-long experiment on 689,003 of its users. Us. Working with researchers from Cornell, they manipulated what appeared in people’s news feeds, reducing positive content for one group and negative content for another, to see whether it would change the emotional tone of what those people posted themselves.

It did. When positive posts were reduced, people wrote fewer positive posts and more negative ones. When negative posts were reduced, the reverse happened.

Emotional states, the researchers concluded, can be transferred through a social network, and people experience those emotions without awareness that anything has been done to them.

None of the 689,003 participants were told. Any of us could have been among them. Those who were weren’t asked, weren’t debriefed, and have never been individually informed. The legal basis was the terms of service.

The paper was published in the Proceedings of the National Academy of Sciences. After the resulting outcry, the journal issued an Editorial Expression of Concern.

That study demonstrated the degree of control Facebook has over what any of us sees. In 2013, Facebook said that of roughly 1,500 possible items that could appear in a feed each day, the algorithm decides which ones do.

Beginnings

In 1998 a Stanford psychologist named B.J. Fogg founded a lab to study one question: how can computers be designed to change what people believe and do?

He called the field persuasive technology and wrote its first textbook in 2002. Then he taught it to his students, one of whom co-founded Instagram.

In 2007, he co-taught a course that became known as the Facebook Class. The assignment was to build an app, get as many people using it as they could, and keep them coming back.

Ten weeks later his students’ apps had millions of users. Several of them were rich before the course ended. Fogg later told the New York Times that 2007 was a year when you could walk in and collect gold.

Stanford’s own student paper ran a piece called How Stanford Profits Off Addiction. It credits this research with inspiring the like button, the red notification badges, infinite scroll, and autoplay video.

He warned the government first

In 2006, a year before the Facebook Class, Fogg made a video for the Federal Trade Commission. The purpose was to warn policymakers about what persuasive technology could do. He even predicted how it would be used to manipulate politics.

A year later, he taught the Facebook Class.

A closer look

The hook

Our brains instinctively snap toward sudden movement, bright color, and sharp sound. Originally this helped us spot predators and kept us alive. Modern predators, however, use knowledge of how these life-saving instincts work to manipulate us. Everything on the phone is designed to trigger this kind of instinctual attention: red notification badges, shifting animations, auto-playing videos, sudden vibrations.

Spend all day responding to micro-triggers like these, including our own ringtone and notification chime, which capture our attention involuntarily, and it wears us out. Our ability to keep our focus where we choose atrophies, and our instincts get played like the proverbial fiddle.

They use our emotions too

Algorithms are built to recognize and act on emotion, as the Facebook emotional contagion experiment above showed.

Two instinctive responses get targeted in particular: outrage and moral contagion. We humans instinctively notice threats. High-arousal negative emotions like anger, moral outrage, and disgust travel faster and wider online than any other content.

Researchers analyzed more than half a million social media posts about gun control, same-sex marriage, and climate change. Every additional moral-emotional word increased a message’s spread by roughly twenty percent, and for negative words specifically the figure was higher. Positive moral language barely registered by comparison.

They called it moral contagion. A later replication using nearly 850,000 posts found the same effect.

The spread was strongest inside political groups and weakest across them. Outrage travels within our own tribe.

It also teaches.

A separate study found that when an outraged post earns likes and shares, its author becomes measurably more likely to post outrage again. Straightforward reinforcement: reward a behavior and you get more of it.

We also tune our outrage to match what’s normal in our network. And in the most ideologically extreme networks, where outrage is already the norm, the reward stops mattering. We keep expressing outrage whether or not it earns likes or shares, because it has simply become the expected way to speak.

Beyond carrying outrage faster than anything else, the platform makes us more outraged than we were.

When a Facebook group posts a meme that is partially true but hyperbolic, as they are most of the time, they’re using outrage as bait. Tristan Harris calls this outrage-ification. The truth makes it believable. The hyperbole triggers a sense of injustice, and tribal affiliation kicks in. Our brains crave belonging. Groups exploit this by framing issues as us versus them.

Sharing a post becomes, subconsciously, a badge of tribal loyalty. If we don’t share or agree, we risk being cast out of the tribe, so to speak, which triggers primal social anxiety.

Brain scans show that being excluded activates the same region that registers physical pain. As far as the brain is concerned, being cast out and being hurt are close to the same event.

The loop

Once an emotion is triggered, the platform’s infrastructure locks us into a behavioral feedback loop.

An outrageous post appears. It sparks anger, and we comment or share it. The algorithm reads that as engagement and pushes the post to more people, who react the same way.

This is the slot machine effect. Every time we open an app, leave a comment, or share a post, we wait. We may go do other things, but on some level, a part of us makes a mental note to check back on notifications, likes, replies.

The unpredictability of when they show up is the point. In behavioral science it’s called a variable ratio reinforcement schedule: a reward that shows up sometimes, on no pattern you can learn. It produces the most persistent behavior of any schedule, and it’s the hardest to stop. It’s what slot machines run on, and it’s what our apps run on.

They don’t need us to click

The algorithm tracks milliseconds of attention.

In 2013 Facebook’s head of analytics told the Wall Street Journal the company was testing new data collection: how long a cursor hovers over part of a page, and whether the feed is actually visible on the screen at a given moment. He called the process never-ending.

That was the desktop era. On a phone there’s no cursor, so the equivalent is dwell time: how long a post sits on the screen before we scroll past. Google had patented the underlying idea years earlier, a system for reordering results based on where people linger without clicking.

So if we pause on an angry political meme for an extra three seconds, that’s noted as interest. We’ll be shown more like it. They are feeding the fury on purpose.

Nice, huh?

The menu was chosen for us

We think we’re exercising sovereignty when we choose what to click on. But choosing item A instead of item B isn’t much of an option, considering the algorithm deliberately put those two on the menu in the first place for maximum engagement.

True cognitive sovereignty means deciding whether we want to look at the menu at all.

Then they shape us

Over time, it stops being about capturing attention and starts reshaping our entire reality.

In the physical world we naturally adjust depending on where we are. We speak differently at a dinner party than in a boardroom. Social media does away with that. danah boyd named it context collapse: online, our boss and our mother and our college friends are all one audience, and there’s no version of us that fits all of them at once.

The same thing happens to the material itself. A complex, layered human issue shows up as a fifteen-second video or a single meme, without any of the conditions that would let us get an honest picture of it. We can’t assess anything without context. Otherwise we’re forming opinions on matters so lacking in critical information they may as well have been made up.

The illusion of consensus

Our feed is a sample of about a thousand things, but it’s designed to make us feel like we’re getting a broad view of information, so when the same outrage appears over and over, it starts registering as reality.

Sometimes it is. Often it’s distorted, and we have no way of seeing what got left out.

Unfortunately, we’re each in a different reality. The algorithm shows different people entirely different worlds based on their data profiles. Two neighbors can look at the same event and see two completely different existential threats. Neither of them knows what the other saw.

That keeps us in a constant state of hypervigilance against a reality tailored specifically for us. No wonder we have a hard time finding common ground.

This is the most insidious part, and the hardest to see from our own shoes.

Shoshana Zuboff, in The Age of Surveillance Capitalism (PublicAffairs, 2019), puts it this way: "It is no longer enough to automate information flows about us; the goal now is to automate us."

If an algorithm determines that someone is slightly more likely to purchase a product or engage with an ad when feeling insecure or anxious, it will subtly feed them content over days and weeks that increases that insecurity or anxiety.

Worse, the person believes their changing mood, or their shifting political views, are an organic part of their own growth.

In reality their personality is being gradually nudged toward a state that is more easily monetized.

Tech Capture: The Audience Farmers ->

Tech Capture: The Chatbots ->

Tech Capture: Staying Sovereign ->