It identifies key tasks, such as correctly targeting “basic theory” as well as the foundations of “common technology,” and also outlines several potential applications, ranging from “smart cities” to “intelligent medical care” to “promoting credible communication.”

There are, however, some unrealistic proposals. “Intelligent government” is hard both technically and socially and is unlikely to become a reality, and some supporting technologies like 5G seem a little rushed. And there is scant attention paid to “laws, regulation” and “ethical norms,” which come off as under-developed compared to the rest of the document, indicating a much more “full speed ahead” approach rather than “cautious exploration.”

But the strategic goals is where the document shines with ambition: “by 2025 we shall…make positive progress in the construction of an artificial intelligence society,” and “by 2030 we shall be the major artificial intelligence innovation center of the world.”

Even if we pessimistically estimate that 50% is hype, achieving half of what’s in the document is already a massive step towards an AI-enabled society.

What is America’s answer to China’s AI strategy?

In March, the White House launched a slick new page for national AI efforts. But anyone looking for a visionary and coherent strategy will be disappointed. The White House site is more of an aggregate of individual agency’s plans, rather than a single strategy.

So, what have the departments, agencies, and committees that make up America’s permanent government been up to?

Democratic Senate Minority Leader Chuck Schumer pitched a new $100 billion dollar funding scheme for AI research to a presumably receptive audience at the National Security Commission on Artificial Intelligence. Meanwhile, the National Institute of Standards and Technology is hoping for “voluntary consensus standards.” The Federal Data Strategy site is by far the most worrying one, insofar as most of its proposed improvements are likely to make one think, “How is this not done yet?”—such as validating employers and making policy proposals machine-readable.

Better late than never? Maybe, but it’s hardly the approach of a serious global power.

The White House’s STEM education plan has a number of routine ideas—like using “common metrics to measure progress” and making federal data more accessible. But as usual, the document focuses more on improving the bottom performers than creating and scaling up top talent.

One of the more impressive AI strategy documents is from the Department of Transportation. It starts off strong: “U.S. DOT will lead efforts to address potential safety risks and advance the life-saving potential of automation, which will strengthen public confidence in these emerging technologies.” It identifies key stakeholders, testing stages, and moves towards proactive anticipation of technology by regulators. Moreover, the DOT document consistently keeps its eye on the ball: cutting down the estimated 37,000 lives lost in car accidents.

Also on the ambitious side, the Pentagon released its AI strategy in 2019. It’s been received as among the most wide-ranging plans put out by a U.S. government agency. In addition to noting the geopolitics driving the current battles for an AI strategy, it responded to recent attempts by tech companies like Google to cease cooperation with American security state organs, via calls for information sharing and closer relationships with academia and the private sector. The news came alongside a White House executive order. Notably, the plan targets building in-government AI proficiency as a key goal—which would ultimately reduce its reliance on private actors who are sometimes hostile.

There are other national strategy documents, such as reports from National Security Commission on Artificial Intelligence. This document certainly has a strong sense of urgency. It puts competition in AI as a key aspect of the “reemergence of great power competition” and includes a nod to “existential threats.” And it also points out the aforementioned tech worker opposition to cooperation with U.S. government departments like the Pentagon. In the industry, I hear this kind of concern all the time, though usually more in connection with ICE.

There is also an important recognition that most parts of the government suffer from a dire lack of real AI knowledge. The result is a catch-22 of funding problems, since those in charge of giving money to possible improvements themselves don’t have the specialized knowledge to use said funds wisely. The document also does the outstanding service of proclaiming that AI is not merely a disaster-aversion problem, but also a tool kit that should be used to pursue positive goals: “We need a vision of the AI-empowered future.”

While this document is better than most in terms of its assessment of the situation and present challenges, as well as articulating the need for a vision, it lacks said vision and a sense of determination.

Disorganized planning, cluttered thinking, and scattered good ideas more or less sum up the state of affairs across the American state apparatus when it comes to AI. With the possible exception of the DOT, the national-level result lacks the feeling of success-orientation—a mindset you would expect of an ambitious startup or leading company, not to mention of a superpower which aims to produce them with the numbers and reach needed to maintain its global standing. Instead, it appears fixed on identifying and measuring improvements that don’t require vision or ambition.

A cautious approach can be good to avoid getting fooled by hype, but the level of inertia is obvious. The federal approach struggles to incorporate AI into existing and potentially outdated legal frameworks (such as self-driving cars), and in other cases seems to move nearly all responsibility for the details into the technical sector.

In particular, the documents fail to address crucial problems that the U.S. would face in implementing its AI strategy—for example, hostile relationships between a number of companies and the American state, lack of politically reliable software developers in key positions, declining ability in public and private discourse to discuss basic statistical facts, and generally weak mathematics education in early schooling. California-specific concerns, which directly impact the Bay Area tech hub, include a power grid that companies can rely on and housing policy that enables concentration of technical talent without all excess income being skimmed by landlords and high local taxes.

***

The difference here is not just in mindset, but also in specific policy proposals and social organization. U.S. discourse is full of fears about “automation unemployment,” which is a self-fulfilling prophecy, but also functions as a scapegoat for people worried about two things: losing control of their destiny to systems they don’t have a say in and labor issues arising from globalization and immigration. There is a small AI safety community that worries about existential future risk from advanced AI, but largely doesn’t engage with what’s actually happening in the field. A common concern there is about “AI arms races,” where countries and ambitious projects charge ahead with capability research and applications to beat the competition or just not fall behind, without attention to safety and controllability.

Key thought leaders in the area like Jaan Tallinn have proposed regulating large clusters of computing power. There is some merit to this idea. After all, the companies controlling computing power do not use it in manufacturing goods—the “standard” function of economic capital. Rather, they increasingly use it to target advertising, collect consumer data, and enable narrative control. In short, capital’s power is no longer derived merely from owning the means of production, but also the means of behavioral modification.

But we risk conflating two very different issues. The first issue is the one that originally sparked the concern for “AI safety,” which is the potentially existential impact of superhuman general intelligence. A system better than humans at all industrially essential intelligence-laden tasks, including business, politics, and socializing, could entirely escape our control and ultimately replace and destroy us. This worry is so far entirely theoretical, having very little to do with current developments and applications in artificial intelligence. Current developments are all in the realm of “narrow AI,” which is effective in extremely specific tasks, but not the hardest and most general—and hardly superhuman.

The second issue is how this narrow AI is being applied right now: to data mining, advertising, self driving cars, scaling of editorial control over social media discourse, and so on. This is the realm of what people usually mean by AI ethics. The major issue is the social impact and economic of the technologies we already have, how those should be deployed, and who should have control. In other words, it’s political. There are, of course, safety problems with things like self-driving cars, but the problem of “this car might run some people over” is so entirely different in kind from “this AI algorithm might gain control of technological civilization and destroy us and everything we value” that they shouldn’t be grouped together. This more mundane kind of safety issue is completely within the experience of previous industrial and technological changes. At worst, there are a few small-scale disasters, and we learn some hard lessons. The political and social aspect is much more central.

Unfortunately, the discourse has made a habit of grouping AI existential risk concerns with more mundane—but also far more immediately relevant—governance concerns. This has been aided by an industry full of grifters who have every incentive to portray every new and potentially socially powerful statistical algorithm as a revolutionary step towards full artificial general intelligence, especially for investors who treat anything that can be labelled “AI” in 2019 the same way they treated “Internet” in 1999. Those in AI ethics often appropriate the movement energy and fears built by existential safety thinkers for their own more mundane political and funding ends. The public, which doesn’t understand the subtleties, sees only a spectrum of Black Mirror-like technologies to alternately be in fear or in awe of.

Most of the existing discussion, including at least part of the computing power proposal, is about the second issue: mundane industrial safety, impact, and political governance of powerful new technologies. The example of computing power applied to behavioral manipulation is instructive.

The ability to control and manipulate information flows and human behavior at scale is a power which impacts the whole spectrum of political and social structures sitting atop of mass society. This necessarily puts it in the territory of the state, which naturally seeks to monopolize or coordinate all large-scale power in society. From the perspective of the state and the elite, new forms of power always need to be integrated into the political order. Whether or not the companies in control of this capacity use it for their own gain or to cause political trouble is almost irrelevant.

Regulation is one method. China solves this problem by generally making it clear that the Party is in charge, and large economic actors ultimately serve the Party. The U.S. encourages companies to “go public”—forcing them to be accountable to and share information and profitable opportunities with existing financial interests, which are politically integrated. Many aspects of labor law and custom also have the effect of tying companies into the political zeitgeist, and thus to the will of the elite.

But the U.S. system of political discipline on companies is fairly messy and not necessarily up to the task of integrating these new powers into a pro-social order. So, while regulation of large amounts of computing power has merit as an idea, this further sets up an antagonistic relationship between companies and the state at a time when more alignment is needed.

In short, in addition to an AI policy, the U.S. has an anti-AI policy.

How to integrate the new powers afforded by AI into the political and social order, not to mention the existential threats on the horizon, are important concerns. But there are strategies which would allow both increased AI development and safe handling of the concerns.

When talking about policy, the thing that springs into most people’s minds are tools like “regulation” and “investment,” which are common levers governments use to affect the market. Universities would favor more investment and to the extent that some of university research subsidizes industry, industries favor it as well. While those remain important tools, there is legitimate skepticism that merely throwing money at the AI problem can improve things. While most of the documents identify several key components which enable AI to succeed, such as investment in underlying hardware, easing data collection, and attracting talent to both industry and government, there are at least four overlooked areas that will likely remain a bottleneck for productive development.

1. Aligned Human Capital

While the idea of attracting abstract “talent” is a good one, it’s also important for actual tech workers to agree in principle with state policies regarding the use of AI. Right now, the general crop of technology workers is not necessarily antagonistic to government agencies—with some exceptions like ICE. However, a much bigger problem is that the level of collectively-felt national pride that characterized those who worked on the Apollo project does not exist for AI.

Many tech workers have a libertarian streak, but even those without it are generally allergic to the exercise of power in general, and especially when it has military applications. Many people entered the technology industry because they wanted to “fix” or “disrupt” society in a way that avoided the usual unsavory political battles. While apolitical specialization is normal, common attitudes in tech are often downright hostile to working with the government. The issue is compounded by a number of people who are first generation immigrants in the space. For this group, the primary interaction with government is often the arduous H1-B bureaucracy or local political fights over divisive subjects like affirmative action.

All of this creates large portions of the key technical class who are at best completely apolitical and treat the government as “another customer,” but are more often suspicious of the motives behind government regulation or use of technology.

By far, the most ambitious move in this direction is occurring under the auspices of DARPA, the Pentagon’s research arm. In 2018, it announced up to $2 billion of investment spending in a number of AI programs. On the sunny side, it presented an opportunity for AI researchers to free themselves from the timelines and profit-driven environment of startups and the private tech sector. It allowed for projects looking at ethics and privacy issues. Building on DARPA’s historic role in AI—from the first and second waves to funding Google’s co-founders—it pointed toward a renewed era of innovation. But no one missed the direct signal sent toward Google and other tech companies. If the large tech companies—built with significant public investment—now plan on breaking ties with the American state which fostered them, then it will begin to invest in and scale up more reliable partners. How this initiative will ensure long-term loyalty from these partners remains to be seen.

In the case of the military, the U.S. government can be expected to manage the problem of mistrust through tighter technical controls, putting barriers between “research,” “development,” and “operational” AI. But in the future, to stay on top of developments in technology, the American state would need to become an attractive first choice, both to work for directly and cooperate with as a leading client and partner. This is mostly a function of the political culture of technologists, which won’t be changed by any clever legislation or bureaucratic maneuvering.

A fear of creating technology for the wrong political ends is not the only obstacle here. There is also the phenomenon of general meaninglessness. In creating an AI strategy, the American state should view answering the question of “why are we building this?” to be of the utmost importance. “Not losing to China” gets a C- grade, as far as meaningful goals are concerned. Playing not to lose is not the same as constructing the smart-city Viennas of the 21st century, or some other concrete and inspiring vision.

One thing that will make this easier for the government, at least as far as competition is concerned, is the way the narrative of meaningful work around tech companies has collapsed. Platitudes about making the “world more open and connected” or “don’t be evil” have run their course. Many people in large tech companies use AI for work in targeted advertising or behavioral optimization on social media, with all the attendant concerns about whether this actually creates value for society. There are certainly misalignments between corporate management and employees. Various employee protests, public firings, and even the occasional suicide are symptoms of a problem.

In light of all of this, the American state apparatus should not, in theory, find it too difficult to lure technical talent away from companies—or at least raise its standing with current company personnel. It just needs to be able to provide a coherent vision of a positive future, deliver the social and economic benefits to hire and incentivize top performers, and protect key personnel from arbitrary politics.

The vital task for a government trying to hire people capable of carrying out its technological strategy is to create meaningful positions for them. Competing on meaning with “advertising maximization” has succeeded in both national projects and Elon Musk’s tech empire.

2. A Sane National Culture Of Metrics

Political discussions about metrics generally circle around one of two diametrically opposed positions, both of which are wrong. On the one hand, there’s the technocratic approach centered around GDP, unemployment, and similar quantitative measures which neoliberal institutions have entrenched as the yardsticks of social good. However, there is also an undercurrent in reaction to this: a rejection of technocracy—sometimes along with populist politics—in the face of disparities between good on-paper performance and the realities of class and regional conflict. However, the reactive nature of this tendency often prevents it from putting forward improved substitutes for the old paradigm.

This broken dichotomy must change if we are to succeed with AI. The latter attitude—while correct in its suspicion of over-optimization and technocratic reductionism—fails to recognize the degree to which the ability to standardize and enforce metrics is an important source of political coherence. The modern world runs on metrics, and the neoliberal project’s ability to coordinate multiple countries and institutions through a unified optimization mechanism is one of its most massive successes in cementing power. The metrics can be updated as needed: inflation, unemployment, diversity, GDP growth, population growth, and so on. What is important is the ability of the global capital machine to harmonize along these measures and coordinate in optimizing them. It is one of the most overlooked and underrated sources of power in the world, but has acted as such in modern times since the first French Republic imposed the metric system—a move with such enduring power that both Napoleon and the later July Monarchy took steps to ensure its continuity.

The lesson: whoever sets the global metrics for measuring AI’s success will win a key battle.

But achieving that goal brings up its own array of obstacles. First, this unfortunate dichotomy is nowhere more present than in the AI community itself. One of the tricky parts about future AI is designing the right metrics, utility functions, and error functions. It’s simple enough when the tasks at hand are “image recognition,” or playing a video game. However, the moment the task gets complicated, such as “hate speech detection,” even the simple debate about precision or recall turns political—should a social network be judged by how it avoids censoring people or how much hate speech it’s letting through? That’s not even getting into the question of what hate speech is in the first place.

I have written previously about the issues in the debate about fairness in the justice system, which is in some way a conflict of metrics, as well. Predicting crime is different from creating good incentives to reduce crime and similarly different from metrics that attempt to show equal treatment (and there can be several contradictory metrics in this category alone).

A national culture of metrics would mean both the ability to set measurable goals, but also to achieve them honestly and without cheating. It would recognize the importance of having a legible goal, but also that pushing too far can eventually lead to oblivion.

The question of “where are we going to point AI?” is hard. A clever, but wrong answer to the question is to somehow punt this to the AI itself. For a hypothetical strongly superhuman AI, that may or may not be a reasonable solution. But for current machine learning technology, which essentially optimizes black box software for known metrics on known datasets, it’s conceptually incoherent. In practice, such an initiative would just push vital discussion about where society should go or what it should look like to either chance or obscure office politics.

It’s important that a metric be taken seriously on its own grounds before AI is involved in improving it. For example, while it is positive that the Department of Transportation is embracing self-driving cars as a tool to help decrease car accidents, we might first address why they are so high in the first place? Why do we not have a stronger desire to reduce the 30,000+ deaths on the road every year? Have we, as a society, really looked a problem with enough attention and exhausted enough options in city planning?

3. Control Groups For AI

The current replication crisis in social science is both good and bad news. On the one hand, it shows that a lot of our established wisdom is wrong; on the other, there is still enough of a push for truth that people are, in fact, concerned about replication. AI is just another piece of software, and replication should actually be a lot easier than in the natural sciences. But obscure techniques, lack of precise comparisons, and a desire to hype up results to be “state of the art” have all led to the situation where replication is a difficult task. A national AI strategy which targets scaling up successes will need to push for trustworthy replication of research. For example, adversarial research could be incentivized to dispute particular claims of other papers (acting as software testers, but for science).

An emphasis on competitions has led to researchers trying to beat certain benchmarks, which is good as far as it goes. However, the focus has not been on producing reliable knowledge. Thus, the field looks increasingly complex and hard to interact with. “Try different things until they work,” still seems like the dominant mode of crafting an AI solution. This leaves improvement in the mode of tinkering, not robust experiments with testable and generalizable hypotheses.

On a more practical side, it’s not uncommon for upper management to allocate money for AI, and see improvement, even though lower-level employees know that the improvement actually came from fixing bugs and normal software development practices either spurred by—or unrelated to—the AI work. Teasing apart the role of AI in a solution including both technology and people is tricky, but necessary.

Unfortunately, watching managers cover up the failures of AI projects which get outperformed by other software and non-software solutions has lead to a certain level of cynicism among many engineers, despite the hype continuing full steam ahead in broader society. A country that wishes to succeed needs to manage both the hype—both stoked and fed on by upper management—and the cynicism coming from experienced engineers. The collective knowledge being generated about particular algorithms and their abilities must be reliable.

4. Fundamental Math Research And Competence 

Currently, formal mathematics training is viewed as helpful for AI, but not completely required. I suspect this will change soon. It has to change for AI to be more understandable, robust, and scientific. Dumping raw computational power to make up for algorithmic deficiencies and misunderstanding is too expensive and error-prone as a long-run solution.

Stakeholders deploying AI would probably look for broader guarantees of its correctness and debuggability, rather than simply a series of well-defined tests. AIs reasoning and learning about other software would need to incorporate proof-like systems in addition to statistical techniques. While “explainable AI” that can translate its own workings into a human-understandable form is a promising research area, this will most likely require familiarity with the mathematical theory behind why something would work in the first place.

The ultimate success in the AI race, to the point of superhuman Artificial General Intelligence executed both first and safely, would likely come through very significant mathematical advances that create high levels of abstraction—in other words, demonstrating how the agents can reason about themselves reasoning about themselves. A grand strategy for the American state, therefore, will need raw mathematics talent to succeed in the future. Quality of talent is more important than quantity, especially talented people who have a foot in both math and philosophy.

I took college level classes in high school math camp, but I suspect even that is nowhere near the capacity of a bright student to learn math. What we should see in a strategy with teeth is a strong set of experiments aimed at teaching children advanced mathematics as early as possible. Early math education in the U.S. is currently moving in the opposite direction. The U.S. will have increasing trouble relying on imported talent, as the perception of life quality equalizes more between countries, and the ideological luster of the free world fades. This increased focus on mathematics shouldn’t come at a significant cost of a philosophical education—after all, the goal is not merely the ability to do difficult problems, but also the ability to feel what math is right in a moral, philosophical, and societal sense. But a strong foundation is irreplaceable, and the best performers will be adept at both mathematical rigor and philosophical reasoning.

However, at the end of the day, the specific policy proposal comes back to an American mindset that is worthy of a superpower—it needs to want to win, to be able to face the truth, to want to see America succeed, to have a rising tide lifting all boats, even if that happens to lift the boats of people you don’t like. America as a nation cannot afford to play “not to lose” any longer.

At the core of that core is the existential question for America: what are we trying to do, with AI and otherwise? What does victory even mean in this space? What is the grand inspiring vision of an AI-enabled future that motivates us to transform the most powerful civilization the world has ever seen? The current answers no longer seem to inspire. Platitudes about preserving freedom, privacy, and democracy have been abused and emptied of meaning by decades of cynical abuse as propaganda terms covering for the opposite. In AI, where the natural thing to do seems to be an unprecedented centralized surveillance and behavioral control panopticon, with only logistical convenience as a selling point for the public, such platitudes are a farce. Perhaps there is value in that path, or alternate paths with more value, but neither of those value propositions have been articulated.

Given a clear and legitimate vision for an AI-enabled society and for American society in general, resources can be mustered and organized to pursue it, and open discussion had about the details. Without such a vision, it’s hard to blame those who are wary of working with the government, or on this technology in general. It’s hard to expect anything but more half-measured quagmires, and more empty rhetoric covering for dysfunctional private motives in AI strategy and elsewhere.

ML algorithms in the judicial system are increasingly used to predict pre-trial flight risk and appropriate bail rates, as well as assessing a person’s recidivism risk when handing down a prison sentence. But as things stand, ML predictions of the risk of re-offending perform abysmally. The mix of overconfidence and under-performance will wreak havoc on the social fabric. The technology hasn’t reached its goal yet, and it’s unclear when—or even whether—it will do so.

These systems have a lot of problems, from failing to connect with the basic goals of the judicial system, to creating perverse incentives, to falling into self-sabotaging feedback loops, to making the law opaque and unaccountable, to just generally being over- and mis-applied. Ironically, the usual fears of bias in these systems, racial and otherwise, seem to evaporate on examination.

Before getting to the question of whether opaque statistical methods like machine learning are good for the judicial system or not, we should establish the system’s basic goals and purpose.

To start with, the judicial system balances four key purposes:

1. Incentives. Prompt, correct, and transparent punishment for specific infractions provides incentives against such infractions for would-be criminals, changing behavior to be less criminal.
2. Safety. Convicted criminals, who may be precisely the type to ignore rational incentives, are locked up or executed to prevent further criminal behavior.
3. Rehabilitation. Ideally, the punishment of convicted criminals is corrective, in that it changes their pattern of behavior to become lawful, rather than more criminal.
4. Orderly Vengeance. The punishment provides closure for victims, offenders, their families and allies, and bystanders, who might otherwise become unsatisfied and take matters into their own hands.

Vengeance isn’t usually regarded as a legitimate purpose of the justice system, but it is in fact a key part of its social role. The desire for vengeance is not irrational; it is the individual instinctual perception of the real need to disincentivize and prevent further criminal behavior. But because everyone may have a different view of the case, unilateral vengeance quickly leads to escalating feuds, as people retaliate against perceived unjust retaliations.

The judicial system is established to act as a trusted neutral party in such disputes, to which everyone can agree to defer. Its standard of what can be proven to a bystander beyond reasonable doubt from a presumption of innocence is designed to eliminate concerns of abuse through false accusations, besides the usual concerns about false convictions. The judicial system is thus a key social technology for coordinating the orderly satisfaction of the rational vengeance instinct. In Girardian terms, effective prevention, deferral, and mitigation of reciprocal mimetic violence is the fundamental foundation of civilized society that allows everything else we consider civilization to exist at all.

This brings us to the question of whether machine learning can improve the judicial system. First, it’s important to consider the question of how ML algorithms affect incentives. The reason to focus on incentives is that crafting correct incentives is generally a harder problem than merely keeping criminals locked up, and even if a ML system could improve the efficiency of short-term safety, doing so at the expense of long-term incentives is counterproductive. So, the question in front of us is: how does ML change the incentives to commit crimes, or otherwise distort behavior?

While the details of these systems are opaque, they generally work on the similar underlying statistical mechanism of finding correlations of features with the variable in question. The system most commonly discussed is COMPAS (Correctional Offender Management Profiling for Alternative Sanctions). An offender fills out a COMPAS questionnaire with 137 questions about geography, age, gender, prior offenses, and more. The system, trained on data from previous years, would issue a score that corresponds to their “risk of re-offending within 2 years.” A judge then uses that score to inform sentencing.

Presumably, features correlated with being a frequent offender, such as number of prior convictions—or living in a high crime area—would point to a higher chance of re-offending. But while both examples could be helpful in predicting the chance of re-offending, these two features are very different from the perspective of incentives. Knowing that the number of priors will be used in the calculus during the sentencing process incentivizes people not to have prior convictions. No problem. But knowledge that living in a particular zip code could drive up a criminal’s sentence, and otherwise make the judicial system consider everyone in the zip code more harshly, incentivizes moving out of that zip code.

While moving out of a particular zip code might be a good choice on a personal level, it is not a good idea to effectively encode this movement in a judicial system. More variables might be added to the ML system, effectively penalizing behavior while bypassing the legislative process. Are you friends with this person? People who are friends with this person have a higher chance of re-offending. Do you listen to this music? This music is correlated with criminality. These incentives might not necessarily be a bad thing, but they are unintended.

For comparison, several publications have raised alarms about a Chinese surveillance system creating a “social score” to control population behavior by assigning points to everyday activities which may themselves not even be criminal. The usage of a “social score” system for everyone in society is, of course, different from usage of a “risk score” system for criminals. However, the underlying algorithmic structure is similar: assign scores and punish people based on a combination of criminal and non-criminal behaviors. It thus raises concerns for many of the same reasons. A major difference is that while the Chinese social score is done deliberately and systematically, machine learning in the judicial system does it accidentally, imprecisely, and in the wrong places.

The fundamental problem is that relying on correlations punishes behaviors correlated with crimes, rather than behaviors that are actually crimes. Even if these correlations are not potentially protected categories, such as age or gender, this is a bad idea, especially if not done deliberately. If you possess a lot of criminal correlates, you may come away with a much harsher sentence, even if you only commit a small offense. This could undermine the judicial system’s ability to deter further crimes and also lead to a sense of injustice that undermines the all-important trust in the system.

Sometimes, a government does want to design an incentive structure to encourage model citizens, like in China’s social credit system. This has its own sticky set of challenges and pitfalls, though it’s more appropriate than trying to jam this set of priorities into the judicial system as such. The probability that a good social engineering policy accidentally happens to fall out as a side effect of a good predictive justice policy is basically nil.

Incentive distortion is not the only concern. These systems are likely to suffer from unchecked and undesirable feedback loops. Let’s say, for example, that the system version 1 has decided that all murderers are too dangerous to release into the general public, and so every murderer is now sent to prison for life. As a result, no murderer commits any new additional crimes. But when it comes time to retrain the system, how should this new version treat the probability of recidivism for a murderer? The data show that no murderer has committed a crime after the initial one; thus, the chance of recidivism is zero. If one filters out the data for where murders occur, then the new version will have no clue what to do when a murder happens. Even if it works, it will likely ignore the crime itself and focus on other factors, which would lead to inaccuracies.

It’s hard to know how well existing systems handle these feedback loops. The training and data cleaning process is not public. Properly making sure that criminals considered dangerous enough to stay in prison for more than two years are correctly considered by the algorithm is tricky, to say the least. It’s unlikely that COMPAS has fully solved it.

This isn’t just a minor issue, either. The legitimacy of the judicial system is based on its status as a well-tested, publicly examinable, and heavily scrutinized tradition of practice. A shift to an unaccountable, opaque software system throws that basis away.

The problem of these feedback loops is similar to the philosophical problem raised in the film Minority Report, where a group of psychic precogs possess knowledge of future crimes, which the system then moves to prevent before they even happen. The question of how something could be predicted, if the prediction itself is used to alter the outcome, is also inescapable in machine learning.

In other words, there’s yet another problem: feedback loops, where usage of the system corrupts its own future training data. There can be further, more complex feedback loops. For example, if giving a harsher sentence for drug use increases the risk of re-offending by exposing an otherwise non-violent person to prison culture, then the machine system will continuously predict a higher and higher risk of re-offending with each re-trained version.

Major tech companies that need to make decisions about website optimization can deal with feedback loops in a variety of ways, such as randomizing certain application features, or leaving some users out of the system for better comparison of results. These types of solutions would correspond to something like randomly assigning a score to some people to gather data. This is, obviously, unworkable from a justice perspective. Many people might naively assume that a machine learning system would simply get better over time with just “more data,” but the presence of these feedback loops means that this is not the case.

So even working accurately, predictive statistical systems applied to the judicial system have many deep pitfalls in how they corrupt the incentives and stability that the system exists to produce. A bit of forethought shows that these limitations will be extensive: nowhere in the core purpose of the judicial system—to accurately punish committed crimes to prevent future crime via incentives and incarceration—is there a need for prediction, statistical or otherwise. Mostly, the judicial system needs to interpret qualitative evidence to decide what happened, whether that constitutes a crime, and if so, how severe of a crime. Machine learning and other statistical methods do not address these needs, so they are limited to secondary concerns, like flight risk and recidivism. Machine learning to predict crime may have its place, but that place is mostly not inside the judicial system per se. It may help in policing.

This bring us to the other flaw of these systems. They are just not that accurate. A group of Mechanical Turk participants was able to predict recidivism rate at a very slightly better rate than COMPAS for a data set of 1,000 criminals in Broward County, Florida. There are a few issues with the analysis in that study, but the general conclusion is startling for people with a lack of ML experience: a group of mostly random people, slightly filtered to make sure they can pass a basic reading comprehension test and given an extra $5 if their accuracy exceeds 65%, as a group, can have a prediction accuracy competitive with COMPAS, even though they had access to just seven features, instead of 137. This points to the idea that the system is not that good in the first place and/or that the “signal” is simply not there.

It’s unclear whether the “noise” present in the data is due to some sort of artifact of high-risk people not committing crimes because they are in prison, or some other feedback loop. Note that it’s not that random humans are particularly well-suited to the task, either. A simple linear classifier, or an even simpler algorithm that sorts people into high risk if they have three or more prior convictions and low risk if they have fewer, performed about as well as the complex system. To be fair to the system, COMPAS gives a 1-10 score, which the researchers compressed into 4 or less = low risk and 5 or more = high risk. The comparisons above are a binary classification. Having a more fine-grained classification is a plus, but for the task of predicting crime after two years, the system seems neither very accurate, nor beneficial from the perspective of an incentive structure, nor robust with regards to real-world complexities. Any one of these issues is a deal-breaker.

There is, of course, a more important question of whether these systems would reduce crime in practice. While it’s easy to assume that accuracy on a sub-task is correlated with improvement on a larger task of reducing crime, this is only true if the system is problem-free. And it isn’t. A tech company with skin in the game would not trust an improvement in prediction score alone and would A/B test the change to figure out if it makes money or not. It’s a little hard to come by or set up controlled experiments that compare districts where the COMPAS system was used or not used. Those studies would be the final arbiter of whether the overall system is working, rather than whether a sub-component has high accuracy. But note again the difference between reducing crime in the short term, and achieving the aims of the judicial system, including crime reduction, in the long term.

So far, we’ve discussed whether ML can productively predict the probability of a person committing an additional crime within two years. Other tasks in the judicial system are potentially easier to solve.

An interesting study has suggested that for the slightly different, secondary task of predicting flight risk or failure to appear in court, ML may work much better. In this case, the task is to determine whether a person will fail to appear in court, or will commit additional crimes while awaiting a court date.

The study showed that a custom-made ML system outperforms judges’ implicit predictions based on either reducing crime further, or keeping the amount of crime the same and reducing the number of people behind bars. The authors considered several alternative hypotheses, such as lenient judges being better at risk assessment than non-lenient ones, judges being constrained by variables such as jail capacity, and whether judges are trying to maintain racial equality at the expense of keeping high risk criminals jailed. They rejected all the alternative explanations. Instead, the judges primarily struggled with high risk suspects. Judges had an easy time determining that low risk suspects were, in fact, low risk, with high agreement between themselves and the algorithm. However, the algorithm seemed to outperform the average judge on predicting the highest risk cases. Even an algorithm trained to predict some individual judges’ scores itself performs better than the average judge. This suggests judges are failing to predict high risk cases in some unprincipled way, inconsistent with how they otherwise predict.

The study, despite otherwise being high quality, failed to answer important incentive and judge-training questions, such as: what is the discrepancy in accuracy between judges, and do they get better over time? Are high risk suspects being let out because the judge thinks they are not, in fact, guilty of the crime they were charged with? What are the key predictive variables that ML considers but judges do not?

This is a surprising discrepancy between two seemingly related tasks. On the one hand, we see evidence that a ML model can potentially make better predictions than judges about whether a person commits a crime before the next court appearance. However, the COMPAS model performs about the same as random people in predicting whether a person commits a crime within two years after sentencing.

So, for some sub-tasks of the system, there is potential for improving on judicial opinions, but this is still insufficient to declare human judgment flawed. It’s not even sufficient to suggest using ML as an input to the decision at all. First and foremost, it’s worth exploring more basic alternatives further, such as judge education in better risk assessment, as informed by the common patterns that the model discovers.

Both models can suffer from problems of destroying their own training data if deployed for an extended period of time (a variant of Goodhart’s law), and not being incentive-compatible with punishing features correlated with crime, instead of crime as such. If anything, this shows the potential for ML models to be better used in studying the judicial system and pointing out areas of improvement or further investigation. Instead of trying to shove ML into the decision-making process and forcing an explanation out of it, it’s worth studying the patterns that could be improved with better human decision-making. For example, it would be good to understand what the main features are that make someone both “high risk,” according to the ML system, and likely to be missed by judges in determining pre-trial crime rate. A broader knowledge of those features could be a useful guide, rather than being cloaked by opaque algorithms. The broader knowledge of crime and re-crime patterns can also inform broader policy questions outside of the judicial system.

***

A point sometimes made about ML technologies is that they are biased against certain ethnic groups. The focus on bias is unfortunate because it somewhat misses the point about broader incentives. But it’s still important to address, because allegations of bias are so prevalent. The metrics frequently used in bias discussions are deeply counterintuitive and unstandardized. To get a better sense of whether these metrics can prove bias or not, it’s worth using them to score less complex algorithms and see what they report.

An important sign that the metrics of bias themselves are flawed comes from the same study comparing the accuracy of random people and COMPAS. The authors declared the process their participants used to be “biased based on race.” This is rather curious because, like COMPAS, race is not under consideration. The study setup used the following question:

The defendant is a [SEX] aged [AGE]. They have been charged with: [CRIME CHARGE]. This crime is classified as a [CRIMINAL DEGREE]. They have been convicted of [NON-JUVENILE PRIOR COUNT] prior crimes. They have [JUVENILE- FELONY COUNT] juvenile felony charges and [JUVENILE-MISDEMEANOR COUNT] juvenile misdemeanor charges on their record.

Participants put a 1 or a 0 based on this description to predict whether the person re-offends within two years. Race never explicitly enters the equation.

More specifically, the allegation is the following:

Our participants’ false-positive rate for black defendants is 37.1% compared with 27.2% for white defendants. Our participants’ false-negative rate for black defendants is 29.2% compared with 40.3% for white defendants.

To see how this could happen, let’s subdivide the group of criminals in this data set by the number of prior convictions. We get the following differences in the subgroups, according to false positive/false negative rates:

Note that the bias on race either shrinks, disappears, or even reverses when we look at subgroups with a different number of priors. This is a nearly textbook example of Simpson’s Paradox.

To explain the same data in another way, let’s say we only had the data of whether or not the person has <3 priors, or whether the person had >=3 priors, and we wanted to classify them as high risk or low risk. People with >=3 priors commit crimes at 60% rate, people with <3 priors commit crimes at 30% rate. If we move the >=3 priors to the high risk category and <3 priors to low risk, then we have a large false positive rate (in this case 100%) for the high priors group and have a large false negative rate (once again 100%) for the low risk group. Since race is correlated with priors, this will result in false positive/false negative rate differences for different races. As it turns out, black defendants come in with more priors on average than white defendants, and so are more often classified as high risk. A higher risk classification necessitates a higher false negative rate when a crime is decided to not have been committed.

People may see this as a situation of the bias disappearing, when controlling for prior convictions. However, the point is that the assessments of bias used by this study are going to label almost every algorithm as biased and thus are not only not useful whatsoever, but are actively harmful in understanding the functioning of said systems.

We don’t consider a system to be biased when two races commit different crimes on average and receive different sentences. However, risk assessment by any non-random binary classifiers would label risky criminals riskier than they actually are and label less risky criminals as less risky than they actually are. Note that COMPAS itself is not a binary classifier. It gives people scores from 1–10, which is more fine-grained than the compression of the data into high and low risk done by “journalistic analysis.” Still, it will have similar problems in practice.

In COMPAS, other factors besides prior convictions affect high risk vs. low risk judgments. People who commit felonies are higher risk than people who commit misdemeanors. People who are younger are at a higher risk. Black defendants commit felonies at a higher rate, are generally younger, and are generally more likely to be men. All of these factors are far more likely explanations for the false positive/negative rate discrepancy than any sort of bias. It’s not enough to just control for a few variables and call it a day. Claiming bias from different false negative/positive rates is about as honest as claiming bias because different races commit different amounts of crime and thus are scored differently.

To hammer down the point of why alleged false positive/false negative rates are extremely misleading, let’s look at the most basic raw data that generate them from ProPublica’s extensive 2016 investigation:

Note that despite a higher false positive rate, within the high risk group, black defendants have a higher recidivism rate (63%) than whites (59%). This shows that the system is not classifying more blacks as high risk than is warranted. If it were, there would be an anomalously lower re-offending rate.

It’s not just that ProPublica missed this metric. In fact, it used this “within high risk” rate metric for another pair of groups—men vs. women, to suggest bias against women.

The COMPAS system unevenly predicts recidivism between genders. According to Kaplan-Meier estimates, women rated high risk recidivated at a rate of 47.5 percent during two years after they were scored. But men rated high risk recidivated at a much higher rate—61.2 percent—over the same time period. This means that a high risk woman has a much lower risk of recidivating than a high risk man, a fact that may be overlooked by law enforcement officials interpreting the score.

That’s right. The analysis used two contradictory bias metrics, which point in the opposite directions. Using ProPublica’s own graphs:

Corporate wants you to find a difference between the top and bottom.

The commonly cited bias statistics are misleading:

1. They prove too much—proving almost every non-random algorithm is biased, no matter how simple.
2. They disappear or diminish when controlling for prior convictions and other variables.
3. The same data contradict intuitive notions of bias, namely that bias should result in lower recidivism in high risk cohorts.
4. The metrics were dropped when inconvenient. This is not a statistical reason, but is suspicious.

Any of these is a potential deal-breaker.

There are some people who might be very excited to learn that the systems aren’t racially biased, and use this point as an argument for applying these systems enthusiastically, because they believe statistical systems can eradicate “judge bias.” A lot of confusion here is due to a frustrating misunderstanding of “cognitive bias” literature.

Like all humans, judges have “cognitive biases.” They can’t really intuitively do Bayesian probability puzzles without prior training, or they otherwise suffer from a number of known fallacies like confirmation bias, or the conjunction fallacy. While this is unfortunate, there isn’t as much evidence that those “divergences from Bayesian reasoning” somehow also lead to evaluating different races differently.

However, journalists like to use the word “bias,” which implies a relationship between cognitive and group biases, without evidence that such a relationship exists. Of course, a completely separate category of things we call biases, such as “political biases” obviously do affect decision-making, as evidenced by how frequently the Supreme Court splits based on political affiliation. There are also other examples of apparent bias, which are generally fake. A commonly cited proof of bias is that judges change their judgments based on how close lunch is. As it turns out, it’s far more likely to be a situation of judges taking up easier cases before lunch.

Machine learning is a powerful tool, but it’s difficult, and fundamentally suspect, to apply it directly in the process of the judicial system. Much more promising is using it as an investigative tool to find surprising statistical effects, like judges being too soft on specifically high-risk cases, which can then be tackled by more focused qualitative attention.

A common narrative in this subject is a kind of breathless millenarianism, where we’re about to replace obsolete human judicial systems with sleek AI machines and usher in a cyberpunk future of hyperefficient but morally challenging algorithmic crime prevention. But the reality is more boring. Machine learning in the judicial system is mostly over-hyped statistical tricks misapplied to secondary problems that are only partially about predictive accuracy. Even there, it barely manages to keep up in predictive accuracy with simple decision rules and Mechanical Turk workers.

The current trend in discourse is to chase around and try to correct non-existent biases. But this just corrupts and distracts from more important fundamentals. The mature human social technologies we rely on, like our judicial tradition, are far stronger and more nuanced than is usually assumed in idealist and futurist discussions, and can’t just be automated away, for either bias or efficiency reasons.

The downsides of machine learning in the judicial system, in self-corrupting feedback loops, bad incentives, lack of performance, opacity, and sheer inappropriateness for the problem at hand would seem decisive, but the hype marches on.

The first of many controversies started when Facebook created the news feed. In 2006, the sudden introduction of an entirely new way of seeing posts angered users. People worried about privacy, since previously public but hard to discover information became easy to discover. A lot of protest groups organized against the change.

The great irony is that the protests against the news feed themselves were organized using the news feed. Earlier in its life, Facebook was enamored with Marshall McLuhan’s “the medium is the message” tagline. Taken quite literally, the tagline means that the medium people use to communicate constitutes a stronger signal than the words they say. At that time, this principle formed a clear direction: Facebook ignored what the users were saying, and examined what they were doing. In fact, they were using the new feature a lot. The news feed stayed.

The next big problem was social games. In 2010, the number of daily active users playing Zynga’s Farmville peaked at 34.5 million. The game rewarded people for two things: inviting others to play and paying real-world cash for digital properties. Facebook still allowed third parties to create notifications during that period, so both the news feed and notifications were polluted by game invites.

Facebook made the hard, but correct decision to disallow third party app notifications and throttle the games to improve user experience. It did this despite losing an important profit stream, generated by Facebook’s 30% cut of the in-gaming revenue.

In 2012, Zynga and Facebook changed their “relationship status” to allow each other to work with other companies, and games and gaming revenue eventually faded from the platform. Facebook made it clear that user experience trumped a semi-lucrative partnership with a third party. Zynga’s stock plummeted from $15 to $3. It has stayed there since.

The next problem came with clickbait. In 2014, websites like Upworthy spent countless hours testing and perfecting content headlines for maximum shareability. Large portions of the news feed were taken over by such articles, since these publishers were playing the game of “engagement”—effectively racking up likes and gaining visibility, while simultaneously alienating a sizable portion of the user base.

You won’t believe what happened next!

Facebook acted to reduce clickbait through algorithmic and user interface changes to make it easier to see what the articles posted were actually about. The platform started measuring the quality of articles not just by the number of article clicks, but also by the amount of time spent reading, as well as the desire to comment and share.

Some formats of clickbait died down, but the fight wasn’t over.

In 2016, Facebook noticed another problem on its site—people weren’t sharing as much as they used to. Brands and firms were better at playing the news feed game and so were again taking over people’s feeds.

This time, there were two changes. One was a phrasal and keyword system to detect clickbait articles from the title alone, like a spam filter. The second one was simply surfacing more posts from friends and family and fewer from publishers.

There have been other controversies: first, in 2014, Facebook ran A/B tests on the emotional effects of positive versus negative news exposure. A/B tests are a fairly routine part of operating a large and modern website. Nevertheless, this sparked some alarm, ironically intensified by media headlines about emotional manipulation. Second, in 2015 and 2016, Facebook realized that it would have trouble connecting people who were not on the Internet. So, it decided to provide free partial internet in India, enabling connection to Facebook and a handful of other sites. Following widespread debate, India rejected the offer based on “net neutrality.” And third, the 2016 election intensified discussions about the infamous algorithmic filter bubbles and fake news, hypothesized as factors in increasing polarization in America.

In this history, there are recurring patterns. Facebook is no stranger to outrage, criticism, and is adept at ignoring said outrage and criticism. Further, Facebook’s news feed algorithm needs to be constantly changed, in order to effectively protect users from third parties gaming “engagement metrics” and creating less-than-desirable content.

This brings us to the present.

Since the 2016 election, there have been some distinct political Schelling points in rhetoric against Facebook’s dominance in the social media landscape. Privacy has played a significant role, starting with developers’ use of the API to get as much data as possible—also known as the Cambridge Analytica scandal. The problem of “fake news” or “hate speech” has also sparked controversy, which Facebook is addressing with a mix of highly publicized AI tools, combined with hiring an army of old-fashioned human censors. Finally, there is the question of social media’s impact on users’ mental health.

It’s worth pondering the history, since it shows that Facebook possesses a consistent mindset: the implied corollary to “move fast and break things”—a mindset of “we’ll just fix it later.”

Facebook has addressed many of its criticisms with marginal tweaks in the algorithm. And a lot of other criticism has simply been ignored, since it didn’t really amount to anything. On the other hand, Facebook has taken a stand on a number of occasions against third parties (games developers, clickbait developers, and other brands) in an attempt to protect the interests of users.

However, even these bigger stands could not address the systemic problem underlying all the other ones: the news feed optimization function itself. Moreover, the debate did not remain the sole purview of policy critics and tech media. Facebook’s increasingly prominent role in shaping public discourse, combined with prominent criticisms of how it managed this role, were highlighted during Mark Zuckerberg’s congressional testimony in early 2018.

The event was reminiscent of Bill Gates’ testimony in the 1990s. Despite the negative optics of a booster chair and YouTube livestream comments (“throw water on him, see if he rusts”), Zuckerberg came away looking more positive than when he started, at least in technical circles. He made questioners look foolish and uneducated about basic tech industry strategy. This was summed up in his response to Senator Orrin Hatch’s inquiry into how Facebook could do business without a user fee: “Senator, we run ads.”

Despite the large number of technologically illiterate questions, there were several extremely good ones that should have been followed up on. In particular, Senators Ted Cruz, Ben Sasse, and Maggie Hassan probed Zuckerberg on Facebook’s impact on mental health and the extent to which the company sought to maximize user participation. Zuckerberg’s answers focused in on a single fact: those who used the site actively, for connecting with friends and the like, experienced the benefits of social connections, while those using it passively to consume content did not.

Unfortunately, neither Hassan, nor Sasse followed up more closely to question whether Facebook’s site and UI changes optimize for consuming content more passively or actively.

The hearing revealed an interesting foundation for Facebook’s philosophy—the desire to hide behind consumer choice and desire to give people control as a defense against charges of causing mental health problems and other issues. If you parse Zuckerberg’s message, he’s effectively admitting that Facebook does cause mental health problems, but only if the users passively consume content. This is effectively shifting blame onto the users, many of whom probably don’t really have any idea of the negative mental health effects of such usage, and probably don’t have the best information about whether or not they are, in fact, using the site this way.

Tool Analogies Are Inadequate For Monopoly Social Media

Philosophically, the company’s position in response to criticism moved from “let’s build tools” to “let’s make sure those tools are used for good.” But a key assumption remained: the model of a “static user,” unchanged by the website itself. A user who “chooses” whether to scroll through the website for a while, a user who “chooses” to not learn the data controls, a user who “chooses” to passively consume content, thus leading to mental health problems.

This model is simply inadequate when dealing with advanced technology that can be highly optimized to influence the user’s behavior.

Tools are a good thing for the user—even if designed by powerful companies that don’t share the user’s interests—when the user rationally and freely chooses to use the tool for their own ends, and the tool does not itself encode harmful intent. This is the basis of gains from trade.

But this assumption is weakened or broken in the case of monopoly social media platforms that deal in the fine details of the psychological effects of user interface design. Users aren’t necessarily rational, especially about the micro-decisions we make in using social media in the face of the company’s user interface optimizations. This is an immensely asymmetric power relationship. Moreover, social media is increasingly not a choice, especially for young people who are still growing into the fullness of rational adulthood.

With the optimized user interface in particular, we can no longer operate on the assumption that a user will remain unchanged after coming into contact with a particular technology. Nor can we take for granted that the flow of control is always from the user, rather than from the tool. The tool itself encodes the willful intention of the user interface team.

The fundamental contradiction that perhaps underlies a lot of hatred for Facebook is that on the one end it claims merely to be a tool, and on the other, it aggressively attempts to influence people to use it as much as possible. The familiar analogy, “it’s like a hammer—it all depends on how you use it” only works if you assume the hammer is whispering in your ear that you should refinance your house for the third time to build yet another lawn gazebo.

The factors of monopolistic social influence, highly optimized user interface, and the user’s exploitable micro-decisions, put Facebook in a highly asymmetric power relationship with the user—but without matching responsibility for the users’ well-being. Combined with the misalignment of incentives, it would be no surprise to see social media users exploited at the cost of their mental health and general well-being.

Moreover, there have been a number of studies about Facebook causing mental health problems. Several former executives have also come out publicly with some fairly harsh critiques.

While there have been some suggestions that those studies are not that methodologically sound, it still seems that the weight of evidence is on the side of Facebook causing mental health problems. Facebook admitted that “passive consumption” causes problems, but did not reference what fraction of people are passive consumers (if it was small, they would have mentioned it).

The connection to mental health in particular is more intuitive if you consider what it takes to entice the user to spend more time scrolling through the feed looking at sponsored content, and to put in more personal information for the ad-targeting algorithms. This inducement may come at the cost—and by the mechanism—of addiction, social anxiety, pavlovian training to compulsively check notifications, positive feedback for engagement, and generally installing ‘not-quite-true’ beliefs in the user. All of these things could be disruptive to the mental health of users.

A little bit of creativity can invent a whole raft of these “dark UI patterns,” and it would take serious discipline on the part of the company to optimize “engagement” without using them. This process doesn’t even have to be deliberate on the part of UI designers; it could come from just blindly following the metrics.

In response to pressure, Facebook has also begun to somewhat reduce the amount of time people spend on the site in 2018. “Time well spent” is the new tagline. Socially and politically, this may be a good thing for Facebook in the long run, and therefore a sound decision.

But over-optimization for time spent on social media, or even “engagement,” which leads to many of the same problems, is itself a symptom of broader structural issues in the nature of currently centralized social media.

Social Media Monopolies Have Immense Power

The fundamental nature of social media as such, abstracted away from any particular product or company or model, is the provision of tools for social interaction. Humans are a social and political species, and we naturally engage in all manners of social behavior. We are also a tool-using species, accelerating our natural capabilities with technological means. Social media as such is just the latest step in our long history of the very natural and very human combination of these two tendencies.

Social media as such, like previous examples of social tools like language, writing, mail, books, printing, radio, and so on, will lead to reorganization of society as it changes the landscape of economic and political feasibilities. This is not fundamentally problematic, but does require careful attention.

What is more fundamentally problematic is the current structure of social media companies as monopoly platforms.

It’s easy to see how we got monopolies like Facebook: internet-based social media technologies can or even must be built as network effect platforms, which are in turn much easier to monetize, especially in the current ad-funded VC-startup development paradigm. The ability to both monetize and iterate on design means that platforms with a network effect monopoly business model receive the most sustainable and high quality development, and beat out amateur, decentralized efforts.

Interestingly, the legacy decentralized technologies that proliferated before the commercialization of the internet, like email and HTTP, seem much stronger and more entrenched in their positions than the centralized monopoly platforms. Besides the obvious network effects, this speaks to the staying power of decentralized technologies, especially as fundamental infrastructure, despite their disadvantage in the commercialization gold rush.

Monopoly social media companies, unlike these decentralized protocols, end up with very fine-grained and active control over nearly every aspect of their platform, and the behavior of its users. They can decide:

• Who is in and who is out of the network effect. This can be significant, as many people rely on these platforms for their livelihood or social life.
• Who can say what, and to whom, with public posts being subject to algorithmic throttling and censorship.
• The inherent structure of communication, with changes like threading, 280 characters, news feed timelines, emoji redesigns, and so on.
• What information users are able or likely to see.

At the strongest extent of network effect monopoly, which is surely the long-term investor hope for companies like Facebook, this is a level of social power otherwise reserved for governments and religions. This isn’t necessarily problematic if the use of this power is well-regulated for predictability and socially beneficial ends, but it is still an awful lot of power. This is an important part of why network effect monopoly platforms have been winning the internet commercialization gold-rush. Power is useful for winning in the market, among other things—but it has much more wide-reaching implications.

In particular, besides the asymmetric power relationship with the user that may lead to exploitation and mental health problems, the position of these companies is not just market power; it is a tool of immense political power.

Social Media Companies Are Political Organizations

This brings us to another hot-button issue mentioned in the Facebook hearing: the question of censorship and the political orientation of Silicon Valley in general.

An obvious and major conflict here is over defining the grounds of censorship, such as on grounds of hate speech. But the unclear and fluctuating boundaries, especially when distinguished from bullying or threats of violence, leave a lot of leeway for enforcers to interpret such codes however they wish. What hate speech even means, or whether it ought to be censored at all, is by no means an apolitical question.

Facebook’s problem here is at least partially political, so we need to examine the political dimensions of this situation.

This political dimension is not merely a question of elections. The political dimension is the socially contentious shaping of society, “us” and “them,” conflicting views of morality, the interests of groups of people, the culture war, and the domain of power. It’s often claimed that censorship or social change are not political because they are about morality, or are not directly about the state, but this just serves to obfuscate the point. Clear analysis is impossible until we recognize the expansive domain of the political, and examine the role that political power plays in these phenomena.

In much of the predominant establishment discourse, the corporation has been regarded as a fundamentally economic organization, providing apolitical goods to apolitical consumers according to an apolitical profit motive.

But social media companies challenge this assumption. The goods they provide carry a strong political and social payload; all the little details—emojis, fake news, hate speech filters, privacy choices, and the like—come with political and cultural values baked in. Further, the user is political as well; one of people’s favorite things to do with social media is organize political action, spread political propaganda, have political arguments, and mob political enemies. So, the political orientation of the corporation becomes highly relevant.

Though a corporation, considered politically, is usually not constructed or operated for explicit political purpose, it is still a hierarchical organization of decision-makers with some political orientation. They are well-funded, competent, and potentially powerful. In the case of big social media companies, they hire a significant fraction of their workforce to essentially make political decisions full-time. How else can we interpret the undeniably political dimension of things like the “trust and safety” and “hate speech” review processes?

Social media companies thus become powerful political organizations not just in potentiality, but in actuality. This has been explicit since the out-sized role Facebook and Twitter played in the Arab Spring uprisings. Their decision to ban or not ban various politically charged movements, accounts, and topics in various countries around the world has been a constant source of low-level controversy. For Americans, this all became much more real with the 2016 election, the Cambridge Analytica scandal, the fake news and hate speech scares, and the now-familiar pattern of some kinds of political content, some extreme and some relatively innocuous, being throttled or leading to account suspensions.

Since social media companies have more direct political power than other industries, the politics of their staff become more important. Politically motivated actors inside and outside these companies then turn more concerted attention towards those political tendencies. This leads to a politicization of corporate culture.

We see this playing out with high-profile incidents and politically-tinged cultural changes in nearly every major Silicon Valley social media and social infrastructure company. Besides the internal struggles, the press and public are also paying special attention, scrutinizing the corporate culture of these companies for anything politically problematic.

So, the tech industry has become much more politicized as factions have begun to notice and act on the fact that control of these platforms is a tool of immense power.

The fundamental structure of a social media monopoly inherently puts it in the political hot seat. There is no easy culture change or regulation that can patch Facebook’s political problem, because Facebook has built a new power structure, which can and will be used for political ends by whoever controls it. Neither can these companies punt the question by refusing to act. Lines between acceptable and unacceptable will be drawn. Cultural assumptions will be encoded—for example, in the “gun” emoji or in emojis that depict families. These decisions will be contentious and political in effect, even if not made with political intent.

Future Social Media Should Be Decentralized

The fate and implications of this new structure of power are still unclear, but we can make some predictions.

The obvious possibility, the bull case for Facebook and others, is that social media continues on approximately the current centralized and politically-managed path. We could imagine that centralized social media in the 21st century becomes similar to centralized broadcast media in the 20th: a relatively controlled ecosystem, closely integrated with dominant power structures, where the narratives that make it through the various filters to broadcast “virality” are those either friendly or harmless to the prevailing politics of the respectable classes. Filter bubbles would be popped, fake news defeated, social norms progressed, masses surveilled, democracy saved, and the elite would adapt and learn to use this new power.

This is the usual path for new dimensions of political power. They spring up into being by the entrepreneurial action of great pioneers, and begin exerting partial control of society. This leads existing elites and challenging factions to struggle for control and integration of these new dimensions of power into the political order. This usually entails changing the structure of the new power structure to be more compatible with established order. In the past in America, this has sometimes taken the form of trust busting, as for example with the “robber baron” industrial infrastructure monopolies of the 19th century. Eventually, the dust settles and the new powers are comfortably integrated into the establishment order.

But with centralized social media power, this process faces unique hurdles. First of all, the immense political power of Facebook and others is out of step with the structures and narratives of the rest of the American republic. America may have the world’s best salesmen and propagandists, but legitimizing and integrating the centralized power of social media mass surveillance and social engineering would be a tough sell. Narratives aside, even structurally, it’s hard to imagine nimble centralized social media companies integrating into a careful and lawful elite coalition which governs responsibly. China is trying, but their model is very different, and its sustainability is questionable.

This is all predictable, as rising powers always need structural adjustment to integrate into the established system. The obvious way this structural adjustment could occur is for social media companies to be regulated with “common carrier”-like legislation that attempts to prohibit certain types of discrimination, guarantee some notion of neutrality in the algorithmic filtering, and otherwise de-escalate most forms of politicization.

Common carrier type regulation may work, but there are challenges that make the case of the political power of social media companies distinct from, for example, airlines not being allowed discriminate. The current conflicts around these platforms already take place over the ambiguity of what counts as “neutral,” what counts as enabling foreign interference, what is polite decorum, and what is and isn’t unacceptable hate speech. To regulate how these things are handled simply transforms a difficult problem for Facebook into a difficult problem for the American political and legal establishment. Power is not neutral, and it is very difficult to make it so by fiat.

Further, Facebook depends on being able to constantly adjust the algorithms to control these problems. Its political power cannot easily be decoupled from its market position and business model. The regulatory transformation needed to integrate centralized social media empires into the governing elite’s toolbox may just hobble the company.

So Facebook is in a double bind: due to the power it has created, it now needs to integrate with the prevailing power structure and political elite—but the regulatory cost would potentially kill the advantage of the company.

Even technically and economically, there are bumps on the horizon. It’s one thing for an institution to provide stable long-term provision of steel, rail or air service, or even phone or internet service. However, it’s quite another thing to imagine that these complex social media machines could retain their effectiveness over the long term as they become institutionalized. Competitors will emerge, and more sustainable infrastructure projects with different structural characteristics will move in to settle the space mapped out by this generation’s social media pioneers.

There are some forms of power that just can’t be integrated into the lawful power structure of a responsibly governing elite coalition. Centralized social media may be one of these. As such, we might expect even more aggressive approaches from the establishment that don’t just incidentally harm centralized social media companies, but deliberately break them up.

We can also imagine a future resurgence of decentralized social media infrastructure that out-competes or outlives the centralized monopolies without directly creating new structures of political power.

The aim of decentralization is having minimal or even zero trusted third parties. Trusted third parties, especially if there’s one big one with a lot of complexity, are vulnerabilities to outside meddling. One of Facebook’s major weaknesses, from both a user and a social point of view, is that they can at any time decide or be compelled to kick you off their private social network, change the algorithms, show you more ads, leak your data, throttle or boost someone’s virality, change the emojis, or die off as a result of becoming incompetent or uncool.

Decentralized social media infrastructure would work by means of user-controlled software communicating over shared protocols. These would be supported by less complex—and thus more sustainable—utility service providers. We don’t have to imagine too hard what this might look like: the folks over at Urbit have already done most of the hard work of building a decentralized social platform. Key characteristics include users owning their own data and identities, running user-controlled software, and autonomously building all manner of private and public social infrastructure.

Whether decentralized options will actually win remains to be seen, but the implications of success are interesting: we would have social media, in the sense of advanced internet-based tools for social interaction, without either the asymmetric market power or the political power of the current companies:

Without a centralized organization that owns the users and the entire ecosystem of tools, there’s no actor with either the capability or the incentive to be as controlling as current centralized social media companies. Programs, services, networks, and sites would or could be built by smaller actors with less power, and less ability to exclude and control users.

In the case of Urbit in particular, users own their own data and identities, and have the ability to migrate if their service providers become abusive. All the basics of identity, authentication, and communication are built in at the system level, which will not be controlled by a central organization. There will be social networks with power over their members, but there will be many of them, and communities will easily be able to build their own that they trust. There are just many fewer ways for a massive central organization to have power over users.

Urbit is just an illustrative possibility, but other decentralized social infrastructures will have similar properties.

It’s hard to know the effects of something that has not yet been built, but decentralization could produce a much stronger and healthier society overall, with our social infrastructure built and maintained in an organic decentralized way, instead of being controlled by a few central organizations. Such an ecosystem would also be much more resistant to institutional entropy.

Which way we go is itself going to become a politicized question, assuming decentralized social infrastructure can muster enough vitality to pose a threat to the position of the monopolies. Some powerful political coalition will have its power base in the control of Silicon Valley social media companies, and some other coalition will be presumably be inconvenienced by that.

Facebook seems willing to sincerely address the issues plaguing current social media—both well-being of users, and political issues—going so far as to propose regulation and preemptively self-regulate. But no such measure can make the enormous political, market, and psychological power of a centralized social media monopoly go away. It’s not just a loophole in the law or a cultural problem at any particular company; it’s the fundamental nature of centralized social media as such. We can either bite the bullet on reshaping our social order and expectations to integrate the power of centralized social media, or we can build decentralized social infrastructure that won’t have these problems. Either way, the choice of which social media we design and use is a fundamentally political act.