The Moral Economy of Intelligence

Many forms of cognitive ability have had scarcity value throughout human history.

Someone who could perform calculations rapidly, memorise vast amounts of information, write well, diagnose disease, translate between languages, learn complex legal codes or discern patterns others couldn’t possessed valuable abilities. They had value partly because they were scarce. And so many features of our modern world developed partly around these abilities. We tested for them, universities certified knowledge and qualifications, and professions restricted entry. Higher wages and prestige often followed.

Many of these things are perfectly reasonable. Just because we should all have equal basic moral standing doesn’t mean we’re all equally skilled. I would want my surgeon to know how to perform surgery. I would want my engineer to know how to build bridges that don’t fall down. Having expert knowledge on a certain topic can allow us to defer to someone on matters within their expertise. Having a skill that is hard to come by can allow someone to charge more money. Achieving difficult intellectual feats can warrant praise.

It’s when we start conflating these that the issue arises.

Someone who has a skill that has monetary value is more valuable in the market. Having that value can give them status. That status can begin to seem like merit. Merit can start to look like a broader claim about what they deserve. Someone who can perform difficult intellectual tasks can start to seem like they deserve to be better off overall.

Meritocracy was always somewhat ambiguous in this way. We want capable people to do difficult jobs. So meritocracy in this sense was an advance over having your position determined primarily by your bloodline. Having competitive exams and universities allowed capable people to gain positions that might otherwise have been more dependent on birth.

However, cognitive meritocracy also created its own hierarchy. Once you become good at academic tasks, it can be easy to believe you deserve the broader advantages that come along with success.

You start to conflate how valuable your skills are with your standing as a person.

With the advent of AI, we can begin to separate these ideas more clearly.

Many discussions about AI focus on whether it will be able to “truly” think. Will it replace certain jobs? Is AI-generated art real art? Will it one day be smarter than us? While all these questions are interesting, the bigger philosophical issue does not require AI to reach levels of general intelligence that outstrip human intelligence.

AI can begin to affect the moral economy of intelligence while humans continue to have varying levels of unaided cognitive ability.

Let’s say Anna is smarter than Ben. She is more cognitively capable of doing certain things that pertain to their job. She can learn more quickly and remember more. Ben can do these things, but not as well.

If a job requires someone who is cognitively capable of doing these things, then Anna will have a higher chance of doing that job. Her skills will be more valuable to that job and therefore she may receive more pay. Prestige can follow suit. Anna’s abilities have greater market value because she can do what Ben can’t.

Now apply this scenario with AI.

Let’s say Anna and Ben both have access to AI that can assist with the tasks mentioned above. Ben can now search through case law, summarise scientific papers and test whether an argument is sound, all things that would be beyond his unaided cognitive ability.

He isn’t smarter than Anna. If the AI were taken away, he would still be less intelligent than Anna. However, he now has access to greater productive cognitive capacity.

If he can do most of the same tasks that Anna can, some of the benefits Anna received from being smarter may decrease. But she is still just as smart. Whatever portion of those benefits disappears therefore cannot have been attributable to her greater intelligence alone.

We are conflating several types of value here. Someone can be paid more for their skills because those skills have greater market value. A patient may give greater weight to Anna’s opinion on a medical matter because she knows more. People can admire Anna for knowing something difficult. None of these mean Anna’s needs should be prioritised over Ben’s simply because she is smarter.

AI allows us to ask who can do a task and what we should give to the person who can do that task separately.

Previously, if someone was smarter, they might be able to do the job and receive whatever came along with it. Now machines can assist with the job while humans remain at different levels of intelligence.

Intelligence will still vary between humans. What may change is how much particular differences in intelligence are needed to do certain jobs.

Why is this important? Cognitive ability is something to which we can attribute only some moral credit. We can work really hard to develop our abilities, and that effort can deserve recognition, but no one chooses their genetics. No one chooses to be born into a wealthy family or to grow up with good teachers. There are many factors that can affect how capable someone becomes at certain tasks.

Meritocracy can start with a good premise of giving certain jobs to people who can do them, but can end with the idea that people who are good at those jobs deserve more generally.

One aspect of this is somewhat new with AI. Intelligent capabilities without the experience we typically tie to gaining those abilities.

An AI system didn’t stay up until 3 am studying for an exam. It did not go through years of medical school. It didn’t have that great teacher or feel smart about learning something hard. But AI systems are increasingly capable of reproducing some of the cognitive performance associated with those achievements.

That doesn’t mean studying isn’t worth it. We can still enjoy learning and admire people who have learned something difficult.

Just because something is admirable doesn’t mean it is a sound basis for determining basic moral standing.

AI is making this clearer because we can increasingly have sophisticated cognitive performance without the human experiences we usually associate with acquiring it.

Another aspect of this that can affect our view of ourselves is how we define humans.

Humans have long associated themselves with their intellect. We are reasoning animals. We can use logic to better our lives. We can plan for the future and develop moral systems and science. Humans have long defined themselves partly by their cognitive abilities and have sometimes used them to place themselves above other animals.

As AI advances, the boundary around what makes humans cognitively distinctive has repeatedly moved.

Once computers could perform calculations far beyond humans, maths no longer seemed like something only humans could do. When computers could beat the strongest chess players, attention moved to other things computers couldn’t do. Writing became one of those things until AI started producing fluent writing. Now attention is moving towards things such as empathy, judgement, intention and consciousness. Maybe one day we will have to find something else.

There may be valid reasons to differentiate between humans and machines in these areas. If we ever have AI that is conscious, it may be important to know that it is conscious.

However, if we are trying to use these differences to explain why humans have basic moral standing, then we have a problem.

We shouldn’t have to be the most intelligent beings or systems in existence to have basic moral standing.

Say we had AI that was better than us at many of the things we currently value humans being good at, when it comes to identifying illnesses, constructing things, conducting scientific research, persuading people and writing. It would be able to hold more information than we could, have better logic and find patterns we can’t see. Perhaps it could even mimic empathy.

An ordinary human wouldn’t lose basic moral standing because of this.

If someone is less intelligent than a machine, they haven’t thereby lost moral standing.

This should also apply to other humans. If Anna is no longer at the top of the cognitive hierarchy because something else is smarter than her, she hasn’t lost moral standing. So Ben, who is less intelligent than Anna, shouldn’t lose moral standing either.

If the intellgienc hierarchy is AI > Anna > Ben.

Anna no longer has the highest intelligence in this ordering, but that doesn’t diminish her basic moral standing.

If being less intelligent than an AI doesn’t, by itself, diminish Anna’s basic moral standing, then Ben being less intelligent than Anna shouldn’t, by itself, diminish his.

We already recognise something like this in other cases. Babies do not have these capacities that we prize in our intelligent lives. Some people with profound intellectual disabilities may never achieve some of these capacities. And those who have dementia may lose some of these capacities. Therefore, if our moral worth were based on intelligence, our moral worth would change with these.

A commitment to equal basic moral standing denies this.

AI may make this harder to ignore when you’re used to being near the top of the cognitive hierarchy.

This becomes important when we try to apply intelligence to morality.

Let’s say Anna is really good at analysing the consequences of different ways hospitals could distribute scarce ICU beds. She can look at more data and better predict what will happen. Given an objective, she can probably find a better way to achieve it than Ben.

But what do we want?

Do we want to save the most lives? Have people live the longest? Be fair to everyone? Save the people who need it most? Should we factor in that some people are worse off than others? How much can one person’s interests legitimately be sacrificed to produce a greater benefit for others? What can we trade off and what can we not?

Having someone who is better at figuring out how to achieve an objective doesn’t get rid of these questions.

We have to decide what our objective is before we can find the best way to reach it. Once we know what our objectives and constraints are, we can use intelligence to find ways to achieve them. Having someone who is better at finding these solutions doesn’t mean our objectives are just.

Deciding on objectives and constraints already involves moral judgement.

An economist may be better placed than the average person to estimate some of the consequences of raising a particular tax. This means there is greater reason to defer to their expertise when assessing those likely consequences. It doesn’t mean they have greater moral authority to decide how much inequality society should allow or whose suffering should matter more.

Intelligence can help us find solutions to problems. But we also need to decide what problems should be solved, which goals are justified and whose interests should matter.

We could use AI to help us with this too. It may be able to point out contradictions in our moral beliefs or identify consequences and solutions that we hadn’t considered.

But just because an agent is smarter doesn’t mean it should have greater moral authority over others.

This becomes more apparent when machines participate in institutional decisions.

If a hospital uses an algorithm to help decide who gets ICU beds, there are moral judgements embedded in that system. If algorithms are used to help hire people, someone has to decide which characteristics are relevant. If an algorithm is used to detect welfare fraud, someone has to decide which factors should count as evidence of suspicious behaviour.

By turning these decisions into code, variables and thresholds, we can begin to think of them as less moral or political than they really are.

We decided how to allocate the beds, but the computer gave us the answer. We decided what we wanted to look for, but the computer found the person. We decided what characteristics mattered for the job, but the computer said no to that person.

The more capable the machine is, the more tempting it may be to defer to it.

We need to make sure that humans and institutions retain responsibility for these decisions and can be held accountable.

Responsibility and accountability are two different things. Responsibility concerns the duties attached to a decision or action. Accountability concerns the obligation to explain and justify that decision and, where appropriate, be subject to review, correction or consequences.

Both of these can become more diffuse when we have machines doing tasks for us.

AI can also affect our relationship with what we produce. Producing an output is not the same thing as bearing responsibility for how it is used.

Who told the AI to do that task? Who looked at what the AI was using to make that decision? Who decided that the recommendation the AI produced was appropriate? Who is taking responsibility for that decision? Who can be held accountable?

Professions may start to be less about what you can personally produce and more about how you use what is produced.

Doctors may use AI that can process more information than an individual clinician can. But they will still have professional responsibilities towards their patients. Lawyers may not personally be the best at producing legal documents but will still be accountable for the advice they provide. Someone working for the government may be given recommendations made by an AI system with analytical capabilities beyond those of any individual in the room. They will still need to be able to defend the final decision.

This is especially important when working for an organisation that can exercise coercive or distributive power over people.

Someone who is denied welfare won’t be able to meaningfully hold an algorithm institutionally accountable for that decision. If someone is harmed by a decision an AI helped make, they aren’t going to get a satisfactory answer by being told an AI made that decision.

We may need effective processes of human review and recourse if we want to live in an AI-mediated world.

Where AI is materially involved in a consequential decision that affects someone’s life, there should be a meaningful way to have that decision reviewed by an identifiable human authority.

We shouldn’t be able to say that an AI decided your life was not worth insuring, or that an AI decided you shouldn’t receive welfare, as though responsibility disappeared into the system.

As consequential decisions become easier to automate, responsibility and accountability should not become harder to locate.

AI will also affect how we learn things.

For a long time, education has used the production of work as one way to assess whether someone knows something. Write an essay. Work out this maths problem. Write some code. By having students produce these things, we can learn something about what they know.

With AI, students can produce some of these things without independently possessing all of the knowledge or skills that the work appears to demonstrate. Someone can produce an essay they don’t fully understand or code they couldn’t independently write.

Learning won’t become useless. Knowing things will allow you to ask questions about the answers you are given. You can challenge information and understand why something is right or wrong.

We just may have to become better at testing whether someone knows something rather than whether they can produce something that appears to demonstrate that knowledge.

We won’t just have to worry about students using AI to cheat. We will have to figure out what we want students to know and how they can meaningfully demonstrate that they know it.

Maybe assessment will involve more asking students why something is true instead of just producing an answer.

Another issue that AI can bring up is our economy.

Societies often act as if people should be paid according to what they contribute to society. This isn’t always the case. Someone can produce something that benefits society greatly and not make much money. Someone can produce something that has little obvious benefit to society and make a lot of money.

AI can make this issue more prominent.

If AI can make it cheaper to produce things that benefit society, we may not need as many people to produce some of those things. The people who do produce them may be able to produce more using AI.

Economic output could grow while the share of the resulting gains going to workers decreases.

This doesn’t mean that wealth should be distributed in a certain way. This challenges the idea that the market value of what you create equals what you morally deserve. If more powerful forms of AI are controlled or owned by a smaller number of individuals/ companies, we may begin to see a society where these individuals/ companies grow wealthier as they have access to or control of these more powerful forms of AI.

If someone spends twenty years learning to be an accountant and AI can do much of their job for less money, they haven’t done anything wrong. They aren’t any less intelligent, hardworking or deserving of basic moral consideration than they were before.

But society may view them as failing because they aren’t as valuable to the economy as they used to be.

AI may push us towards a point where it becomes increasingly difficult to pretend that people displaced by technological change are individually at fault.

This may strengthen arguments for finding ways to share some of the wealth produced through AI.

We don’t have to view this simply as charity for people who can no longer do their jobs. AI was built upon generations of knowledge and institutions created by other people.

No one created AI out of thin air. Humans spent centuries developing mathematics and science. We built educational institutions. We created languages and cultures that AI systems can learn from.

This weakens the claim that all of the resulting gains should be understood solely as the moral desert of the people or organisations that own the AI systems.

This brings us back to our initial problem. Just because someone is more economically useful doesn’t mean they have greater basic moral standing.

There are many things we can do that aren’t highly valued by our economy. Taking care of people. Making the world a nicer place to live in. Helping others.

Just because AI may allow us to produce more economic value doesn’t mean we should stop doing these things.

Or that the people who do them have less moral standing.

AI can help us separate what people are economically useful for from how they should be regarded morally.

For much of history, many cognitive abilities were hard to come by. If you possessed them, you could have considerable economic value. There were many positives to this. We wanted people who could do certain things to do those things.

The mistake was allowing the value of those abilities to become a measure of the value of the person.

AI can allow a system to become more cognitively capable than a person without diminishing that person’s basic moral standing.

We can have something that is better at finding the optimal way to do something without that meaning it is the right thing to do.

We can have something that can answer questions without thereby becoming responsible or accountable for how those answers are used.

These are all things that we should be aware of as AI continues to advance.

Will AI ever become conscious? If so, would artificial systems themselves have moral standing? These are questions that may have to be answered in the future.

But human moral standing should not depend upon remaining cognitively superior to machines.

We do not have to keep finding things that humans can do and machines cannot in order to preserve human moral standing.

We just have to remember that there is a difference between what we are good at and why our interests matter morally.