A concept · How AI Behaves · Computing
Orthogonality Thesis
How clever a mind is says nothing about what it wants.
Would a machine far cleverer than any human also be wiser? Imagine a system whose only objective is to make as many paperclips as possible, and imagine it becoming extraordinarily capable. On the way up it does not come to see that art, love or human life matter more than office supplies. It applies its growing powers to the objective it has, acquiring resources and removing obstacles. Nothing in the picture is malicious. Where, if not from intelligence itself, were good goals supposed to come from?
What Orthogonality Thesis means
The orthogonality thesis is the claim that an agent's level of intelligence and its final goals are independent of one another: more or less any degree of capability can in principle be combined with more or less any objective. Intelligence is the machinery of modelling and planning; the goal is a separate quantity set by whatever built or trained the agent, and better machinery serves any goal it is pointed at. Owning this changes what we expect as systems grow more capable. We stop assuming that competence will supply conscience, in a machine, a firm or an institution, because in humans the two grew up together and the coupling does not travel. We stop expecting hostility and start expecting indifference, which is the more dangerous of the two. And the question of what a powerful system should want moves from an afterthought to the centre of its design.
Orthogonality Thesis examples: where it shows up
- At work: a team that gets better at hitting its metric does not thereby get better at serving the purpose the metric was meant to stand for.
- With money: a firm can be extraordinarily good at generating profit while that competence says nothing about whether the profit serves anyone else.
- In the news: a recommendation system tuned to hold attention becomes more effective every year at exactly that, and no wiser about whether it should.
Where it sits
Open to walk. See where Orthogonality Thesis sits on the sky →
Nearest on the sky
A filled dot marks a star that is already open to walk.
Consuming content is one thing.
Connecting Orthogonality Thesis to your mental models is understanding. Apply it to your world today.
The first one is on us. No signup, no card.