In the never-ending debate between AI optimists and pessimists, one thing that the naysayers have always been (fairly, in my opinion) accused of is moving the goalpost. Doing large numeric calculations was once thought to be the hallmark of intelligence, until the early computers managed it in the mid-20th century. Then it was chess, until Deep Blue defeated Garry Kasparov in 1996. But Go is way more complex, there is no way computers could beat humans at that! Until exactly 20 years after Kasparov’s loss, Lee Sedol was defeated by AlphaGo.

To their credit, the smarter cynics had long shifted their conception of intelligence from these abstract, bounded, perfect-information games in the pure mathematical space to things that more closely resemble what a typical human deals with. Things like recognizing images, driving, drawing art, and holding a conversation with another person. In recent years, even these domains have be encroached by AI systems. With releases like DALL-E, Stable Diffusion, GPT-3, and the latest ChatGPT, it’s starting to become unclear where to move the goalpost to.

Ever since the dawn of computing we have been coming up with ways to quantify or measure intelligence (starting with the Turing test formulated by Alan Turing himself), then building systems to beat those measures versus humans. But what if we got it backwards? What if the former step of measuring intelligence is actually the harder problem?

I used to ponder about the possibility that we might never be able to build a true human-level AGI, because of some mathematical or cosmic limitation (like the second law), where a system can only produce other systems simpler than itself. It isn’t that ridiculous to imagine this as reality, as many other things already work this way. No engine is 100% efficient, no communication channel is 100% error-free, no (non-trivial) program is 100% bug-free. Maybe no intelligence can produce a system that is equally or more intelligent.

It is obvious to me now that this hard limit does not exist. The single definitive proof came as a rather mundane realization, that we are able to produce intelligent systems at least as smart as ourselves: children. This is somewhat “cheating”, as young people are not considered technologically created AIs. But it at least shows that there is no fundamental physical limit preventing us from creating systems as smart as ourselves.

But perhaps the limit does apply when it comes to understanding such intelligent systems, and by extension to the measurement problem of AI. Explicit, holistic understanding is always more difficult than the initial informal, intuitive understanding. That is, if we can understand it at all, even intuitively. It feels like proper understanding requires a somewhat external viewpoint. The classic parable of the fish that cannot see the water that surrounds it comes to mind. Just like how whether our universe is a simulation cannot be answered from within our own universe, it’s entirely plausible that understanding and measuring a system’s intelligence requires an intelligent (far) greater than it.

Evidence for the difficulty of measuring intelligence is abundant. The continuous moving of goalposts by AI detractors is not even the most convincing. We need to look no further than how bad we are at measuring our own intelligence, despite years of rigorous effort. IQ and the more general g factor are the best we could manage, and while they have been shown to correlate with socioeconomic success (potentially one of the best hallmarks of human intelligence we have, in a relatively free and open society at least), the effect is relatively weak. They are nowhere close to actually quantifying intelligence.

Still, it seems unusual for the measurement problem to be harder than the creation problem. Normally in engineering, software included, a lack of understanding prevents us from being able to build it. Even Richard Feynman famously said “what I cannot create, I do not understand”. This definitely applies to most classical engineering problems, such as building a car or a website. But note that he did not say “what I can create, I understand”. There are lots of things that we can create that we do not really understand. The free market economy is probably the most salient (and clichéd) example. In fact, most distributed, emergent systems kind of fall into this category of “easier to create, harder to understand”.

AI systems of today, based on deep neural network and transformers, definitely feel more like the global economic system than a car to me. On the implementation level - things like gradient descent, backpropagation, dot-product attention - we completely understand how they work. But this is like knowing the rules and regulations of a market economy. It isn’t on the same level of abstraction as understanding what each weight value in the neural network mean, how the whole thing works, or why it behaves the way it does. Training a deep learning AI system based on a particular architecture feels very much like setting up the rules of a market economy and letting it run freely.

If the measurement problem is indeed harder, what implications does it have? For one I think we should be less obsessed about coming up with better Turing tests. Maybe the goalpost for AGI can be in no other form than the comparative survivability in the real human world, versus actual people. Using AI systems to gauge the intelligence of other AI systems could be the only way out of the measurement problem (this is sort of what GANs already do).

Unfortunately it also means that the AI safety problem is probably futile to tackle, as we will almost certainly build an AGI before being able to understand, measure, and control one.