No Singularity

The broken math of superintelligent AI promises

When we hear of “AI”, it’s hard not to imagine science fiction: super-intelligent computers on starships, androids like Commander Data of Star Trek contemplating what it means to become more human, or the more foreboding stories of rogue AI spiralling out of control, gaining superhuman intelligence and taking over the world! But what is the science behind these stories? Is it real? And why does it often sound like that’s exactly what tech companies are building (and selling) today?

We’re going to look at the basic math behind an idea you may have heard of: the “singularity.” In AI speak, the idea is that if a learning machine can get smart enough to improve its own systems, then it can make itself smarter still, and then even smarter, and again, and again, until soon you have this infinitely smart machine. Exciting, right? But why is it called a “singularity” and not, I don’t know, the SuperSmart5000?

In math and physics, a singularity is an unreachable point, such as where some pattern quickly shoots up infinitely high and stops making sense. A physical example of this is a black hole, where the crushing density of a collapsed star makes the usual forces that hold matter apart just sort of fail and it all squishes into an infinitely tiny speck of infinitely-dense matter!

Now, there are plenty of reasons to suspect that this is not how intelligence is going to work out. But just to give us a good visual understanding of what this means mathematically, we’re going to play with the idea first.

Mathematically, we have a few options as to how we model this. We could use an exponential function. This model fits if our SuperSmartThing is getting some specific percentage “smarter” at every cycle of improvement. Even if the increase isn’t exactly the same percent better each time, this is still probably about right.

Now teeeeeeechnically an exponential function isn’t a mathematical “singularity” despite how impressive it is. There’s no point of infinite and instant growth in the same way as a black hole has infinite density at the center. But this is still speedy enough that I think it’s really what most AI fans think is going to happen. And there is a way in which it does show a singularity, if we change our thinking and graph how quickly this new SmartyPantsMachine learns.

There aren’t any real “smartness” units that we can use to make this number-precise (and we’ll talk more about that later). But we can probably get away with defining “more smarts” as “can learn a specific thing quicker.” So we can model that mathematically by treating “smarts” kind of like a speed: the faster you’re going, the shorter the time it takes to get there.

Imagine we were graphing how long it would take to drive somewhere that’s 500km away. If we’re going 50km/h, it’ll take 500km / 50km/h = 10 hrs. If we’re going 100km/h, it’ll take 500km / 100km/h = 5 hrs. In other words, we’re dividing the distance by the speed.

So to graph our time-to-learn-a-thing, we can make up some number to represent “learning the whole thing”, and divide that by our made-up “number representing how smartsypants you are.” The specific numbers don’t matter, it’s the pattern of how they change as you get “smarter” that should give us a useful picture.

No matter how “smart” we get, we can’t make it take zero seconds to learn something, but we can get awfully close. Infinitely close, in fact. Hey, it’s a singularity!

So, is this possible? I mean, black holes sounded impossible and we’ve found those, right?

Now you’ve probably guessed by this point that I’m a math teacher. Who else would bring this many graphs into a conversation? But as both a math teacher and a programmer who’s worked with these AI systems hands-on, I’ve had a bunch of time to think about how the learning process works: how kids learn, how computers “learn,” how programmers learn to make computers “learn” better, and what it all has in common. And I think our Simple Singularity above needs a bit more detail so we can find out exactly why it doesn’t make sense.

Imagine you’re learning how to do something new. Not just learning some fact, but learning a skill like, oh I don’t know, writing an essay, or solving an equation, or making a skateboard deck, or baking muffins. While there’s a lot that’s different in these examples, we can also say they have some things in common. You have to come up with a Plan For What You’ll Do To Do The Thing; you have to Try To Do The Thing; and then you have to Find Out If It Worked. I can probably save on capital letters if we give these shorter names, so how about:

1. Plan
2. Do
3. Evaluate

and if you didn’t make it perfectly, or if you notice problems along the way and adjust your technique or plan, then we also have:

4. Adjust

The interesting thing is, this is also true of how “AI” systems learn. They have some kind of ‘model’ of how to do the thing, and they run inputs through their model to Do the thing, and then they have to Evaluate if they did it well and Adjust the model based on what kind of errors it made.

Part of what makes AI systems so impressive is that if you just give them a metric frackton of example data, they can spin through this process millions or billions of times until they get their model pretty dang good. This is why AI fans are pretty sure that the whole singularity thing is possible!

But this is missing something, something kinda huge: we need two layers of this cycle.

Imagine you had some kind of super-speed power and you were going to freeze time and spend as much time as you want writing an incredible essay. You write it, and you read it over, and you see it has problems, and you write it again, and you review it again, and rewrite, over and over dozens or maybe even hundreds of times, all while the rest of the world lies frozen in time. You are in procrastinator HEAVEN, nothing can stop you now!

Finally, after the 732nd revision, you unfreeze time, take your amazing essay, and walk to class. You hand it into the teacher, and since you’re a little early they take a few minutes and read it right away.

And they see problems.

You kind of misunderstood a major theme in the novel, or misunderstood a major idea in your sources, or ... well the point is, oof. No amount of SuperTimeGod rewriting would change the fact that you needed to get feedback from somebody who knew better than you. Somebody who could evaluate whether you were actually reaching the correct target.

Computer learning is exactly like this, except even moreso. A machine learning “AI” system can optimize itself and get increasingly good “scores” ... but only if it actually has a perfect way to “score” what it was doing in the first place. Or in other words: how does an “AI” system know how to do the Evaluate step in the first place? What if it’s doing Evaluate ... wrong?

I hear you ask: come on, how could this be a real problem? They’ll just learn to evaluate better! By, uh, learning more! Like they learn! And to this I say, nononono. If you are going to learn better “Evaluate”, then you have yet another learning cycle, and you still have the same problem.

But, but but! Then how come computers can get so good at things like, being a chess grandmaster without having to study under a grandmaster? How can they learn to do things like that better than any human can?

AHA! Now we get to the good stuff.

A chess game is just rules. Physical pieces are kind of optional to actually learning How To Chess Better; you can simulate the whole experience. If something learns to win at Chess-Inside-A-Computer, they are learning the entire whole game and all they have to do to win at Chess-On-A-Board is have someone move the pieces for them. No nuance lost.

This means that chess skill can be fully evaluated in a digital, abstract way. The entire learning process can live (almost) entirely inside that orange zone:

I’m exaggerating, because you still might have to debug things, or try different model architectures, etc etc. But it’s totally possible to set it all up correctly, have it spin through that orange cycle a bajillion times, and when it’s done you’ve got a chess grandmaster.

Because chess isn’t real.

How about learning to cure cancer? Cancer is real, it’s messy, and we don’t even know what all of its “rules” are yet. If we could perfectly simulate how cancer behaves inside a human body, we would have Solved Cancer and this wouldn’t be an issue. But we can’t! We’re not even remotely close, because that would require simulating molecular interactions on a scale that makes even the largest terabyte-parameter “AI” model look like a tiny speck.

The only way we can evaluate the challenge of “cure cancer” is by interacting with the real world.

This is where the trick lies. It’s not just that today’s so-called “AI” systems don’t have any lived experience, as though all we need to do is give them more access to sensors and lived data to make the Singularity. It’s that the Evaluate step in a learning cycle depends on interacting with the thing you’re learning about. If you’re learning how to communicate in an essay, you have to evaluate it by having someone else receive that communication and see if it makes sense. If you’re learning about woodworking, you have to evaluate by actually working with real wood and see if your actions made it into the shape you wanted. If you’re learning about curing cancer, you can only evaluate your solution by doing actual trials of your cure with living beings who have cancer, or even samples of living cancer cells.

Ironically, this applies to computer learning as well! Programmers often have to tweak how a learning model works to avoid problems like it overfitting to the sample data, or failing to converge to a solution at all, or seemingly acing all the automated success scores only to find out it’s generating super biased or racist content. It’s kind of like the hypothetical teacher in the earlier example, grading the essay and finding out it’s only a C+.

Computer scientists can set up a digital learning task that will run a digital-only learning cycle really quickly for a time. Periodically, however, they need to look at the results, see if it’s working (“Evaluate”), and make changes to how things are set up (“Modify”).

Okay, we still have our hypothetically-infinitely-fast digital learning, so probably that just means our exponential growth singularity is just a little bit slower, right?

Let’s look at the graph again.

To model this, I’m going to assume that our digital learning system gets faster and faster, but our human or real-world interaction step stays about the same. Making our cancer-solver smarter isn’t going to make real-world treatment trials happen in milliseconds, no matter how many GPUs we feed it. Even if the real-world side of things did ‘improve’, it will be so much slower than digital changes that it hardly matters in this simple model.

Try playing with these two parameters here.

Hmm. Our ‘time to learn’ graph on the left looks similar to before, except it’s shifted up a bit. The ‘smartness’ graph on the right, however, looks completely different. Where did the Singularity go? Why does it flatten out instead of continuing to improve?

We’re living in the middle of this effect right now. We have seen tremendous growth in what these systems can accomplish, so much so that it’s been easy to believe that we really were seeing the start of an infinitely exponential growth in learning that would magically save us all. But as long as there’s a non-digital, real-world interaction required, the whole learning system is limited by how long those real-world actions take. We can’t infinitely improve the whole process into a “black hole” of smarts because the real-world interactions slow us down, limiting our growth until eventually it just levels off.

In summary: we can’t learn any faster than we can interact with the system or thing we’re learning about. And this is why an “artificial intelligence” singularity is impossible, no matter how we build our machine learning system.

Thanks for reading this far. I hope this helps clarify things for someone out there, and maybe, just maybe, we can help this AI bubble pop a little bit faster.

- josh giesbrecht
josh@thoughtlost.org


FAQ:

Q: What if we replace the human evaluators with other AI?
A: This is bad logic. If an AI evaluator could do the job already, then there’s nothing to learn. Again, for simple rule-based systems like games or simulations, this works, but the only way to find out if a simulated solution works in the real-world is to test it in the real world. (And you wouldn’t believe the weird ways in which learning systems can come up with glitchy kludge “solutions” within a simulation that are useless in real-life.)

Q: Can we just give AI agents direct access to real world tools and let them learn that way?

A: We could, but it doesn’t solve that real-world interactions take real-world amounts of time. If I was feeling fancy, I would say that you can’t learn anything from a system in less time than it takes for you to see the results of changes to the system.

Q: What about learning from all of the writings, observations, and data that humanity has collected? Doesn’t that count for a lot?

A: Okay, so. Sure it does. You can learn a lot about the past by processing existing media, data, and artifacts. But it’s not going to create new solutions, which is what the whole Singularity thing is promising. Like you can’t read scientific papers summarizing past experiments and then magically know what would happen on experiments that haven’t actually taken place yet.

Q: What if we just let them learn how to program themselves? Then it’s all digital, so can’t they do that super-quickly and still get exponential growth?

A: Congrats! You’ve found exactly the line of thinking that’s led to OpenAI, Google, and countless other groups pouring trillions of dollars into “coding agents”. Sure, they market it as a tool for programmers, but I’m quite sure that what they’re really hoping for is exactly this.

So what’s the problem?

Well, think about what these coding agents are able to change, and what they aren’t. It’s actually the same problem but on a slightly different level! Even if these “coding agents” end up self-improving, they’re still only improving the digital-world aspect of their own abilities. They can’t program the real world into interacting faster. They’re just improving the digital learning cycle, not removing the real-world bottleneck entirely.

Q: Can we speed up the real-world interaction in some other way?

A: Kind of. One example of this is how many “AI” companies are trying to get as many people as they can to use AI systems and rate their outputs. If the systems can learn from thousands or even millions of people all at once, then that should mean improving the learning speed by a factor of thousands or even millions, right?

The problem is, if you crowd-source your evaluation, you’re only going to get results as good as the average user is able to evaluate. You’re learning how to create what people think they want; that’s not going to get you new solutions to big problems, that’s just going to get you mediocrity that tells you what you want to hear. Oh hey, that’s exactly what these systems already do! Oopsie.

At some point, you’re going to require experts. I’ve already seen job posts where “AI” startups are trying to hire engineers in various disciplines to basically grade the output of their “AI” systems at solving basic engineering problems in their field. How many iterations of paid evaluation are you going to need just for it to do ... about as much work as you could have paid that engineer to do in the first place? And in the meantime, shouldn’t we be just a bit concerned about trusting a “good-enough” machine to solve engineering calculations that might impact people’s lives and safety?

Q: Isn’t all this talk of “time to learn” missing the point of an artificial general intelligence? Like, if it’s generally smart, then it’s smart at everything, isn’t it?

A: Aha! Good question. This is a tricky thing to make sense of because it hits the same spot as how we get kind of weird when we think of “smart people” (or the opposite). Many of us have grown up learning that some people have higher IQ than others, are smarter than others, etc. And it’s hard to dismiss this because some people do have an easier time learning certain things than others! It’s not like it’s all made-up.

The problem is that you don’t get there by just figuring out a secret formula for “IQ number go up”. In fact, even just the IQ measurement itself has a pretty ugly history; it’s got a lot of ties to eugenicists, ie. Nazi-like mindsets that want to rate some people as “more valuable” and then get rid of the “less valuable” people. (Nasty, evil stuff.)

Modern psychology, on the other hand, has found that there is some truth to measuring how well people can perform certain kinds of mental tasks, like verbal processing or spatial reasoning. But beyond that? You get smarter at something by working at learning it. And even those “natural” abilities with words or geometry can be improved through practice and training, so they’re not a thing you can only get by being “gifted” with it.

Ok ok, what does that have to do with the question? Mostly, that there is no easy road to “generally good at everything” intelligence. Nobody just wakes up one morning automatically able to do something with smartness-aura like chess or rocket surgery or whatever. They work at it - maybe with some natural advantage, or maybe with a weirdly focused sense of joy in studying it, who knows, but it takes time. And all of that learning requires interacting with the thing-you’re-learning, whether it’s through books, hands-on experience, listening to instructors, whatever.

So, no, there’s no magic shortcut by which a machine learning system can just kind of get “smart at everything”. And even if it does manage to improve some digital version of our verbal or spatial skills, that’s still just the digital-only side of a learning cycle. It would still have to interact with the outside world to advance truly new ideas or learn new things.

Q: Okay, but what if we develop new AI technologies that are better than the LLMs we use today?

A: The thing is, if you go back and read carefully, this entire argument doesn’t depend on the digital system being an LLM, neural network, or any other kind of modern machine learning algorithm. This really applies to absolutely any type of learning, artificial or otherwise. Learning speed is going to be limited by how quickly you can interact with the thing you’re learning about.