How smart could an AI become?

I got an interesting question in a comment from Jim T on my last blog.

What is your opinion now on how powerful machine intelligence will become?

Funny, but my answer relates to the old question: how many angels can sit on the head of a pin?

The brain is not a digital computer, and don’t think a digital processor will be capable of consciousness (though that doesn’t mean it can’t be very smart and help make huge scientific progress). I believe a conscious AI will be mostly analog in nature, probably based on some fancy combo of adaptive neural nets. as suggested decades ago by Moravec.

Taking that line, and looking at how far miniaturisation can go, then adding all the zeros that arise from the shorter signal transmission paths, faster switching speeds, faster comms, and the greater number of potential pathways using optical WDM than electronic connectivity, I calculated that a spherical pinhead (1mm across) could ultimately house the equivalent of 10,000 human brains. (I don’t know how smart angels are so didn’t quite get to the final step). You could scale that up for as much funding, storage and material and energy you can provide.

However, what that quantifies is how many human equivalent AIs you could support. Very useful to know if you plan to build a future server farm to look after electronically immortal people. You could build a machine with the equivalent intelligence of the entire human race. But it doesn’t answer the question of how smart a single AI could ever be, or how powerful it could be. Quantity isn’t quality. You could argue that 1% of the engineers produce 99% of the value, even with only a fairly small IQ difference. 10 billion people may not be as useful for progress as 10 people with 5 times the IQ. And look at how controversial IQ is. We can’t even agree what intelligence is or how to quantify it.

Just based on loose language, how powerful or smart or intelligent an AI could become depends on the ongoing positive feedback loop. Adding  more AI of the same intelligence level will enable the next incremental improvement, then using those slightly smarter AIs would get you to the next stage, a bit faster, ad infinitum. Eventually, you could make an AI that is really, really, really smart.

How smart is that? I don’t have the terminology to describe it. I can borrow an analogy though. Terry Pratchett’s early book ‘The Dark Side of the Sun’ has a character in it called The Bank. It was a silicon planet, with the silicon making a hugely smart mind. Imagine if a pinhead could house 10,000 human brains, and you have a planet of the stuff, and it’s all one big intellect instead of lots of dumb ones. Yep. Really, really, really smart.