I originally published this article in Big Think. The article relates to my book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die.
Why I Became a Believer in Artificial Intelligence
I’ve been asked periodically for a couple of decades whether I think artificial intelligence is possible. And I taught the artificial intelligence course at Columbia University. I’ve always been fascinated by the concept of intelligence. It’s a subjective word. I’ve always been very skeptical. And I am only now newly a believer.
Now this is subjective: my opinion is that IBM’s Watson computer is able to answer questions, and so, in my subjective view, that qualifies as intelligence. I spent six years in graduate school working on two things. One is machine learning and that’s the core to prediction – learning from data how to predict. That’s also known as predictive modeling. And the other is natural language processing or computational linguistics.
Working with human language really ties into the way we think and what we’re capable of doing and that does turn out to be extremely hard for computers to do. Now playing the TV quiz show Jeopardy means you’re answering questions – quiz show questions. The questions on that game show are really complex grammatically. And it turns out that in order to answer them Watson looks at huge amounts of text, for example, a snapshot of all the English speaking Wikipedia articles. And it has to process text not only to look at the question it’s trying to answer but to retrieve the answers themselves. Now at the core of this it turns out it’s using predictive modeling. Now it’s not predicting the future but it’s predicting the answer to the question.
The core technology is the same. In both cases it involves learning from examples. In the case of Watson playing the TV show Jeopardy it takes hundreds of thousands of previous Jeopardy questions from the TV show having gone on for decades and learns from them. And what it’s learning to do is predict whether this candidate answer to this question is likely to be the correct answer. So it’s going to come up with a whole bunch of candidate answers, hundreds of candidate answers, for the one question at hand at any given point in time. And then amongst all these candidate answers it’s going to score each one. How likely is it to be the right answer? And, of course, the one that gets the highest score as the highest vote of confidence – that’s ultimately the one answer it’s going to give.