
The Prism, the Pointing Arm, and What Hinton Got Right (and Wrong).
In a recent talk, Geoffrey Hinton offered a thought experiment he thinks settles a long-running argument about machine consciousness. The setup is simple. You have a multimodal chatbot with a camera, a robot arm, and language. You place an object in front of it and ask it to point. It points. You then sneak a prism in front of the camera lens. You ask again. It points off to one side, because the prism has bent the light. You tell it about the prism. The chatbot replies: “Oh, I see. The prism bent the light. The object is actually straight in front of me, but I had the subjective experience that it was off to one side.”
Hinton’s claim is that the chatbot, in saying this, is using the phrase “subjective experience” exactly the way you and I use it. Therefore the chatbot had a subjective experience. Therefore the line we draw between human and machine experience is, in his words, rubbish.
I had to think about this, because Hinton is that guy, and the example is doing more work than it first appears. But I also want to say where I think the argument is weaker than it is being sold, and where I think it is stronger.








