Ask Susan Schneider whether ChatGPT is conscious and she will tell you the question is badly formed. A program, she argues, is an abstract object, like an equation, and equations do not feel anything. If consciousness is present at all, it lives in the physical system running the code, which means the same model could in principle be conscious on one machine and not on another. It sounds like a technicality. It is not, because it cuts against both the companies hinting that their products are waking up and the critics who wave the possibility away.
Schneider is the William F. Dietrich Distinguished Professor at Florida Atlantic University, where she founded and directs the Center for the Future of AI, Mind and Society, holds a joint appointment in the university’s brain institute and co-directs a laboratory that builds AI systems informed by neuroscience. She studied economics at Berkeley and took her PhD at Rutgers under Jerry Fodor, who spent the early 2000s arguing against connectionism, the approach that became deep learning. Before Florida she held the NASA Baruch S. Blumberg Chair in Astrobiology and Technological Innovation, spending three years on the proposition that the most intelligent minds in the cosmos are likely to be postbiological; held the Distinguished Scholar Chair at the Library of Congress, working with members of Congress on AI; and was a fellow at the Institute for Advanced Study in Princeton. She writes for the New York Times, the Financial Times, the Wall Street Journal and Scientific American, has explained her work on PBS, the History Channel and Neil deGrasse Tyson‘s StarTalk, and advises Prism, the Partnership for Research into Sentient Machines. Her book Artificial You: AI and the Future of Your Mind (Princeton University Press, 2019) has been translated into 15 languages, and with the Princeton astrophysicist Edwin Turner she devised the AI Consciousness Test, a way of probing a machine for an inner life without simply taking its word for it. Her earlier The Language of Thought (MIT Press, 2011) argued for a neurosymbolic approach, the kind of hybrid she suspects lies behind today’s mathematical breakthroughs. Her most recent work, published this year in the Journal of Consciousness Studies with Mark Bailey, is Superpsychism, and she has been warning about what she calls the megasystem problem: AI services knitting together into structures nobody designed and nobody can align.
You might expect a self-described transhumanist and former NASA chair to be among the believers. On chatbots she is a sceptic. She calls today’s systems “savant systems”, brilliant and fast and built from a tonne of human work, and she reads their uncanny familiarity as evidence of training rather than of experience: they are human-like because they are made of us. The radical claims she keeps for physics. Drawing on her time at Princeton, sitting with string theorists reading papers on how space-time might be recovered from quantum entanglement, she and Bailey ask what follows if the panpsychists are right, and land on a layer of reality outside space and time in which consciousness is an ingredient. Their answer turns the question of what is conscious into a mathematical one. And when I asked what she would say to the chief executives building these systems, she did not reach for extinction. She talked about privacy, and about the infrastructure of surveillance being laid now, which she suspects will be very hard to pull up later.
It sits naturally alongside earlier conversations in the archive with Anil Seth on the constructed self, Tom McClelland on whether machines could be conscious, Michael Levin on cognition beyond brains and Nita Farahany on the battle for the brain. In this conversation, we spoke about why AI has become a laboratory for the mind, the category mistake at the heart of the chatbot debate, superpsychism and the consciousness beneath space-time, transhumanism and the slow arrival of brain implants, what it means to be human once intelligence, rationality and consciousness come apart, and the message she would give to the people running the AI companies.
Q: Thank you for making the time. I suspect you’re receiving a tsunami of these requests, so I’m glad I made it through the filter. You look at the whole technology space through a philosophical lens, so let me start there: how significant is this moment of AI in the story of our species?
[Susan Schneider]: It’s very significant. For the first time, we are interacting with another linguistic intelligence, one that exhibits a good deal of acumen, both in conversation and in what it produces. We are seeing amazing science and very interesting mathematics. We’re also seeing a multiplicity of forms of intelligence. We tend to focus on the chatbot, but there is a whole range of AI systems in which the advances, relative to where we were 20 years ago, are immense.
Q: Has AI already changed our understanding of what intelligence is? It seems unlike any intelligence, or pseudo-intelligence, that we’ve encountered before, and I wonder whether it has quietly rewritten some of our assumptions.
[Susan Schneider]: Some people have never really thought about what intelligence is. But for those who have, and for the many people now interacting with chatbots, I think it has affected our impression of it. Traditionally, we bundle several mental properties together: we think of consciousness and of human-like behaviour, and we correlate those with linguistic intelligence. When you interact with an AI, those things come apart, or at least they potentially can.
[At this point, our video screens were hit by some bizarre interference, with the picture flashing in and out.] Your screen is flashing in and out, by the way. I’m just making sure you’re not a deepfake. You never know any more, and then I’d be interacting with a different form of intelligence. Actually, that’s an example of the kind of world we’re in. We have these intelligences, but we don’t know what their other mental properties are, if any. Even using the word “mental”, as a philosopher I’d say that to use it legitimately, the thing may need to be conscious. And we’re still trying to figure that out.
It’s as if AI has become a laboratory for understanding the nature of the mind. One of the most fascinating things is the conceptual representations inside LLMs. Anthropic has been doing some really interesting work on circuit tracing, for example, which illustrates how human-like those representations are. If you map out a semantic space, in which some concepts sit closer together than others, you find that an LLM’s concept of a dog sits closer to cat than to, say, black hole. Overall, that conceptual space mirrors human conceptual thinking, because the systems have been trained on us.
It’s a really interesting time, but it’s also a very scary time in terms of AI safety. We’ve got all these bots populating the internet that don’t get along, to say the least.
Why asking whether a chatbot is conscious is a category mistake
Q: The consciousness question is the one I find most interesting, and I see three possibilities. Trained in our image, the LLMs simply mimic us. Or consciousness has emerged from what we’ve built. Or it is conscious and we wouldn’t know. You’ve approached this through the AI Consciousness Test and the chip test. Where is your thinking now, and how would we even know?
[Susan Schneider]: Here’s the elevator pitch on where I am now. The first thing we need to understand is that consciousness may not inhere in a model. Consciousness isn’t primarily something we should ask about a program type, because a program or a model is an abstract entity, like an equation. Those are not the things that are conscious. When we think about human consciousness, or the consciousness of a non-human animal, we ask about a particular being: what is it like to be my dog? And if you were able to write a program specifying dog consciousness, we wouldn’t think the program itself was conscious.
So, first, we need to stop making category mistakes and look at system dynamics: what is running the program? That is the concrete thing that is potentially conscious, or not. If you’re asking whether a chatbot or a data centre could be conscious when it runs a particular program, the question should be about the system implementing the program, not the program itself. To give an example, who knows, the same GPT, say 6.1 or whatever the latest model is, could be conscious when it’s run on one kind of system and not conscious on another. The issues are very nuanced and subtle; these are highly distributed systems to begin with. But the distinction between a model and its dynamical implementation is the first step.
“Consciousness may not inhere in a model.”
The second part of the pitch is that, even with all that in mind, the question of synthetic consciousness is far richer than chatbots. Chatbots as a class aren’t conscious, for various reasons; they emulate consciousness because they’re trained on our data. But there could be other classes of AI that are conscious, and there are all kinds of nuances to that. You can have hybrid systems, for example. I’m particularly interested in biological AIs, or biologically inspired AIs, which could in principle run LLM programs. It’s very complicated.
Superpsychism and the consciousness beneath space-time
Q: You talk about the system, and I think that’s an important point. I might assume you’re conscious, but only because of the vast biological system we share, which gives me the context, and even that system we can’t fully understand. It leads to your notion of superpsychism, in which our entire universe might be generated by, or be the result of, compute happening elsewhere.
[Susan Schneider]: There’s a lot in what you’ve just said. One thing I’d call attention to, which everybody should be thinking about, is the extended mind: whether, to understand consciousness, we need to go beyond the nervous system and the brain and include the world. I think that’s a very real and important possibility, and it’s connected with the work of Andy Clark, David Chalmers and others over the years.
The bigger question is the metaphysical one. Nothing I’ve said so far, about AI consciousness, about dynamics versus models, or about which AIs might be conscious, concerns the metaphysics, really. But your question goes into my work in physics and metaphysics, and there’s some really interesting stuff going on there.
I’d start with panpsychism. Panpsychists say everything is conscious. That may seem funny: how can a glass of water or a smartphone be conscious? There are a lot of panpsychists out there, and what they really mean is that the universe and its fundamental constituents must have some element of experience, perhaps dispositions of the fundamental particles or strings, whatever you take to be basic, that give rise to consciousness, and that a purely materialist view can’t explain. I’m thinking here of Galen Strawson, who pointed out that if you want to say consciousness emerges, it has to emerge from something. You can’t get apples from oranges. The base from which it emerges must itself be experiential. Galen thinks that’s still physicalism, and you know how philosophers are, they debate definitions. But the crucial thing is that it’s panpsychist, and a good indication of what panpsychists think: reality has to be fundamentally experiential. A fundamental particle would have only the tiniest bit of consciousness. Panpsychists agree with neuroscience that sophisticated consciousness is a matter of configuration and complexity, and some think artificial consciousness is possible while others don’t. So it’s not that they’re being crazy. They’re not.
I’m something of a panpsychist myself. With Mark Bailey, I developed an approach called superpsychism. It’s separate from much of my other work; my current research in neuroscience and consciousness doesn’t depend on it. The starting point is physics. There’s a lot of interest in resolving the contradiction between relativity and quantum mechanics, which is a very difficult problem. During my time at the Institute for Advanced Study in Princeton, I remember sitting with the string theorists reading papers on space-time emergence, which tried to recover space-time from quantum entanglement. So this isn’t loopy, woo-woo stuff. It’s an area of physics dealing with the fact that relativity and quantum mechanics are currently contradictory, and with what I think is one of the most interesting problems in physics: spooky action at a distance. Entanglement, I think, calls into question our normal understanding of space and time.
So I roped in Mark, my dear collaborator, and I said: let’s work with panpsychism, because I think it’s interesting. If the panpsychists are right that consciousness is a basic ingredient of reality, how would that work with physics itself? Drawing on my work at the Institute, I said, let’s look at the string theorists’ recovery of space-time from entanglement and think about how that works together with panpsychism. It was a really fruitful collaboration.
“So this isn’t loopy, woo-woo stuff.”
If you move in that direction, and I think it’s an all-things-considered judgement spanning several fields, it turns out that reality has a fundamental, non-spatiotemporal element. I don’t think it’s unquantifiable or necessarily mysterious. You recover space-time from it, and consciousness is an ingredient of that recovery. What is conscious and what isn’t then becomes a mathematical question.
You might say that leaves us with two separate things. We have conscious entities in the physical world. Hopefully we’re right that creatures with brains have some level of consciousness, though there’s plenty we probably don’t understand, and we can connect that with our science of consciousness through a nice panpsychist account. That account also fits the recovery of space-time, which is what I was interested in. But it raises a further possibility, which I think is just suggestive and interesting: that the fundamental reality underlying space and time is itself a form of consciousness. It has a tremendous amount of complexity. You can ask questions there about the emergence of the universe, and about whether it is a form of intelligence. That connects with really interesting work by David Grinspoon and others on planetary intelligence, which let me link these issues to my work at NASA, where I held an astrobiology chair.
So it gives a sweeping picture of the universe. I’m writing a book on it, a very overdue book. I think it helps with some of the questions about machine consciousness, and with understanding human consciousness.
Q: Anyone who has used these systems at any depth comes to see them as, in a way, non-temporal and non-local. With enough compute they run far faster than biological time, and they exist nowhere, yet everywhere. I think that fundamentally changes our understanding of what an intelligence is.
[Susan Schneider]: I don’t think they’re everywhere in the sense that non-locality works, and I don’t think they’re outside space and time. When you’re interacting with GPT, you can ask where it is, and physics gives an actual answer: those activation patterns are situated in space-time. Think of it more like the televising of the Super Bowl. It’s an event happening in multiple locations, transmitted to different homes through fibre-optic cables and other technology before it reaches your TV screen. Ultimately, we’re talking about processing in one or more data centres. It’s pretty cool science, and you can watch documentaries on how it works. But it’s nothing like something emerging from fundamental ingredients to which the normal dimensions of space and time literally do not apply.
The hype cycle and the question of human primacy
Q: Let’s return to consciousness. I don’t think it’s improbable that within a couple of years we build something that passes close to every test we could throw at it. If we can no longer make the case that these systems aren’t conscious, what does that mean for us as a species? Our consciousness has given us a sense of primacy since the beginning of our story. We could create something that breaks it, and may even exceed us.
[Susan Schneider]: I think we really need to avoid hype cycles. There’s a lot of talk, and it suits the AI companies to have amplificatory talk about how superintelligent and how conscious their systems are. I think major assumptions have to be backed by extraordinary science. Right now, there’s no science I’ve seen that’s publicly accessible indicating these systems are conscious, but there’s good science to indicate that they emulate us, because they’re trained on our data. AI safety researchers are right to be concerned, and right to ask whether these systems are conscious. But I see no motivation for a yes answer where chatbots are concerned. Again, I stress that if you’re dealing with a biological system, the situation is quite different. But we have to be really careful not to distract ourselves from other facets of what these chatbots are doing.
Q: The biological piece interests me because, alongside the LLMs, there’s a whole other world of development blurring the barrier between us squishy meat robots and the digital, through Neuralink, its many peers and other technologies that take us into what was once the philosophical domain of transhumanism. How do you see that?
[Susan Schneider]: That’s definitely the case. I was a transhumanist when I was young, and to an extent I still am, so these are fabulously intriguing questions for me. I don’t know if you’ve read Ray Kurzweil‘s The Singularity Is Near, or any of those books. They were thoughtful books, actually. Ray thought the singularity would be utopian, and that highly intelligent machines would be conscious. You see that assumption in a lot of transhumanists. Nick Bostrom tends to assume that AIs are conscious, that superintelligence and consciousness go hand in hand.
I don’t want to call what we’re seeing now superintelligence. I’d say we have very smart, fast-processing savant systems; that’s what I call them. We’re seeing really impressive capabilities in, say, mathematics, although I very seriously doubt that’s just an LLM. I’d bet it’s a hybrid system with symbolic processing, drawing on a tonne of human work. These are really interesting systems, but it doesn’t follow that the other properties Ray promised will come to pass in the way we transhumanists thought.
One thing I find fascinating goes back to the nuances of the singularity. Ray argued that as soon as machine intelligence reached a certain level, we would see rapid progress in brain-machine interfaces. We now know that’s a very slow scientific process, so everyday brain-machine interfaces will probably take a lot longer. But could they be developed? Absolutely, especially biological implants. It’s a really exciting space because of what it could do to help people with disabilities and to heal the damaged mind, if you will. These are all wonderful things. But we’ll have to regulate, or at least watch the geopolitics very carefully, because if it turns into a super-soldier hype cycle, that won’t be good.
“I’d say we have very smart, fast-processing savant systems”
What it means to be human when intelligence and consciousness come apart
Q: Transhumanism has always carried a debate about what it is to be human once we’ve entered a different realm. So what, if anything, are we losing of ourselves to technology? And do we need a more coherent definition of what it means to be human first? I do wonder whether we’re adequately aware of who and what we are before we start navel-gazing about what we may be losing.
[Susan Schneider]: Great question. We’re not going to be able to slow this down, so it’s imperative that we grapple with the questions: what is it to be sentient? What is it to be human? What is it to be intelligent? How do these things come apart? How far will people want to interact with AI, or to augment their own intelligence? These are rich, deep questions, and different cultures and different individuals will answer them differently. We used to think that one distinguishing characteristic of humans was that we are more rational than non-human animals, which perhaps lack rationality altogether. Now I think we realise that other systems may exhibit a degree of it, though philosophers will debate that. A philosopher like John Searle, who was a biological naturalist, built consciousness into his account of mental properties such as understanding itself. By extension, you could do the same with rationality. I’d like to think of these properties as potentially coming apart.
Q: That separation matters. With technology outside us in LLMs and inside us through implants, are we better placed to solve for consciousness? We used to talk about the God of the gaps. These systems may be explaining more of the envelope of the body, with intelligence as a correlate of mind, but a gap remains: why do I feel like something rather than nothing? Perhaps that is where consciousness now sits.
[Susan Schneider]: That’s the hard problem, and I think it’s a great problem. I hope we’ll learn more about it by working out whether machines can be conscious. We could potentially realise that consciousness extends beyond brains. If we learned that it didn’t depend solely on neurons, that would give us some insight into the fundamental nature of consciousness.
Surveillance, not doom: a message to the AI companies
Q: As we come towards the end, I’d like your message as a philosopher and scientist in this space. The AI businesses are either calling for regulation or sounding the alarm, perhaps partly because it’s great PR, but they have built something very unusual, with consequences for humanity that could be grave or profoundly brilliant. What would you say to their chief executives and founders about what they’ve made?
[Susan Schneider]: I don’t think a philosopher’s understanding of the implications is as central to them as one might think. It would be like me trying to tell them how to build an AI cheaply, or about the latest chip developments. But dialogue is very important. We’re all dealing with geopolitical realities, tech races and so on, which I find quite scary, and we have to move very, very carefully with risk. The AI risk people in these businesses are on the right track in worrying, and many of them are trying to be responsible. My message to the CEOs, though, is to challenge them about techno-surveillance. Their companies flourish on a surveillance economy. I do worry that, at the end of the day, we might be laying down the tendrils of surveillance and that they will be hard to remove. And we may not like the people who end up with that data and access to it. This is moving very quickly. I probably sound like a doomer, but I think the doomers are right about a lot.
Q: I don’t think you sound like a doomer. I’ve been careful in how I frame AI interviews, because the doom side is compelling: we like to talk about the existential threat. But, to use a well-worn example, nuclear power brought huge benefits, and it required exactly that dialogue: be careful about what you’ve created, and build the mechanisms to put an envelope around it.
[Susan Schneider]: And there were catastrophes, many of them, which I think is why people are so nervous. We certainly don’t want AI to be involved in a pandemic, and some of the biological risks are, I think, the worst. The AI safety community has been working really hard on an international scale, and Godspeed to them. I just worry about the privacy implications: digital privacy and techno-surveillance. That’s my sober thought. I also don’t want our work to be part of a hype cycle that distracts from other issues.
—————
Professor Susan Schneider is the William F. Dietrich Distinguished Professor at Florida Atlantic University and director of its Center for the Future of AI, Mind and Society. Her book Artificial You: AI and the Future of Your Mind is published by Princeton University Press, and her papers, including Superpsychism with Mark Bailey, are collected at schneiderwebsite.com.
