Ask Susan Schneider whether ChatGPT is conscious and she will tell you the question is badly formed. A program, she argues, is an abstract object, like an equation, and equations do not feel anything. If consciousness is present at all, it lives in the physical system running the code, which means the same model could in principle be conscious on one machine and not on another. It sounds like a technicality. It is not, because it cuts against both the companies hinting that their products are waking up and the critics who wave the possibility away.
Schneider is the William F. Dietrich Distinguished Professor at Florida Atlantic University, where she founded and directs the Center for the Future of AI, Mind and Society, holds a joint appointment in the university’s brain institute and co-directs a laboratory that builds AI systems informed by neuroscience. She studied economics at Berkeley and took her PhD at Rutgers under Jerry Fodor, who spent the early 2000s arguing against connectionism, the approach that became deep learning. Before Florida she held the NASA Baruch S. Blumberg Chair in Astrobiology and Technological Innovation, spending three years on the proposition that the most intelligent minds in the cosmos are likely to be postbiological; held the Distinguished Scholar Chair at the Library of Congress, working with members of Congress on AI; and was a fellow at the Institute for Advanced Study in Princeton. She writes for the New York Times, the Financial Times, the Wall Street Journal and Scientific American, has explained her work on PBS, the History Channel and Neil deGrasse Tyson’s StarTalk, and advises Prism, the Partnership for Research into Sentient Machines. Her book Artificial You: AI and the Future of Your Mind (Princeton University Press, 2019) has been translated into 15 languages, and with the Princeton astrophysicist Edwin Turner she devised the AI Consciousness Test, a way of probing a machine for an inner life without simply taking its word for it. Her earlier The Language of Thought (MIT Press, 2011) argued for a neurosymbolic approach, the kind of hybrid she suspects lies behind today’s mathematical breakthroughs. Her most recent work, published this year in the Journal of Consciousness Studies with Mark Bailey, is Superpsychism, and she has been warning about what she calls the megasystem problem: AI services knitting together into structures nobody designed and nobody can align.
You might expect a self-described transhumanist and former NASA chair to be among the believers. On chatbots she is a sceptic. She calls today’s systems “savant systems”, brilliant and fast and built from a tonne of human work, and she reads their uncanny familiarity as evidence of training rather than of experience: they are human-like because they are made of us. The radical claims she keeps for physics. Drawing on her time at Princeton, sitting with string theorists reading papers on how space-time might be recovered from quantum entanglement, she and Bailey ask what follows if the panpsychists are right, and land on a layer of reality outside space and time in which consciousness is an ingredient. Their answer turns the question of what is conscious into a mathematical one. And when I asked what she would say to the chief executives building these systems, she did not reach for extinction. She talked about privacy, and about the infrastructure of surveillance being laid now, which she suspects will be very hard to pull up later.
It sits naturally alongside earlier conversations in the archive with Anil Seth on the constructed self, Tom McClelland on whether machines could be conscious, Michael Levin on cognition beyond brains and Nita Farahany on the battle for the brain. In this conversation, we spoke about why AI has become a laboratory for the mind, the category mistake at the heart of the chatbot debate, superpsychism and the consciousness beneath space-time, transhumanism and the slow arrival of brain implants, what it means to be human once intelligence, rationality and consciousness come apart, and the message she would give to the people running the AI companies.
Q: Thank you for making the time. I suspect you’re receiving a tsunami of these requests, so I’m glad I made it through the filter. What I really like is that you look at the whole technology space through a philosophical lens, and so I thought the starting point should be this: how significant is this moment of AI in the story of our species?
[Susan Schneider]: I think it’s very, very significant. For the first time, we are interacting with another linguistic intelligence, one that exhibits a good deal of acumen, both conversationally and in terms of its results. We are seeing amazing science. We’re seeing very interesting mathematics; the mathematical space is great. And we’re also seeing a multiplicity of different forms of intelligence. We tend to focus on the chatbot, but in reality there is a range of different kinds of AI in which we are seeing immense advances relative to where we were, say, 20 years ago.
Q: In your view, has AI already changed our fundamental understanding of what intelligence is? It seems very unlike any other type of intelligence, or pseudo-intelligence, that we’ve encountered before, and I wonder whether it has quietly rewritten some of our assumptions.
[Susan Schneider]: Some people have never really thought about the issue of intelligence. But for those who have, and for the people in the general public who are interacting with chatbots, I think it has affected our impression of intelligence. The standard way of understanding intelligence piles several mental properties together. We think of consciousness, we think of human-like behaviour, and we correlate those with linguistic intelligence. Now, all these things come apart when you’re interacting with an AI, or at least they potentially can. I think that’s quite interesting.
Your screen is flashing in and out, by the way. I’m just making sure you’re not a deepfake. You never know any more, and then I’d be interacting with a different form of intelligence. Actually, that’s an example of the kind of world we’re in. We have these intelligences, but we don’t know what their other mental properties are, if any. Even using the word “mental”, as a philosopher I’d say that to use that expression legitimately, the thing may need to be conscious. And we’re still trying to figure that out.
So I just think it’s a really interesting time. It’s as if AI has become a laboratory for understanding the nature of the mind. One of the most fascinating things is the conceptual representations inside LLMs. Anthropic has been doing some really interesting work on circuit tracing, for example, which illustrates how human-like the linguistic representations are. If you were to set up a semantic space, where some concepts sit closer together than others, you would get an LLM whose dog concept is closer to cat than, say, to black hole. Overall, that conceptual space mirrors human conceptual thinking, because the systems have been trained on us. I think that’s one of the most fascinating things about these chatbots: how human-like they are.
In general, it’s a really, really interesting time. It’s also a very scary time in terms of AI safety. We’ve got all these bots populating the internet that don’t get along, to say the least.
Why asking whether a chatbot is conscious is a category mistake
Q: The consciousness question is the one I find really interesting, because there seem to be a few possible routes. One is that the LLMs have been taught in our image, so perhaps they are able to mimic us and give the impression of consciousness. Another is that we have created something whose emergent property appears to be consciousness. And the third is that, even if it is conscious, would we know anyway? I know you’ve looked at this through the AI Consciousness Test and the chip test, so where are you in your thinking about whether these models are conscious? And how would we even know?
[Susan Schneider]: To give the elevator pitch on what I think about these issues now: the first thing we need to understand is that consciousness may not inhere in a model. A question about consciousness is not primarily a question we should ask about a program type, because a program or a model is an abstract entity, like an equation. Those are not the things that are conscious. When you think of human consciousness, or the consciousness of a non-human animal, we ask questions about a particular being. What is it like to be my dog? And we don’t think that, if you were able to write a program specifying dog consciousness, the program itself would be conscious.
So first of all, we need to stop with the category mistakes. Second, we need to look towards system dynamics: what is running the program? That is the concrete thing that’s potentially conscious, or not. This is the interesting part, I think. If you’re thinking of a chatbot or a data centre and asking whether it could be conscious when running a particular program, you really should be asking the question about the system that’s implementing the program, not about the program. To give an example, who knows, the same GPT, say 6.1 or whatever the latest model is, could be conscious when it’s run on one kind of system and not conscious on another. The issues are very nuanced and subtle; these are highly distributed systems to begin with. But I think that’s the first step: the distinction between the model and its dynamical implementation.
“Consciousness may not inhere in a model.”
To conclude the elevator pitch, the second thing I want to suggest is that, even bearing all that in mind, the group of potentially conscious entities that are AIs, the synthetic consciousness question, is far richer than chatbots. Chatbots as a class aren’t conscious, for various reasons. They’re emulating consciousness because they’re trained on our data. But there could be another class of AIs that are conscious, and there are all kinds of nuances to that. You can have hybrid systems, for example. I’m particularly interested in biological AIs, or AIs that are biologically inspired, and they can run LLM programs in principle. You see? Very complicated.
Superpsychism and the consciousness beneath space-time
Q: You talk about the system there, and I think that point is really interesting. I might ask you whether you are conscious, or I might simply assume you are, but that only matters because of this big biological system we’re in, which gives me the context. And even for us, it’s impossible to understand the true nature of that biological system. I know you’ve written about this before, and it leads to the notion of superpsychism, where our entire universe might be generated by, or might itself be the result of, compute happening elsewhere.
[Susan Schneider]: There are a lot of things in what you just said. One thing I want to call attention to, which everybody should be thinking about, is the extended mind: whether, to understand consciousness, we need to go beyond the nervous system and the brain and include the world. I think that’s a very real and important possibility, connected with the work of Andy Clark, David Chalmers and others over the years. So that’s one thing I wanted to tag.
The bigger question is this. Nothing I said at the beginning of our discussion, about AI consciousness, about dynamics versus a model and about which AIs are conscious, concerns the metaphysics, really. Now you’ve asked me a question that goes into my work in physics and metaphysics, and I think there’s some really interesting stuff going on there.
I like to think of it, first off, in terms of panpsychism. Panpsychists say everything’s conscious. That may seem funny: how can we talk about a glass of water or a smartphone as being conscious? Here’s what they really mean, in general, and there are a lot of panpsychists out there. They mean that the universe itself and its fundamental constituents have to have some element of experience, maybe dispositions of the fundamental properties, of fundamental particles or strings, whatever you think is basic, that give rise to consciousness, and that a purely materialist vantage point would not explain that. Here I’m thinking of the work of Galen Strawson, for example, who pointed out that if you want to say consciousness emerges, it has to emerge from something. You can’t get apples from oranges. You have to have a base of emergence that is itself experiential. Now, Galen thinks that’s still physicalism, and you know how philosophers are, they debate definitions. But the crucial thing is that it’s panpsychist, and it’s a good indication of what panpsychists are thinking. They’re thinking that reality has to be fundamentally experiential. If a fundamental particle has some consciousness, it has just a tiny bit. They agree with neuroscience that sophisticated consciousness is a matter of configuration and complexity. Some will say it’s possible to build artificial consciousness, others won’t. So it’s not that they’re being crazy. They’re not.
Now, I’m kind of a panpsychist. Together with Mark Bailey, I developed an approach called superpsychism. This is separate from a lot of my other work; I’m doing some work in neuroscience and consciousness right now that doesn’t depend on it. The idea is this. In physics there’s a lot of interest in dealing with the contradiction between relativity and quantum mechanics. It’s a very difficult issue. During my time at the Institute for Advanced Study in Princeton, I remember sitting around with the string theorists reading papers on space-time emergence, which tried to recover space-time from quantum entanglement. So this isn’t loopy, woo-woo stuff. It’s an area of physics trying to deal with the fundamental problem that relativity and quantum mechanics are currently contradictory, and with what I think is one of the most interesting problems in physics, which is the fact that you have spooky action at a distance. Entanglement does, I think, call into question our normal understanding of space and time.
So what I did, and I roped Mark in, my dear collaborator, was to say: let’s work with panpsychism, because I think it’s interesting, and let’s ask ourselves, if the panpsychists are right that consciousness is a basic ingredient of reality, how would it work with physics itself? From my work back at the Institute, I said, let’s look at their recovery of space-time from entanglement and think about how that works together with panpsychism. That’s what I did with Mark, and it was a really fruitful collaboration. It was really, really interesting.
“So this isn’t loopy, woo-woo stuff.”
If you were to move in that direction, and I think it’s an all-things-considered judgement spanning a range of different fields, it would turn out that you have a non-spatiotemporal element, a fundamental element, to reality. I can tell you more about that. I don’t think it’s unquantifiable or necessarily mysterious, and I’d be happy to talk about it. You recover space-time from that, and consciousness is an ingredient of that recovery. So it becomes a mathematical question, what’s conscious and what’s not.
Now, interestingly, you might say it seems as though we have two separate things here. We have conscious entities in the physical world. Hopefully we’ve got it right that the ones with brains have some level of consciousness, but there are plenty of other things we probably don’t understand. Somehow that connects up with our science of consciousness, and I think we can connect it with a nice panpsychist account of what’s going on. It also works well with the recovery of the space-time picture, which is what I was interested in. But then you also have the potential, and this is, I think, just suggestive and interesting, to ask whether the fundamental reality that underlies space and time is itself a form of consciousness. It does have a tremendous amount of complexity. You can ask questions there about the emergence of the universe. You can ask whether it’s a form of intelligence. And you can connect that with really interesting work done by people like David Grinspoon and others on planetary intelligence. So there I was able to connect these issues with my work at NASA, where I held an astrobiology chair.
So I think it gives a sweeping picture of the universe, really. I’m writing a book on it, a very overdue book. It helps with some of the questions about machine consciousness, and I think it helps with understanding human consciousness.
Q: There’s something unique there, because anyone who has interacted with chatbots or AI at any depth comes to understand that these systems are, in a way, non-temporal and non-local. The amount of compute you throw at a problem means they can operate much faster than biological time, and they don’t exist anywhere, yet they exist everywhere. I think that fundamentally changes our understanding of what an intelligence is.
[Susan Schneider]: I don’t think they’re everywhere in the same sense that non-locality works, and I don’t think they’re outside space and time. Consider what’s happening when you’re interacting with GPT. You can ask where it is, and there’s an actual answer in physics: it situates those activation patterns in space-time. Think of it more like the televising of the Super Bowl. It’s an event, it’s happening in multiple locations, and it’s being transmitted to different homes through fibre-optic cables and other kinds of technology before it reaches your TV screen. It’s like that. Ultimately, we’re talking about processing in one or more data centres, and you can watch documentaries on how that works. It’s pretty cool science, actually. But it’s nothing like something emerging from fundamental ingredients where the normal space-time dimensions literally do not apply. So I think that’s a different thing.
The hype cycle and the question of human primacy
Q: Let’s go back to the other question, then, because I don’t think it’s improbable that within the next couple of years we create something that meets close to every test we could possibly throw at it, in terms of whether we are dealing with a being or an it. Suppose we reach a point where we find it difficult to make a case that the systems we’re building are not conscious. Is there an implication for us as a species? Philosophically, our sense of being and consciousness gives us a sense of primacy, which has been part of the story of humanity from the beginning. All of a sudden we could create something that not only breaks that story but might even exceed our capabilities, and we are now something different.
[Susan Schneider]: I think we really need to not fall into any hype cycles. There’s a lot of talk, and it suits the AI companies to have amplificatory talk about how superintelligent and how conscious their systems are. I think major assumptions have to be backed by extraordinary science. Right now, there’s no science I’ve seen that’s publicly accessible indicating these systems are conscious, but there’s good science to indicate that they emulate us, because they’re trained on our data. I think AI safety researchers are right to be concerned, and I think they’re right to ask whether these systems are conscious. But I don’t see any motivation for a yes answer in the space of chatbot consciousness. Again, I stress that if you’re dealing with a biological system, the situation is quite different, and I’m happy to talk about that. But we have to be really careful not to distract ourselves from other facets of what these chatbots are doing.
Q: The biological piece is interesting because, alongside this world of LLMs and systems, there’s a whole other world of development in which we are starting to blur the barrier between us squishy meat robots and the digital world, whether through Neuralink, its many peers or the other technologies that are enabling us to enter what was previously the philosophical domain of transhumanism. I’d love your views on that as well.
[Susan Schneider]: That’s definitely the case. I was a transhumanist when I was young, and I still am, to an extent. These are the really fabulously intriguing questions. I don’t know if you’ve read Ray Kurzweil‘s The Singularity Is Near, or any of those books. They were thoughtful books, actually. Ray thought the singularity would be utopian, and that machines that were highly intelligent would be conscious. You see this with a lot of transhumanists. Nick Bostrom tends to make the assumption that the AIs are conscious, that superintelligence and consciousness go hand in hand.
What we’re seeing right now, I don’t think I want to call superintelligence. I’d say we have very smart, fast-processing savant systems; that’s what I call them. We are now seeing really impressive capabilities with, say, maths, although I doubt very seriously that’s just an LLM. I bet you it’s a hybrid system with symbolic processing, drawing on a tonne of human work. These are really interesting systems, but it doesn’t mean that the other properties Ray promised are going to come to pass in the way we transhumanists thought.
One thing I find fascinating goes back to the nuances of the singularity issues. Ray talked about how, as soon as machine intelligence got to a certain level, we would see great movement in the space of brain-machine interfaces. We now know that’s a very slow scientific process, so the timing is probably going to be different. It will probably take a lot longer to develop everyday brain-machine interfaces. But could those systems be developed? Absolutely, especially the biological implants. I think that’s a really exciting space because of what it can do to help individuals with disabilities and to heal the damaged mind, if you will. These are all wonderful things. But we’ll really have to regulate, or at least watch the geopolitics very carefully, because if it just turns into another super-soldier hype cycle, that’s not going to be good.
“I’d say we have very smart, fast-processing savant systems”
What it means to be human when intelligence and consciousness come apart
Q: Within the transhumanist movement there has always been that philosophical debate about what it constitutes to be human once we’ve entered a slightly different realm. So, philosophically or otherwise, what are we losing of ourselves to technology, if anything? And do we need a more coherent definition of what it means to be human before we embark on some of these discussions? I do wonder whether we are adequately aware of what and who we are before we start navel-gazing about what we may be losing.
[Susan Schneider]: Great question. I think we have to grapple with this, because we’re not going to be able to slow this down. It’s imperative that we grapple with the questions: what is it to be sentient? What is it to be human? What is it to be intelligent? How do these things come apart? To what extent will people want to interact with AI? To what extent will they want to augment their intelligence? These are all very rich and deep questions, and different cultures will answer them differently. Different individuals will answer them differently. We used to think one characteristic of humans is that we are more rational than non-human animals; maybe non-human animals don’t even have rationality. Now I think we realise that other systems may exhibit a degree of it, although philosophers will debate these issues. If you had asked a philosopher like John Searle, who was a biological naturalist, he would build consciousness into his understanding of mental properties like understanding itself: could you understand? By extension, I imagine you could do that with rationality as well. I’d like to think of the different properties as potentially coming apart.
Q: That separation matters, because the interface between technology and humanity in all its forms, outside us in LLMs and inside us with implants, makes me wonder whether we are now better able to solve for consciousness. We used to talk about the God of the gaps. Perhaps the power of these systems is giving us a better understanding of the envelope of the body, and intelligence is a correlate for what mind might be, but there’s still a gap: what does it mean to be me, and why do I feel like something rather than nothing? Perhaps that gap is where consciousness now sits.
[Susan Schneider]: That’s the hard problem, and I think it’s a great problem. I hope we’ll learn more about it through understanding whether machines can be conscious. That would be fascinating, because we could potentially realise that consciousness extends beyond brains. If we learned that it was not just something that depended upon neurons, I think that would give us some insight into the fundamental nature of consciousness. That would be fascinating.
Surveillance, not doom: a message to the AI companies
Q: As a roundup, I’d like a message from you as a philosopher and scientist in this space. We’re at a point where the businesses are either calling for regulation themselves or sounding the alarm, perhaps because it’s great PR, but nevertheless I think they have built something very unusual, with either grave or profoundly brilliant consequences for humanity. What would be your message to the chief executives and founders of these AI businesses when they think about what they’ve made?
[Susan Schneider]: I don’t think a philosopher’s understanding of the implications is as central to them as one might think. It would be like me trying to tell them how to build an AI cheaply, or about the latest chip development. So I think dialogue is very important. We’re all trying to deal with geopolitical realities, tech races and so on, and I think that’s quite scary. We have to move very, very carefully with risk. I think the AI risk people in business are on the right track in worrying, and a lot of them are trying to be responsible. My message, though, is to challenge them about techno-surveillance. They flourish on a surveillance economy. I do worry that, at the end of the day, we might be laying down the tendrils of surveillance and that they will be hard to remove. And we may not like the people who have that data and who have access to it. That is my worry. This is moving very quickly. I probably sound like a doomer. I think they’re right about a lot, though.
Q: I actually don’t think you sound like a doomer. I’ve been careful in how I frame some of the AI interviews, because the doom side is compelling: as human beings we like to talk about the existential threat of humanity’s destruction. At the same time, and it may be a well-worn example, nuclear power has brought society huge benefits, but it required people to have that dialogue: be careful about what you’ve created, and create the appropriate mechanisms to put an envelope around it.
[Susan Schneider]: And there were catastrophes, many catastrophes. I think that’s why people are so nervous. We certainly don’t want an AI to be involved in a pandemic, and I think some of these biological risks are the worst. The AI safety community has been working really hard on an international scale, and Godspeed to them. I just worry about the privacy implications: digital privacy and techno-surveillance. That’s my sober thought. I also don’t want our work to be part of a hype cycle that distracts from other issues. I don’t like that at all.
———
Professor Susan Schneider is the William F. Dietrich Distinguished Professor at Florida Atlantic University and director of its Center for the Future of AI, Mind and Society. Her book Artificial You: AI and the Future of Your Mind is published by Princeton University Press, and her papers, including Superpsychism with Mark Bailey, are collected at schneiderwebsite.com.
