I recently read a great tip: If you are being pursued by a killer robot, your best defense is to close the door. Robots have trouble opening doors. (So far.)
Lesson being: Perhaps our best defense against AGI is to build and inhabit a world ENTIRELY uninhabitable by AGI. Unplug. Go steampunk.
I'm leading a class for Christians in STEM this spring, and I assign students full hours outdoors, with no phone, just praying and being silent. The real world is always much more interesting than the one we can model with data.
I was wondering when you would take this one on. Hope this article stands the test of time a bit better than that NY Times one from the 50s fretting about the "thinking machines". On the plus side, I believe they are still covering the topic enthusiastically decades later.
Some thought/questions I would love to get your impressions on:
1. Maybe Michael Tomasello's "The Evolution of Agency" is still dominating my perspective, but his emphasis on the _how_ versus the _what_ of intelligence feels pertinent. Particularly his simple cybernetic/control-system framing of agency. His focus on flexibility of feedback and control feels like the right counterbalance: While conversation context does modify the LLM's output, the underlying model is a large "offline optimized" static data structure. Where is the flexibility? How do you relate Agency to AI?
2. Could you describe what you noticed to observe this: "ChatGPT doesn’t just learn; it learns how to learn". In a sense, I can see how any non parametric statistical tool, say even a humble histogram, is capable of learning how to learn? Or do you mean something more?
3. On FEP correspondences: I don't see it. Where is the epistemic foraging? Where is the active inference?
4. I was wondering if LLMs are a new kind of medium. The most they can hope for I believe! If so, let's think with Gutenberg and Licklider. Mass printing and the internet have a different character now than when they first showed up -- just hundreds/tens of years later. There was surely a McLuhanist effect from them both, and their character today is related to quite noticeable shifts in our patterns of activity. Both technologies changed power and flow dynamics, but I don't believe their effects were predictable based on their original use/content/message (e.g. Bible, Remote telnet). Do you really think Altman could be a bigger deal than Licklider/Gutenberg? If true, it would be cool to be living so proximate to greatness.
5. Lastly, I was surprised you didn't make more of the bug-features of language that Rappaport alerted us to: "the lie" and "the alternate". ChatGPT seems to me one of the grandest monument we have yet built to the gift-curse of symbolic communication. Folks were stressing about writing scriptures down back in the day, maybe it is okay for us to stress about LLMs.
I might be super wrong on this. But as folks who have been working the area since the seventies will tell you, winter is never far behind when those AI summers fully kick in. They can be exciting while they last.
1. If I'm reading you right, I think I agree LLMs like GPT aren't yet capable of the flexibility that would be needed to simulate a true cybernetic agency (cybernetic in Wiener's and Rappaport's sense). But the idea *is* to produce systems that do have that level of flexibility, which I think is probably necessary for the kind of general intelligence people are hoping for. If they solve that problem, then we've got a true artificial agent, and then the challenges I've written about here become live issues.
2. Maybe I'm guilty of imprecision. I just meant that the conceptual model-building that we assume GPT does in order to predict language is massively iterative. It builds on itself in training so that "learning" is basically autonomous. Not sure what the dividing line would be there for "learning to learn."
3. You're right, as I say above, GPT isn't a free energy–minimizing system yet. It doesn't do active inference (as far as we know). But if Friston is right about natural cognition as axiomatically a free energy–minimizing process, then naturalistic future AIs will move toward full active inference because that's just what the "intelligence" part of AI means. This is the road it seems to me we're on, given current assumptions and goals, not the destination I think we've arrived at yet.
4. I won't even venture a guess as to the comparative greatness of AI researchers.
5. Good point. I've written about this before at Patheos — digital spaces are almost wholly de-ritualized and symbol-based, so there's not an obvious way to solve the problem as there is in the 3D world. Maybe a topic for another future post here.
Thanks for clarifying in detail. A few follow ups.
1. I read Tomasello as suggesting intelligence (behavioral patterns) and agency (behavioral organizational structure) as being related but on different dimensions (what Vs how). So there might well be super intelligent existential problem solvers with less agency than lizards. Maybe artificial general agency is what we are really worried about? If so, then LLMs do not seem to be a cause for existential concern.
2. From my narrow engineering perspective: Statistical methods help summarize, describe and simulate data. This is statistical learning. Isn't an LLM basically a statistical learner? Where is it updating its learning method based on what it learnt earlier?
3. Got it: We agree. LLMs are not FEP optimizers. But I still don't get the big break ahead. We already have natural and artificial FEP optimizers of varying levels of intelligence (behavioral complexity): molds, trees, thermostats, squirrels, advertisers, corporations. And I am still most fearful of a faster spreading pandemic, runaway inflation, tornadoes, and nuclear war. We seem to be most at risk of damage from phenomena that are neither agentive, not intelligent.
4. What would AI/LLMs be according to McLuhan? Will we see something different from the writing and print technology induced cultural logorrhea of earlier times?
5. I look forward to reading that future post. De-ritualized and symbol-based systems may just not be sustainable. Why would societies have not tried to fully symbolize before? They are so quick, energy efficient, easy and infectious. Surely they've been tried --and have failed-- many more times than we can remember.
All good points. As for the Tomasello intelligence/agency distinction, I'm not sure I see it. I think an active inference model of intelligence or cognition suggests that the one is necessarily pegged to the other — more complex intelligences are more agentic, because intelligence is biologically *for* action, not simply pattern recognition. We as biological agents learn about the world into to exercise and enhance our agency — to maximize model evidence for our self-models. The "active" in active inference is important.
In the same way, you could make the same critique re: "learning to learn" about us, as humans. A human, too, is "basically a statistical learner." And yet we learn to learn — as our conceptual trees get deeper and more complex, we apply that acquired knowledge to make better and more sophisticated predictions about future experiences. Maybe this resolves to simply Bayesian statistical learning of associations, but it sure seems like "learning how to learn."
Generally, I think the point I'm interested in is that LLMs *are* just statistical learners, but the thing is, this doesn't distinguish them from humans. The most sophisticated models of human intelligence seem to be converging with the most powerful AI models: they're both Bayesian optimization machines. Every hesitation about the fundamental limitations of an AI based on this principle would seem to apply also to us, yet we seem to have agency and intelligence, and we seem to learn how to learn. I'm not an engineer, so maybe I'm missing something important. But this is the question I'm getting at.
The other free energy minimizers you mention are many orders of magnitude less cognitively powerful than some of the future AIs we're likely to develop. So while I agree there are plenty of other serious threats to worry about, I think the possibility of developing a potential agency that's enormously more intelligent than anything we've yet encountered should be the occasion for debate — and a lot of smart people are saying the same thing!
I really don't know McLuhan well enough to have anything smart to say about AI-induced logorrhea. Maybe it would be different because it will be more recycled — the secondary products of previous communications, rather than the printing press's introduction of masses of new, original communications.
And yes, I agree! Societies that try to de-ritualize probably just get weeded out. Again, cue the stinging violin music…
So much to think about with this one. AI pushes us to think about ourselves and our relationship with the creation and our own creations in new ways with real implications on our world-view and "self-view". Only one correction, Microsoft's AI chat in Bing is built on top of ChatGPT. It's not a competitor to ChatGPT or its successor technology, GPT-4
Good point! We still hadn't learned that Bing was based on GPT-4 when I first started writing this post, and I apparently didn't update the draft after that news broke. I'll add a correction and mea culpa.
I recently read a great tip: If you are being pursued by a killer robot, your best defense is to close the door. Robots have trouble opening doors. (So far.)
Lesson being: Perhaps our best defense against AGI is to build and inhabit a world ENTIRELY uninhabitable by AGI. Unplug. Go steampunk.
I'm leading a class for Christians in STEM this spring, and I assign students full hours outdoors, with no phone, just praying and being silent. The real world is always much more interesting than the one we can model with data.
I was wondering when you would take this one on. Hope this article stands the test of time a bit better than that NY Times one from the 50s fretting about the "thinking machines". On the plus side, I believe they are still covering the topic enthusiastically decades later.
Some thought/questions I would love to get your impressions on:
1. Maybe Michael Tomasello's "The Evolution of Agency" is still dominating my perspective, but his emphasis on the _how_ versus the _what_ of intelligence feels pertinent. Particularly his simple cybernetic/control-system framing of agency. His focus on flexibility of feedback and control feels like the right counterbalance: While conversation context does modify the LLM's output, the underlying model is a large "offline optimized" static data structure. Where is the flexibility? How do you relate Agency to AI?
2. Could you describe what you noticed to observe this: "ChatGPT doesn’t just learn; it learns how to learn". In a sense, I can see how any non parametric statistical tool, say even a humble histogram, is capable of learning how to learn? Or do you mean something more?
3. On FEP correspondences: I don't see it. Where is the epistemic foraging? Where is the active inference?
4. I was wondering if LLMs are a new kind of medium. The most they can hope for I believe! If so, let's think with Gutenberg and Licklider. Mass printing and the internet have a different character now than when they first showed up -- just hundreds/tens of years later. There was surely a McLuhanist effect from them both, and their character today is related to quite noticeable shifts in our patterns of activity. Both technologies changed power and flow dynamics, but I don't believe their effects were predictable based on their original use/content/message (e.g. Bible, Remote telnet). Do you really think Altman could be a bigger deal than Licklider/Gutenberg? If true, it would be cool to be living so proximate to greatness.
5. Lastly, I was surprised you didn't make more of the bug-features of language that Rappaport alerted us to: "the lie" and "the alternate". ChatGPT seems to me one of the grandest monument we have yet built to the gift-curse of symbolic communication. Folks were stressing about writing scriptures down back in the day, maybe it is okay for us to stress about LLMs.
I might be super wrong on this. But as folks who have been working the area since the seventies will tell you, winter is never far behind when those AI summers fully kick in. They can be exciting while they last.
Thanks for the detailed comment!
1. If I'm reading you right, I think I agree LLMs like GPT aren't yet capable of the flexibility that would be needed to simulate a true cybernetic agency (cybernetic in Wiener's and Rappaport's sense). But the idea *is* to produce systems that do have that level of flexibility, which I think is probably necessary for the kind of general intelligence people are hoping for. If they solve that problem, then we've got a true artificial agent, and then the challenges I've written about here become live issues.
2. Maybe I'm guilty of imprecision. I just meant that the conceptual model-building that we assume GPT does in order to predict language is massively iterative. It builds on itself in training so that "learning" is basically autonomous. Not sure what the dividing line would be there for "learning to learn."
3. You're right, as I say above, GPT isn't a free energy–minimizing system yet. It doesn't do active inference (as far as we know). But if Friston is right about natural cognition as axiomatically a free energy–minimizing process, then naturalistic future AIs will move toward full active inference because that's just what the "intelligence" part of AI means. This is the road it seems to me we're on, given current assumptions and goals, not the destination I think we've arrived at yet.
4. I won't even venture a guess as to the comparative greatness of AI researchers.
5. Good point. I've written about this before at Patheos — digital spaces are almost wholly de-ritualized and symbol-based, so there's not an obvious way to solve the problem as there is in the 3D world. Maybe a topic for another future post here.
Thanks for clarifying in detail. A few follow ups.
1. I read Tomasello as suggesting intelligence (behavioral patterns) and agency (behavioral organizational structure) as being related but on different dimensions (what Vs how). So there might well be super intelligent existential problem solvers with less agency than lizards. Maybe artificial general agency is what we are really worried about? If so, then LLMs do not seem to be a cause for existential concern.
2. From my narrow engineering perspective: Statistical methods help summarize, describe and simulate data. This is statistical learning. Isn't an LLM basically a statistical learner? Where is it updating its learning method based on what it learnt earlier?
3. Got it: We agree. LLMs are not FEP optimizers. But I still don't get the big break ahead. We already have natural and artificial FEP optimizers of varying levels of intelligence (behavioral complexity): molds, trees, thermostats, squirrels, advertisers, corporations. And I am still most fearful of a faster spreading pandemic, runaway inflation, tornadoes, and nuclear war. We seem to be most at risk of damage from phenomena that are neither agentive, not intelligent.
4. What would AI/LLMs be according to McLuhan? Will we see something different from the writing and print technology induced cultural logorrhea of earlier times?
5. I look forward to reading that future post. De-ritualized and symbol-based systems may just not be sustainable. Why would societies have not tried to fully symbolize before? They are so quick, energy efficient, easy and infectious. Surely they've been tried --and have failed-- many more times than we can remember.
All good points. As for the Tomasello intelligence/agency distinction, I'm not sure I see it. I think an active inference model of intelligence or cognition suggests that the one is necessarily pegged to the other — more complex intelligences are more agentic, because intelligence is biologically *for* action, not simply pattern recognition. We as biological agents learn about the world into to exercise and enhance our agency — to maximize model evidence for our self-models. The "active" in active inference is important.
In the same way, you could make the same critique re: "learning to learn" about us, as humans. A human, too, is "basically a statistical learner." And yet we learn to learn — as our conceptual trees get deeper and more complex, we apply that acquired knowledge to make better and more sophisticated predictions about future experiences. Maybe this resolves to simply Bayesian statistical learning of associations, but it sure seems like "learning how to learn."
Generally, I think the point I'm interested in is that LLMs *are* just statistical learners, but the thing is, this doesn't distinguish them from humans. The most sophisticated models of human intelligence seem to be converging with the most powerful AI models: they're both Bayesian optimization machines. Every hesitation about the fundamental limitations of an AI based on this principle would seem to apply also to us, yet we seem to have agency and intelligence, and we seem to learn how to learn. I'm not an engineer, so maybe I'm missing something important. But this is the question I'm getting at.
The other free energy minimizers you mention are many orders of magnitude less cognitively powerful than some of the future AIs we're likely to develop. So while I agree there are plenty of other serious threats to worry about, I think the possibility of developing a potential agency that's enormously more intelligent than anything we've yet encountered should be the occasion for debate — and a lot of smart people are saying the same thing!
I really don't know McLuhan well enough to have anything smart to say about AI-induced logorrhea. Maybe it would be different because it will be more recycled — the secondary products of previous communications, rather than the printing press's introduction of masses of new, original communications.
And yes, I agree! Societies that try to de-ritualize probably just get weeded out. Again, cue the stinging violin music…
So much to think about with this one. AI pushes us to think about ourselves and our relationship with the creation and our own creations in new ways with real implications on our world-view and "self-view". Only one correction, Microsoft's AI chat in Bing is built on top of ChatGPT. It's not a competitor to ChatGPT or its successor technology, GPT-4
Good point! We still hadn't learned that Bing was based on GPT-4 when I first started writing this post, and I apparently didn't update the draft after that news broke. I'll add a correction and mea culpa.
What do you propose?