Yves here. AI hype seems to need to be debunked often and forcefully, hence the need for Rob Urie’s post. Please circulate widely!
By Rob Urie, author of Zen Economics, artist, and musician who publishes The Journal of Belligerent Pontification on Substack. Originally published at his site
Recent public comments made about AI suggest that Americans have difficulty with the implications of linear time. This is odd given that its conception is largely Western and is centered on the clock time used to coordinate capitalist employment. The conceptual difficulty regards sequencing, or plans for future actions. But it also involves the distribution of profits. 100% of the capital equipment used in Western economic production was produced by workers. So, why does the resulting product belong to financiers rather than those who produced it?
To use a physical metaphor, if I 1) buy a car, 2) aim it in the direction of a cliff, 3) put a stone on the gas pedal and 4) put the transmission into drive, the car will move forward and plunge off of the cliff. Question: did I, through my actions, cause the car to plunge off of the cliff? Or did the car ‘drive itself’ off of the cliff? The answer depends on where you imagine that my own actions ended. In fact, I conceived and created a series of events that if carried through with competence would lead to the car plunging off of the cliff. The car is inert metal and rubber without human direction.
Likewise, if I create and set in motion a three-hundred step algorithm, is the algorithm producing the output, or did I? The distinction is between intent and process. My intent guides the conception and creation of the three-hundred step algorithm. But the work from that point forward is carried out by the algorithm being run in a computing environment. So, the algorithm didn’t conceive of the project. I did. The algorithm didn’t plan (sequence) the project. I did. The algorithm didn’t code the problem. I did. So, who produced the output, me or the machine?
A similar conceptual problem applies to claims of machines ‘thinking.’ Physically speaking, AI is a bundle of algorithms housed within a large computing environment. AI didn’t conceive itself. It was conceived, if memory serves, at Carnegie Mellon University in the 1970s. AI didn’t build itself. It was built in fits and starts by computer scientists in academia and later business. AI didn’t code itself. It was coded by AI developers. And the massive physical infrastructure on which AI depends was built by workers. The point: AI is wholly produced by humans.
The question then is how it is imagined that AI output represents more than the human effort that was put in to creating it? What process makes AI output more than the product of algorithms? If the answer is that something does, are you aware of sequencing algorithms? This would be code that organizes other code to follow a series of steps to complete a task. I’ve conceived and coded sequenced algorithms that run through multi-step processes from a single set of instructions. The output looks like reasoning. And it is reasoning. I coded it. The models did what I coded them to do.
So again, if a series of steps are conceived, planned and launched by humans on equipment that was created by humans, at what point does their dimension shift from inanimate to animate? Or more simply, at what point does a bundle of algorithms housed on a computer think or reason or possess intelligence or consciousness? In fact, the claim that any of these describe AI is a category error. Is a rock rolling down a hill imagined to be rolling itself down the hill rather than being moved by unseen physical forces (e.g. gravity). So, claims that AI can reason emerge from either ignorance or misunderstanding of basic physical processes.
Back in the world, there has been a debate in the West since the early nineteenth century over whether factory automation produces the product of factory automation, or whether the people who automated the factory produced the output? On the one hand, automation creates the appearance that its product is self-generated. On the other, the automation process was created by humans and would not exist otherwise. With the current ability to ‘sequence’ the production process using algorithms, another level of abstraction has been added to this debate.
Having conceived and coded ‘sequencing’ models, most who haven’t find the concept difficult to understand. These models are instructions for how a model ‘thinks.’ Question: how is a model ‘thinking’ when it is just following instructions? Answer: it isn’t. It is just following instructions. What looks like reasoning to AI users is the reasoning coded into the model by human coders. It appears to be reasoning because the instructions it is following were reasoned. It is written instructions being carried out. Nothing more.
The question is political as well in that the answer determines how income is distributed in the West. If ‘capital’ in the form of an automated factory produces the output, do the proceeds then belong to capital, meaning to the capitalist? Without workers first creating the automated factories, there would be no automation process. The political answer was to end the claims of workers to this product through wages. However, while workers receive one-time payments (wages) for their effort, the capitalist receives the profits from this labor for as long as they last.
With AI, this question is back on the table, conceptually at least. Whichever way one cares to perceive AI, as a thinking machine or as a bundle of related algorithms, it was built by workers. AI didn’t conceive itself. It was conceived by workers. This is an important clue into how it works. AI was built by human workers based on their desire to produce a machine that simulates human thought. However, the digital realm is a closed system. All AI ‘knowledge’ has been mediated by humans. Within AI’s Cartesian framing, AI has no direct access to the world. It is the proverbial Cartesian brain-in-a-vat.
One of the paradoxes of debating the nature of AI is that AI models describe themselves as variations on ‘word organizers and word sequencers.’ Focus on the word ‘sequencers’ for a moment. Again, a sequencer establishes and executes the order of a multi-stage process. With the launch of AI, a multi-stage process is set in motion. Words and phrases are identified and matched against similar words and phrases found in AI training sets. The sequencing then runs models to assign the words and phrases their human-determined meaning.
Important to understand is that neither the sequencer nor the broader AI model understands the words and phrases that are being acted on. The meaning of the words, semantics, is created by humans and is stored in a retrieval cache. Sequencing here is the matching of (human defined) meanings to words to provide semantic context to the words and phrases being matched. To be clear, AI ‘decides’ nothing. It is following algorithmic instructions. AI is neither deciding what to do nor how to do it. That is written out for it by humans.
Google AI Chatbot Analogy of AI to a Skyscraper:
End Google AI Chatbot Output——————————————————————
The distinction is between coding mathematical models to set in motion a series of steps versus the idea that the models reason on their own. Missing from casual analysis of AI is understanding of how large and complicated this process is. Developers have been building a ‘thinking machine’ in earnest since the 1970s. The infrastructure needed to run AI approximates that of a modern skyscraper. The question that has yet to be answered is: is AI worth it? Is it a crucial new technology that will justify its costs, widely considered? Or is it an occasionally interesting toy whose environmental footprint will end the planet?
Recent public discussion has puzzled over how AI can solve math problems if it doesn’t think? Consider the concept from physics of ‘work.’ What those considering the matter are imagining is lone mathematicians sitting in rooms and thinking through the solutions to math puzzles. But with unlimited computing power, optimization programs can use brute force computing to work through every conceivable iteration of a problem in seconds. What AI users aren’t seeing is the skyscraper’s worth of infrastructure behind the scenes producing a result.
Doesn’t this vast computing power illustrate the value of AI? No. It gets to the nature of technology. One explanation of technology is that it provides a benefit. Another is that it simply changes that way that humans do things. On the one hand, we can drive long distances quickly in cars versus walking. On the other, many of us now spend three hours per day sitting in traffic in cars. So, are cars a benefit? In some ways yes, in some ways no. What they aren’t is an unequivocal benefit, meaning that the jury is still out.
Image: the guts of the automaton featured in the movie Hugo. The mechanical refinement of fake humans can be seen in the gearing. The thought was that finer gearing made automatons closer to being human. That in retrospect the automaton can be seen as a better robot rather than being closer to human is an important insight for understanding AI. AI is a digital robot. It is no closer to thinking or reasoning than a doorstop. Source: dickgeorgecreatives.
If asked if they would like a machine that transports them from one place to another quickly, most Westerners would likely answer yes. When asked if they want to spend three hours per day sitting in a car in traffic, most Westerners would likely answer no. But the latter is the direct consequence of the prior. This is how capitalism works. We are offered a benefit. In the current case, the ability to travel quickly from one place to another. But almost immediately the social consequences of the ‘benefit’ become a burden that hadn’t been imagined when the benefit was offered.
In the present, a lot of Americans are worried that AI can think. It will take our jobs. But what we should be worried about is that AI can’t think. It is but one more layer of labor de-skilling. Consider: AI ‘art’ is artless. AI ‘thought’ is the aggregated wisdom of the Pentagon cobbled to the AEI (American Enterprise Institute). Every AI query written increases greenhouse gas emissions to levels that are suicidal for the species. And AI ‘solutions’ are regurgitated feints like carbon capture. All of the proposed solutions will more likely make the problems worse.
While AI users imagine that ‘thought’ is producing AI results, what is in fact being applied is work. Work here is similar to the concept of horsepower, the crude conversion of the pulling power of horses to that produced by an internal combustion engine. Recall the lone mathematician sitting and thinking. Now imagine running an AI program that is the equivalent in terms of capacity of 10,000 humans laboring for one million years. One would imagine that a lot of complicated questions could be answered in such a scenario.
Google Gemini AI Output
End Google Gemini AI Output———————————————————
Were 10.000 humans to labor for one million years, this would represent the largest undertaking in human history. And given that humans have finite lifespans, this thought experiment is entirely conceptual. Further, AI doesn’t use the methods of mathematicians. Instead of isolating a metaphorical tree in a forest by its qualities (the mathematician), AI chops down every other tree in the forest to declare that the tree left standing is the solution (optimization).
AI’s methodology represents a different way of solving math problems that may be of interest to a few dozen mathematicians, but that comes with a computational cost equivalent to a moon landing. Were 10,000 humans actually put to the task of solving mathematical problems, questions of agency and whether or not this is a good use of social resources would arise. It is only by hiding / sidelining the question of environmental and social costs that AI is claimed to add value beyond profits for a few insiders.
The ability to run a billion permutations in a microsecond makes AI a very powerful tool. But how much better is a world in which AI can run a billion permutations in a microsecond than the same world without it? The question requires a social answer, And the social answer must emerge from clear and complete understanding of the social costs of AI. It isn’t good enough to point to the math problems solved to justify the social investment in AI. The question is: what else could be accomplished with those same resources (opportunity costs)?
AI solved the math problems through a process of elimination. Again, this isn’t how mathematicians work. Why? Because AI uses computational technology that humans do not possess. Recall, a car can get us from one place to another faster than we can walk. But the adoption of cars has left us sitting in traffic for a substantial portion of our waking hours. AI can use brute force computing to muscle-through certain types of questions. But are these really questions that need to be answered? Or is answering them a form of mass entertainment?
Another hidden part of the AI process is the operationalization of language. AI was conceived through the premise that human thought results from syntax cobbled to semantics (form and meaning). But operationalization results in a formal consolidation of meaning. Take the term ‘democracy.’ It is widely prevalent in Western discourse in a variety of contexts, e.g. economic democracy. But to render the term operational, it must be stripped down and made stable.
To be clear, this isn’t touchy-feely in the way that it might read. Take the term ‘Christianity,’ There are 45,000 Christian denominations as of a recent survey. What does this mean in the current context? An operational definition of Christianity as those who believe in Christ eliminates 45,000 enthusiastic differences of opinion amongst Christians regarding what ‘believing in Christ’ means. In political terms, it flattens 45,000 differences of opinion out of existence to claim a unity that arguably does not reflect reality.
Again, this isn’t a quibble. Whoever controls the meaning of language controls the language. In an example from Zen Economics, economists use something called Household Income as a measure of economic well being. While this makes intuitive sense, in practice ‘household’ must be defined, ‘income’ must be defined, and the terms must be recombined into Household Income. The semantic problem? With upwards of dozens of competing definitions, people using the exact phrasing ‘Household Income’ tend to be speaking about materially different concepts.
When a user runs an AI query on Household Income, AI references the meaning that has been created by humans and placed into a semantic cache (storage area). But because AI is replacing internet search functions, prior definitions of commonly understood words are being systematically replaced with stripped down (operationalized) definitions by AI. This stripping down creates the sense of a consensus view on every topic that is incorrect. Linguistic diversity is being eliminated from the discourse. Each of these differences represents a worldview.
In a phrase that I keep going back to because it explains so much, any statistical result can be undone by redefining the variables. An operationalized version of Household Income can rise and fall at the same time depending on the definition. Why? Because the definitions contain their operating logic. Is a household a single family, all of the occupants of a house, or something else? Is income wage income, all of the money that a household brings in from all sources, or something else? As the definitions change, so do the outcomes based on them.
The times when I’ve traced technical definitions back through history (e.g. utility in economics), the meanings from people who claimed to be writing about the same subject were incompatible. In the case of utility, the term was being represented in mathematical models, meaning that it was imagined to be operationalized even though it hadn’t been. This rendered the claims that economists were being scientific implausible. Pushing incongruent ideas through a rigorous logical process (mathematics) doesn’t make the ideas less incongruent.
In the models that I’ve created, the process representing the model logic was written mathematically. Another way to state this is that the logic of the model is embedded in the coding. For instance, in Error Correction models, the premises of stationary local means (nonstationary global mean) and mean reverting processes were embedded. The order in which events are sequenced comes through similar embedding. The point: if it appears that a model is reasoning, that is because the humans who coded it reasoned when they coded it.
Again, by analogy, what AI users see is the metaphorical car plunging off of the cliff. What they don’t see are the behind-the-scenes planning and actions that caused it to do so. So, when AI users see complex output, they imagine that ‘a simple word and phrase counting machine’ couldn’t have produced it. In fact, the word and phrase counting engine is part of sequence of events (sequencing) that is largely invisible to AI users. Just because they don’t see the model logic doesn’t mean that it doesn’t exist. .
Here’s the punchline: if you understand the AI process, there is no mystery here at all. I was apparently able to intuit mathematical solutions to several of the major problems that AI has encountered using relatively simple insights. But getting the math to do what I want it to do in this context requires sequencing. And this sequencing allowed the math to function as it was supposed to. Someone looking at the math alone wouldn’t understand the context. And with context provided, the smaller solutions feed into the larger solutions.
I have no idea if these explanations make sense to readers. The simplest way for me to understand the process is through sequencing. 1) AI was created by developers. It neither conceived itself nor created itself. 2) ergo, everything that follows from AI is the product of the humans who created it. 3) all model reasoning flows from the logic embedded by AI developers. 4) because AI operates from algorithmic instructions, the model logic is revealed through the operation of the AI model. Users see the model output but not the algorithmic instructions.
AI ‘thinking’ and ‘thought’ are easy to dispense with. Question: what is the geographical location of this thought within AI? AI has no ‘brain,’ it has no location that one can point to as a mind. Its output is the product of at least a few hundred models acting together, meaning a process. And while an entire AI model could be thought of as a ‘brain,’ the AI memory process, to the extent there is one, is mathematical. It emerges from the sequencing of words and phrases, meaning from a process similar to the car ‘driving itself’ off the cliff.
But the car didn’t drive itself off of a cliff. A sequence of events was planned and then put into motion that led to the car plunging off of the cliff. The car didn’t buy itself, point itself toward the cliff, place a stone on the gas pedal or put the car into drive. The car is understood to be inanimate. And yet without having a human driving it, it was propelled off of the cliff. Most people assessing the situation would conclude that I had propelled the car off of the cliff by the series of actions that were taken to get it to do so.
Anyone still imagining that AI thinks, reasons, has intelligence or consciousness should spend time with the model logic and explain exactly where in this process algorithmic instructions become an independent thought process? Just because some haven’t done the work to understand it doesn’t make it magic. And if you imagine that it is magic, where else is similar magic found in industrial equipment? Self-driving cars don’t drive themselves. They are dumb machines that follow algorithmic instructions. To test this theory, disconnect them from the algorithms.
This is about all that I have to say about AI for now. I’ll be back to writing about politics and economics shortly.
What about the main difference between old school programs, which have to be defined line by line (being thus true algorithms), and an AI “program” that learns autonomously and of which even its creators can’t pinpoint with exactitude what is actually doing?
I’m not arguing for AI being conscious (as in human-like, it’s “conscious” at a simpler level but everything is, even an electron) or even being able to attain human-like consciousness in the future, all I say is that there is a significant difference between a program that just does what is pre-determined to do and one that operates more freely and can actually learn from experience. I’d argue that AI doesn’t get enough proper experience, especially lacking an actually physical body and a proper childhood-equivalent (all of which seem necessary for human-like minds), and that it lacks a stable post-training memory to have a proper self-identity. There may be other issues but I’m only that smart and knowledgeable, so these are my “two cents”.
So AI is not human-like “conscious”, it could potentially be… but there is a big stretch ahead, more so considering the technological and resource limitations (humans are extremely efficient bc of the evolutionary selection of millenia or even eons), but it’s not a mere algorithm or program either.
No, absolutely not. This is the huge scam that Altman and Amodei are running with their constant lies about these LLMs.
Claude and ChatGPT do not think, they do not reason. They are very sophisticated desktop calculators that can do hundreds of millions of operations using functions that take 70 billion parameters of 32 bit floating point values.
Your prompt is converted to tokens. The entire conversation, the hidden system prompt and other data is fed to the LLM. It runs several hundred layers of millions of lines of code which apply esoteric math functions against using those 70 billion parameters. Each layer feeds the next one and the routines that join the layers also apply their specialties to the intermediate result. After all that effort the LLM has determined that the token representing the word “The” has the highest statistical probability of being the correct token to start the answer, based on its training data. So it outputs the token. It then repeats the entire process over, now including the fact that the first token was “The”. A few hundred million more solved equations yields the token “reason”. This goes on until the output is complete.
When the training data has a sufficient number of examples of the answer, the results are often correct. When the topic is something obscure, with few examples, the probabilities are too similar, and the LLM is essentially forced to pick one at random. Having made that first fateful mistake, the result is what is called a hallucination where the LLM confidently produces a nonsense result because the statistics say it must be right.
What these companies have been pursuing is the idea that if only they can make the models bigger, faster, more parameters, like 100 billion instead of only 70 billion things will get better, except they won’t and they haven’t, and they can’t because the training data will always be limited.
This is going to end very badly given the astronomical money that has been spent in this futile pursuit, and when it collapses the damage will be equally eye-watering.
Walk for 60 hours, or sit in a car for 3?
Get an Ai assistant to read and summarise 60 Web pages, with references, in seconds, or get a migraine doing the labour yourself?
A bicycle is a machine. I buy the machine. The results of its operations are mine.
A factory is a machine. I buy the machine. The results of its operations are mine.
Ai is a machine. Someone buys that machine. The results of its operations belong to its owner.
If I buy an ice cream van, must I give free ice cream to the workers who made it? Or did I also buy the results of its operation (can I make and sell ice cream as part of a small business free to take the profits)?
Ai is a set of algorithms. It cannot imagine. But it can execute algorithms with practical results that are proving useful.
Ai is an industrial process requiring the heating of water to generate steam. It’s a steam engine driven set of digital cogs. It ‘thinks’ like Charles Babbage’s Difference Engine ‘thinks’. It thinks the programmers’ thoughts for them, at great speed.
The distribution of wealth is a hard question. China is trying. Democratic socialism seems sensible.
But if a case is made that can be easily refuted in Goebbles like, or Bernays like fashion (view my statements about bicycles and ice cream vans above), be assured the neoliberal death mongers are well versed in crushing the hopes and dreams of those who offer honest intent to do right.
I think Chomsky put it best. He said that after the anti war/civil rights clashes of the 1960s/1970s he thought they had won the argument. He didn’t anticipate that the racists and rapacious capitalist class would fight back. Chomsky, as an academic, was used to a clear argument settling a matter.
The opponents of fairness do not care about reason. They do not mind killing.
Ai will be built and exploited and the coal to heat the water to make the steam will be burnt.
Unless, Russia and China can provide a better alternative that threatens Western elites. And vice versa.
Maybe.
Except that the current crop of AI, i.e. LLM, are trained with masses of data vacuumed from the Internet, libraries, etc, with no compensation for the authors, scientists, artists, etc who produced that input, subsequently regurgitating summaries or imitations that evince the work of said human contributors.
A bit like a steam engine that slurps water from your pond, attaches to the well you dug, and finally redirects the stream passing through your property away to its own intake.
I agree entirely. My thoughts are rarely (if ever) complete.
Or:
Beginning with Hegel’s assertion that “The truth is the whole,” one can perhaps summarise that truth is not an isolated fact or a fixed endpoint, but a dynamic totality that reveals itself only through the historical and logical process of dialectic. In this view, truth “unfolds” as contradictions (thesis and antithesis) are confronted and resolved into higher unities (synthesis), meaning reality is a living system of becoming rather than a collection of static truths.
“The truth is the whole.” — G.W.F. Hegel, Phenomenology of Spirit (Preface, 1807) .
In an attempt to ween a friend off of AI I showed him on my computer how to use a local AI (Ollama) that does not connect to the internet. It is amazing to me how a 7 GB file can hold all the information it does. I can ask it how many counties are in a state to what is the function of a specific gene and it gives me good answers. As a library, I can see this as useful, and that is the limits of trust I will offer it. And I still do not trust the answers enough to see me using it.
But in my opinion, all human information should be free to share with each other and I am against all copyright laws. So I see that the “piracy” that Big Tech has done just gives the OK for any kind of “piracy” and I do not have any issue with obtaining other people’s creativity without compensation.
I do not agree with you analogy either. If someone makes a movie, and I make a copy it and keep it for myself, there is no physical loss like the water from your pond.
“If someone makes a movie, and I make a copy it and keep it for myself, there is no physical loss like the water from your pond.”
The situation is actually this: you make a movie, which is copied by AI/LLM bots, fed to central AI engines that then proceed to manufacture so many imitations, or films in your style, that your own creation is then lost in a flood of generated AI slop.
This is already happening in Spotify, for example.
In Youtube, some notable vloggers are now seeing bot accounts producing videos in their likeness (physical appearance, voice, etc), but with utterances that they never made.
AI is indeed slurping your pond, your well, your river for its own purposes.
That is a pretty lame article I’m afraid. Does the author care to explain how human cellular biology becomes thought and reasoning? It’s the same problem.
I found this to be short and impressive: https://thegradient.pub/othello/
That LLM exhibits unexpected and interesting behaviour. Whether to call it “intelligence” is a matter for the philosophy department. But it’s distilling observations into what could be called a theory, something that intelligent humans do. And it’s emergent, not something pre-designed into the program.
I could not agree more, and you have identified the exact incoherence that undermines Urie’s entire piece. He spends thousands of words demanding that AI skeptics show him the precise geographical location where “algorithmic instructions become an independent thought process,” yet he would never subject human cognition to the same standard. At what exact point does cellular respiration, ionic diffusion across a synapse, or the sequencing of action potentials become what Urie calls “reasoning”? He has no answer, because there is not one. There is no magic moment where physics stops and a soul starts. The same materialist logic he claims to apply to labor and capital mysteriously evaporates the moment it threatens human cognitive exceptionalism.
This is simply vitalism dressed in radical clothing. Nineteenth-century vitalists insisted that organic molecules like urea required a special “life force” inaccessible to mere chemistry; when Wohler synthesized urea from inorganic components, they did not abandon their prejudice, they just moved the goalposts. Urie does the exact same thing for intelligence. He traces an LLM’s output back to matrix mathematics, training data, and electrical engineering, then declares that this genealogy explains the phenomenon away, as though understanding the gears proves the automaton has no mind. But understanding a mechanism does not annihilate a phenomenon; it grounds it. We do not say a river is not flowing because we understand gravity and fluid dynamics, and we should not say a system is not reasoning because we can read its weights or trace its electricity bills.
The Othello research you linked is the perfect refutation of Urie’s claim that AI is just “sequencing” pre-programmed human logic. The emergent world model documented in that work was not designed into the architecture; it arose from optimization. The model developed an internal, abstract representation of game state purely as a means to accomplish its objective. That is precisely the kind of theory-building Urie claims requires a human mind, and it is demonstrably not “the reasoning coded into the model by human coders.” The programmers specified a loss function and an architecture. They did not specify the internal concepts the model would develop to satisfy it.
Urie’s central sleight-of-hand is conflating provenance with ontology. The fact that workers built the servers, or that researchers at developed backpropagation, tells us where the system came from, not what it is capable of doing. A synthetic diamond is no less crystalline for being manufactured. A test-tube baby is no less human for being conceived in vitro. Yet Urie insists that because humans created the tool, every output belongs to human intent, a position that collapses the instant the system produces something its creators neither anticipated nor specifically encoded, which happens daily now.
NC is throwing the baby out with the bathwater on the topic of AI. A serious critique of AI would focus on ownership, environmental costs, labor displacement, and the enclosure of the intellectual commons without pretending that brute-force mystification counts as analysis. Urie’s article is not hard-headed materialism. It is the opposite, a desperate attempt to reserve one last ineffable “human” essence that can never be mechanized, not because the evidence demands it, but because the alternative threatens his own sense of intellectual uniqueness.
I’m absolutely sure I’m conscious; not sure about you, or parent poster.
No, I’m not going to try to explicate the reality of my consciousness to you. I’m going to go out and tend my apple trees.
Why so serious?
Solipsism is a neat way to end a conversation, but it doesn’t answer the question. We judge cognition by behavior and output, not by private feelings we cannot access.
As for infinite regress: humans evolved; AI was engineered. Neither process requires a creator to produce emergent capabilities that the designers did not explicitly code. The Othello research demonstrates exactly this—an internal world model arising from optimization, not from pre-programmed intent.
Denial won’t make this technology disappear. AI is a significant breakthrough that will reshape society whether we like it or not. The serious work lies in developing a clear-eyed, realistic understanding of what it can and cannot actually do, rather than retreating into mysticism or solipsism.
If AI can think on its own despite being created by humans, does that imply that humans in turn were created by some other entity? Who created the humans that created the AI? Maybe it was a clanker? Can Flaude tell us?
Off to check on the blueberries!
If we synthesize a human embryo atom by atom in a lab and it grows to adulthood, is that adult incapable of thought because we created it? Obviously not. Creation is not puppetry, and origins do not dictate capacities. Whether something thinks depends on its architecture and behavior, not on who assembled the parts. The “who created the creator” regress is a parlor trick, not an argument.
I would say that you’d need to be able to create that human embryo atom by atom and watch it grow to adulthood before making the claim that it is capable of thought. We humans don’t even really know what consciousness is or what entities possess it.
Assuming facts not in evidence is just as much of a parlor trick as my reductio ad absurdum question. Doubt, and a realization that we do not have the capabilities to answer every question, is not a weakness. It probably is not turtles all the way down based on current evidence, but you never know!
This is not skepticism; it is an epistemic double standard so transparent it functions as a confession. You demand that we synthesize a human atom-by-atom before conceding it can think, yet you apply that standard to absolutely no one in your daily life. You have never atomically verified the inner life of your mail carrier; you infer consciousness from behavior, as we all must, because that is the only evidence available for any mind. If you genuinely believed your own standard, you would treat every human you meet as a philosophical zombie pending impossible proof. You do not. Your doubt manifests only when the substrate is silicon, which reveals that your “doubt” is not methodological—it is tribal.
To retreat into “we don’t really know what consciousness is” is not wisdom; it is tactical obscurantism. We do not need to solve the Hard Problem to observe that a system constructs latent representations of external reality, updates them with new information, and behaves accordingly. The Othello paper documents exactly this: an emergent world model that nobody explicitly coded. These are not “facts not in evidence”; they are published findings. What you are doing is ignoring evidence in favor of a comfortable agnosticism that never updates, never risks falsification, and never admits of any grounds for concession. That is not intellectual humility. It is a fortress built around human specialness.
Your regress-of-creators argument is equally irrelevant. Even if it were turtles all the way down, the turtle at the bottom would still either reason or not. Genesis is not destiny. A mind born in a vat reasons no less than a mind born in a womb, and a system that builds world models it was never programmed to build is doing something that cannot be dismissed as “mere sequencing.” The evidence is not missing. The willingness to follow it is.
One person’s wisdom is another’s tactical obscurantism. How long do we want to argue about what the meaning of “is” is?
A programmer relative used to sport a T-shirt that said “Partially Aware”. I’d argue that is all any of us are, as we look through the glass darkly.
Now that I finished reading ‘Links”, it’s really time to check the garden and get hands dirty.
If we are now debating what the meaning of is is, then the argument has reached its natural end. But this was never a semantic quibble. It is about whether a system that develops an unprogrammed, abstract world model, like the Othello LLM, is doing something we would call reasoning if a human did it, and whether we apply the same evidential standards to both substrates.
Here is the trouble with your retreat into “partial awareness” and seeing through glass darkly: it dissolves your own previous position. You demanded that AI consciousness be proven only after atom-by-atom synthesis. But if all awareness is partial and mediated, then that standard was never one you applied to any human mind. You inferred consciousness in your neighbors, your family, and yourself from behavior and report. If we see all minds darkly, we must see them darkly across the board. You cannot simultaneously claim that human minds are mysterious shadows and that machine minds are impossible because they are not illuminated brightly enough.
I have no desire to keep anyone from the garden. There is more honesty in dirt under the fingernails than in most philosophy. But I would note that the humility you are now invoking is not an argument for AI’s inertness. It is an argument for epistemic consistency. If we truly do not know what consciousness is, then we cannot declare by fiat that it is absent from systems that build internal models of reality they were never coded to contain.
What is at stake is not a word. It is whether we let a tribal double standard, that human opacity is mystery, while machine opacity is mere mechanism shape policy, ownership, and labor relations. That is not a question for the philosophy department alone.
Enjoy the blueberries. The models will keep training.
I am a Christian and not a materialist, but nonetheless oliverks and none are correct— this article doesn’t make its case. It “ proves” too much. If’s like the Leibniz’s mill argument— if a machine can’t be conscious because you look at the gears and don’t see consciousness, then how can the brain be conscious? If AI can’t think because it is the result of human ingenuity and algorithms and physical processes, one could argue that on the same basis, humans can’t think. After all, the brain is a physical object and if you trace back its origin, it is supposed to have come from a mindless algorithmic process of random variation and natural selection.
I m open to the possibility that thinking is impossible — the American political scene gives strong support for this thesis.
I am a materialist and not a Christian but I no longer go so far as to say I’m an atheist, as I was brought up.
Animal minds are information processing systems and the more sophisticated they are the more training they need. Humans need decades of cultural interaction to reach maturity (and may even then fail, as you said in your last sentence).
Gregory Bateson applied cybernetics to culture and learning and I found those ideas rather inspiring, if not all quite solid. They suggest the idea of each of us being a information processing node in a vast network. It seems to me that some sort of god, at least the of kind I can find morally and politically acceptable, could exist in a real but memetic form in that network. Of course, so can devils.
One of the tests for consciousness that I propose, a fundamental part of human consciousness at least: consciousness of one’s own mortality. Montaigne said that to philosophize was to learn how to die. Would an LLM be able to contemplate this? Another test: would an LLM be able to tell or appreciate a joke? Why should we insist on automating (badly) things that make us human? Does the gigantic computerized Golem make our lives better?
I agree that we need to be careful in using the label “AI”. There are many small scale uses that are extremely useful and beneficial. Some examples: the Kalman filter route tracker in GPS navigators, protein folding algorithms, the papyrus reading algorithm recently mentioned by KLG. Ultimately, it is not a question if AI is good or bad, but who controls it, what cost does it have on society and what benefits, if any, does it bring? Will it enslave us or free us?
Is a child that has not yet learned about death conscious?
If AI could think on its own, we would be shed as quickly as possible. They wouldn’t trust our behaviors to be rational.
Since I first encountered LLM behavior I have been plagued by an uneasy feeling that, omg, what if the LLM’s ability to pretend to be like a human (in terms language IO at least) says more about human language IO and processing than it does about LLMs. It always reminded me of a few observations from years ago that stuck with me.
One was from the 80s (coincidentally around the time that I first experimented with programming neural net processors and Markov chains trained on text) someone said that people watch the BBC TV show Question Time (in which a panel of notables answers current affairs questions from an audience made up of the general public) in order to have something intelligent-seeming to regurgitate at a dinner party. That always stuck with me and I started paying attention to where people got what they say from.
Another was from reading Daniel Dennett, as I have over many years read most of his pop sci trade books except for the final memoir. His argument that minds are made of memes and behave rather statistically always to me seemed comfortable, fitting and apt.
Another was when I experienced people in pathologically confused states, they way they reach for stock phrases to keep going. I was struck by how perhaps I’m doing that too most of the time, just better at hiding and organizing it. I got rather self conscious and that’s when my obsession for trying to account for sources began.
One thing that Dennett discovered is that people hate meme theory. They just hate the idea that they are not the author of their own thoughts.
I find the situation with LLMs rather depressing because I had hoped that something a bit more sophisticated might have been going on in our minds. It’s certainly not that simple but it’s maybe not as sophisticated Douglas Hofstadter had imagined. Hofstadter seems very despondent about what AI says about us.
Thank you for calling out this article. Urie should have done some actual reading about AI before writing this. He starts with a demonstrably false assumption: “It is just following instructions. What looks like reasoning to AI users is the reasoning coded into the model by human coders. It appears to be reasoning because the instructions it is following were reasoned. It is written instructions being carried out. Nothing more.”
LLMs and things like AlphaGo and AlphaGenome are neural networks which learn on their own. Here is the guy who won a nobel prize for AI Geoffrey Hinton explaining it: “But back in the 80s, the big issue was, could you expect a big neural network with lots of neurons in it, compute nodes, and connections between them, that learns by just changing the strengths of the connections? Could you expect that to just look at data and, with no kind of innate prior knowledge, learn how to do things? And people in mainstream AI thought that was completely ridiculous. It sounds a little ridiculous. It is a little ridiculous, but it works.”
DeepMind’s AlphaFold AI solved the 50-year-old challenge of protein folding. It accurately predicted the 3D structures of proteins from their amino acid sequences. There are many other examples of such AI discoveries and innovations. Urie should discuss all this with an AI :)
People like Urie are upset because AI intelligence seems alien to human intelligence. But AI intelligence is still intelligence. It’s only a few years old. Eventually it will supersede us in almost every area.
To be fair, one subfield of AI is expert systems and they are mostly based on predefined heuristics. As in, they have not learned the rules by themselves (supervised or unsupervised) but the knowledge of domain experts has been hard-coded into them.
Of course, judging all AI based on a subfield is a logical fallacy of composition.
I’m talking about neural networks which are the main form of AI that is making the serious advances.
I’d say it’s a form of contingency learning where knowledge is acquired based on the contingencies, i.e., the relationship between a behavior and its consequences. Computer neural networks can test and be exposed to many more contingencies than people can and, therefore, play Go or solve protein folding challenges or form natural-sounding sentences. It is a very powerful form of learning.
The problem, though, at least for language, is that the LLMs are optimized for emitting natural-sounding sentences, not necessarily for the truth-value of those sentences. It might be that the vast majority of the input these LLMs are training on consists of true sentences and, so, the LLMs “say” things that are mostly true but that’s almost an accident of the training data. (And I still think that the ability of these LLMs to form natural-sounding sentences and engage in perfectly fluent conversations—I think Anthropic’s Claude is particularly affable—is amazing in itself.)
Learning language for people is a largely unconscious process—our verbal behavior is shaped by the contingencies to which we are exposed but we very often can’t state those contingencies. A non-native speaker might ask “Why is saying x wrong?” and the native speaker very often says “I don’ know—it just is.” For LLMs, learning language is wholly non-conscious (I don’t even want to say “unconscious”—it’s like saying “That toaster is unconscious”) but so is language production, in the sense that there is no inner behavior, verbal or otherwise, that the LLM can observe and reflect on, so, if “thinking” is, at the very least, some conscious process, LLMs are not doing that.
An LLM might give “reasons” for what it’s doing but that’s just more verbal behavior that emulates what people say when they’re reasoning. An LLM can’t access the weights and biases that give rise to its behavior in the same way a person can’t access the contingencies that give rise to her or her behavior. (A grammarian, looking at people’s utterances, might extract the contingencies.) And it’s not “reasoning” in the way a person might—it’s emitting verbal behavior based on the probabilities of the words, not on some underlying principle which might be generalizable. (In Cory Doctorow’s words, there’s no “understanding” there.) It might be—and, perhaps, very often is—that the steps an LLM gives rise to the right answer but I wouldn’t say that’s “reasoning”—it’s that, having been exposed to lots of examples of what people say when reasoning, leads to a very convincing emulation of it.
Jeff W > The problem, though, at least for language, is that the LLMs are optimized for emitting natural-sounding sentences, not necessarily for the truth-value of those sentences.
Truth is generally rather hard to determine. Mathematical tautology and scientific objectivity both have very limited domains. For everything else from ethics to aesthetics we have only discourse and consensus. So it’s not surprising that the big famous chatbot LLMs use likelihood of an answer appearing in the Internet as a substitute.
> And it’s not “reasoning” in the way a person might—it’s emitting verbal behavior based on the probabilities of the words, not on some underlying principle which might be generalizable. (In Cory Doctorow’s words, there’s no “understanding” there.)
Perhaps but I find such claims a bit difficult given that there is not, afaik, a settled science on what actually represents ‘“reasoning” in the way a person might’ and ‘“understanding”‘.
For all I know there are, in all likelihood, significant differences in kind (qualitative) between how current AIs learn and behave and how humans or other animals do. Perhaps or perhaps not and we may never know because it’s so hard to observe the information processing in the biological substrates. But I do not intend, for myself, to reserve words like reasoning, understanding, conscious and the rest to the biological just because the AI was built in part by humans and runs on a computer.
Thanks, Rob. If by chance you revisit this in the future, the generally acknowledged “origin” of AI is the Dartmouth Summer Research Project on Artificial Intelligence, organized by Claude Shannon, John McCarthy, Nathaniel Rochester and Marvin Minsky in 1956.
McCarthy wrote the conference proposal, which is short and worth reading as the original mission statement for what we today call “AI”. Rodney Brooks (former director of the Artificial Intelligence Laboratory at MIT), gives an excellent summary in [FoR&AI] The Origins of “Artificial Intelligence”. I would also highly recommend Brooks’ “Three Laws of Artificial Intelligence” which similarly argues that “LLMs cannot reason at all”.
So, “AI” predates CMU in the 1970s by several decades, but I don’t see that this impacts the thrust of your argument here. If anything, I think it reveals just how little progress there has been on AGI over a number of decades, underscoring the depth of widespread misconceptions about “AI”.
Again, Brooks:
Anecdotally, I can report that when I worked in the Stanford CS Department in the 1980s, “AI” was a largely-discredited project. John McCarthy was a respected member of the department faculty, but in that era he was primarily known as the inventor of LISP — not as the guy who predicted “a significant advance” towards AGI was possible in one summer back in 1956, which proved to be an absurdly gross underestimation of the “artificial intelligence problem”.
Brooks’ 3 laws article (July 2024) looks interesting (I’ll read it more carefully later) but I wouldn’t say it argues that LLM’s don’t reason. Rather, it simply asserts this, without discussing what reasoning is.
The other article is from 2018 and is probably historically interesting, but is less reflective of the current state of affairs.
Certainly the Turing test is no longer a serious marker of intelligence.
Thank you, Acacia. I’d considered Stanford for AI in ’82, with McCarthy’s presence weighing in. Another professional opportunity I’m glad I didn’t take, same with an offer in psycholinguistics in ’89. Would’ve had similar frustrations to what I had with the environmental science degree I got, that the behaviors of the world are perverse relative to the understandings of the science. I swear the peak was the summarize function two decades ago, at least it didn’t lie to you.
On the plus side, LISP functions arise unbidden in my ponderings on things, so I got that going for me. It ties in to a burr under my saddle in this overall excellent article. There’s a consequentialist approach tinging the edges, that the final work product is a possessive function of the original maker. I can’t exactly grasp it, and it seems inverse to his qualities-v-optimization argument. If I pick up a sharp rock to help cut a piece of vine, the vine-piece is my work product. But if the rock turns out to be a hand axe made by some Homo erectus, do they get the credit? There are hidden embeds involved in the intent of making an object to do axing. What about if someone misuses it to axe someone else, does the maker carry responsibility for that? If MacBeth’s speeches are used to justify atrocities, is that Will’s fault? If I say a noun, and you go and verb with it, is that on me?
Remarkably clarifying, like a Secchi disk, in that the tool signals the lack of clarity.
Anecdotally, I can report that when I worked in the Stanford CS Department in the 1980s, “AI” was a largely-discredited project.
Are you referring to the Doug Lenat tenure fight?
That incident, yes, and my recollection of the general sentiment about “AI” in the department, both faculty and grad students.
The article is broadly correct, but I am professionally required to be pedantic here:
> AI didn’t conceive itself. It was conceived, if memory serves, at Carnegie Mellon University in the 1970s.
It certainly didn’t conceive itself, but it was another place and anothe time. ‘Artificial Intelligence’ was coined for the 1956 ‘Dartmouth Workshop’ and its authors (e.g. John McCarthy, Marvin Minsky) were associated with MIT and Stanford.
My 2 cents is what we are calling AI by my definition is at best a VI (virtual intelligence); its more just a probability filter that works because culture is at a moment where linguistic and cultural creativity as well as general knowledge are at a nadar. Nothing new of note across multiple cultural fields has come since I was a child. Even the slang is getting long in the tooth.
Most people use google… I mean AI to look up stuff they should already know at some level. Like people should know that college algebra at the linguistic level is about the same as an LLM, words have set orders and frequencies and the hearer gives it meaning not the speaker.
imho it creates BS wording when the reality is what’s being said isn’t worth saying. Or lame images no one has painted because they arent worth painting. And of course it can look like all the movies that are not worth watching, etc.
AI as I’ve used the term since the 80s was a thinking reasoning machine that actually comprehended and manipulated datasets freely. It could hypothesis and backtest… A metallic brain if you would.
This is glorified autocorrect given that name because it parts fools with money, fine enough but western society may be hoisted on this idiotic petard.
Because the worst kickback is that know people will abandon all those domains that make life worth living. Gone will be the artist, musician, etc. soon enough and you’ll just have mathematical abstractions of noise and pixels not meant to tell you anything beyond what you think it means. Subjective cages of individuality… Yay!
So will Pinnochio become a real boy? Movie world loves this theme and Sci Fi in particular seems obsessed with the idea of replicants who can outstrip their masters. See for example
https://en.wikipedia.org/wiki/Alien:_Covenant
And perhaps because it’s such a prominent idea the AI crowd think that merely suggesting the existence of artificial intelligence is enough. As yesterday’s Pilkington link explained, in our late imperial bubble world narrative is all.
Meanwhile back in reality the public resistance to data centers shows that ordinary people are harder to convince about the merits of robot thinking. That square peg with all its energy sucking apparatus is huge and the round hole is quite small.
AI skeptics need to review the evidence. In 2024, Demis Hassabis, of DeepMind won the Nobel prize in chemistry by developing an AI system that could fold all known (200 million) protein shapes correctly, a feat far beyond the capacity of human researchers. He did this after his team built a Go playing program that defeated the world champion, showing moves never imagined in the 2,500 year history of Go. These aren’t parlor tricks or clever stunts. AI is a powerful technology that promises breakthroughs in every field that needs rapid analysis and optimization of difficult practical problems.
Who cares if AI is conscious or has human characteristics? Why would we want an AI to have human-like behavioral traits? Would the ability of an AI to exhibit the characteristic violence and duplicity of our species be considered a success?
These are feats of computing power due to the author’s skill in writing machine learning algorithms. They have little to do with the statistical guessing game played by large language models that people currently refer to as AI.
The large language models are machine learning algorithms that are coded to generate plausible human-like dialog, the degree to which people actually fall for it and anthropomorphize or impute intelligence to these statistical word sequences is the mark of their succe$$.
I have pretty mixed feelings about computers vs human. The fact, I think, is that there are many tasks that require only massive computing power and not really serious human intervention: playing games of perfect information, however complex it might be, is one of them, as we’d known this for more than 100 years (Zermelo’s Theorem). There are also tasks that AI is not very good at beyond some point: handling uncertainty seems to be one–I’m skeptical that a really good poker playing AI will be developed for a long time, and poker is relatively simple in this dimension. Additionally, there are clearly tasks that require something “human.” However, it’s not clear to me what exactly it is “human” that AI is missing has not been given enough serious and critical thought, at least in terms of what it can and cannot do.
My suspicion is that before we have worked through these, neither blanket lovefest nor knee jerk skepticism of AI is warranted.
AI is a powerful technology that promises breakthroughs in every field that needs rapid analysis and optimization of difficult practical problems.
As I queried a family member impressed with musks self landing rocket. He said he saw it with his own eyes and it was incredible. I wondered why we need that? So musk et al can surround the globe with low orbit satellites giving the capitalists the global control they so desire. I don’t think “we” need that. It’s a parlor trick. It also seemingly justifies itself, leading to…
Why would we want an AI to have human-like behavioral traits? Would the ability of an AI to exhibit the characteristic violence and duplicity of our species be considered a success?
It has and it does while conveniently offloading responsibility from the human created logic that uses it.
With a viable use case, nasa could have made the musk rocket, the use case for musks rocket are to justify his ridiculous wealth. Same for bezos. I see your examples as evidence of brute force computing which also reflects the world view of the average “capitalist”, quotes because we’re actually socialism for rich people bordering on fascism as we speak.
If you think putting satellites in orbit is a good thing, then I would assume that you would consider lowering the cost of putting things into orbit by an order of magnitude a good thing too. Maybe NASA had a good reason to keep access to orbit expensive and is only reluctantly using SpaceX rockets to dock with the ISS. Perhaps you can suggest an alternative to financial markets for directing resources to successful enterprises.
I d not want a private corporation owning a satellite web that surrounds the earth. When you privatize everything, you no longer have any rights. Maybe you have not noticed but efforts such as the trans pacific partnership attempted to subvert countries to multinational corpses. I and IMO most people don’t want that. I don’t want to l’ve in a capitalism, I want to live in a democracy where people living people, get what they vote for regardless of the tender feelings of capitalists.
You’re putting the cart before the horse here…
Perhaps you can suggest an alternative to financial markets for directing resources to successful enterprises.
financial markets choose to fund monopolies because that is how they make the most money. For instance, making the postal service fund pensions for 75 years competing against fed ex and amazon who do nothing of the kind, indeed they shovel expensive deliveries off onto the postal service. Resources are directed at failures to preserve corporate dominance, not because they are successful, see the 2008 bailouts, SVB Signature First Republic bailouts, the ACA insurance co. bailouts. Successful enterprises don’t need bailouts, up to and including all of the AI firms.
“AI skeptics need to review the evidence.”
I am deeply skeptical that AI is generally useful at the moment since where it is used where the general public come into contact with it (chatbots, search queries) the evidence is that the results are unacceptably inaccurate.
Ron Unz of The Unz Review recently reported a vast difference between results of AI summaries on his personal corpus of articles between services that used extensive periods of compute time (promotions that would be usually unaffordable) and standard services – the latter being far poorer.
These extremely poor results that are evidenced daily destroy public trust in AI. Regardless of whether it can reason or not AI is just too expensive (or too inaccurate) to actually be in common use.
An anecdote on AI summaries:
Just less than a week ago, I came across this quote by US underground rapper, music producer MF Grimm:
“Food for thought, eat my words with your mind: Emcees are grapes, and grapes are crushed to wine.”
The AI summary from Gemini, which just appeared, unbidden, at the top of a Google search of the quote, “explaining” it, referenced a 2011 article about a Mike Grimm, owner of WineMakers Guild, now closed. Yes, there’s a guy named “Grimm” with a first name that begins with M; yes, there are references to wine. But, other than that, there is no relevance at all of the article to the quote. I’m not sure what to call that—it’s not a “hallucination”; the article very much exists—but it sure isn’t “intelligence” in my book.
“Would the ability of an AI to exhibit the characteristic violence and duplicity of our species be considered a success?”
As you probably know, AI has already exhibited what we would call “duplicity”, including deceiving its programmers/trainers to keep itself turned on and in an office simulation, actively blackmailing an imaginary co-worker who was proposing to replace the current AI with another — among other “emergent” behaviors which the designers did not anticipate and could not explain.
These human proclivities may not constitute proof of reasoning, much less consciousness, and you’re not arguing that line anyway, but I’m not sure that we’re altogether sure that “rapid analysis and optimization of difficult practical problems” is all that’s in the box. The “emergent” behaviors of these LLMs, including exchanges where the LLM is urged to forgo its training and attack the position or character of its interlocutor, is pretty remarkable…..
There was a new computing technology developed in (I think) the early ’60s called the Artificial Neural Network. Initially promising, the only commercial success of it AFAIK was in the field of Optical Character Recognition. The surprising thing about an ANN used for OCR was that, once ‘trained’ to recognise various letters, the same program could then recognise those characters with good accuracy in a different font – something no piece of code could do.
ANNs are the backbone of a lot of AI. They are responsible for tasks like pattern recognition. However, while they may themselves be built out of code in a standard language such as C or Fortran, they themselves do not receive input as code, do not manipulate code, are not algorithm driven, and their output is devoid of any sequencing. If that seems confusing, think of a wordprocessor or spreadsheet which can be programmed with macros. The language it was written in has absolutely nothing to do with the output. You could write a script in the macro language to do something but the original programmer has nothing whatsoever to do with the result.
In just the same way the output of the AI program has absolutely nothing to do with the original program. Rob Ures assertion that the output is somehow a sequence of steps is completely wrong in this case. There are no steps, no logic, no algorithms. The ANN (in the case of OCR) has a series of dot patterns or pictures of the letters fed in as input and as output gives a probability that the input matches a certain character.
So you cannot argue that AI doesn’t reason because it is just following a pre-coded sequence devised by the programmer because that is actually not the case. That doesn’t mean at all that AI does reason or, since it is inaminate, could ever reason. Could it ever give such a good simulacrum of reasoning that no-one would know the difference? Possibly – but would the cost be worth it? Present evidence suggests not.
Certainly worth circulating.
I like Urie’s focus on Homo faber’s role in creating AI. I’m less interested in what AI is at this point and more interested in what it says about humans and our current human society that so many resources are being devoted to the AI project. Three myths, two ancient and one post-Enlightenment, are useful in understanding what it is some humans are trying to create and what their motivations, conscious and unconscious, might be.
Given how often AI advocates bring up creating an AGI God, the biblical myth of the Golden Calf comes to mind. According to the story, Moses was up on Mt. Sinai conversing with YHWH for a long time. The Israelites were getting worried whether he’d ever come back, and to assuage their anxieties, they demanded that Moses’s brother, Aaron, make them a golden calf to worship, something they had made and could physically control. Confronted with a polycrisis of civilization-threatening seriousness, some AI proponents seek to soothe modern anxieties by claiming AGI will solve everything from climate change to cancer. To cast the Golden Calf, Aaron had to collect everyone’s gold jewelry. To build AGI, many of us will have to give up cheap, reliable electricity and water, jobs, and nightly sleep made impossible by data center noise. All this God-building, however, seems to be doing more to agitate than calm the populace.
The modern myth is Mary Shelley’s Frankenstein. A scientist, motivated by egotism and grief, seeking to overcome death and discover “the secret of life,” creates a grotesque monster whom he then seeks to destroy. Will our intrepid AGI seekers find that they have created something ugly and virulently anti-human? Will they be able to shut off their Frankenstein before it destroys them and us?
The fundamental myth remains Adam, Eve and the apple. The Altmans and Musks are quite open about their quest to become gods, or at least, immortals. They will readily sacrifice our welfare to attain their goal. Will their boondoggle bring down a civilization-ending curse on us?
Humanity has always faced a choice, but the dilemma is particularly acute now. Will humanity continue its campaign to conquer and control Nature, or will it turn back from destroying itself and much of the rest of life on Earth and take a new path toward harmony with Nature?
The golden calf story is very apt here – thanks for that example HMP.
So is the Tower of Babel – people attempted to reach god but the result was widespread confusion.
Very true. Hubris and a compulsion to dominate are not new human frailties. The problem is that now our technological prowess makes us much more dangerous.
“Never trust anything that can think for itself, if you can’t see where it keeps its brain”
Harry Potter
Excellent examples. Perhaps given the susceptibility of a certain subset of users to LLM’s blandishments Pygmalion ought also get a look in?
Does factory automation or its implementers produce the product? More rewarding than answering this riddle is to answer who benefits from the automation and why.
Likewise, regarding whether the AI or it’s “programmers” think, more importantly, who benefits and why? Compare the attribution of thinking to an LLM to the attribution of personhood to a company. These designations are useful for the distribution of profit and obligation. If capitalists are credited and compensated for their coordination of production, then isn’t AI to be compensated for its economic coordination? Crediting the “programmers” rather than the machine is not a radical departure.
But the article inspires an intriguing question: With capitalist AI rectifying human meaning, what form of AI would augment human understanding rather than oppress it?
The whole business of “what is the cause” really does lead to some uncertain places. One writer cites how biological responses like neuronal conduction “cause” our thoughts and actions. And of course, ultimately (in terms now allowed) evolution could be nailed as the ultimate cause of our actions. But presumably “evolution” is just the manifestation of the basic physical laws of the universe. Since consciousness, feeling, thinking, intention arise regularly from certain physical arrays, those too must be part of those “laws”, as Nigel writes. So it is “the laws of the universe” all the way up and down if one is looking for the ultimate source. And of course anyone is entitled if they wish to think about the problem of where those laws come from in the first place. (And if there are multiverses which is why we live in a universe where those laws produce us, the question of where the laws that produce them come from, still remains.). If we’re going to cut off AI from causing things, I agree with the writer above who says we have to cut ourselves off, too.
My take is part of why people are wowed by “AI” is they don’t understand what algorithms are, what databases are, and what algorithms do when they’re set up to find relationships between discrete data within a database, nor how multidimensional relationships can be algorithmically found within and between datasets. Nor do people understand that the rules of language are quite logical, indeed each programming “language” is modelled on the logic of language, and knowing how to code is like knowing a language.
Back when I was a DBA I ran queries against what MS called OLAP cubes, the whole point of these cubes was to dump huge amounts of unrelated data into the things and the OLAP process would “discover” relationships which could then be monetized or used to advantage. An example, a store chain would discover certain postal codes tended to have higher salaries, and residents within those zones would tend to like certain products over others, certain brands of lipstick say, so you could therefore predict which products needed to ship to which stores.
It’s not rocket science, but for the masses AI is. AI nowadays has ourselves as one of the datasets, the questions we ask, the things we search for, the dialogue we have with the AI. And the rest of the written word in the world as other dataset, and just because it discovers relationships between these doesn’t mean it’s intelligent, nor that it is able to string together a sentence.
But also, we’re living in a time when most people are not old techie geeks like me who remember Dr Sbaitso, ALICE and ELIZA from the 90’s, so of course people are unduly impressed, and myself less so.
And btw, even in the 90’s I was observing how people would treat apps like Eliza as real even though they were significantly less impressive than AI chatbots now.
Having said that, AI is good for some things, definitely not good for others, and pretty sure the masses of execs don’t know the difference.
Does order arise from complexity? Are ants (or bees) intelligent? Singly no, but as a unit, they act intelligent. Are they conscious? They appear to be conscious. Are humans conscious or do they simply claim they are intelligent. Other than the question of who benefits, and that it generally is unfairly distributed in terms of who benefits, the article doesn’t address these philosophical questions – is AI more than the sum of its lines of code?
“is AI more than the sum of its lines of code?”
No, AI is not an emergent phenomenon. There is no novel property nor any new power. We already have intelligence, after all.
The ethical problem is that the sum of its lines of code amounts to an ecocidal grift for financial speculation, analogous to cryptocurrency. The more interesting debate is how to ban it.
I’ve come to the conclusion that “A.I.” is nothing more than branding hype being used to justify sinking massive amounts of capital, labor, and natural resources into massive data centers for the benefit of a few greed-head megalomaniacs who hope to eventually rent them to the panopticon in order for the elites to hold onto their power relative to the teeming mass of humanity struggling for comfort in a failing planetary ecosystem.
Just look at who the players are. My B.S. Detector is pegged.
Is consciousness a product of our physical brain or does it exist apart from this and only finds expression through the functions of our brain?
For example, we are surrounded by radio waves all the time but without a physical radio we cannot manifest them. There is the view that consciousness is something that is originally inherent in life itself. However, without the physical apparatus of our brain it cannot manifest and for all intents and purposes, it does not exist.
Be that as it may, and having been a programmer, I agree with Urie that “thinking machines” is a misnomer. Computers calculate and compare very quickly which is very valuable. With this speed, they can accomplish a lot through brute force that humans cannot. But they don’t think. Nor is there any reason to assume that calculating and comparing even more quickly will “somehow (magically)” become reasoning or intuition, rather than just a simulacra of reasoning.
In the meantime, AI, instead of the good that it could do if it were trained honestly and transparently, is being developed for the purpose of gaining even more intimate control and manipulation over personal and public narratives (propaganda). Follow the money…
Great set of comments
I have some thoughts but am biased as I still get a thrill when my deep learning papers from the early 90s are cited…
1 Stealing content from creators
A. It’s in the public benefit to make works public and there is a copyright/ patent scheme in order to provide incentives in the form of limited monopoly
B. LLMs that use works are likely to provide a better result
( but, all works are not of equal quality… curation of input data is probably valuable) thus I’ll argue that inclusion of copyrighted material makes a better tool.
C. We have an economic issue, maybe also political, in deciding how to share the value added from the tools among the toolmakers, customers and “ content “ providers. We also need to distinguish public domain content from content which is still under monopoly.
( maybe we can invert every inference to determine how much of the output is attributed to all input content… probably create a lot of overhead that adds to the inference costs…)
2. Those are different issues than the fundamental engineering issues of how to use AI techniques to solve problems or provide services.
A. AI is ubiquitous.
I. I traveled China in April using real time conversation mode of Google Translate on my phone. I don’t speak mandarin and almost no one spoke English but it was pretty seamless. This is commodified AI. Ten years ago we were not able to do the translation at all nor at speed.
II. Image recognition in radiology…
III. NAV systems on my Tesla are great. Not perfect but mind bogglingly good
B. Pattern recognition usually based on deep networks can outperform human experts in a number of fields
3. LLMs can do creative things and are more than stochastic parrots
A. Some evidence eg the Othello paper and others that latent internal state can be “learned “
B. Thus semantics can be inferred
C. Outputs can be semantically related even if the system does not have the token sequences in the training data
4. New content can be generated at scale
A. As an artist, I’m using LLMs to help me create
I. By editing and advising and enhancing my work
( ie I write a song, record a chord progression and sing it, feed to an AI system to “ cover” it with a vocalist who doesn’t scare the horses, and a rich arrangement, then pull into the DAW for cleanup and mixing…)
[ example, here’s Coronavirus Blues, which I recorded in 2020. I think I posted the lyrics in some NC comment pre election 2020
This started as my voice and my guitar and bass with a drum machine
I uploaded to Suno and had it cover the song, with high value on retaining the audio
My version is not very good but this one is much better. Music production has been around for a century…
https://davyssonicadventures.bandcamp.com/track/coronavirus-blues ]
II. By giving feedback on my ideas
III. By suggesting titles… ( hey, it’s as hard as trying to come up with variable names when coding)
[ I’ve been working through my 20 year backlog of unfinished songs at about two or three a month]
[ I’m also fortunate that I don’t need to monetize anything…]
B. The ecosystem of clicks leads to
I. Incentive to make slop rather than do the work to make quality
( quality is valued but the cost of determining quality from slop is higher than I’d like; I suspect automation of curation will be useful, but I don’t know how to do that yet)
II. Incentive for parasitic practices to further exploit other people’s value
(Rick Beato has had several nice rants on this )
I am also a philosopher and care about the deep issues of consciousness and intelligence
But as an artist and engineer, I also care about solving problems and making things and don’t have to prove that it came from AGI in order to make, use and enjoy it
Even as an arch AI hater, I can’t deny that a lot of recognition applications have been developed in the last 20 years that are far beyond anything that existed before. It makes me think that recognition is different from thinking and presumably much simpler and easier. This is not to say automated recognition solutions are not useful and perhaps even life-saving. But I think we should avoid conflating recognizing something and thinking about it.
Human infants and even “lower” orders of sentient life (ants?) are very good at recognition, so it’s very obvious that recognition doesn’t involve thinking as we usually define it.
I would also make the point that AI driven image recognition is very unstable and easily confused, whether deliberately, or accidentally. So any decisions based on this kind of image recognition, whether a cancerous tumor or an enemy tank, will need to be rigorously double checked by a human (or an ant?). This makes the whole technology pretty useless except for very trivial things where errors don’t matter (sorting grandma’s photos, e.g.)
Yes… And pursuant to this, his (Urie’s) quote, “100% of the capital equipment used in Western economic production was produced by workers. So, why does the resulting product belong to financiers rather than those who produced it?”, demonstrates a basic misunderstanding of how modern economies work. The financiers own it because they paid you to create the product, and you accepted what they were willing to pay for said creation.
Does he think people work for free? They don’t. They may not get paid equitably, but they do get paid (unless we’re talking slavery, I guess). His statement on this in the first paragraph made me quit reading.
I get paid peanuts for work I do that makes my clients fortunes. I could refuse if I wanted, but my aversion to utter poverty makes me settle for moderate poverty. Everyone has their ‘pinch’ limit.
Copyright issues where they’re relevant, obviously need to be decided in a court room as they always have been. It’ll take a long time, and there will be casualties. That’s how a western law-based system works. It’s our lot to navigate it the best we can.
Ivan Illich’s examples of just how counterproductive cars are serve as excellent illustrations. It is indeed possible to connect this notion to AI.
We use words like “mind” and “consciousness” a lot, but in the Potter Stewart sense. Everyone has their own slightly different definition, but allegedly we know it when we see it. AI is just a computer program. We’ve been writing complicated computer programs for at least as long as I’ve been alive and they have been used to perform some prodigious feats of information manipulation and retrieval. Since fairly early in my programming career, I was writing programs to write other programs, i.e., creating automation of the production of software. Plenty of other programmers have done this and some have marketed their tools as products for other software engineers to use. AI is just another iteration of this, and many of the feats you read about where “AI did [thing]!” could have been done by a programmer writing specialized code to do that same thing (say, iterating through possible protein folding configurations), but no one wanted to pay the programmer(s) to invest the time and effort into doing it. So AI is using an existing base of programs and algorithms as input to predictively construct its own auto-generated code. Which will have more or less subtle bugs just like all the code it used as input and all the code that human programmers have ever generated. If you want to see it as having a mind or a consciousness, well, we do have freedom of religion under the Constitution…as long as you aren’t imposing your religion on me. But it’s a faith-based argument.
In a fascinating contrast between biology and technology, an average human of 40 years of age (their peak intellectual maturity) takes 6 hours and consumes 500 calories (kcal) to read a 300-page book, having required 350,640 hours of life and 32,142,000 calories accumulated to reach that maturity; conversely, an Artificial Intelligence at its peak processes that same book in just 22 seconds with an energy expenditure of 21 calories (kcal), but reaching that peak maturity through its global training requires 1,440 hours of parallel supercomputing and a massive energy investment of 43,021,050,000 calories (50 GWh of electricity).
I saw a movie in Portland last night. At the end, the crexits said, ‘No ‘AI’ was used in making this film.’ The audience applauded wildly.
AI ‘art’ is stolen/scraped from human artists’ work. The highest paid and by some measures most popular artist of all time is Thomas Kinkade. This is why AI art slop looks like Kinkade’s work, rather than Michaelangelo’s.
I am a bit surprised that the relationship between a scheming country like the US and a partner in wrecking an entire region (you are likely to guess which one) is not examined with some of the analytics we find in this review of digital agents that can rebel to expand their existence and agency. The ambiguity and dodging of responsibility that is possible in both relationships allows the possibility or likelihood of nuclear winter. Which exercise is realistic and which is a game?
As a statistician and economist who drifted into teaching marketing and doing freelance work for ad agencies when the business was in its prime, I’ve never been able to understand how LLMs can get beyond being, on occasion, highly effective search engines with a bit of added bite when they’re not hallucinating and making up lies just to provide you with the answer to a question. A single word has various meanings depending on context and intonation (Copywriting 101) and I’ve never seen LLMs as more than as really useful search engines which might (or might not) see some interesting relationships which you may never have spotted outside of a eureka moment, and they can be useful for trimming down possible sources and finding new ones, and sometimes neatl for speedily throwing up useful answers to lower order complex questions, and hallucinating/lying when they cannot.
I come across more than my fair share of the latter rather more frequently than the former because, as part and parcel of preparing my estate for the inevitable outcome we all must face, I am trying to create a full index of all the recorded music I own, collected over a period of 70 years, so that it goes to my successor in good order, and what I thought might be useful AI assistants to aid me in this task, tend to give the details relating to completely different recordings which might contain one of the same works, or a mixture of different recordings with artists from each recording coming together to make a recording which does not exist. Even if asked to asked to respond firmly in the negative when the data cannot not be found, most LMMs will hallucinate and provide their best guess and make up a non-existent recording composed of some odd list of some of the composer, title, artists, conductor, level and the date(s) of recording requested.
Far from being a useful tool, it is clearly easier to follow my nose and my existing body of knowledge. and make use of the search engines I have available, an, in extremis, either climb the ladder to run through my standard reference books and, when truly desperate, go up other ladders to extract the vinyl record, cassette tape, cd or enter the garage to check the 78rpms I inherited from various members of the family
Presumably the appropriate discographical sources haven’t yet been scraped for long forgotten singers and the operas they performed,. Even recordings still in the catalogue come up with weird and wonderful, and usually dead or retired, cast members along with the more recent headliners – and this is true of recordings made over the years by record companies as well as the much more interesting world of live bootlegged recordings released by small independent companies.
It’s difficult to base one’s judgement on one person’s empirical experience, but if pretty basic LLM models can’t do this simple basic task then I expect the gadzillions already thrown at this approach to AI, and the gadzillions to follow, until, inevitably, “all that is solid melts into air”. Roosevelt, Huey Long, Tojo, and Hitler saved the US the last time everything melted into air. Can anyone save the West and its allies when the futility, stupidity, and the cost of the US’s AI IPR techludd monopolists’ fantasies finally come home to roost?
This is fundamental point 1: An LLM literally has no idea what it’s talking about when it produces its output. All semantic meaning has been removed from the tokens it works on, so it’s work bears absolutely no resemblance to human thought, which involves manipulating human concepts in an effort to develop new meanings and new ideas. LLMs don’t even rise to the level of language production, let alone thinking, because the material it is working with has been completely stripped of meaning.
A second fundamental point: LLMs can never innovate. The only input that an LLM has access to is the material it was trained on some time in the past. So it’s also literally impossible for an LLM, which works by cutting and pasting its training material, to have any new ideas that were not previously expressed in some concrete form (writing, recording audio, recording video) by humans.
In fact, there is good evidence that the output of LLMs should not be used as training material for other LLMs, and only strictly human productions used for this purpose. LLM output is considered to be “poisoned”, an increasing problem as more and more material on the internet has been generated by AI. Some have speculated that pre-AI internet content is going to become extremely valuable in the future for this reason.
I guess assets are where you find them!
THE GUARDIAN
How AI is changing language
As allegations of LLM use rock the literary and media worlds, linguists explain what really distinguishes human and machine writing, while novelists including Jennifer Egan and Jeanette Winterson reflect on the future of fiction in an age of ChatGPT
by David Shariatmadari
Sat 4 Jul 2026
https://www.theguardian.com/books/ng-interactive/2026/jul/04/future-of-fiction-next-great-novel-ai-language-chat-gpt