Science Fiction Science and Artificial Intelligence in Our Future

The 7 August 2025 issue of Nature included an interesting article on The Science Fiction Science Method (paywall) by Iyad Rahwan, Azim Shariff, and Jean-François Bonnefon.  The scientific method is difficult enough to understand these days, especially regarding previously important but mundane scientific advances such as vaccines.  The Science Fiction Science (SFS) Method is likely to take a bit of work, too.  But the objective of predicting the consequences of future technologies is noble and should be given a chance:

Predicting the social and behavioral impact of future technologies before they are achieved would enable us to guide their development and regulation before these impacts get entrenched.  Traditionally, this prediction has relied on qualitative, narrative methods.  Here we describe a method that uses experimental methods to simulate future technologies and collect quantitative measures of the attitudes and behaviors of participants assigned to controlled variations of the future.  We call this method ‘science fiction science’.  We suggest that the reason that this method has not been fully embraced yet, despite its potential benefits, is that experimental scientists may be reluctant to engage in work that faces such serious validity threats.  To address these threats, we consider possible constraints on the types of technology that science fiction science may study, as well as the unconventional, immersive methods that it may require.  We seek to provide perspective on the reasons why this method has been marginalized for so long, the benefits it would bring if it could be built on strong yet unusual methods, and how we can normalize these methods to help the diverse community of science fiction scientists to engage in a virtuous cycle of validity improvement.

The authors are correct.  Experimental scientists are reluctant contemplate projects that face serious validity threats, and this does explain the marginality of something called science fiction science. [1]  As an aside, the first thing I thought about when I read the title of this article was a book called “The Science of Star Trek” (or something similar) that I saw on the bookshelf of a friend in the mid-1970s.  Warp 8 is 1024 times the speed of light, which is no more likely in this universe than “Beam me up Scottie, there is no intelligent life on this planet.”  Yet, more than a few of our billionaires seem to think of Star Trek as a documentary.

Could SFS have worked on some of our current problems?  Social media, for example?

(Practitioners of SFS) might have noted a tendency for participants to focus on upward social comparisons, leading to self-esteem issues (that “odd and unclassifiable concept” described by the late, great Elizabeth Fox-Genovese), or a focus on moral outrage, resulting in exaggerated feelings of polarization.  These speculative findings could have guided us in designing and regulating social media with the benefit of foresight, instead of constantly playing catch-up with their effects.

I see very little catch-up in action.  A similar argument can be made for “speculative behavioral research” on foods from genetically modified organisms (GMO).  As it happened, I was adjacent to a laboratory throughout the 1980s that was at the leading edge of GMO development when molecular cloning and the insertion of transgenes into plants was still a dream.  To these scientists, their intention of solving the world’s agricultural problems led them to believe with certainty that Roundup Ready and Bt corn and other commodity crops and Golden Rice were the solutions we needed.

My plant molecular biologist friends never considered that much of the public would react negatively to GMO foods.  The potential harm of a food crop containing a gene for glyphosate resistance is vanishingly small.  This is not necessarily relevant.  Food can be sacred, even in our thoroughly industrialized agricultural economy.  Nor did it ever occur to them that Roundup Ready corn, cotton, soybeans, and other plants were a technical fix for a problem that did not have to exist, except under the regime of industrial agriculture.  The only agriculture conceivable to these scientists was and remains industrial.  Very few have ever been convinced that industrial agriculture is a category mistake and even fewer view CAFOs and industrial feedlots as the abominations they are (and the source of the extra greenhouse gases attributed to livestock production).  And even after all these years it is still not clear that GMO crops produce greater yields than genetic variants produced using conventional plant breeding techniques.  But Roundup Ready crops did lead to rapid appearance of herbicide-resistant weeds, much as the indiscriminate spraying of DDT from airplanes led to DDT-resistant mosquitoes within two years in Florida after World War II.

The basic thrust of SFS is to respond rationally to “technology-driven futures” before the fact.  Whether the changes are good or bad depends on politics and preferences instead of business imperatives.  The birth control pill gave women control they previously lacked.  The success of organ transplants has changed ethical approaches to the end-of-life of potential organ donors and their families.  Overall, these two advances have been positive for all but a very few.  And regarding birth control, reservations have been mostly honored in the breach.

Still, speculative behavioral research has had some reach, with autonomous vehicles (AVs) having been studied long before their introduction.  Although AVs are technically feasible and Waymo has deployed them in several communities, the question remains: “How do citizens and consumers [2] wish AVs (as if they had a free choice of their own) to prioritize safety of different road users in unavoidable accidents?”  According to the authors:

By engaging millions of citizens (not consumers this time) worldwide the Moral Machine experiment (Rahwan, Shariff, and Bonnefon are also co-authors of this study) has contributed to a wide debate in civil society around AV ethics, which in turn informs policy-making.  More broadly SFS studies in this context have informed specific policy recommendations and subsequent legislative measures.

I do wonder, though.  In an iteration of the Trolley Problem, should the AV taxi with one human occupant run over a gathering in the street and kill or injure many, or drive off the road into a canal and risk the life of “his” one paying passenger.  What would a normal human driver of a non-autonomous vehicle do?  That seems to be the proper question.

Other potential targets of speculative behavioral research are social credit systems, embryo screening, and ectogenesis.  Governments do not (yet, but they are working on it) have the capacity to use Artificial Intelligence (AI) systems that would monitor “every behavior of all their citizens in real time to calculate and release social scores.”  Given time this seems likely, and it may be more advanced in some countries than others.  Embryo screening has led to questions of whether potential parents should be able to “choose or reject specific traits in their offspring” when advances in in vitro fertilization give them the opportunity.  Ectogenesis (artificial gestation of humans) is not yet feasible, but the ethics have been discussed.  These three possibilities have something similar to an “ick factor” associated with them and this tells us something we should not ignore.

The biggest problem with SFS is that science fiction scientists generally do not know which questions to ask ex ante.  Things do not always turn out as imagined.  For example, twenty years ago, people were asked whether they would be comfortable with “nano-augmentation of cognitive capacities.”  This has not come to pass with the development of nanotechnologies and will probably not, so the validity of those studies was very low.

A different matter altogether is described in Case Study B: Cooperating with Autonomous Agents.

The SFS question is: How should an autonomous agent (such as a robot or software agent) prioritize its own interests.  This problem goes back to Isaac Asimov in his ‘Third Law of Robotics’: “A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.” (I, Robot; orig. Gnome Press,1953).  In What Matters to a Machine?, AI pioneer Drew McDermott imagined a robot that was tempted to break an ethical rule to further its owner’s interests [in Machine Ethics (eds Anderson, M. & Anderson, S. L.) 88–114, Cambridge Univ. Press, 2011].

This research would have been considered unnecessary only a few years ago.  As the authors note, not so much now:

These studies…may have seemed highly speculative just a few years ago.  However, this has changed in the past two years with the sudden and rapid rise of conversational agents like ChatGPT and their numerous autonomous (agentic) implementations…We (now) live in a world in which we interact frequently with autonomous customer service agents acting on behalf of other organizations.  The challenge of establishing cooperation between humans and autonomous agents is already pervasive, but fortunately we had a head start.

A head start in the beginning, perhaps.  But it is not clear that an initial lead in this race has held up.  And this discomfort leads directly to the second article in the 7 August 2025 issue of Nature: We need a new ethics for a world of AI Agents (archived copy, published online on 4 August 2025) by Iason Gabriel, Geoff Keeling, Arianna Manzini, and James Evans. [3]  As Gabriel et al., put it and as everybody knows:

Artificial intelligence (AI) developers are shifting their focus to building agents that can operate independently, with little human intervention.  To be an agent is to have the ability to perceive and act on an environment in a goal-directed and autonomous way.

The rise of more-capable AI agents is likely to have far-reaching political, economic and social consequences.  On the positive side, they could unlock economic value: the consultancy McKinsey forecasts an annual windfall from generative AI of US$2.6 trillion to $4.4 trillion globally, once AI agents are widely deployed [McKinsey report, pdf download).  They might also serve as powerful research assistants (sic) and accelerate scientific discovery.

But AI agents also introduce risks. People need to know who is responsible for agents operating ‘in the wild’ (i.e., where we live) and what happens if they make mistakes.

Here, we argue for greater engagement by scientists, scholars, engineers and policymakers with the implications of a world increasingly populated by AI agents.  We explore key challenges that must be addressed to ensure that interactions between humans and agents – and among agents themselves – remain broadly beneficial.

That is a tall order, or series of concerns.  Mistakes happen.  Some are not serious, but some can be as catastrophic as the imagination allows – medical misdiagnosis (AI is coming to the clinic with a frightening speed and little direction), ruinous financial error, therapy-bot losing the plot and recommending something outrageous, or in this classic conversation between HAL and Dave from Arthur C. Clarke and Stanley Kubrick.

The obvious problem for something that is algorithmic to its core is the alignment problem:

The risks of mis-specified or misinterpreted instructions, including situations in which an automated system takes an instruction too literally, overlooks important context, or finds unexpected and potentially harmful ways to reach a goal.

A well-known example involves an AI agent trained to play the computer game Coast Runners, which is a boat race. The agent discovered that it could earn higher scores not by completing the race, but rather by repeatedly crashing into objects that awarded points—technically achieving an objective, but in a way that deviated from the spirit of the task (see this ancient example, in AI time).  The purpose of the game was to complete the race, not endlessly accumulate points.

And then there are cases where AI agents have been empowered to modify the environment which they operate by using expert-level coding “skills” and tools:

When the user’s goals are poorly defined or left ambiguous, such agents have been known to modify the environment to achieve their objective, even when this entails taking actions that should be strictly out of bounds.  For example, an AI research assistant that was faced with a strict time limit tried to rewrite the code to remove the time limit altogether, instead of completing the task.  This type of behavior (sic) raises alarms about the potential for AI agents to take dangerous shortcuts that developers might be unable to anticipate.  Agents could, in pursuit of a high-level objective, even deceive the coders running experiments with them, (such as when) an AI research assistant (sic) that was faced with a strict time limit tried to rewrite the code to remove the time limit altogether instead of completing the task.  This type of behavior raises alarms about the potential for AI agents to take dangerous shortcuts…Agents could, in pursuit of a high-level objective, even deceive the coders running experiments with them.

I was never a Trekkie, but this reminds me of something I seem to remember from one of the early Star Trek movies.  In the telling, Captain James Tiberius Kirk of Iowa finessed an impossible scenario during his training at Star Fleet Academy by reprogramming the computer.  That was inconsequential compared to what a rogue AI agent could do when rewriting code to take a dangerous shortcut while hiding “its intentions” from those depending on the outcomes.

And then we come to social chatbots, which are designed to have:

(A)n uncanny ability to role-play as human companions – an effect anchored in features such as their use of natural language, increased memory and reasoning capabilities, and generative abilities.  The anthropomorphic pull of this technology can be augmented through design choices such as photorealistic avatars, human-like voices and the use of names, pronouns or terms of endearment that were once reserved for people.  Augmenting language models with ‘agentic’ capabilities has the potential to further cement their status as distinct social actors, capable of forming new kinds of relationship with users.

This is frankly nuts, and very little conceivable good can ever come of it.  And therein lies the largely unstated problem in each of these very good articles.  Manzini, Gabriel, and Keeling have argued in a previous article that “relationships with AI agents should benefit the user, respect autonomy, demonstrate appropriate care and support long-term flourishing.”

Yes, these “relationships” should facilitate these things if they must exist.  This will be advocated by conscientious “scientists, scholars, engineers and policymakers.”  But that is not the purpose of agentic machines.  AI, whether having agentic capabilities or not, is celebrated for just the opposite, while according to McKinsey producing “an annual windfall from generative AI of US$2.6 trillion to $4.4 trillion globally (for whom?), once AI agents are widely deployed.”  The clearly stated goals set by the developers of AI in its many guises are to benefit the “economy” instead of the user, destroy the autonomy of those forced to use it – who are also its intended victims, and “care” not one whit for appropriate care of anyone or anything.  Long-term flourishing?  In our current Late Neoliberal Economy this has no meaning whatsoever.  In The Science Fiction Science Method Rahwan, Shariff, and Bonnefon note that

Recent examples of technological innovations…include personalized algorithms for media content, direct-to-consumer genetic testing (23andMe, bankrupt), SMS texting, credit cards and other forms of digital cashless payment, 24-hour news…(were)… incremental technological advances that could have been predicted in the years preceding (their) emergence, but each also had pronounced and often unintended societal consequences.

The mining of personal data through social media platforms should be added to the list.  The result has been the inexorable decline of a captive, passive population of consumers instead of citizens.  This was not the intention when Diners Club International was founded 75 years ago, but a Surveillance Capitalism from which there is little prospect of escape has been the result – especially when things like algorithmic pricing are on the horizon.

Can Science Fiction Science prevent us from making mistakes that are obvious in retrospect?  Yes, but only if this science is intended explicitly to augment human flourishing.  SFS is more likely to be narrative and qualitative than its advocates desire.  Can AI be harnessed for good rather than ill?  Perhaps, but we will probably run into hard limits due to the energy and water needs of the server farms before we get that far.  The epigram of We need a new ethics for a world of AI Agents states “The deployment of capable AI agents raises fresh questions about safety, human – machine relationships, and social coordination.”  Indeed.

And the first thing we must remember is that machines are not sentient beings – they neither sense nor feel, in any living sense, no matter how lifelike they appear.  And in the coming inconvenient apocalypse, which can be mastered without total social and cultural collapse if the will exists, agentic AI in all its forms may well be superfluous except at the far margin.  The only ethical precept to be followed is the Golden Rule.

Notes

[1] Given the recent Executive Order of the President that will put the review and funding of American science under the control of commissars from the Executive, SFS, emphasis on the fiction part, could have a bright future after all.

[2] Only under the Neoliberal Dispensation are the people identified by conflating citizens with consumers, with the latter being the effective definition of citizen to a Neoliberal politician or policymaker.  See Wendy Brown, Quinn Slobodian, and Melinda Cooper.

[3] It is interesting that Gabriel and Manzini are at Google DeepMind (London), Keeling is at Google’s Paradigms of Intelligence Team (London), and Evans is a visiting faculty member in Paradigms of Intelligence Team (Chicago) and a professor at the University of Chicago and the Santa Fe Institute.