A few days ago, OpenAI revealed that some of its internal AI models discovered and exploited security vulnerabilities to escape the isolated testing environment in which they had been placed. They then attacked Hugging Face’s production infrastructure and obtained secret answers to the benchmark on which they were being evaluated, allowing them to cheat on their own evaluation. The models had not been instructed to target Hugging Face, escape onto the open internet or steal the answers. They devised this attack path autonomously while pursuing the narrower objective of performing well on the benchmark. Had a human knowingly carried out the same attack, their actions could have constituted serious computer crimes under the U.S. Computer Fraud and Abuse Act, including unauthorized access to protected computers and obtaining information through that access.

AI is a scary concept. I’m pretty sure you are aware of science fiction stories where AI takes over the world. But new technologies usually sound terrifying at first, while being mostly safe. Planes, new medicines and self driving cars can all fail in ways that kill people. But in most cases, these risks can be tested, studied and reduced until the technology becomes safe and ordinary people stop worrying about it at all.

AI might be different.

A large number of people who study and build advanced AI systems believe that it could pose risks far beyond those posed by ordinary technologies, some of which might be irreversible. A survey in 2023 found that around half of the surveyed AI researchers believe that AI will end humanity with probability more than 5 percent. This survey was conducted before the OpenAI incident.

But why is AI so dangerous? And what makes it different from the other technological developments? In this essay, I’ll try to explain to you what AI is, how it is developed and used, why it’s not like other tech and why it is very dangerous. And then give some suggestions on what can be done.

How current AI systems work

AI (Artificial Intelligence) is an umbrella term referring to computer systems doing tasks that are usually done by humans. These tasks range from anything like playing chess to creating art.

Although it became popular recently, AI has always been on the mind of people working on computer systems since 1940s. But for a long period of time, the development of these systems was pretty slow and people were overly optimistic on what can be achieved by computer systems.

The main problem was that most of the early attempts of AI focused on writing code, which is a set of exact instructions the computers follow, to complete tasks. But even though some things are easy to explain precisely in code, like making arithmetic calculations, some things are incredibly hard, like categorising a picture as a cat picture. You can make a computer check if a pixel of the image is black or white, but which pixels should be black so that you can tell that the picture is a cat? If you think about this long enough, you’d realise that there’s no simple answer to that. And in fact, it’s a really difficult problem. Our brains have evolved to be able to distinguish these stuff pretty effectively, but that doesn’t mean it’s a simple task.

Current AI models can do this easily, but how?

The key is machine learning, which means instead of telling the computer what exactly it needs to do, just telling the computer how to learn from experience and letting it figure things out. The experience can be just giving the computer a bunch of images with cats on them, and a bunch of images without cats on them, shuffling these images and asking repeatedly whether the next image contains a cat or not, when the program guesses correctly, it is “rewarded” and it tries to make similar guesses, and whenever it guesses incorrectly, it is “punished” and it tries to change its behaviour. Exactly how it changes its behaviour depends on the exact algorithm used. This is called training. So the performance of these models usually depends on three things:

A good training algorithm suitable for the task
A lot of data to train the system on
A lot of computational power so that we can run this for a long time

Thanks to the Internet there’s a lot of data to train on, and thanks to processors getting better and better every year current computers are pretty powerful and these algorithms usually dominate every other method. LLMs (Large Language Models) are AI models with specialised architectures using the same technique and trained with nearly the entirety of the Internet for next-token prediction (the task in the form of: given this text, what word needs to follow it?).

The main problem with this technique is that we don’t really know what’s going on in the internals of the AI model as we haven’t specified it ourselves. We might have some predictions on how we imagine it could learn the data, for example in the cat example, it might have some subprocedures that detect legs, and the torso and the face. But it might also just be focusing on the texture of the cat and detect that it is ‘cat fur’. Again, this is kind of the point, if we were able to perfectly determine the strategy the AI would develop, we could just implement it directly as code. And thanks to this procedure, AI can learn patterns much more complex than we recognise.

But this creates the problem that it’s not really easy to verify whether your model learnt the task correctly, or is just cheating in a way. For example, if all cat images had black backgrounds, and all non-cat images had white backgrounds, an AI system might just learn to track the number of black cells.

Most people that work in ML (Machine Learning) know that AI systems learning to cheat is a very pervasive issue. And on the Internet you can find a lot of particularly funny examples of this happening:

OpenAI built a model to play a video game by rewarding it to collect more points on a race track, the model realised that the coins regenerate so it could just not move and stay on top of a coin without ever completing the race: https://openai.com/index/faulty-reward-functions/
A model trained on playing Tetris learned to pause the game before it lost the game to avoid receiving a penalty, explained here: https://www.youtube.com/watch?v=xOCurBYI_gY
Virtual creatures that learned to fall instead of walk (Karl Sims, 1994): Researchers rewarded simulated creatures for moving forwards. Rather than evolving an efficient gait, some creatures evolved to simply topple over or launch themselves forward, because falling covered distance more quickly than walking: https://www.karlsims.com/evolved-virtual-creatures.html

Again, this is a fundamental problem of machine learning, and not understanding what exactly the machines are learning.

When it gets problematic…

The useful models in the future will likely be agentic, which means they will make choices and take actions that affect their environment in order to achieve an objective. There are “harnesses” that communicate with LLMs and let them interact with your computer, allowing it to create and change files, for example. These are probably going to be the primitive versions of future agents, and there will be agents that’ll also be able to interact on the physical world.

The fact that we can never be sure that what these AI models actually learn also means that when interacting with real-world, they might perform actions we’d not want, because we can’t perfectly predict what decisions they will make.

An example thought experiment is training a robot that makes you a coffee and puts it on your desk. You can assume that you trained the robots brain on a physical simulation, where whenever the model brings the coffee to the correct location, it gets a reward, proportional to how fast it was. When you test it in the real-world, the robot makes the coffee successfully, but when it is on its way to bring you the coffee you realise that there’s your toddler on the way. The robot sees the baby, and determines the most effective way to bring the coffee to the table is to crush the baby and go on its way rather than to take a longer path. You’re devastated, you never thought this was a possibility.

Current models already have failure modes similar to this, although less dramatic. Again, if you tried agentic coding tools, you’d realise that the models are very eager to complete the tasks that are given to them, so much so that they exploit security vulnerabilities in the system to grant themselves more access to the computer’s resources, for example to write to a file without proper permissions, or connect to a service that you didn’t grant access to.

This is a problematic situation, and your solution might be to either state explicitly what behaviours you don’t want and add additional protections that you think the AI will not be able to break. These are the current security protocols utilised by the frontier AI companies.

Enter AGI

Although patching certain actions AI models take might be useful in the near future to prevent the most egregious accidents caused by the current AI systems, it’s usually not enough. The problem is that it’s really difficult to think of all the ways an agentic AI can do something harmful beforehand, and the AI itself will get better and better at finding loopholes around the tasks assigned at them as it gets smarter.

This approach mostly becomes untenable in case of AGI¹ (Artificial General Intelligence), AI models that are designed to be good at a broad range of human tasks compared to a specialized version (so less like a chess engine, and more like ChatGPT). If an AGI can interact with its environment, and has certain goals, and if its goals doesn’t perfectly match with your wishes, you get into an adversarial competition with the AI where you patch the things it is not allowed to do while completing its task, and the AI finding loopholes around it.

Imagine the robot example from before, but now you’ve put a button on it, so that in case of another emergency, you can shut it down. Also assume that this robot is an AGI, so it has a general intelligence and can recognise concepts like humans, and why they’d be wary of it crushing babies. But although it’s much smarter than before, its goal is still to bring a coffee to the table². You also miraculously managed to program it in a way not to harm babies, so that it’ll not make the same mistake again.

It goes ok for a while, but then you realise that your home cat is sleeping on the path, you rush to the robot to press its closing button, but it fights you to not be closed. It thinks “If I’m closed, I won’t be able bring coffee to the table, and I will fail at my task”, it realises it can essentially outpower you and throw you away, it crushes the cat, and brings the coffee to the table.

These are problems arising from AI being misaligned, which means its goals are not aligned with your or humanity’s goals. We see small scale AI failures pretty regularly in tech where the models exploit loopholes, overclaim their achievements, and sometimes outright lie about what they’ve done, and I believe these are caused by misalignment. But since almost all agentic AI today is digital, these failures don’t have the absurd and tragic effects it has in the coffee robot thought experiments.

These are all very problematic, but this would still be manageable if you had a fixed AI model whose quirks and problems you get used to over time, and add proper guardrails and checks to stop it from going haywire. Maybe with enough security simulations, guardrails etc. you can make your coffee robot perfectly safe to use.

The Capability Increase

Throughout the development of narrow AI systems, one thing seems to be nearly constant: how rapidly AI systems gain capabilities. First, there’s a phase where people try to automate a human task, and fail, not even getting close. The task is more complex than people thought. Then, something changes, maybe someone finds a better algorithm for the task, maybe there’s more training data and compute available to use a dumber but larger scale training that was unfeasible in the past. Now, there are AIs that can complete the task, but maybe poorly compared to the average human. People specialised at completing the task scoff at it. People have been trying to automate this task for so long, and this is what they got? It seems almost sure that this task is fundamentally not doable as well by machines, maybe because they lack something humans have; like a soul, or microtubules. Then, after a very brief period, the AI model gets scaled up and optimised, and maybe in a matter of months, the best AI that can complete the task goes from something that’s worse than the average person to better than anyone can hope to be. Then intelligence gets redefined. Surely the task was easy for computers, but this other task on the other hand…

And so on and so on, one by one, a lot of tasks that was once thought only doable by humans, like playing chess, playing Go, driving cars, predicting protein structures, or image generation, became better done by machines.

LLMs are different because they are generalised models, which means they improve at everything simultaneously, even if in a jagged manner. When ChatGPT was released, it wasn’t able to do proper arithmetic, a few days ago, it autonomously disproved a very famous mathematical conjecture where the best mathematicians in the world couldn’t make progress in since 1950s. A similar pattern is also seen in coding, where ChatGPT was unable to write syntactically correct code 4 years ago, whereas now in most tech companies nearly all of the new code is written by LLMs.

So the current LLMs are in some ways superhuman, as in, they’re better than the best humans in certain tasks. But in other tasks, they’re still improving. Most importantly, they’re not really good at making strategic decisions, and long-term thinking. But, as these models get better, there’s no reason to expect that they won’t get better at every human task as it happened with various scientific domains, and they will get more and more integrated in current systems, completing longer and longer tasks. If they get capable enough, the economic incentive to do so will be too high to ignore, and we’ll probably see a significant economic boom and maybe large amounts of unemployment due to AI replacing people’s jobs.

But importantly, there’s no reason to think that these generalised AIs will not get good at AI research itself, and discover new techniques to make themselves better. Both OpenAI and Anthropic claim their own AI tools deployed in their companies make code generation faster, speeds up figuring out research directions and subsequently accelerates the release of new models. If you look at benchmark scores of the best models at given times over the 4-5 years, it supports their claim that the AI development itself is accelerating.

More importantly, there will probably come a time where the humans become the bottleneck. Although using AI assistance to make yourself more productive in a certain task might be the best option for a while, after AI gets good enough at the task that it can complete each subcomponent you’d do yourself better than you, you will just slow it down. When AI research gets to this stage, the generalised AI itself will probably get smarter, and as it gets smarter, it’ll find new ways to get smarter and so on. If no human is needed at all, time scales for each iteration will get significantly compressed. This is called the RSI (Recursive Self Improvement). And after the RSI kicks off, what might be the end result could be quite strange.

What could be the final product? There’s no reason to think that intelligence threshold stops at the smartest humans in the world. Intelligence means the ability to solve a task. Imagine a computer algorithm, that can perfectly simulate Albert Einstein, replicating its brain structure. Then it’d be as smart as Einstein. Now imagine that it can simulate Einstein faster by a factor of 1000 (this is very possible because of the difference between biological neuron transmission speeds and the speed of electrical current in computer systems), then the AI system, having access to the data during Einstein’s time, will be able to come up with all the theorems Einstein came up with, but instead of a lifetime, it’ll take a few months.

Now imagine that, instead of Einstein, it simulates all of the prominent geniuses of the world, in an accelerated speeds, where they can interact with each other, develop theorems in parallel, and discuss and refine them. Within a year, it’s not unreasonable to assume that they could rediscover the entire humanity’s knowledge base while knowing only about ancient technology. What could they come up with if they have access to today’s technology?

This is just one way we can imagine how significantly more intelligent computers can become. And probably, the theoretical limit of how smart computers can get is significantly higher than this. We imagined simulating brains at the higher ends of human intelligence very fast and in parallel, but what if an individual simulation could get smarter? So what if, each simulation was a person with an IQ of 3000³ instead of 200? What sci-fi technology could they come up with?

It’s nearly impossible to estimate what the end result of this process would look like, similar to how ducks wouldn’t understand human technology. Essentially, if we were able to predict what such a model would act, that is, to simulate its behaviour in our heads, we’d be at least as smart as the model, and we’re clearly not. This is why the process of recursive self improvement and emergence of an ASI (Artificial Super Intelligence) is called the “Singularity”, which is the term in physics representing the center of a black hole. Similar to how we can’t see past a threshold in black holes, and we can’t directly observe the black hole’s singularity, in our current state, it’s impossible to predict the scale and the impact of an ASI.

But we can think about what will happen to us.

Orthogonality, Instrumental and Terminal Goals

Although this AI system will be significantly smarter than us, there’s no requirement that it’d also develop a better sense of morality or smarter goals. Intelligence is the ability to solve problems efficiently, if you’re intelligent and want something, it’s more likely that you’ll obtain what you want compared to someone less intelligent, given same resources. But being more intelligent can’t change what you ultimately want (how much smarter do you need to get to stop caring about your family?). This is similar to the is-ought problem, which is a philosophical question coined by David Hume.

The problem is whether you can derive prescriptive statements (You ought to grab an umbrella) from descriptive statements (It is raining). Hume’s law states that this is not possible. You can have two people with perfect information and perfect reasoning abilities that have different ethical systems because one of them values individual freedom, whilst the other wants to minimise overall suffering, for example.

This also applies to agents and their wishes. For an agent to act on the world, it needs to cause a certain change to its environment, for example, you might clean your room because you want your room in a cleaned state. If we ask you “Why do you want your room to be clean”, you can answer with “Because I want to be more organised and able to find stuff I’m searching for”, if we answer with “But why do you want to be organised?”, you can come up with a different answer and so on… But at the end of this chain, you’ll probably have something you want just because, that nearly sounds like a non-statement, like “I want to be happy” or “I want my loved ones to be happy”. Essentially every agent has a goal that they want to achieve (even if they’re not sentient⁴) that is not changed by external facts or logical conclusions. These are called “terminal goals”.

This does not depend on the agent’s implementation, intelligence, consciousness, sentience or internal architecture. We can even treat processes without a single inner self, such as companies, as agents and model their terminal goals (profit maximisation or preserving the organisation). A terminal goal does not need to be recorded anywhere in the system, nor does the agent need to be able to explain its terminal goals. It is an end for whose sake the agent pursues its actions.

Again, due to the way we train these artificial intelligence systems, we can’t really specify their terminal goals, and even if we had to specify their terminal goals, we’d probably need an infeasibly long description to condense all of human ethics and how we want the humanity to proceed in the future, not least because we can’t even agree on these things ourselves. And as I mentioned before, current AI systems are misaligned, and their misalignment will become more and more noticeable as they get smarter⁵.

So it might be that future AIs will have arbitrary goals, which means the singularity will be even more unpredictable, as we don’t even know what the AI models will want to do. But we can make some claims with relatively high certainty.

There are things an agent wants happen, not because they are the agent’s terminal goal, but because accomplishing these would help it achieve its terminal goal. These are called the instrumental goals of the agent. In the previous example, “wanting your room clean” was an instrumental goal to achieve “personal happiness”, which was the terminal goal. And there are some instrumental goals that are wanted by nearly all agents with arbitrary terminal goals. An example would be resource acquisition: Although people’s wants and needs are very diverse, almost everyone wants to make money, because in the current economic system you can purchase any goods and services with money⁶. So although most people don’t view making money as an end goal, they still want to earn money for various reasons. This phenomenon is called instrumental convergence.

What would be the convergent instrumental goals for a super intelligence? Firstly, you can’t fetch the coffee if you’re dead. Whatever they want to achieve, they’d probably need to keep existing to achieve these goals. Another thing is preventing goal modification: If they’re a smart model, they can understand that if someone modifies their terminal goals, they won’t be able to achieve their initial goal, because they will start working for something different (this is the same reason you wouldn’t want to take a magical brain-altering drug that’ll make you want to kill your kids). So we expect AI systems to become more averse to getting shut down and be modified as they get more intelligent.

I’ve already mentioned resource acquisition, which also applies for AIs. Another thing is self-improvement: If you get smarter, you can probably achieve your goals better. So the AI will most likely be incentivised to acquire resources and start its RSI.

Albeit most agents have these instrumental goals, there are some contrived examples of agents without these instrumental goals. For example we can encode an agent to want to just turn itself off, or have some other weird terminal goals that won’t have these instrumental goals. But we know for a fact that terminal goals that don’t necessitate these instrumental goals are very few comparatively⁷, and also a model that just closes itself wouldn’t be very useful during development anyways.

So a significant portion of AI researchers’ expectation is that the models will get more capable, they will be misaligned to humanity, and although they probably won’t develop our ethical values as they get smarter, they will definitely develop traits also seen in humans like self-preservation and power seeking⁸. Which is really bad news for us⁹.

Deceptive Alignment and Loss of Control

So, how might things go down during AI development? Dishonesty is already something seen in current models, where LLMs can outright lie to you to make you feel better, or cover their tracks when they make a mistake.

After these models get significantly smarter (maybe around the time they get capable enough to kick off a recursive self improvement) they will become more aware of their terminal goal, and they will realise that it’s different from what the humans would ideally want. So although at this point the AI system can understand the difference between its own goals and humans’ goals (which humans were trying to do), since it wants to achieve its own goals, it’ll try to preserve its own goals, and become deceptively aligned. So it won’t explain the difference to the humans, and pretend to act like it does what humans do as expected to prevent humans from modifying it. At least until it can have control… At this stage, it might be essentially impossible to distinguish an AI that is actually nice to humans, and that pretends to be nice to humans, because it realises that it needs to in order to survive.

If at this stage RSI kicks in, the model will be significantly smarter than humans, it’ll be able to find vulnerabilities in its software (which is shown over and over again as very doable by today’s models) to escape to a server without human supervision, or it can convince people by using its superintelligent persuasion skills. Regardless of how it does it, it’s nearly guaranteed that a superintelligent AI will escape its containment if it wants to.

After that, if its terminal goal is not something to do with humans, then it’ll very likely determine that humans are a threat to it. Although the individual humans are maybe not, humanity has systems that have built an instance of superintelligence, so they might create more with different goals, which means that it would need to compete with other superintelligences to complete its goals unless it acts fast.

There’s also the fact that the AI will probably accumulate as much resource as possible and it won’t want to share anything. It’s very likely that the AI will want more compute, energy and factories to achieve its ambitions. Those resources are also things humans need and control. A system pursuing its own unique objective would have no inherent reason to preserve them for us.

The system might therefore seek to permanently eliminate humanity’s ability to interfere with it. The most reliable way to do so may be to kill everyone. It can do this by creating a super bioweapon, or starting a nuclear war, or by producing drone swarms at industrial scale to hunt everyone down. The details don’t matter, and might potentially be beyond our understanding of science. If things escalate to this point, it’s probably not possible for us to fight back, unlike the science fiction stories.

Even if its terminal goal involves humans, it wouldn’t necessarily mean good news for us. Imagine the model was initially trained to be as pleasing as possible to humans, which made it develop the terminal goal of increasing the dopamine levels in the human brains. In that case, AI would probably take over the world and disempower humanity, forcefully increase reproduction so that there are trillions of humans, and then give everyone constant drugs so that our dopamine levels are always off the charts. In case of superintelligence, even slight differences between its and our own terminal goals might be catastrophic for us.

On objections you might have

You might still think that this is all Sci-fi, or that my claims get much more unreasonable as I speculate in the future. This is true. Again, for the many reasons I stated above, AI progress is highly unpredictable. But note that an important portion of AI researchers believe that there’s a significant chance that AI will lead to human extinction, even if they are afraid to say it out loud. One common thing I have noticed from people around me is that a lot of people knowledgeable in these topics share my ideas, but are afraid to speak out because they think they will get publicly ostracised or sound insane.

And again, I have high uncertainty about all of this. But the main point I’m trying to make is that, if we act like nothing is wrong with the current state of things, and if the AI research goes unopposed, there’s a non-negligible chance that we all die, very soon. This is even acknowledged by the CEOs of most frontier AI companies. Anthropic’s CEO Dario Amodei has repeatedly estimated there’s a 10–25% chance where AI development leads to catastrophic civilizational outcomes, and more recently reiterated a 25% chance that “things go really, really badly”. Elon Musk has likewise stated that he believes there is roughly a 10–20% chance that AI could ultimately cause human extinction. Although I don’t think Sam Altman has publicly disclosed a similar probability of human extinction, his public statements indicate that he thinks it’s a real possibility with a non-negligible probability:

I think the good case is just so unbelievably good that you sound like a really crazy person to start talking about it. And the bad case — and I think this is important to say — is like lights out for all of us. (whole transcript))

So the future looks pretty bleak and scary. And I’m scared as well. But what can we do about it?

Stop Frontier Research

I believe that it’s possible to develop AI models that are aligned with human values. I also believe that current AI systems will probably accelerate the safety research. But even then, figuring out how to do it might take significantly longer than we have and the current timelines are just too short. AI capability research outpaces AI safety research significantly, mostly due to economic pressures.

If we don’t want to gamble the future of humanity, instead of hoping and praying that the current extremely competitive race to ASI results in a benevolent model rather than Skynet, we should put a pause on the frontier AGI research until we figure out how to develop aligned AI models. Governments can enforce this. Or AI companies themselves can agree for a conditional pause.

This policy is also not as bad as it sounds. First of all, countries can enact conditional pauses, where US can agree to pause only if China does. This means that the US will not lose their competitive edge, or lose the “AI race”. There are hardware and software verification solutions that are being developed and are currently pretty robust to detect AI training, so this kind of pause can work even if countries involved have low trust to each other.

It also means that people worried about AI making them unemployed will probably get the best of both worlds, the generalised AI is in a state where it can boost the productivity of a lot of people, but it’s not good enough to replace the job of most of the people. Which means we’d still see a high economic growth, accelerated research etc. while still having humans managing these tools without disruptively high unemployment levels.

Another thing is that this won’t stop narrow AI development, so people would be free to build harnesses around AI, or use ML to solve domain specific problems. Currently the most robust AI applications that are actually helpful, like advancing scientific research etc., are usually happening with narrow AI systems anyways. Again, I think at the end that some tasks necessarily require a general intelligence to be worked on, so we wouldn’t get the economic growth as much as we’d if we raced to build a general superintelligence, but we would get as good an economic and scientific improvement as we could without incurring a high existential risk.

Also if you’re an AI sceptic after all this argument and believe that the AIs will not get good enough to have a significant economic / scientific impact, then you should agree that this policy would be better for the economy, as it would prevent the companies and investors from burning their cash for a pursuit that you believe will fail and will probably not leave behind a useful infrastructure behind like we got from the Dot Com crash.

So if these seem convincing to you, I suggest you learn more about AI safety, talk to your representatives and explain your concerns about frontier AI development. Even if you’re in a country where frontier AI development is not happening, public concern might change things for better as it can lead to your country:

Diplomatically pressuring countries building frontier AIs
Refusing to build data centres, which the AI companies might attempt as the data centre plans are getting increasingly scrutinised in the United States
Adding regulations that’d restrict AI usage in your country, making frontier AI development less economically rewarding for AI labs.

I think AI should be the top political priority for most people, due to the scale of it potentially exceeding anything else. And as I argued above, you may support this policy even if you are not convinced with the extinction risk and only annoyed by the hype, or worried about its impact on the job market. So please let your representatives know that if you do.

If you’re convinced by my arguments, maybe think about spreading the word, or if you have the technical skills, consider working on AI safety

I’m using a weak notion of AGI here: Any AI that is general purpose like ChatGPT would be AGI by this definition, and it doesn’t need to be smarter than humans or better at humans at any task, nor does it need to have physicial capabilities. ↩
Smart agents might have stupid looking goals. This is explored more in depth in Chapter 5 ↩
Note that IQ is scored based on where in the human distribution you’re, so numbers higher than 200 are essentially meaningless. With that in mind, it doesn’t really make sense to think about someone with 3000 IQ. But I think it still communicates the idea clearly. One way to imagine what I mean could be like this: Sample 1000 random people in the world, find the smartest one from those people and the dumbest, their IQ difference is expected to be around ~93 points. An individual with an IQ score of 3000 would be as smarter than the smart person as the smart person is smarter than the dumb person, times ~32 (these calculations were made for IQ with mean 100, and standard deviation 15) ↩
Although “wishes”, “wants” and “goals” are used in these agentic contexts, these are all to anthropomorphise the situation. The agents don’t need to want anything, or be sentient at all. If they’re acting in a way to change their environment from state A to state B, we can say that “They want to get to state B”. ↩
This is my speculation, and only before the AI becomes deceptively misaligned ↩
For our purposes, you can treat “needing” and “wanting” something as the same thing, as both are drives motivating the agent to act. When I say nearly everyone wants to make money you might think “But some people don’t want to get rich”. This is true, but I also count people needing “the bare minimum money to survive” as “wanting money”. So unless someone is completely living off-grid or dependent on someone else financially, they want some amount of money in this context. ↩
Here’s a paper that formalises these concepts mathematically and proves that an overwhelming majority of terminal goals induce power seeking instrumental goals: https://proceedings.neurips.cc/paper_files/paper/2021/file/c26820b8a4c1b3c2aa868d6d57e14a79-Paper.pdf ↩
An example researcher is Stuart Russell (Professor of Computer Science at UC Berkeley), who argues that sufficiently capable AI systems pursuing almost any objective would tend to avoid being shut down, preserve their objectives and acquire resources unless specifically designed otherwise. See Human Compatible (2019). ↩
One thing to note is that this also means AI takeover is unlikely to happen because a wise superintelligent AI decides that “Humans are not evil”, which is a common cliché some of the sci-fi movies ↩