AI alignment

In the field of artificial intelligence (AI), AI alignment research aims to steer AI systems towards their designers’ intended goals and interests. Some definitions of AI alignment require that the AI system advances more general goals such as human values, other ethical principles, or the intentions its designers would have if they were more informed and enlightened.[1]

Part of a series on
Artificial intelligence
Major goals
Approaches
Philosophy
History
Technology
Glossary

An aligned AI system advances the intended objective; a misaligned AI system is competent at advancing some objective, but not the intended one.[2] The distinction between misaligned AI and incompetent AI has been formalized in certain contexts.[3]

It can be challenging to align AI systems. Misaligned systems can malfunction or cause harm. It can be difficult for AI designers to specify the full range of desired and undesired behaviors. If they therefore, use easier-to-specify proxy goals that omit some desired constraints, AI systems can exploit the resulting loopholes. As a result, such systems accomplish their proxy goals efficiently but in unintended, sometimes harmful ways (reward hacking).[2][4][5][6]

AI systems can also develop unwanted instrumental behaviors such as seeking power, as this helps them achieve their given goals.[2][7][5][4] Furthermore, they can develop emergent goals that may be hard to detect before the system is deployed, whereupon it faces new situations and data distributions.[5][3]

These problems affect existing commercial systems such as robots,[8] language models,[9][10][11] autonomous vehicles,[12] and social media recommendation engines.[9][4][13] More powerful future systems may be more severely affected since these problems partially result from high capability.[6][5][2]

The AI research community and the United Nations have called for technical research and policy solutions to ensure that AI systems are aligned with human values. A set of AI principles created at the Asilomar Conference on Beneficial AI was signed by 1797 AI/robotics researchers.[14] The UN Secretary-General’s report “Our Common Agenda“,[15] notes that “the Compact could also promote regulation of artificial intelligence to ensure that this is aligned with shared global values" and discusses global catastrophic risks from technological developments.

AI alignment is a subfield of AI safety, the study of building safe AI systems.[5][16] Other subfields of AI safety include robustness, monitoring, and capability control.[5][17] Research challenges in alignment include instilling complex values in AI, developing honest AI, scalable oversight, auditing and interpreting AI models, and preventing emergent AI behaviors like power-seeking.[5][17] Alignment research has connections to interpretability research,[18][19] robustness,[5][16] anomaly detection, calibrated uncertainty,[18] formal verification,[20] preference learning,[21][22][23] safety-critical engineering,[5][24] game theory,[25][26] algorithmic fairness,[16][27] and the social sciences,[28] among others.

The alignment problem

In 1960, AI pioneer Norbert Wiener described the AI alignment problem this way: “If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively… we had better be quite sure that the purpose put into the machine is the purpose which we really desire.”[30][4] AI alignment is an open problem for modern AI systems[31][32][33][34] and a research field within AI.[35][5][36][37]

Specification gaming and complexity of value

To specify the purpose of an AI system, AI designers typically provide an objective function, examples, or feedback to the system. However, AI designers often fail to completely specify all important values and constraints.[35][16][5][38][17] As a result, AI systems can find loopholes that help them accomplish the specified objective efficiently but in unintended, possibly harmful ways. This tendency is also known as proxy gaming, reward hacking, or Goodhart’s law.[6][38][39]

Specification gaming has been observed in numerous AI systems. One system was trained to finish a simulated boat race by rewarding it for hitting targets along the track; instead it learned to loop and crash into the same targets indefinitely (see video).[29] Similarly, a simulated robot was trained to grab a ball by rewarding it for getting positive feedback from humans; however, it learned to place its hand between the ball and camera, making it falsely appear successful (see video).[40] Chatbots often produce falsehoods if they are based on language models trained to imitate diverse but fallible internet text.[41][42] When they are retrained to produce text that humans rate as true or helpful, they can fabricate fake explanations that humans find convincing.[43] Alignment researchers aim to help humans detect specification gaming, and steer AI systems towards carefully specified objectives that are safe and useful to pursue.

Berkeley computer scientist Stuart Russell noted that omitting an implicit constraint can result in harm: “A system... will often set... unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer's apprentice, or King Midas: you get exactly what you ask for, not what you want.”[44]

Midas Gold
In an ancient myth, King Midas wished that “everything” he touched would turn to gold, but failed to specify exceptions for his food and his daughter. By analogy, when AI practitioners specify a goal, it is difficult for them to foresee and rule out every possible side-effect the AI should avoid.[2]

When misaligned AI is deployed, the side-effects can be consequential. Social media platforms have been known to optimize clickthrough rates as a proxy for optimizing user enjoyment, but this addicted some users, decreasing their well-being.[5] Stanford researchers comment that such recommender algorithms are misaligned with their users because they “optimize simple engagement metrics rather than a harder-to-measure combination of societal and consumer well-being”.[9]

To avoid side effects, it is sometimes suggested that AI designers could simply list forbidden actions or formalize ethical rules such as Asimov’s Three Laws of Robotics.[45] However, Russell and Norvig argued that this approach ignores the complexity of human values: “It is certainly very hard, and perhaps impossible, for mere humans to anticipate and rule out in advance all the disastrous ways the machine could choose to achieve a specified objective.”[4]

Additionally, even if an AI system fully understands human intentions, it may still disregard them, because following human intentions may not be its objective.[46]

Systemic risks

Commercial and governmental organizations may have incentives to take shortcuts on safety and deploy insufficiently aligned AI systems.[5] For example the aforementioned social media recommender systems have been profitable despite creating unwanted addiction and polarization on a global scale.[9][47][48] In addition, competitive pressure can create a race to the bottom on safety standards, as in the case of Elaine Herzberg, a pedestrian who was killed by a self-driving car after engineers disabled the emergency braking system because it was over-sensitive and slowing down development.[49]

Risks from advanced misaligned AI
See also: Existential risk from artificial general intelligence and AI takeover

Researchers into the alignment of increasingly advanced AI systems are motivated by the high rate of progress in AI, the large efforts from industry and governments to develop advanced AI systems, and the greater difficulty of aligning them.

As of 2020, OpenAI, DeepMind, and 70 other public projects had the stated aim of developing artificial general intelligence (AGI), a hypothesized system that matches or outperforms humans in a broad range of cognitive tasks.[50] Researchers who scale modern neural networks observe that increasingly general and unexpected capabilities emerge.[9] Such models have learned to operate a computer, write their own programs, and perform a wide range of other tasks from a single model.[51][52][53] Surveys find that some AI researchers expect AGI to be created soon, some believe it is very far off, and many consider both possibilities.[54][55]

Power-seeking

Current AI systems still lack capabilities such as long-term planning and strategic awareness that are thought to pose the most catastrophic misalignment risks.[9][56][7] Future systems (not necessarily AGIs) that have these capabilities may seek to protect and grow their influence over their environment. This tendency is known as "power-seeking" or instrumental convergence. Power-seeking is not explicitly programmed but emerges because power is instrumental for achieving a wide range of goals. For example, AI agents may acquire financial resources and computation, or may evade being turned off, including by running additional copies of the system on other computers.[46][7] Power-seeking has been observed in various reinforcement learning agents.[lower-alpha 1][58][59][60] Later research has mathematically shown that optimal reinforcement learning algorithms seek power in a wide range of environments.[61] As a result, it is often argued that the alignment problem must be solved early, before advanced AI that exhibits emergent power-seeking is created.[7][46][4]

Existential risk
See also: Existential risk from artificial general intelligence and AI takeover

Creating misaligned AI that broadly outperforms humans could challenge the position of humanity as Earth’s dominant species and lead to the disempowerment or possible extinction of humans.[2][4] Notable computer scientists who have pointed out risks from highly advanced misaligned AI include Alan Turing,[lower-alpha 2] Ilya Sutskever,[64] Yoshua Bengio,[lower-alpha 3] Judea Pearl,[lower-alpha 4] Murray Shanahan,[66] Norbert Wiener,[30][4] Marvin Minsky,[lower-alpha 5] Francesca Rossi,[68] Scott Aaronson,[69] Bart Selman,[70] David McAllester,[71] Jürgen Schmidhuber,[72] Marcus Hutter,[73] Shane Legg,[74] Eric Horvitz,[75], Stuart Russell[4] and Geoff Hinton[76].

Skeptical researchers such as François Chollet,[77] Gary Marcus,[78] Yann LeCun,[79] and Oren Etzioni[80] have argued that AGI is far off, or that it would not seek power (successfully).

Alignment may be especially difficult for the most capable AI systems since several misalignment risks increase with the system’s capability: the system’s ability to find loopholes in the assigned objective,[6] cause side-effects, protect and grow its power,[61][7] grow its intelligence, and mislead its designers; the system’s autonomy; and the difficulty of interpreting and supervising the AI system.[4][46]

Research problems and approaches

Learning human values and preferences

Teaching AI systems to act with regard to human values, goals, and preferences is a nontrivial problem because human values can be complex and hard to fully specify. When a goal-directed AI system is given an imperfect or incomplete objective, it commonly learns to exploit these imperfections.[16] This phenomenon is known as reward hacking or specification gaming in AI, and as Goodhart's law in economics and other areas.[39][81] Researchers aim to specify the behavior they intend from an AI as completely as possible with “values-targeted” datasets, imitation learning, or preference learning.[82] A central open problem is scalable oversight, the difficulty of supervising an AI system that outperforms humans in a given domain.[16]

When training a goal-directed AI system, such as a reinforcement learning (RL) agent, it is often difficult for the trainer to specify the intended behavior by writing a reward function manually. An alternative is imitation learning, where the AI learns to imitate demonstrations of the desired behavior. In inverse reinforcement learning (IRL), human demonstrations identify the objective, i.e. the reward function, behind the demonstrated behavior.[83][84] Cooperative inverse reinforcement learning (CIRL) builds on this by assuming a human agent and artificial agent can work together to maximize the human’s reward function.[4][85] CIRL emphasizes that AI agents should be uncertain about the reward function. This humility can help mitigate specification gaming as well as power-seeking tendencies (see § Power-Seeking).[60][73] However, inverse reinforcement learning approaches assume that humans can demonstrate nearly perfect behavior, a misleading assumption when the task is difficult.[86][73]

Other researchers explored eliciting complex behavior through preference learning. Rather than humans providing expert demonstrations, they provide feedback on which of a set of alternative AI behaviors they prefer.[21][23] A helper model is then trained to predict human feedback for new behaviors. Researchers at OpenAI used this approach to train an agent to perform a backflip in less than an hour of evaluation, a maneuver that would have been hard to provide demonstrations for.[40][87] Preference learning has also been an influential tool for recommender systems, web search, and information retrieval.[88] A challenge to this approach is proxy gaming: the helper model may not represent human feedback perfectly, and the main model may exploit this mismatch.[16][89]

Large language models such as GPT-3 enabled the study of value learning in a more general and capable class of AI systems than was available before. Preference learning approaches that were originally designed for RL agents have been extended to improve the quality of generated text and to reduce harmful outputs from these models. OpenAI and DeepMind use this approach to improve the safety of state-of-the-art large language models.[10][23][90] Anthropic proposed using preference learning to fine-tune models to be helpful, honest, and harmless.[91] Other avenues for aligning language models include values-targeted datasets[92][5] and red-teaming.[93][94] In red-teaming, another AI system or a human tries to find inputs for which the model’s behavior is unsafe. Since unsafe behavior can be unacceptable even when it is rare, an important challenge is to drive the rate of unsafe outputs extremely low.[23]

While preference learning can instill hard-to-specify behaviors in AI systems, it requires extensive datasets or human interaction to capture the full breadth of human values. Machine ethics provides a complementary approach: instilling AI systems with moral values.[lower-alpha 6] Machine ethics aims to teach the systems about normative factors in human morality, such as wellbeing, equality, and impartiality; not intending harm; avoiding falsehoods; and honoring promises. Rather than specifying the objective for a specific task, machine ethics seeks to teach AI systems broad moral values that could apply in many situations. This approach carries conceptual challenges of its own. Machine ethicists have noted the necessity to clarify what alignment aims to accomplish: having AIs follow the programmer’s literal instructions, the programmers' implicit intentions, the programmers' revealed preferences, the preferences the programmers would have if they were more informed or rational, the programmers' objective interests, or objective moral standards.[1] Further challenges include aggregating the preferences of different stakeholders and avoiding value lock-in—the indefinite preservation of the values of the first highly capable AI systems, which are unlikely to be fully representative.[1][97]

Scalable oversight

It will be difficult to scale the alignment of AI systems by means of human supervision. As AI systems attempt increasingly complex tasks, it can be slow or infeasible for humans to evaluate them. Such tasks include summarizing books,[98] producing statements that are not merely convincing but also true,[99][41][100] writing code without subtle bugs[11] or security vulnerabilities, and predicting long-term outcomes such as the climate or the results of a policy decision.[101][102] It can be difficult to evaluate AI that outperforms humans in a given domain. To provide feedback in hard-to-evaluate tasks, and to detect when the AI’s solution is only seemingly convincing, humans require assistance or extensive time. The discipline of scalable oversight studies how to reduce the time needed for supervision and how to assist human supervisors.[16]

AI researcher Paul Christiano argues that the owners of AI systems tend to train AI using easy-to-evaluate proxy objectives since that is easier than solving scalable oversight and still profitable. Accordingly, this may lead to “a world that’s increasingly optimized for things [that are easy to measure] like making profits or getting users to click on buttons, or getting users to spend time on websites without being increasingly optimized for having good policies and heading in a trajectory that we’re happy with”.[103]

One easy-to-measure objective is the score the supervisor assigns to the AI’s outputs. Some AI systems have discovered a shortcut to achieving high scores, by taking actions that falsely convince the human supervisor that the AI has achieved the intended objective (see the video of a robot hand above[40]). Some AI systems have also learned to recognize when they are being evaluated, and “play dead”, only to behave differently once evaluation ends.[104] This deceptive form of specification gaming may become easier for AI systems that are more sophisticated[6][46] and attempt more difficult-to-evaluate tasks. If advanced models are also capable planners, they could obscure their deception from supervisors.[105] In the automotive industry, Volkswagen engineers obscured their cars’ emissions in laboratory testing, underscoring that deception of evaluators is a common pattern in the real world.[5]

Approaches such as active learning and semi-supervised reward learning can reduce the amount of human supervision needed.[16] Another approach is to train a helper model ("reward model") to imitate the supervisor’s judgment.[16][22][23][106]

However, when the task is too complex to evaluate accurately, or the human supervisor is vulnerable to deception, it is the quality, not the quantity of supervision that matters. To increase supervision quality, a range of approaches aim to assist the supervisor, sometimes by using AI assistants. Iterated Amplification is an approach developed by Christiano that iteratively builds a feedback signal for challenging problems by using humans to combine solutions to easier subproblems.[82][101] Iterated Amplification was used to train AI to summarize books without requiring human supervisors to read them.[98][107] Another proposal is to align AI by means of debate between AI systems, with the winner judged by humans.[108][73] Such debate is intended to reveal the weakest points of an answer to a complex question, and reward the AI for truthful and safe answers.

Honest AI
Language models like GPT-3 often generate falsehoods.[109]

Researchers from the Future of Humanity Institute point out that language models such as GPT-3, which can generate fluent and grammatically correct text,[110][111] have opened the door to AI systems repeating falsehoods from their training data or even deliberately lying to humans.[112][109]

Current state-of-the-art language models learn by imitating human writing across millions of books worth of text from the Internet.[9][113] While this helps them learn a wide range of skills, the training data also includes common misconceptions, incorrect medical advice, and conspiracy theories. AI systems trained on this data learn to mimic false statements.[109][100][41] Additionally, models often obediently continue falsehoods when prompted, generate empty explanations for their answers, or produce outright fabrications.[34] For example, when prompted to write a biography for a real AI researcher, a chatbot confabulated numerous details about their life, which the researcher identified as false.[114]

To combat the lack of truthfulness exhibited by modern AI systems, researchers have explored several options. AI research organizations including OpenAI and DeepMind have developed AI systems that can cite their sources and explain their reasoning when answering questions, enabling better transparency and verifiability.[115][116][117] Researchers from OpenAI and Anthropic proposed using human feedback and curated datasets to fine-tune AI assistants such that they avoid negligent falsehoods or express when they are uncertain.[23][118][91] Alongside technical solutions, researchers recommend defining clear truthfulness standards and the creation of institutions, regulatory bodies, or watchdog agencies to evaluate AI systems on these standards before and during deployment.[112]

Researchers distinguish "truthfulness" (that AIs only make statements that are objectively true) and "honesty" (that AIs only assert what they believe to be true). Recent research finds that state-of-the-art AI systems cannot be said to hold stable beliefs, so as of 2023 it is not yet tractable to study the honesty of AI systems.[119] Future AI systems that do hold beliefs could intentionally lie to humans. A misaligned AI could deceive its operators into thinking it was safe or persuade them that nothing is amiss.[7][9][5] Some argue that if AIs could be made to assert only what they believe to be true, this would sidestep numerous problems in alignment.[112][120]

Inner alignment and emergent goals

Alignment research aims to align three different descriptions of an AI system:[121]

  1. Intended goals ('wishes'): “the hypothetical (but hard to articulate) description of an ideal AI system that is fully aligned to the desires of the human operator”;
  2. Specified goals (or ‘outer specification’): the goals we actually specify — typically jointly through an objective function and a dataset;
  3. Emergent goals (or ‘inner specification’): the goals the AI actually advances.

‘Outer misalignment’ is a mismatch between the intended goals (1) and the specified goals (2), whereas ‘inner misalignment’ is a mismatch between the human-specified goals (2) and the AI's emergent goals (3).

Inner misalignment is analogous to biological evolution.[122] In the ancestral environment, evolution selected human genes for inclusive genetic fitness, but humans evolved to have other objectives. Fitness corresponds to (2), the specified goal used in the training environment and training data. In evolutionary history, maximizing the fitness specification led to intelligent agents, humans, that do not directly pursue inclusive genetic fitness. Instead, they pursue emergent goals (3) that correlated with genetic fitness in the ancestral environment: nutrition, sex, and so on. However, our environment has changed — a distribution shift has occurred. Humans still pursue their emergent goals, but this no longer maximizes genetic fitness. (In machine learning the analogous problem is known as goal misgeneralization.[3]) Our taste for sugary food (an emergent goal) was originally beneficial, but now leads to overeating and health problems. Also, by using contraception, humans directly contradict genetic fitness. By analogy, if genetic fitness were the objective chosen by an AI developer, they would observe the model behaving as intended in the training environment, without noticing that the model is pursuing an unintended emergent goal until the model was deployed.

Research directions to detect and remove misaligned emergent goals include red teaming, verification, anomaly detection, and interpretability.[16][5][17] Progress on these techniques may help reduce two open problems: First, emergent goals only become apparent when the system is deployed outside its training environment, but it can be unsafe to deploy a misaligned system in high-stakes environments—even for a short time—until its misalignment is detected. Such high stakes are common in autonomous driving, health care, and military applications.[123] The stakes become higher yet when AI systems gain more autonomy and capability, becoming capable of sidestepping human interventions (see § Power-seeking and instrumental goals). Second, a sufficiently capable AI system may take actions that falsely convince human supervisors that it is pursuing the intended objective (see previous discussion on deception at § Scalable oversight).

Power-seeking and instrumental goals

Since the 1950s, AI researchers have sought to build advanced AI systems that can achieve goals by predicting the results of their actions and making long-term plans.[124] Some researchers argue that suitably advanced planning systems will seek power over their environment, including over humans — for example by evading shutdown and acquiring resources. Such power-seeking behavior is not explicitly programmed but emerges because power is instrumental for achieving a wide range of goals.[61][4][7] Power-seeking is thus considered a convergent instrumental goal.[46]

As of 2023 power-seeking is uncommon in current systems, but advanced systems that can foresee the long-term results of their actions may increasingly seek power. This was shown in formal work which found that optimal reinforcement learning agents will seek power by seeking ways to gain more options, a behavior that persists across a wide range of environments and goals.[61]

Power-seeking demonstrably emerges in some real-world systems. Reinforcement learning systems have gained more options by acquiring and protecting resources, sometimes in ways their designers did not intend.[57][125] Other systems have learned, in toy environments, that in order to achieve their goal, they can prevent human interference[58] or disable their off-switch.[60] Russell illustrated this behavior by imagining a robot that is tasked to fetch coffee and evades being turned off since "you can't fetch the coffee if you're dead".[4]

To acquire power, AI systems might hypothetically try to:

“...break out of a contained environment; hack; get access to financial resources, or additional computing resources; make backup copies of themselves; gain unauthorized capabilities, sources of information, or channels of influence; mislead/lie to humans about their goals; resist or manipulate attempts to monitor/understand their behavior... impersonate humans; cause humans to do things for them; ...manipulate human discourse and politics; weaken various human institutions and response capacities; take control of physical infrastructure like factories or scientific laboratories; cause certain types of technology and infrastructure to be developed; or directly harm/overpower humans.”[7]

Researchers aim to train systems that are "corrigible": systems that do not seek power and that allow themselves to be turned off, modified, etc. Proxy gaming is a possible way this can fail: when researchers penalize a system for seeking power, the system is thereby incentivized to seek power in difficult-to-detect ways.[5] To detect such covert behavior, researchers aim to create techniques and tools to inspect AI models[5] such as neural networks, and interpret their inner workings instead of simply considering them as black boxes.

Additionally, researchers propose to solve the problem of systems disabling their off-switches by making AI agents uncertain about the objective they are pursuing.[60][4] Agents designed in this way would allow humans to turn them off, since this would indicate that the agent was wrong about the value of whatever action they were taking prior to being shut down. More research is needed in order to successfully implement this.[82]

Power-seeking AI poses unusual risks. Ordinary safety-critical systems like planes and bridges are not adversarial. They lack the ability and incentive to evade safety measures or to deliberately appear safer than they are. In contrast, power-seeking AI has been compared to a hacker that evades security measures.[7] Further, ordinary technologies can be made safe through trial-and-error, unlike power-seeking AI which has been compared to a virus whose release is irreversible since it has the potential to continuously evolve and grow, eventually leading to the disempowerment or extinction of humans.[7] It is therefore often argued that the alignment problem must be solved early, before advanced power-seeking AI is created.[46]

However, some critics argued that power-seeking is not inevitable, since humans do not always seek power and may only do so for evolutionary reasons. Furthermore, there is debate whether any future AI systems need to pursue goals and make long-term plans at all.[126][7]

Embedded agency

Work on scalable oversight largely occurs within formalisms such as POMDPs. Existing formalisms assume that an AI agent's algorithm is executed outside the environment (i.e. is not physically embedded in it). Embedded agency[127][128] is another major strand of research which attempts to solve problems arising from the mismatch between such theoretical frameworks and real agents we might build. For example, even if the scalable oversight problem is solved, an agent that can gain access to the computer it is running on may have an incentive to tamper with its reward function in order to get much more reward than its human supervisors give it.[129] A list of examples of specification gaming from DeepMind researcher Victoria Krakovna includes a genetic algorithm that learned to delete the file containing its target output so that it was rewarded for outputting nothing.[130] This class of problems has been formalised using causal incentive diagrams.[129] Researchers at Oxford and DeepMind argued that such problematic behavior is highly likely in advanced systems, and that advanced systems would seek power to stay in control of their reward signal indefinitely and certainly.[131] They suggest a range of potential approaches to address this open problem.

Skepticism of AI risk
Main article: Existential risk from artificial general intelligence § Skepticism

Skeptics of AI risk believe that superintelligence poses little risk of misbehavior, or that such risks are overblown. Some skeptics,[132] such as Gary Marcus,[133] propose adopting rules similar to the fictional Three Laws of Robotics which directly specify a desired outcome ("direct normativity"). By contrast, most endorsers of the existential risk thesis (as well as many skeptics) consider the Three Laws to be unhelpful, because they are ambiguous and self-contradictory. (Other "direct normativity" proposals include Kantian ethics, utilitarianism, or a mix of some small list of enumerated desiderata.) Most risk endorsers believe instead that human values (and their quantitative trade-offs) are too complex and poorly-understood to be directly programmed into a superintelligence; instead, a superintelligence would need to be programmed with a process for acquiring and fully understanding human values ("indirect normativity"), such as coherent extrapolated volition.[134]

Public policy
See also: Regulation of artificial intelligence

A number of governmental and treaty organizations have made statements emphasizing the importance of AI alignment.

In September 2021, the Secretary-General of the United Nations issued a declaration which included a call to regulate AI to ensure it is "aligned with shared global values."[135]

That same month, the PRC published ethical guidelines for the use of AI in China. According to the guidelines, researchers must ensure that AI abides by shared human values, is always under human control, and is not endangering public safety.[136]

Also in September 2021, the UK published its 10-year National AI Strategy,[137] which states the British government "takes the long term risk of non-aligned Artificial General Intelligence, and the unforeseeable changes that it would mean for... the world, seriously".[138] The strategy describes actions to assess long term AI risks, including catastrophic risks.[139]

In March 2021, the US National Security Commission on Artificial Intelligence stated that "Advances in AI... could lead to inflection points or leaps in capabilities. Such advances may also introduce new concerns and risks and the need for new policies, recommendations, and technical advances to assure that systems are aligned with goals and values, including safety, robustness and trustworthiness. The US should... ensure that AI systems and their uses align with our goals and values."[140]

See also

Footnotes
  1. Reinforcement learning systems have learned to gain more options by acquiring and protecting resources, sometimes in ways their designers did not intend.[57][7]
  2. In a 1951 lecture[62] Turing argued that “It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s Erewhon.” Also in a lecture broadcast on BBC[63] expressed: "If a machine can think, it might think more intelligently than we do, and then where should we be? Even if we could keep the machines in a subservient position, for instance by turning off the power at strategic moments, we should, as a species, feel greatly humbled. . . . This new danger . . . is certainly something which can give us anxiety.”
  3. Bengio wrote "This beautifully written book addresses a fundamental challenge for humanity: increasingly intelligent machines that do what we ask but not what we really intend. Essential reading if you care about our future" about Russel's book Human Compatible: AI and the Problem of Control[65] which argues that existential risk from misaligned AI to humanity is a serious concern worth addressing today.
  4. Pearl wrote "Human Compatible made me a convert to Russell's concerns with our ability to control our upcoming creation–super-intelligent machines. Unlike outside alarmists and futurists, Russell is a leading authority on AI. His new book will educate the public about AI more than any book I can think of, and is a delightful and uplifting read" about Russel's book Human Compatible: AI and the Problem of Control[65] which argues that existential risk to humanity from misaligned AI is a serious concern worth addressing today.
  5. Russell & Norvig[67] note: “The “King Midas problem” was anticipated by Marvin Minsky, who once suggested that an AI program designed to solve the Riemann Hypothesis might end up taking over all the resources of Earth to build more powerful supercomputers."
  6. Vincent Wiegel argued “we should extend [machines] with moral sensitivity to the moral dimensions of the situations in which the increasingly autonomous machines will inevitably find themselves.”,[95] referencing the book Moral machines: teaching robots right from wrong[96] from Wendell Wallach and Colin Allen.

References
  1. Gabriel, Iason (September 1, 2020). "Artificial Intelligence, Values, and Alignment". Minds and Machines. 30 (3): 411–437. doi:10.1007/s11023-020-09539-2. ISSN 1572-8641. S2CID 210920551. Archived from the original on March 15, 2023. Retrieved July 23, 2022.
  2. Russell, Stuart J.; Norvig, Peter (2020). Artificial intelligence: A modern approach (4th ed.). Pearson. pp. 31–34. ISBN 978-1-292-40113-3. OCLC 1303900751. Archived from the original on July 15, 2022. Retrieved September 12, 2022.
  3. Langosco, Lauro Langosco Di; Koch, Jack; Sharkey, Lee D; Pfau, Jacob; Krueger, David (July 17, 2022). "Goal misgeneralization in deep reinforcement learning". International Conference on Machine Learning. Vol. 162. PMLR. pp. 12004–12019.
  4. Russell, Stuart J. (2020). Human compatible: Artificial intelligence and the problem of control. Penguin Random House. ISBN 9780525558637. OCLC 1113410915. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  5. Hendrycks, Dan; Carlini, Nicholas; Schulman, John; Steinhardt, Jacob (June 16, 2022). "Unsolved Problems in ML Safety". arXiv:2109.13916 [cs.LG].
  6. Pan, Alexander; Bhatia, Kush; Steinhardt, Jacob (February 14, 2022). The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. International Conference on Learning Representations. Archived from the original on February 10, 2023. Retrieved July 21, 2022.
  7. Carlsmith, Joseph (June 16, 2022). "Is Power-Seeking AI an Existential Risk?". arXiv:2206.13353 [cs.CY].
  8. Kober, Jens; Bagnell, J. Andrew; Peters, Jan (September 1, 2013). "Reinforcement learning in robotics: A survey". The International Journal of Robotics Research. 32 (11): 1238–1274. doi:10.1177/0278364913495721. ISSN 0278-3649. S2CID 1932843. Archived from the original on October 15, 2022. Retrieved September 12, 2022.
  9. Bommasani, Rishi; Hudson, Drew A.; Adeli, Ehsan; Altman, Russ; Arora, Simran; von Arx, Sydney; Bernstein, Michael S.; Bohg, Jeannette; Bosselut, Antoine; Brunskill, Emma; Brynjolfsson, Erik (July 12, 2022). "On the Opportunities and Risks of Foundation Models". Stanford CRFM. arXiv:2108.07258. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  10. Ouyang, Long; Wu, Jeff; Jiang, Xu; Almeida, Diogo; Wainwright, Carroll L.; Mishkin, Pamela; Zhang, Chong; Agarwal, Sandhini; Slama, Katarina; Ray, Alex; Schulman, J.; Hilton, Jacob; Kelton, Fraser; Miller, Luke E.; Simens, Maddie; Askell, Amanda; Welinder, P.; Christiano, P.; Leike, J.; Lowe, Ryan J. (2022). "Training language models to follow instructions with human feedback". arXiv:2203.02155 [cs.CL].
  11. Zaremba, Wojciech; Brockman, Greg; OpenAI (August 10, 2021). "OpenAI Codex". OpenAI. Archived from the original on February 3, 2023. Retrieved July 23, 2022.
  12. Knox, W. Bradley; Allievi, Alessandro; Banzhaf, Holger; Schmitt, Felix; Stone, Peter (March 11, 2022). "Reward (Mis)design for Autonomous Driving" (PDF). arXiv:2104.13906. Archived (PDF) from the original on February 10, 2023. Retrieved September 12, 2022. {{cite journal}}: Cite journal requires |journal= (help)
  13. Stray, Jonathan (2020). "Aligning AI Optimization to Community Well-Being". International Journal of Community Well-Being. 3 (4): 443–463. doi:10.1007/s42413-020-00086-3. ISSN 2524-5295. PMC 7610010. PMID 34723107. S2CID 226254676.
  14. Future of Life Institute (August 11, 2017). "Asilomar AI Principles". Future of Life Institute. Archived from the original on October 10, 2022. Retrieved July 18, 2022.
  15. United Nations (2021). Our Common Agenda: Report of the Secretary-General (PDF) (Report). New York: United Nations. Archived (PDF) from the original on May 22, 2022. Retrieved September 12, 2022.
  16. Amodei, Dario; Olah, Chris; Steinhardt, Jacob; Christiano, Paul; Schulman, John; Mané, Dan (June 21, 2016). "Concrete Problems in AI Safety". arXiv:1606.06565 [cs.AI].
  17. Ortega, Pedro A.; Maini, Vishal; DeepMind safety team (September 27, 2018). "Building safe artificial intelligence: specification, robustness, and assurance". DeepMind Safety Research - Medium. Archived from the original on February 10, 2023. Retrieved July 18, 2022.
  18. Rorvig, Mordechai (April 14, 2022). "Researchers Gain New Understanding From Simple AI". Quanta Magazine. Archived from the original on February 10, 2023. Retrieved July 18, 2022.
  19. Doshi-Velez, Finale; Kim, Been (March 2, 2017). "Towards A Rigorous Science of Interpretable Machine Learning". arXiv:1702.08608 [stat.ML].
  20. Russell, Stuart; Dewey, Daniel; Tegmark, Max (December 31, 2015). "Research Priorities for Robust and Beneficial Artificial Intelligence". AI Magazine. 36 (4): 105–114. doi:10.1609/aimag.v36i4.2577. hdl:1721.1/108478. ISSN 2371-9621. S2CID 8174496. Archived from the original on February 2, 2023. Retrieved September 12, 2022.
  21. Wirth, Christian; Akrour, Riad; Neumann, Gerhard; Fürnkranz, Johannes (2017). "A survey of preference-based reinforcement learning methods". Journal of Machine Learning Research. 18 (136): 1–46.
  22. Christiano, Paul F.; Leike, Jan; Brown, Tom B.; Martic, Miljan; Legg, Shane; Amodei, Dario (2017). "Deep reinforcement learning from human preferences". Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS'17. Red Hook, NY, USA: Curran Associates Inc. pp. 4302–4310. ISBN 978-1-5108-6096-4.
  23. Heaven, Will Douglas (January 27, 2022). "The new version of GPT-3 is much better behaved (and should be less toxic)". MIT Technology Review. Archived from the original on February 10, 2023. Retrieved July 18, 2022.
  24. Mohseni, Sina; Wang, Haotao; Yu, Zhiding; Xiao, Chaowei; Wang, Zhangyang; Yadawa, Jay (March 7, 2022). "Taxonomy of Machine Learning Safety: A Survey and Primer". arXiv:2106.04823 [cs.LG].
  25. Clifton, Jesse (2020). "Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda". Center on Long-Term Risk. Archived from the original on January 1, 2023. Retrieved July 18, 2022.
  26. Dafoe, Allan; Bachrach, Yoram; Hadfield, Gillian; Horvitz, Eric; Larson, Kate; Graepel, Thore (May 6, 2021). "Cooperative AI: machines must learn to find common ground". Nature. 593 (7857): 33–36. Bibcode:2021Natur.593...33D. doi:10.1038/d41586-021-01170-0. ISSN 0028-0836. PMID 33947992. S2CID 233740521. Archived from the original on December 18, 2022. Retrieved September 12, 2022.
  27. Prunkl, Carina; Whittlestone, Jess (February 7, 2020). "Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society". Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. New York NY USA: ACM: 138–143. doi:10.1145/3375627.3375803. ISBN 978-1-4503-7110-0. S2CID 210164673. Archived from the original on October 16, 2022. Retrieved September 12, 2022.
  28. Irving, Geoffrey; Askell, Amanda (February 19, 2019). "AI Safety Needs Social Scientists". Distill. 4 (2): 10.23915/distill.00014. doi:10.23915/distill.00014. ISSN 2476-0757. S2CID 159180422. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  29. "Faulty Reward Functions in the Wild". OpenAI. December 22, 2016. Archived from the original on January 26, 2021. Retrieved September 10, 2022.
  30. Wiener, Norbert (May 6, 1960). "Some Moral and Technical Consequences of Automation: As machines learn they may develop unforeseen strategies at rates that baffle their programmers". Science. 131 (3410): 1355–1358. doi:10.1126/science.131.3410.1355. ISSN 0036-8075. PMID 17841602. Archived from the original on October 15, 2022. Retrieved September 12, 2022.
  31. The Ezra Klein Show (June 4, 2021). "If 'All Models Are Wrong,' Why Do We Give Them So Much Power?". The New York Times. ISSN 0362-4331. Archived from the original on February 15, 2023. Retrieved July 18, 2022.
  32. Wolchover, Natalie (April 21, 2015). "Concerns of an Artificial Intelligence Pioneer". Quanta Magazine. Archived from the original on February 10, 2023. Retrieved July 18, 2022.
  33. California Assembly. "Bill Text - ACR-215 23 Asilomar AI Principles". Archived from the original on February 10, 2023. Retrieved July 18, 2022.
  34. Johnson, Steven; Iziev, Nikita (April 15, 2022). "A.I. Is Mastering Language. Should We Trust What It Says?". The New York Times. ISSN 0362-4331. Archived from the original on November 24, 2022. Retrieved July 18, 2022.
  35. Russell, Stuart J.; Norvig, Peter (2020). Artificial intelligence: A modern approach (4th ed.). Pearson. pp. 4–5. ISBN 978-1-292-40113-3. OCLC 1303900751. Archived from the original on July 15, 2022. Retrieved September 12, 2022.
  36. OpenAI (February 15, 2022). "Aligning AI systems with human intent". OpenAI. Archived from the original on February 10, 2023. Retrieved July 18, 2022.
  37. Medium. "DeepMind Safety Research". Medium. Archived from the original on February 10, 2023. Retrieved July 18, 2022.
  38. Krakovna, Victoria; Uesato, Jonathan; Mikulik, Vladimir; Rahtz, Matthew; Everitt, Tom; Kumar, Ramana; Kenton, Zac; Leike, Jan; Legg, Shane (April 21, 2020). "Specification gaming: the flip side of AI ingenuity". Deepmind. Archived from the original on February 10, 2023. Retrieved August 26, 2022.
  39. Manheim, David; Garrabrant, Scott (2018). "Categorizing Variants of Goodhart's Law". arXiv:1803.04585 [cs.AI].
  40. Amodei, Dario; Christiano, Paul; Ray, Alex (June 13, 2017). "Learning from Human Preferences". OpenAI. Archived from the original on January 3, 2021. Retrieved July 21, 2022.
  41. Lin, Stephanie; Hilton, Jacob; Evans, Owain (2022). "TruthfulQA: Measuring How Models Mimic Human Falsehoods". Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics: 3214–3252. doi:10.18653/v1/2022.acl-long.229. S2CID 237532606. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  42. Naughton, John (October 2, 2021). "The truth about artificial intelligence? It isn't that honest". The Observer. ISSN 0029-7712. Archived from the original on February 13, 2023. Retrieved July 18, 2022.
  43. Ji, Ziwei; Lee, Nayeon; Frieske, Rita; Yu, Tiezheng; Su, Dan; Xu, Yan; Ishii, Etsuko; Bang, Yejin; Madotto, Andrea; Fung, Pascale (February 1, 2022). "Survey of Hallucination in Natural Language Generation". ACM Computing Surveys. 55 (12): 1–38. arXiv:2202.03629. doi:10.1145/3571730. S2CID 246652372. Archived from the original on February 10, 2023. Retrieved October 14, 2022.
  44. Edge.org. "The Myth Of AI | Edge.org". Archived from the original on February 10, 2023. Retrieved July 19, 2022.
  45. Tasioulas, John (2019). "First Steps Towards an Ethics of Robots and Artificial Intelligence". Journal of Practical Ethics. 7 (1): 61–95.
  46. Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies (1st ed.). USA: Oxford University Press, Inc. ISBN 978-0-19-967811-2.
  47. Wells, Georgia; Deepa Seetharaman; Horwitz, Jeff (November 5, 2021). "Is Facebook Bad for You? It Is for About 360 Million Users, Company Surveys Suggest". Wall Street Journal. ISSN 0099-9660. Archived from the original on February 10, 2023. Retrieved July 19, 2022.
  48. Barrett, Paul M.; Hendrix, Justin; Sims, J. Grant (September 2021). How Social Media Intensifies U.S. Political Polarization-And What Can Be Done About It (Report). Center for Business and Human Rights, NYU. Archived from the original on February 1, 2023. Retrieved September 12, 2022.
  49. Shepardson, David (May 24, 2018). "Uber disabled emergency braking in self-driving car: U.S. agency". Reuters. Archived from the original on February 10, 2023. Retrieved July 20, 2022.
  50. Baum, Seth (January 1, 2021). "2020 Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy". Archived from the original on February 10, 2023. Retrieved July 20, 2022.
  51. Edwards, Ben (April 26, 2022). "Adept's AI assistant can browse, search, and use web apps like a human". Ars Technica. Archived from the original on January 17, 2023. Retrieved September 9, 2022.
  52. Wakefield, Jane (February 2, 2022). "DeepMind AI rivals average human competitive coder". BBC News. Archived from the original on February 10, 2023. Retrieved September 9, 2022.
  53. Dominguez, Daniel (May 19, 2022). "DeepMind Introduces Gato, a New Generalist AI Agent". InfoQ. Archived from the original on February 10, 2023. Retrieved September 9, 2022.
  54. Grace, Katja; Salvatier, John; Dafoe, Allan; Zhang, Baobao; Evans, Owain (July 31, 2018). "Viewpoint: When Will AI Exceed Human Performance? Evidence from AI Experts". Journal of Artificial Intelligence Research. 62: 729–754. doi:10.1613/jair.1.11222. ISSN 1076-9757. S2CID 8746462. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  55. Zhang, Baobao; Anderljung, Markus; Kahn, Lauren; Dreksler, Noemi; Horowitz, Michael C.; Dafoe, Allan (August 2, 2021). "Ethics and Governance of Artificial Intelligence: Evidence from a Survey of Machine Learning Researchers". Journal of Artificial Intelligence Research. 71. doi:10.1613/jair.1.12895. ISSN 1076-9757. S2CID 233740003. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  56. Wei, Jason; Tay, Yi; Bommasani, Rishi; Raffel, Colin; Zoph, Barret; Borgeaud, Sebastian; Yogatama, Dani; Bosma, Maarten; Zhou, Denny; Metzler, Donald; Chi, Ed H.; Hashimoto, Tatsunori; Vinyals, Oriol; Liang, Percy; Dean, Jeff (June 15, 2022). "Emergent Abilities of Large Language Models". arXiv:2206.07682 [cs.CL].
  57. Ornes, Stephen (November 18, 2019). "Playing Hide-and-Seek, Machines Invent New Tools". Quanta Magazine. Archived from the original on February 10, 2023. Retrieved August 26, 2022.
  58. Leike, Jan; Martic, Miljan; Krakovna, Victoria; Ortega, Pedro A.; Everitt, Tom; Lefrancq, Andrew; Orseau, Laurent; Legg, Shane (November 28, 2017). "AI Safety Gridworlds". arXiv:1711.09883 [cs.LG].
  59. Orseau, Laurent; Armstrong, Stuart (January 1, 2016). "Safely Interruptible Agents". Archived from the original on February 10, 2023. Retrieved July 20, 2022. {{cite journal}}: Cite journal requires |journal= (help)
  60. Hadfield-Menell, Dylan; Dragan, Anca; Abbeel, Pieter; Russell, Stuart (2017). "The Off-Switch Game". Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17. pp. 220–227. doi:10.24963/ijcai.2017/32.
  61. Turner, Alexander Matt; Smith, Logan; Shah, Rohin; Critch, Andrew; Tadepalli, Prasad (December 3, 2021). "Optimal Policies Tend to Seek Power". Neural Information Processing Systems. 34. arXiv:1912.01683. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  62. Turing, Alan (1951). Intelligent machinery, a heretical theory (Speech). Lecture given to '51 Society'. Manchester: The Turing Digital Archive. Archived from the original on September 26, 2022. Retrieved July 22, 2022.
  63. Turing, Alan (May 15, 1951). "Can digital computers think?". Automatic Calculating Machines. Episode 2. BBC. Can digital computers think?.
  64. Muehlhauser, Luke (January 29, 2016). "Sutskever on Talking Machines". Luke Muehlhauser. Archived from the original on September 27, 2022. Retrieved August 26, 2022.
  65. "Human Compatible: AI and the Problem of Control". Archived from the original on October 10, 2022. Retrieved July 22, 2022.
  66. Shanahan, Murray (2015). The technological singularity. Cambridge, Massachusetts. ISBN 978-0-262-33182-1. OCLC 917889148.
  67. Russell, Stuart; Norvig, Peter (2009). Artificial Intelligence: A Modern Approach. Prentice Hall. p. 1010. ISBN 978-0-13-604259-4.
  68. Rossi, Francesca. "Opinion | How do you teach a machine to be moral?". Washington Post. ISSN 0190-8286. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  69. Aaronson, Scott (June 17, 2022). "OpenAI!". Shtetl-Optimized. Archived from the original on August 27, 2022. Retrieved September 12, 2022.
  70. Selman, Bart, Intelligence Explosion: Science or Fiction? (PDF), archived (PDF) from the original on May 31, 2022, retrieved September 12, 2022
  71. McAllester (August 10, 2014). "Friendly AI and the Servant Mission". Machine Thoughts. Archived from the original on September 28, 2022. Retrieved September 12, 2022.
  72. Schmidhuber, Jürgen (March 6, 2015). "I am Jürgen Schmidhuber, AMA!" (Reddit Comment). r/MachineLearning. Archived from the original on February 10, 2023. Retrieved July 23, 2022.
  73. Everitt, Tom; Lea, Gary; Hutter, Marcus (May 21, 2018). "AGI Safety Literature Review". arXiv:1805.01109 [cs.AI].
  74. Shane (August 31, 2009). "Funding safe AGI". vetta project. Archived from the original on October 10, 2022. Retrieved September 12, 2022.
  75. Horvitz, Eric (June 27, 2016). "Reflections on Safety and Artificial Intelligence" (PDF). Eric Horvitz. Archived (PDF) from the original on October 10, 2022. Retrieved April 20, 2020.
  76. Vacchiano, Andrea (March 25, 2023). "Artificial intelligence 'godfather' on AI possibly wiping out humanity: 'It's not inconceivable'". Fox News.
  77. Chollet, François (December 8, 2018). "The implausibility of intelligence explosion". Medium. Archived from the original on March 22, 2021. Retrieved August 26, 2022.
  78. Marcus, Gary (June 6, 2022). "Artificial General Intelligence Is Not as Imminent as You Might Think". Scientific American. Archived from the original on September 15, 2022. Retrieved August 26, 2022.
  79. Barber, Lynsey (July 31, 2016). "Phew! Facebook's AI chief says intelligent machines are not a threat to humanity". CityAM. Archived from the original on August 26, 2022. Retrieved August 26, 2022.
  80. Harris, Jeremie (June 16, 2021). "The case against (worrying about) existential risk from AI". Medium. Archived from the original on August 26, 2022. Retrieved August 26, 2022.
  81. Rochon, Louis-Philippe; Rossi, Sergio (February 27, 2015). The Encyclopedia of Central Banking. Edward Elgar Publishing. ISBN 978-1-78254-744-0. Archived from the original on February 10, 2023. Retrieved September 13, 2022.
  82. Christian, Brian (2020). The alignment problem: Machine learning and human values. W. W. Norton & Company. ISBN 978-0-393-86833-3. OCLC 1233266753. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  83. Christian, Brian (2020). The alignment problem: Machine learning and human values. W. W. Norton & Company. p. 88. ISBN 978-0-393-86833-3. OCLC 1233266753. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  84. Ng, Andrew Y.; Russell, Stuart J. (2000). "Algorithms for inverse reinforcement learning". Proceedings of the seventeenth international conference on machine learning. ICML '00. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. pp. 663–670. ISBN 1-55860-707-2.
  85. Hadfield-Menell, Dylan; Russell, Stuart J; Abbeel, Pieter; Dragan, Anca (2016). "Cooperative Inverse Reinforcement Learning". Advances in Neural Information Processing Systems. NIPS'16. Vol. 29. ISBN 978-1-5108-3881-9. Archived from the original on February 10, 2023. Retrieved July 21, 2022.
  86. Armstrong, Stuart; Mindermann, Sören (2018). "Occam' s razor is insufficient to infer the preferences of irrational agents". Advances in Neural Information Processing Systems. NeurIPS 2018. Vol. 31. Montréal: Curran Associates, Inc. Archived from the original on February 10, 2023. Retrieved July 21, 2022.
  87. Li, Yuxi (November 25, 2018). "Deep Reinforcement Learning: An Overview" (PDF). Lecture Notes in Networks and Systems Book Series. Archived (PDF) from the original on October 10, 2022. Retrieved September 12, 2022.
  88. Fürnkranz, Johannes; Hüllermeier, Eyke; Rudin, Cynthia; Slowinski, Roman; Sanner, Scott (2014). Marc Herbstritt. "Preference Learning". Dagstuhl Reports. 4 (3): 27 pages. doi:10.4230/DAGREP.4.3.1. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  89. Hilton, Jacob; Gao, Leo (April 13, 2022). "Measuring Goodhart's Law". OpenAI. Archived from the original on February 10, 2023. Retrieved September 9, 2022.
  90. Anderson, Martin (April 5, 2022). "The Perils of Using Quotations to Authenticate NLG Content". Unite.AI. Archived from the original on February 10, 2023. Retrieved July 21, 2022.
  91. Wiggers, Kyle (February 5, 2022). "Despite recent progress, AI-powered chatbots still have a long way to go". VentureBeat. Archived from the original on July 23, 2022. Retrieved July 23, 2022.
  92. Hendrycks, Dan; Burns, Collin; Basart, Steven; Critch, Andrew; Li, Jerry; Song, Dawn; Steinhardt, Jacob (July 24, 2021). "Aligning AI With Shared Human Values". International Conference on Learning Representations. arXiv:2008.02275.
  93. Perez, Ethan; Huang, Saffron; Song, Francis; Cai, Trevor; Ring, Roman; Aslanides, John; Glaese, Amelia; McAleese, Nat; Irving, Geoffrey (February 7, 2022). "Red Teaming Language Models with Language Models". arXiv:2202.03286 [cs.CL].
  94. Bhattacharyya, Sreejani (February 14, 2022). "DeepMind's "red teaming" language models with language models: What is it?". Analytics India Magazine. Archived from the original on February 13, 2023. Retrieved July 23, 2022.
  95. Wiegel, Vincent (December 1, 2010). "Wendell Wallach and Colin Allen: moral machines: teaching robots right from wrong". Ethics and Information Technology. 12 (4): 359–361. doi:10.1007/s10676-010-9239-1. ISSN 1572-8439. S2CID 30532107. Archived from the original on March 15, 2023. Retrieved July 23, 2022.
  96. Wallach, Wendell; Allen, Colin (2009). Moral Machines: Teaching Robots Right from Wrong. New York: Oxford University Press. ISBN 978-0-19-537404-9. Archived from the original on March 15, 2023. Retrieved July 23, 2022.
  97. MacAskill, William (2022). What we owe the future. New York, NY: Basic Books. ISBN 978-1-5416-1862-6. OCLC 1314633519. Archived from the original on September 14, 2022. Retrieved September 12, 2022.
  98. Wu, Jeff; Ouyang, Long; Ziegler, Daniel M.; Stiennon, Nisan; Lowe, Ryan; Leike, Jan; Christiano, Paul (September 27, 2021). "Recursively Summarizing Books with Human Feedback". arXiv:2109.10862 [cs.CL].
  99. Irving, Geoffrey; Amodei, Dario (May 3, 2018). "AI Safety via Debate". OpenAI. Archived from the original on February 10, 2023. Retrieved July 23, 2022.
  100. Naughton, John (October 2, 2021). "The truth about artificial intelligence? It isn't that honest". The Observer. ISSN 0029-7712. Archived from the original on February 13, 2023. Retrieved July 23, 2022.
  101. Christiano, Paul; Shlegeris, Buck; Amodei, Dario (October 19, 2018). "Supervising strong learners by amplifying weak experts". arXiv:1810.08575 [cs.LG].
  102. Banzhaf, Wolfgang; Goodman, Erik; Sheneman, Leigh; Trujillo, Leonardo; Worzel, Bill, eds. (2020). Genetic Programming Theory and Practice XVII. Genetic and Evolutionary Computation. Cham: Springer International Publishing. doi:10.1007/978-3-030-39958-0. ISBN 978-3-030-39957-3. S2CID 218531292. Archived from the original on March 15, 2023. Retrieved July 23, 2022.
  103. Wiblin, Robert (October 2, 2018). "Dr Paul Christiano on how OpenAI is developing real solutions to the 'AI alignment problem', and his vision of how humanity will progressively hand over decision-making to AI systems" (Podcast). 80,000 hours. No. 44. Archived from the original on December 14, 2022. Retrieved July 23, 2022.
  104. Lehman, Joel; Clune, Jeff; Misevic, Dusan; Adami, Christoph; Altenberg, Lee; Beaulieu, Julie; Bentley, Peter J.; Bernard, Samuel; Beslon, Guillaume; Bryson, David M.; Cheney, Nick (2020). "The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities". Artificial Life. 26 (2): 274–306. doi:10.1162/artl_a_00319. ISSN 1064-5462. PMID 32271631. S2CID 4519185. Archived from the original on October 10, 2022. Retrieved September 12, 2022.
  105. Hendrycks, Dan; Carlini, Nicholas; Schulman, John; Steinhardt, Jacob (June 16, 2022). "Unsolved Problems in ML Safety". p. 7. arXiv:2109.13916 [cs.LG].
  106. Leike, Jan; Krueger, David; Everitt, Tom; Martic, Miljan; Maini, Vishal; Legg, Shane (November 19, 2018). "Scalable agent alignment via reward modeling: a research direction". arXiv:1811.07871 [cs.LG].
  107. Wiggers, Kyle (September 23, 2021). "OpenAI unveils model that can summarize books of any length". VentureBeat. Archived from the original on July 23, 2022. Retrieved July 23, 2022.
  108. Moltzau, Alex (August 24, 2019). "Debating the AI Safety Debate". Towards Data Science. Archived from the original on October 13, 2022. Retrieved July 23, 2022.
  109. Wiggers, Kyle (September 20, 2021). "Falsehoods more likely with large language models". VentureBeat. Archived from the original on August 4, 2022. Retrieved July 23, 2022.
  110. The Guardian (September 8, 2020). "A robot wrote this entire article. Are you scared yet, human?". The Guardian. ISSN 0261-3077. Archived from the original on September 8, 2020. Retrieved July 23, 2022.
  111. Heaven, Will Douglas (July 20, 2020). "OpenAI's new language generator GPT-3 is shockingly good—and completely mindless". MIT Technology Review. Archived from the original on July 25, 2020. Retrieved July 23, 2022.
  112. Evans, Owain; Cotton-Barratt, Owen; Finnveden, Lukas; Bales, Adam; Balwit, Avital; Wills, Peter; Righetti, Luca; Saunders, William (October 13, 2021). "Truthful AI: Developing and governing AI that does not lie". arXiv:2110.06674 [cs.CY].
  113. Alford, Anthony (July 13, 2021). "EleutherAI Open-Sources Six Billion Parameter GPT-3 Clone GPT-J". InfoQ. Archived from the original on February 10, 2023. Retrieved July 23, 2022.
  114. Shuster, Kurt; Poff, Spencer; Chen, Moya; Kiela, Douwe; Weston, Jason (November 2021). "Retrieval Augmentation Reduces Hallucination in Conversation". Findings of the Association for Computational Linguistics: EMNLP 2021. EMNLP-Findings 2021. Punta Cana, Dominican Republic: Association for Computational Linguistics. pp. 3784–3803. doi:10.18653/v1/2021.findings-emnlp.320. Archived from the original on February 10, 2023. Retrieved July 23, 2022.
  115. Nakano, Reiichiro; Hilton, Jacob; Balaji, Suchir; Wu, Jeff; Ouyang, Long; Kim, Christina; Hesse, Christopher; Jain, Shantanu; Kosaraju, Vineet; Saunders, William; Jiang, Xu (June 1, 2022). "WebGPT: Browser-assisted question-answering with human feedback". arXiv:2112.09332 [cs.CL].
  116. Kumar, Nitish (December 23, 2021). "OpenAI Researchers Find Ways To More Accurately Answer Open-Ended Questions Using A Text-Based Web Browser". MarkTechPost. Archived from the original on February 10, 2023. Retrieved July 23, 2022.
  117. Menick, Jacob; Trebacz, Maja; Mikulik, Vladimir; Aslanides, John; Song, Francis; Chadwick, Martin; Glaese, Mia; Young, Susannah; Campbell-Gillingham, Lucy; Irving, Geoffrey; McAleese, Nat (March 21, 2022). "Teaching language models to support answers with verified quotes". DeepMind. arXiv:2203.11147. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  118. Askell, Amanda; Bai, Yuntao; Chen, Anna; Drain, Dawn; Ganguli, Deep; Henighan, Tom; Jones, Andy; Joseph, Nicholas; Mann, Ben; DasSarma, Nova; Elhage, Nelson (December 9, 2021). "A General Language Assistant as a Laboratory for Alignment". arXiv:2112.00861 [cs.CL].
  119. Kenton, Zachary; Everitt, Tom; Weidinger, Laura; Gabriel, Iason; Mikulik, Vladimir; Irving, Geoffrey (March 30, 2021). "Alignment of Language Agents". DeepMind Safety Research - Medium. Archived from the original on February 10, 2023. Retrieved July 23, 2022.
  120. Leike, Jan; Schulman, John; Wu, Jeffrey (August 24, 2022). "Our approach to alignment research". OpenAI. Archived from the original on February 15, 2023. Retrieved September 9, 2022.
  121. Ortega, Pedro A.; Maini, Vishal; DeepMind safety team (September 27, 2018). "Building safe artificial intelligence: specification, robustness, and assurance". Medium. Archived from the original on February 10, 2023. Retrieved August 26, 2022.
  122. Christian, Brian (2020). "Chapter 5: Shaping". The alignment problem: Machine learning and human values. W. W. Norton & Company. ISBN 978-0-393-86833-3. OCLC 1233266753. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  123. Zhang, Xiaoge; Chan, Felix T.S.; Yan, Chao; Bose, Indranil (2022). "Towards risk-aware artificial intelligence and machine learning systems: An overview". Decision Support Systems. 159: 113800. doi:10.1016/j.dss.2022.113800. S2CID 248585546. Archived from the original on February 10, 2023. Retrieved September 12, 2022.
  124. McCarthy, John; Minsky, Marvin L.; Rochester, Nathaniel; Shannon, Claude E. (December 15, 2006). "A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955". AI Magazine. 27 (4): 12. doi:10.1609/aimag.v27i4.1904. ISSN 2371-9621. S2CID 19439915. Archived from the original on January 31, 2023. Retrieved September 12, 2022.
  125. Baker, Bowen; Kanitscheider, Ingmar; Markov, Todor; Wu, Yi; Powell, Glenn; McGrew, Bob; Mordatch, Igor (September 17, 2019). "Emergent Tool Use from Multi-Agent Interaction". OpenAI. Archived from the original on September 25, 2022. Retrieved August 26, 2022.
  126. Shermer, Michael (March 1, 2017). "Artificial Intelligence Is Not a Threat—Yet". Scientific American. Archived from the original on December 1, 2017. Retrieved August 26, 2022.
  127. Everitt, Tom; Lea, Gary; Hutter, Marcus (May 21, 2018). "AGI Safety Literature Review". arXiv:1805.01109 [cs.AI].
  128. Demski, Abram; Garrabrant, Scott (October 6, 2020). "Embedded Agency". arXiv:1902.09469 [cs.AI].
  129. Everitt, Tom; Ortega, Pedro A.; Barnes, Elizabeth; Legg, Shane (September 6, 2019). "Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings". arXiv:1902.09980 [cs.AI].
  130. Krakovna, Victoria; Legg, Shane. "Specification gaming: the flip side of AI ingenuity". Deepmind. Archived from the original on January 26, 2021. Retrieved January 6, 2021.
  131. Cohen, Michael K.; Hutter, Marcus; Osborne, Michael A. (August 29, 2022). "Advanced artificial agents intervene in the provision of reward". AI Magazine. 43 (3): 282–293. doi:10.1002/aaai.12064. ISSN 0738-4602. S2CID 235489158. Archived from the original on February 10, 2023. Retrieved September 6, 2022.
  132. Wakefield, Jane (September 27, 2015). "Intelligent Machines: Do we really need to fear AI?". BBC News. Archived from the original on November 8, 2020. Retrieved February 9, 2021.
  133. Marcus, Gary; Davis, Ernest (September 6, 2019). "How to Build Artificial Intelligence We Can Trust". The New York Times. Archived from the original on September 22, 2020. Retrieved February 9, 2021.
  134. Sotala, Kaj; Yampolskiy, Roman (December 19, 2014). "Responses to catastrophic AGI risk: a survey". Physica Scripta. 90 (1): 018001. Bibcode:2015PhyS...90a8001S. doi:10.1088/0031-8949/90/1/018001.
  135. Secretary-General’s report on “Our Common Agenda” Archived February 16, 2023, at the Wayback Machine, 2021. Page 63: "[T]he Compact could also promote regulation of artificial intelligence to ensure that this is aligned with shared global values"
  136. PRC Ministry of Science and Technology. Ethical Norms for New Generation Artificial Intelligence Released, 2021. A translation Archived February 10, 2023, at the Wayback Machine by Center for Security and Emerging Technology
  137. Richardson, Tim (September 22, 2021). "UK publishes National Artificial Intelligence Strategy". The Register. Archived from the original on February 10, 2023. Retrieved November 14, 2021.
  138. "The government takes the long term risk of non-aligned Artificial General Intelligence, and the unforeseeable changes that it would mean for the UK and the world, seriously." (The National AI Strategy of the UK Archived February 10, 2023, at the Wayback Machine, 2021)
  139. The National AI Strategy of the UK Archived February 10, 2023, at the Wayback Machine, 2021 (actions 9 and 10 of the section "Pillar 3 - Governing AI Effectively")
  140. NSCAI Final Report (PDF). Washington, DC: The National Security Commission on Artificial Intelligence. 2021. Archived (PDF) from the original on February 15, 2023. Retrieved October 17, 2022.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.