How the US and China can cooperate to reduce urgent AI risks

In the aftermath of the recent meeting between Presidents Donald Trump and Xi Jinping, U.S. Treasury Secretary Scott Bessent said that China and the U.S. would “set up a protocol” regarding “best practices for AI to make sure nonstate actors don’t get a hold of these models.” On May 19, China’s Foreign Ministry Spokesperson Guo Jiakun confirmed the two leaders agreed to conduct an “intergovernmental dialogue” on AI.

This move, which Bessent called “wholesome discussions” on AI, is a significant outcome of the summit, despite the lack of attention. As we urged in a recent commentary, the Trump-Xi meeting was an opportunity to restart a government-to-government dialogue on AI risk reduction, and the two nations seized the chance. The Trump administration now has something to bring to the table with the signing of a new executive order establishing a framework for AI companies to work with the government in managing risks inherent in AI models, although the recent Mythos–Fable debacle is a setback to stable U.S. AI governance.

Now the question is how to proceed with this dialogue. Former Biden administration official Chris McGuire, for instance, writes that a goal should include “agreements related to AI safety that would likely place some constraints on China’s development of AI” and “modest constraints on U.S. AI capabilities.” But this would be to approach the AI risk reduction dialogue as if it were a negotiation aiming for a mutual commitment, like an arms control deal, that would limit or regulate the development and deployment of AI models. 

Industry analyst Paul Triolo recognizes that the dialogue will not be about arms control, but he thinks it could set up a “crisis management mechanism” in which the two governments “define red lines around AI-enabled cyber operations, biosecurity, autonomous military escalation, model theft, and third-party/non-state actor use.” This too exceeds what can be reasonably hoped for in the next phase of these discussions.

Much more modest expectations are in order. The dialogue framework should provide ongoing ties between specialized technical experts working in key government agencies in both countries, involving frequent interaction, structured dialogues, and regular meetings. The exchange should not be a negotiation where one side gives something up in return for the other side taking certain actions; rather, it should be a regular process of sharing information, ideas, plans, and research in the area of AI model risk assessment and mitigation. Red lines and release restrictions might be part of the dialogue eventually, but only after a gradual process of confidence-building restores a minimum level of trust between the parties. If the parties want to pursue international regulatory cooperation on AI or further dialogue on export controls, these initiatives should proceed separately.

This raises the question of what should be discussed in these bilateral conversations. AI researchers Christina Knight and Scott Singer suggest that the twin topics should establish “a shared understanding of risk” and “best practices for reducing the risks of AI models.” We agree. Each side should feel free to share the approaches and tools they have developed to deal with a set of narrowly defined AI model risks that can be approached as technical problems. An initial short list of model risks might include enabling offensive cybersecurity threats; easing the creation or deployment of biological, chemical, or nuclear weapons; and producing reliability challenges such as hallucinations, defective code, and mistaken recommendations that could result in catastrophic harm to critical infrastructure. A key exchange would concern testing protocols, red teaming, and other techniques and strategies for assessing these risks prior to model release. Another area ripe for discussion and the exchange of best practices would be mitigation measures intended to reduce these risks, including alignment, and strategies used to ensure that when AI models are made available to the public, they do what their users intend them to do. It could also include control techniques that keep AI models within safe environments and harden attack surfaces.

Matt Sheehan, a senior fellow at the Carnegie Endowment, has usefully phrased the right discussion framework as “parallel” discussions and actions. This approach would rule out one side adopting testing, implementing safety protocols, or restraining AI development only because the other side agrees to do it as well or makes some other concession. The dialogue is not a negotiation at all. Rather, both sides would share their perceptions, evaluations, and mitigation techniques for specific risks—their best practices for identifying, assessing, and addressing a concrete problem that both sides recognize. The dialogue would be ongoing, so each side would receive regular updates on how things are going, perhaps with a commitment to reach out outside of these scheduled exchanges when an extraordinary or urgent development takes place.

Mutual understandings might come out of these discussions that stop short of formal commitments. In 2024, then-President Joe Biden and Xi both affirmed the need to “maintain human control over the decision to use nuclear weapons.” But this mutual declaration did not amount to a formal agreement, nor was one side’s decision to keep humans in control of nuclear weapons dependent on the other side’s acceptance of a similar constraint. What was negotiated was the timing and context of the announcement, not the adherence to the policy. Rather, each side thought it was a good idea to exercise this restraint on the role of AI for their own, or perhaps similar, reasons and said so in the context of a meeting between their country’s leaders. That might be a model for any understandings that come out of the U.S. dialogue.

A number of more specific issues also need to be addressed to set the discussions on the right path, including the nature of the participants, the approach to open-source models, the question of China’s access to U.S. AI models, the dialogue’s relationship to AI chip export controls, and the expansion to international cooperation.

A U.S.–China AI meeting in Geneva in May 2024 focused on AI military applications, but it was a mismatch. The U.S. sent a team of technical experts headed by White House Special Assistant Tarun Chhabra and Department of State official Seth Center while the PRC delegation included political and foreign policy officials. As a result, the discussions veered into political and trade concerns rather than staying focused on the key issues of AI risks and countermeasures.

This mismatch could happen again. It has been rumored that China is thinking of sending Politburo Standing Committee member Cai Qi, giving the first meeting a political tinge rather than an aspect of a narrow technical exchange. This might provide some political authority on the Chinese side, but as Triolo points out “that only helps if both sides bring real cyber, AI, intelligence, and lab-level expertise into the room.” The U.S. should be sure to match any political weight the Chinese bring to the table by sending appropriately authorized senior officials, but they should also send qualified technical experts from key government agencies.

Discussion of measures to improve the security of open-source AI models would make sense in the context of these technical exchanges. As the Biden administration noted in a July 2024 report, users can download open-source models to modify and use them without the knowledge or control of the model developer, which makes it difficult to maintain the risk reduction controls the original developer built into them. Security and successful alignment for these modes is an unsolved technical problem and would benefit from a collaborative approach.

But it would be a mistake for the U.S. to use AI risk reduction talks to pressure China to abandon or restrict its embrace of open-source models. China’s models are almost all open source, and they have focused on this development path to embed their models widely in domestic and international tech infrastructure. China would see such pressure as a backdoor way to use risk reduction talks to restrain its national and international AI competitiveness.

It would also run counter to current U.S. policy toward its own open-source models. The U.S. has no restrictions on open-source models as was recommended in the July 2024 report. Indeed, in its July 2025 AI Action Plan, the current administration encouraged them as having great “geostrategic value” and said it wanted to “create a supportive environment for open models.”

The turning point for the movement to restart bilateral AI risk reduction talks was Anthropic’s April 7 announcement that its new AI model, Claude Mythos, had found previously unknown vulnerabilities in software and that the company would not release the model publicly. It set up Project Glasswing, a collaborative framework—which has since expanded to 150 organizations across 15 countries—to help participants harden their systems against attacks newly made possible by Mythos. No Chinese institutions were included in Project Glasswing and Anthropic CEO Dario Amodei refused a request from a Chinese think tank to provide access to Mythos. 

On June 9, Anthropic released Claude Fable 5, a version of Mythos with safeguards to block sensitive cybersecurity and biology queries. And then on June 12, the U.S. government sent Anthropic an export control order restricting foreign access to Fable and Mythos, and in response, Anthropic withdrew the model from public availability. The U.S. government has since lifted export controls on Fable 5, but continues to restrict the more capable Mythos 5 to approved U.S. users.

While it might be useful for Chinese institutions to have access to the latest U.S. models for defensive cyber security purposes, such a move is now “politically radioactive” on the U.S. side, as Triolo put it, and should not even be part of the agenda for these technical discussions. 

Export controls should be kept separate from these ongoing AI risk reduction talks. This forum for cooperation to address AI risks is independent of the question of export controls. Many think export controls need to be tightened, including members of Congress who are seeking to legislate tough new limits on what can be sold to China and who can use U.S. technology in third-country data centers. But these China hawks can still embrace government-to-government cooperation to improve common AI hygiene. Former Biden administration official Chris McGuire, who argues that the U.S. needs to ratchet up tech export controls to maintain the maximum lead over China, agrees that even export control escalation would not “preclude co-operation with China on AI safety.” He also wants the AI risk reduction dialogue to be “narrowly focused on AI safety issues and not cover export controls.”

We agree that export controls are not at stake in, and should not be part of, AI risk reduction talks. But at the same time, these safety discussions should not be leveraged to achieve an aim like export control, which is to limit the capabilities of Chinese AI models.

Over time these bilateral discussions could morph into a more structured global institution, perhaps modeled after what the Organization for Economic Co-operation and Development (OECD) calls trans-governmental networks. These networks provide ongoing ties among specialized regulators developed through frequent interaction, structured dialogues, and regular meetings.

In November 2024, the International Network of AI Safety Institutes was launched with a mission to advance and share research on AI risks and capabilities and to build best practices for testing advanced AI systems and facilitate shared approaches. Initial members included the AI safety institutes and similar government offices from a broad range of countries, unfortunately excluding China.

While these institutions and initiatives are instructive models for expanding the U.S.-China dialogue to include other countries, the structure and functioning of any global AI risk mitigation forum should remain rigidly non-regulatory. The Chinese proposal for a “world AI cooperative organization” and the idea floated at the recent G7 meeting for a U.S.-led coalition to shape AI rules should proceed, if they do at all, on separate tracks.

AI’s benefits are only beginning to be felt in the U.S. and China and indeed around the world. But the message from Claude Mythos-Fable incident is that international cooperation is needed to control AI’s downside risks. The administration successfully seized the moment of the Trump–Xi meeting to start a process for ongoing government dialogue on this urgent problem. Now the next step is moving forward with the right discussion framework.

The goal should not be arms control or foreign policy; the parties should not aim at a binding treaty, or even shared norms. The U.S. should not even be contemplating a transactional quid pro quo, where, for instance, it would allow China access to more powerful chips in return for China doing something to constrain their open-source models. The focus instead should be on achieving a common understanding on a short, manageable list of AI risks to be controlled and on technical mitigation strategies to do the job.