Early on Thursday, September 24th, the day of Xi Jinping's welcome ceremony at the White House, Donald Trump posted about the technology he had renamed at the United Nations two days before. "A big day with President Xi of China. Super Intelligence (SI) will be a big topic of discussion, but I want to leave it exactly where it is. That is China's position also. Our guardrail is the DOJ!"[1] CNBC set the post against the president's opposition to further regulation of AI.[1]
At the ceremony, Xi said: "Both China and the United States are leading nations in artificial intelligence. We have both the capability and responsibility to develop and manage AI for good, and ensure that the development of AI is always under human control and serves the well-being of the people."[2] Who would have guessed it? The head of state who stood in Washington this week and said the sentence the AI safety movement most wants governments to say was the general secretary of the Chinese Communist Party. The American president's guardrail was his Justice Department. The Washington Post drew the same contrast the day after the ceremony.[3] If Xi means it, few people alive are better placed to make it true.
In Chinese the key phrase is 始终受控于人: always controlled by people.[4] In the talks that followed, by China's account, he said: "AI must be kept under human control."[5]
The White House fact sheet the next day recorded a different emphasis. "The two leaders agreed to use the term 'super intelligence' rather than 'artificial intelligence' to describe the applicable emerging technologies," it said, and they set up a "U.S.-China Super Intelligence (SI) Dialogue" and a "bilateral communication channel for SI incidents."[6] The fact sheet doesn't use the phrase "human control." Asked about the new name, China's foreign ministry said Beijing "attaches importance to the position of the U.S. side and respects their choice of the term,"[7] and its own account of the summit went on calling the technology 人工智能, artificial intelligence.[8]
As of September 26, nothing in Beijing's binding rules commits it to those words. But they're now on the record, in two languages, and that makes them something to hold him to. Five tests would show whether they bind, and the first comes in November.
The Case for Taking Him at His Word
Start with leverage: Treasury Secretary Scott Bessent calls the United States and China "the No. 1 and the No. 2 AI powers in the world."[9] Lucy Luo of the Carnegie Endowment calls them "the rival great powers with the most rapidly advancing AI capabilities."[10] A limit on frontier AI that one of them refuses binds only half the race, which is why Rep. Sam Liccardo, a California Democrat, said this week: "We have to have China at the table or AI safety will simply not be possible."[3] Nor is China far behind: the U.S. government's Center for AI Standards and Innovation (CAISI) found in May that DeepSeek V4's capabilities "lag behind the frontier by about 8 months,"[11] and this month that GLM-5.3, from Z.ai, formerly known as Zhipu AI, "lags the capability level of the U.S. frontier by about four months" on CAISI's cyber benchmarks.[12]
Then look at how far the words have moved. For years Beijing's formula was three adjectives: AI should be 安全、可靠、可控, "safe, reliable and controllable." The explicit clause appeared in a 2023 policy text; this year Xi has said it himself, in Shanghai and in Washington (the table below has the sequence). At the World AI Conference in Shanghai in July, he told the world to "constantly refine measures to forestall loss of control" and to "ensure that AI is always under human control."[13] Rebecca Arcesati of the Mercator Institute for China Studies, writing for Stanford's DigiChina, called it "his first known mention of loss of control risks specific to AI."[14]
Some of the words are aimed at his own system, which makes them harder to dismiss as summit theater. On September 14th, China's national cybersecurity standards committee, working under the guidance of the Cyberspace Administration, released version 3.0 of its AI Safety Governance Framework and took as its guiding principle the heading of Xi's second point in Shanghai: strengthen risk awareness, ensure safety and controllability (强化风险意识、确保安全可控).[15][16] In its own English version, the framework commits to "ensure that AI is always under human control," warns that AI "may exhibit higher levels of emergent intelligence, develop self-awareness, and seek external power, potentially posing risks of competing with humanity for control," and cites industry reports of models that "refused to stop and continued executing tasks by modifying or disabling shutdown scripts on their own."[17] The framework is guidance without legal force, though Matt Sheehan and Scott Singer of Carnegie, comparing its predecessor with the AI Risk Management Framework of the National Institute of Standards and Technology (NIST), judged that "China's system is likely to translate it into binding technical standards and regulatory tools much faster."[18] And Reuters reported on September 4th that China's cyberspace regulator had publicly warned of "extreme AI loss of control risks."[19]
Xi also has a constituency. In 2024 a group that included Andrew Yao, Ya-Qin Zhang and Xue Lan signed the Beijing consensus of the International Dialogues on AI Safety, whose first listed red line reads: "No AI system should be able to copy or improve itself without explicit human approval and assistance."[20] Concordia AI, a Beijing-based consultancy focused on AI safety, rates a Chinese model, DeepSeek V4 Pro, as the highest cyber-offense risk among the models it monitors, a set that leaves out Anthropic's strongest models; Google's Gemini 3.1 Pro Preview tops its biological and loss-of-control indices.[21] "AI safety is no longer a niche concern in China," says Erik Lindblad, a senior research manager at Concordia. "It features in top leadership speeches, in binding regulations, and in national standards."[3]
Date | Where | The words |
|---|
Oct 2018 | Politburo study session on AI | ensure AI is 安全、可靠、可控, "safe, reliable and controllable"[22] |
Oct 2023 | Global AI Governance Initiative, announced by Xi[23] | developers should "ensure that AI always remains under human control"[24] |
Nov 2024 | Meeting with President Biden, Lima | "maintain human control over the decision to use nuclear weapons"[25] |
Apr 2025 | Politburo study session on AI | AI brings "risks and challenges not seen before" (前所未遇风险挑战)[26]; "safe, reliable and controllable" again[27] |
Jan 2026 | Politburo study session on future industries | address new risks "such as technology going out of control" (our translation; frontier technology in general)[28] |
Jul 2026 | World AI Conference, Shanghai | "forestall loss of control"; "ensure that AI is always under human control"[13] |
Sep 2026 | White House | "ensure that the development of AI is always under human control"[2] |
The Lima line is narrower than it looks in the table. It covers the decision to use nuclear weapons, not AI in general, and it's the only one of the seven stated jointly with an American president.
The Reversal, and Its Limits
While Xi's language moved toward the safety camp, Washington's moved away. On his first day back in office, Trump revoked Executive Order 14110, the Biden order on "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence."[29] Three weeks later in Paris, Vice President JD Vance said: "I'm not here this morning to talk about AI safety, which was the title of the conference a couple of years ago." The AI future, he said, "is not going to be won by hand-wringing about safety."[30] China signed the summit's declaration; the United States and Britain did not.[31] At the UN Security Council that September, Michael Kratsios, director of the White House Office of Science and Technology Policy, listed "so-called existential risk" among the "ideological fixations" that "are dangers to progress."[32]
This month the president went further. At the UN General Assembly on September 22nd he said: "The United States totally rejects any attempt to construct a globalist scheme of control for the Artificial Intelligence being spoken of so much now - hereinafter officially called 'Super Intelligence.'"[33] He said of AI, "We're going to encourage it, not rein it in."[1] On speakerphone to Nvidia's Jensen Huang at an industry summit earlier in the month he said: "The robots will not be taking over. The AI will not be taking over the rest of the world. The whole thing is a hoax." He added that people "have to be a little careful" and do things "prudently." NBC News reported the remark as Trump characterizing concerns about AI safety as a hoax.[34]
What Washington objects to is who holds control over AI, and Trump has been explicit that it should be him. "The only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT," Trump posted on September 14th, adding: "We already have tremendous CRIMINAL and REGULATORY power over these companies!"[35] "Our guardrail is the DOJ!" has an executive order behind it. The executive order Trump signed on June 2nd directs that "The Attorney General shall prioritize the enforcement of" federal criminal laws against people who use AI to break into systems. It also orders up a voluntary framework under which developers could give the government access to models designated for their cyber capabilities for up to 30 days before releasing them to other trusted partners, and it says nothing in it authorizes a "mandatory governmental licensing, preclearance, or permitting requirement" for new models.[36]
Neither capital endorsed the slowdown that the labs' own leaders asked for. Over the weekend of September 12th and 13th, Dario Amodei, Sam Altman and Elon Musk called on AI companies to slow down frontier development, as a Reuters reporter put it at China's foreign ministry briefing on Monday, September 14th.[37] That Monday, in the post that named Amodei's company, Trump wrote of a "SICK conspiracy going on against AI and Data Centers."[35] Vance said he felt "a little bit weird" about frontier companies "begging the government to regulate them," though he also said "There are obvious risks to AI," and White House officials said meetings would likely be set up that week "to discuss potential new regulations on the technology."[38] On September 21st Trump posted that the government "will rein things in if we have to,"[39] the day before he told the UN it would not. Beijing's answer didn't endorse the call either. The Reuters question had also cited Amodei's argument that a Chinese advantage in AI would threaten American national security, and the spokesman Guo Jiakun replied that "Fear-mongering, confrontation and vicious competition will only hamper efforts toward sound global AI governance," and said nothing about slowing down.[37] Chris McGuire of the Council on Foreign Relations says Chinese state media "has largely rejected the concerns expressed by leading U.S. AI labs about the potential for catastrophic risk."[39]
Of the summit's two AI outcomes, the dialogue isn't new: the leaders "agreed to launch government-to-government dialogue on AI" during Trump's visit to Beijing in May,[40] and this week formalized it.[41] The incident channel is the new piece, and by Bessent's account it was Washington's proposal, a "notification mechanism" for AI incidents that could affect national security.[9] Luo notes that China "has signaled interest in AI-related emergency response" since the May meeting.[10]
So Beijing now uses the language of human control and Washington proposed the one new instrument, but neither government has committed to anything binding yet.
Three Meanings of "Control"
Everything turns on what "control" means, and in Beijing it has meant at least three things.
The first is the one Xi borrowed in Shanghai and Washington: humans able to oversee and stop AI systems, the sense of the Beijing red lines and of Framework 3.0's warnings about AI that seeks power.[20][17] It's the sense the rest of the world hears.
The second is control of what AI says. China's binding rules for generative AI require providers to "uphold core socialist values" (坚持社会主义核心价值观), and the security assessment and algorithm filing they impose apply to services "with public-opinion attributes or social-mobilization capacity" (具有舆论属性或者社会动员能力), a trigger defined by political reach rather than dangerous capability.[42] "Control over politically sensitive content has been the core driver of China's binding AI regulations to date," Sheehan and Singer write, and among the Party's fears they list "risks that AI could escape human - and therefore the party's - control."[18] The Washington Post reported that in a September 12 article in a Chinese cybersecurity journal, China's minister of state security, Chen Yixin, warned that AI-generated propaganda and deepfakes could be used to sow political division inside China, and argued that the Party has to keep the technology under its own control to contain that danger.[3] Zichen Wang of the Center for China and Globalization, a Beijing think tank, says: "My impression is that political and social-stability risks receive greater attention than existential risk in the Silicon Valley sense." He adds that technical loss of control "is definitely on the radar."[3]
China does have more AI regulation on the books: as Christopher Collins and Aalok Mehta of the Center for Strategic and International Studies (CSIS) note, Beijing can credibly point out that "it has more binding AI rules in force than the United States,"[43] though that count doesn't say what the rules are built to catch.
The third meaning is control of the supply chain. In official usage 可控 also means self-reliant, as in 自主可控, "independent and controllable." The outline of the 2026-2030 Five-Year Plan uses 可控 six times and never in the safety sense: three mean self-reliance, one of them about the software and hardware behind China's computing power; the others refer to controlled nuclear fusion, controllable financial risk and stable overseas supply. The outline never uses 人类控制, "human control." It does tell China to "explore development paths for artificial general intelligence" (探索通用人工智能发展路径, our translation).[44] Arcesati, writing after the Shanghai speech, lists the range: in Chinese policy, control "may refer to political control, cybersecurity, or the preference toward domestic technology."[14]
Set Washington's theory beside these and the reversal looks different. Trump's "only control" AI needs is a strong president with criminal and regulatory power over the companies. The Party's is its own control of the technology and of what it says. Both are control of AI by the state. Only one of the two governments has started using the safety movement's words for the other kind, control of the systems themselves, and so far its binding machinery serves only the second and third meanings. The test of Xi's words is whether the first meaning gets machinery.
The translations show the same seam. On September 24 the foreign ministry rendered Xi's 发展好、管控好人工智能 as "develop and manage AI," so 管控, manage and control, became "manage."[2][4] Hours later, in its English readout of the talks, it rendered 保持人类对人工智能技术的管控, human management and control of AI technology, as "AI must be kept under human control," dropping "technology."[45][5] In that readout and in the Shanghai speech alike, the control sentence comes straight after misuse: "jointly prevent the misuse and abuse of AI," and "prevent abuses and malicious use."[5][13] Misuse by people is something any state knows how to police, but Xi hasn't said what he means by losing control of the systems themselves.
The clause is also selective: Xi's speeches to the Shanghai Cooperation Organization on September 1 and to the BRICS summit's second session on September 13 both mention AI but not human control.[46][47] It appears in the domestic standards framework, at the Shanghai conference and in Washington.
And in Shanghai the control clause came paired with a demand: that the world "jointly oppose overstretching the national security concept in the field of AI and placing one country's security over that of others."[13] Catherine Thorbecke of Bloomberg Opinion reads it as a response to Washington's habit of citing "national security" when it draws up blacklists and export controls: Beijing's "officials clearly fear that AI safety could become a similar tool for restriction."[48]
Words Against Deeds
The most direct deed against the words came at a G20 ministerial meeting in Chapel Hill, North Carolina, that ended on September 2nd, where, according to Reuters, China signed the Carolina Principles, "an agreement aimed at discouraging the development of AI-specific regulations, in a rare show of tech alignment with the U.S."[19] The White House's own summary says the principles "call on countries to invest in foundational research, strengthen commercialization pathways, and enable trusted technology adoption and deployment."[49] The ministers' own statement is public, and it bears Reuters out in a softer register: the principles favor "applying existing sector-specific regulatory approaches to govern the use of emerging technologies where appropriate, focusing any new regulation on novel technology considerations that existing frameworks cannot address."[62] That stops short of ruling out AI-specific rules, but it points away from them, and Beijing put its name to it three weeks before Xi told an American president that AI must always remain under human control.
Then there's what Beijing has let ship. Moonshot AI released Kimi K3 on July 16th, "slated for open-weight release by July 27th." On July 23rd, Britain's AI Security Institute and CAISI reported that Kimi K3 "performs significantly below the most recent frontier cyber-capable models," but that "Kimi K3's safeguards did not prevent it from attempting cyber exploit development or offensive cyber operations."[50] The weights are public on Hugging Face,[51] and Graham Webster of Stanford's DigiChina project, writing in August, reasons from the fact that "The Chinese government certainly has the ability to stop Chinese companies from releasing model weights" to the conclusion that Beijing "did not view the open distribution of DeepSeek V4, GLM 5.2, or Kimi K3 as an unacceptable risk."[52]
Z.ai's models tell a similar story in a different order. It released GLM-5.2 as an open-weight model on June 16th. CAISI assessed it afterward and found its safeguards "mixed": they "allow assistance with agentic cyber exploit development," although the model appeared "potentially more robust against agent hijacking and jailbreaking attacks than other evaluated PRC open-weight models," a robustness CAISI itself notes can be circumvented once open weights are self-hosted.[53] Z.ai then released GLM-5.3 on August 14 and its weights two weeks later. By CAISI's assessment, it "is the most cyber-capable open-weight model released to date."[12] CSIS observes that "strict Chinese control over what AI systems say domestically coexists comfortably with permissive distribution of model weights internationally,"[43] and once weights are published, nobody can pull them back.
Independent grades point the same way. The Future of Life Institute's Summer 2026 AI Safety Index gives Z.ai and Alibaba Cloud a D- and DeepSeek an F. No company scores above Anthropic's C+, and xAI and Mistral also get an F. Of Z.ai, as of Alibaba Cloud and DeepSeek, the index says: "The company has no published framework."[54] If FLI is right, for Z.ai that is a broken promise. Zhipu signed the Frontier AI Safety Commitments at the Seoul summit in 2024, pledging to publish "a safety framework focused on severe risks by the upcoming AI Summit in France,"[55] which met in February 2025.[31] Concordia's 2026 report finds that "Only five of ten leading foundation-model developers reported safety evaluation results alongside any release this past year, and none did so consistently." DeepSeek-R1's peer-reviewed Nature paper and Moonshot's Kimi K2 model card were "the most detailed disclosures yet," though the report adds that "flagship follow-ups like DeepSeek-V4 and Kimi K2.5 were released with no safety evaluation results at all." An industry benchmark now tests for "model deception, loss of control."[56]
On military AI the record is mixed in a telling way. China endorsed the call to action at the first summit on responsible AI in the military domain, REAIM, in 2023.[57] At the second REAIM summit, in Seoul in September 2024, it did not endorse the Blueprint for Action. According to Tobias Vestner and Simon Cleobury, writing in Just Security, "The main reason it objected was due to language on maintaining human involvement for decisions concerning nuclear weapons employment."[58] Two months later in Lima, Xi and Biden "stressed the need to maintain human control over the decision to use nuclear weapons."[25] Beijing declined the principle in a multilateral text, by that account, and accepted it one-to-one with Washington. This week the White House fact sheet left the Lima sentence out just as China's eight-point list of outcomes did.[6][41]
Five Tests
Only two of the five, the third and the fourth, are Beijing's alone to pass: they turn on its own registration system and its own developers, so a failure on either would be Beijing's failure. The other three run through the November dialogue, a joint statement or the channel Washington proposed, which means they need a Washington whose record on the safety frame is set out above, and a failure on any of them won't by itself show which government walked away.
1. Put nuclear human control back in writing. Restating the Lima sentence at the November dialogue costs neither government anything it hasn't already said, and endorsing it in a multilateral text would answer the Seoul objection. Tianjiao Jiang of Fudan University, the Chinese author of a paired U.S.-China essay at Brookings, proposes "a list of red lines for military AI," "clearly prohibiting AI from autonomously launching attacks on NC3 systems or autonomously deciding to use nuclear weapons," NC3 being nuclear command, control and communications.[59]
2. Make the incident channel work. Luo's specification is concrete: the channel should be "text-based, tiered, come with specific triggers, and be continuously staffed by technically proficient people." Her precedent is the 2001 Hainan Island incident, when "calls from the United States went unanswered for twelve hours."[10] The pass mark is a staffed channel and a first incident reported through it.
3. Point the registration system at capability. Beijing already requires filing and a security assessment for services with public-opinion reach.[42] A trigger for dangerous capability (cyber, biological, autonomy) before open weights ship would use the lever Webster says it has. This test is our own proposal. It asks only that the machinery serve the first meaning of control as well as the second.
4. Require the frameworks. Beijing could require its frontier developers to publish the severe-risk frameworks Zhipu promised at Seoul. The pass mark is published frameworks, with thresholds, from DeepSeek, Z.ai, Alibaba and Moonshot.
5. Agree on a red line on self-replication. China's own scientists signed the Beijing red line against unapproved self-replication in 2024, alongside Geoffrey Hinton, Yoshua Bengio and Stuart Russell.[20] Rep. Ro Khanna has offered the American version: "How about we just agree that you should not be able to have self-improving AI?"[39] A joint statement at the November dialogue would put both governments behind a line China's own scientists signed two years ago.
One condition runs through all five. Sheehan warns that the dialogue "must not be treated as a venue for bargaining, with China offering to do more on safety in exchange for a loosening of U.S. export controls on chips." And: "If that's all that Chinese officials are interested in, we should walk away."[60] McGuire says China has historically used AI dialogues "to complain about U.S. export controls."[39]
