#25 Society and Technology Bulletin

Artificial intelligence debates have long revolved around superintelligence, labor markets, regulations, or security concerns. Among these discussions, a topic that has received less attention especially from industry figures is the question of artificial intelligence and consciousness. The issue highlighted by Microsoft executive Mustafa Suleyman in his recent bulletin falls precisely within this dimension: “Seemingly Conscious AI” (SCAI), or Seemingly Conscious Artificial Intelligence. This issue mirrors the stance of a nearly institutionalized camp within both our Society and Technology bulletins and broader AI discourse.

According to this perspective, the popular conceptualization of “conscious AI” contains a subtle but significant error. From the standpoint of technical and practical progress, whether AI has truly become conscious does not appear to be the primary concern. When we think functionally, what matters is not whether AI is conscious but whether it is functional. Yet when we turn to examining how AI impacts our daily lives, progress becomes impossible without addressing this issue. There is a crucial difference between assuming AI is conscious and assuming it is not.

Attributing intelligence and consciousness to machines, starting from the human model, has almost become a reflex. Yet this reflex also exposes us to the risk of projecting onto machines many attributes traditionally associated with humans. Although every new model released to market claims to be approaching organic intelligence, what is actually achieved is a flawless imitation of organic intelligence.

Suleyman argues that within a short time, AI systems will be developed that behave as if they are conscious, feel emotions, remember past interactions, and possess a personality. According to him, these systems will not be truly conscious, but they will simulate the human mind so convincingly that perceiving them as conscious entities will become nearly inevitable.

Human nature inclines us to search for consciousness in whatever entity we encounter. Whether interacting with a person, an animal, or even an object, we tend to perceive them as subjects. We observe the same behavior when dealing with language models: we greet them, check on their well-being, and treat them with many human-like attitudes.

Suleyman labels the danger this process may bring as the “psychosis risk”: individuals coming to believe in a consciousness that does not exist, forming emotional bonds with AI, and ultimately demanding AI rights, AI welfare, and even AI citizenship.

This risk is not confined to individuals with fragile mental health. On the contrary, it is a widespread illusion emerging from the daily interactions of millions of users. Even today, many users report forming deep emotional connections during conversations with large language models. Some perceive them as divine beings, fictional characters, or real friends.

Suleyman believes the components necessary to build Seemingly Conscious AI already exist or can be developed in the near future. These include the ability to generate emotionally resonant, persuasive, and fluent communication; to develop distinctive personalities through training and guidance; to remember past conversations and reference them; and to make claims of subjective experience, self-awareness, and autonomy.

The problem here is that the concept of consciousness sits at the heart of modern civilization. Rights, freedoms, and legal systems largely rest on the assumption that humans are conscious beings. If people come to believe AI is conscious, then these systems must be granted the same rights.

Suleyman’s warning becomes concrete here. These debates may appear as purely scientific matters, leading to the neglect of their social risks. Yet a conscious AI poses a societal risk—not because machines might become rights-holders, but because humans might lose their rights.

For this reason, Suleyman says “we must build AI for humans, not to make it a digital person”. As humanity prepares to confront machines that behave as if conscious, what must be done is not to humanize them, but to ensure they remain tools.

Nvidia has long been at the center of geopolitical bargaining among major powers. U.S. President Trump recently announced permission for Nvidia and AMD to sell certain AI chips to China. However, this authorization is not based on conventional export licenses or regulatory frameworks, but on a direct revenue-sharing arrangement. Under the agreement, companies will transfer 15 percent of the revenue generated from chip sales to China to the U.S. government.

This decision marks a significant departure from Washington’s policies in recent years. Recall that since 2022, the U.S. imposed strict export controls to restrict China’s access to advanced semiconductors, aiming to prevent these technologies from enhancing China’s military capabilities. During this period, Nvidia developed a specially slowed-down chip called H20 for the Chinese market. However, in April, the Trump administration suddenly halted exports of the H20 by imposing licensing requirements, blocking approximately $2.5 billion in revenue.

At this point, Nvidia CEO Jensen Huang intervened. Huang announced that Nvidia would invest $50 billion within the U.S. and launched an intensive lobbying campaign. Ultimately, after negotiations with Trump in the Oval Office, permission was granted to resume H20 exports to China—but only on the condition of revenue sharing. Trump stated he initially demanded 20 percent, while Huang negotiated it down to 15 percent.

The arrangement offers Nvidia a chance to return to the Chinese market. For the U.S., it provides short-term revenue and preserves corporate competitiveness. Yet this “pay-to-play” model creates the impression that export controls can be determined not by principles but by negotiation. Both Republican and Democratic members of Congress have expressed concern, as the idea that geopolitical concerns can be eased for a fee sends a message to U.S. allies that security is negotiable.

Reactions in China are mixed. Beijing has advised companies not to use H20 chips in national security-related projects, citing fears that U.S. intelligence could access backdoors in the chips. Even if Chinese firms continue purchasing Nvidia chips, state pressure may lead them to reduce orders or favor domestic alternatives. Nevertheless, experts emphasize that even the slowed-down versions of U.S. chips remain world-class, meaning demand will not vanish entirely.

Another dimension of the deal involves future uncertainty. Trump currently limits the arrangement to the H20 and AMD’s MI308 chips. Yet he signals that more advanced chips could be sold under a similar “revenue share” model. He has stated that Nvidia’s most advanced Blackwell series chips will not yet be permitted, describing them as “too new and strategic.”

Some observers see this as a rational short-term solution: the U.S. preserves its presence in the Chinese market while securing direct revenue. Yet in the long term, treating security policies as negotiable commodities risks undermining U.S. credibility and normative power internationally, eroding trust among allies and projecting weakness to rivals.

Chips have acquired a strategic status comparable to energy or oil in earlier eras. Trump’s latest deal reveals that this strategic resource can also function as a bargaining chip—and even as a direct “trade good.”

If the Trump administration continues this approach, other companies may join the queue for similar arrangements. Yet it remains unclear whether this serves America’s long-term interests or creates new risks by compromising security principles for economic gain. One thing is clear: the agreement with Nvidia marks the beginning of a new and contentious phase in the U.S.-China technology war. The issue is not merely granting one company export permission; it is about how Washington is redefining its security principles in light of economic interests. While this approach may appear profitable in the short term, it opens the door to a transformation that could profoundly destabilize international balances and erode perceptions of U.S. reliability.