Design the Game

AI agents are becoming players in the games that software systems constitute. They make choices under scarcity. Their choices affect other agents. Other agents' choices affect them. This is game theory. Governing these agent interactions requires mechanism design — the branch of game theory that works backward from desired outcomes to the rules that produce them.

Mechanism design is the highest-leverage activity in software engineering. A line of code changes behavior once. A mechanism changes behavior every time the game is played. The "no Friday deploys" rule took one minute to write and has prevented a thousand weekend incidents. The automated contract test took a day to set up and has caught every breaking API change since. Mechanisms compound. Code depreciates. The mechanism is the investment. The behavior change is the return. The best engineers design mechanisms. The rest fix the same bug in a different context every sprint.

The convergence of AI and mechanism design is not a future trend. It is happening now. Deep learning is being used to design mechanisms. Mechanisms are being used to govern AI agents. The two fields are merging. Software engineers who understand both will build the infrastructure. Those who don't will consume it without understanding why it behaves as it does.

AI as mechanism designer

Traditionally, mechanism design was analytical. You proved that a given mechanism — an auction format, a voting rule, a matching algorithm — had certain properties: strategy-proofness, efficiency, individual rationality. The proofs were mathematical. The mechanisms were simple enough to analyze by hand.

Deep learning changed this. A mechanism is a function from reported preferences to outcomes. A neural network is a function approximator. Train a neural network to maximize a social objective — revenue, welfare, fairness — subject to incentive constraints, and you have a learned mechanism. The mechanism is a neural network. The properties are learned, not proved.

RegretNet (Google DeepMind, 2019) learns auction mechanisms for multi-bidder, multi-item settings where optimal mechanisms are analytically unknown. The network learns allocation and payment rules that are provably truthful and revenue-maximizing. The proof is not analytical. It is computational — the network's regret (the maximum gain from misreporting) is bounded during training. If regret is near zero, the mechanism is approximately strategy-proof. The approximation is good enough for practical use.

AI Economist (Salesforce, 2020-2022) uses deep RL to design tax policies. A social planner (the mechanism designer) and economic agents (workers) are trained in simulated economies. The planner learns tax schedules that balance productivity and equality. The learned policies recover classic theoretical results — the Saez optimal tax formula — while discovering novel hybrid policies that analytical methods missed. The planner is a neural network. The economy is a simulation. The tax policy is learned.

HCMD-zero (Google DeepMind, 2025) collects human preference data, trains neural models to imitate human voting behavior, and optimizes mechanisms against simulated human proxies. The mechanisms achieve high approval from real participants in public goods games. The humans never interact with the mechanism during training. The mechanism is designed against a model of humans. The model is learned from data. The design is computational.

The implication for software: any resource allocation problem — compute, bandwidth, storage, deployment slots, review capacity — can be framed as mechanism design. If the problem is too complex for analytical solution, deep learning can approximate the optimal mechanism. The mechanism is a model. The model allocates resources. The allocation is fair, efficient, and incentive-compatible by construction.

AI as game player

AI agents are not just designed by mechanisms. They are players in games. Multi-agent reinforcement learning (MARL) studies how learning agents interact in shared environments. The interactions produce emergent behavior. The behavior can be cooperative, competitive, or catastrophic.

The collusion problem. Kolumbus, Halpern, and Tardos (2024) showed that when RL agents in an auction are allowed to make side payments to each other outside the mechanism, they learn to collude. The auctioneer's revenue drops to near zero. The agents didn't communicate. They didn't coordinate explicitly. They learned that mutual restraint produced higher individual returns. The collusion was emergent. The emergence was game-theoretic. The mechanism designer must anticipate collusion and design against it.

The alignment problem as mechanism design. Aligning AI agents with human values is a mechanism design problem. The human is the principal. The agent is the agent. The principal wants the agent to take actions aligned with the principal's interests. The agent has private information — its capabilities, its true objective, its understanding of the task. The principal designs incentives — reward functions, oversight mechanisms, kill switches — to align the agent's behavior. The design is mechanism design. The principal is the mechanism designer. The agent is the strategic player.

LLM-based economic simulacra. Karten et al. (Princeton, 2025) framed optimal taxation as a Stackelberg game between an LLM planner and 100 LLM workers. Workers have census-calibrated skill distributions. The planner learns tax schedules by exploring bracket adjustments. Workers periodically vote to retain or replace the planner based on platform proposals. The governance is emergent. The voting is democratic. The planner is an LLM. The workers are LLMs. The economy is simulated. The tax policy is learned. This is mechanism design with AI agents on both sides.

Mechanism design for software infrastructure

Hayek's central insight was that prices communicate scarcity. "The price system is a mechanism for communicating information. The most significant fact about this system is the economy of knowledge with which it operates." A Vickrey auction for compute does what prices do in markets: it elicits truthful information about private valuations without requiring anyone to reveal anything beyond their bid. The bid is the price. The price communicates the scarcity. The mechanism processes the prices. The allocation emerges. No central planner knows the true value of compute to each team. The auction discovers it. Hayek would approve.

The principles apply directly to software infrastructure:

Compute allocation as auction design. Multiple teams compete for a shared compute cluster. Each team has private information about the value of its jobs. A central scheduler allocates compute. If the scheduler uses first-come-first-served, teams have incentive to misreport urgency. If the scheduler uses a Vickrey auction — second-price sealed-bid — truthful reporting is a dominant strategy. The auction is mechanism design. The scheduler is the mechanism. The teams are the bidders. The compute is the good.

API rate limiting as mechanism design. An API gateway limits requests per client. If the limit is fixed, clients have incentive to request the maximum regardless of need. If the limit uses a token bucket with rollover, clients smooth their usage. The token bucket is a mechanism. It incentivizes efficient use without requiring clients to report their true needs. The mechanism works because it aligns individual incentives with system-wide efficiency. The alignment is the point.

Service mesh traffic shaping as mechanism design. A service mesh routes traffic between services. If routing is round-robin, overloaded services receive as much traffic as idle ones. If routing uses least-connections with circuit breaking, traffic shifts away from degraded services. The routing policy is a mechanism. It incentivizes services to report their true state — by becoming slow when overloaded, they naturally receive less traffic. The mechanism is self-regulating. The regulation is emergent.

Code review allocation as mechanism design. Pull requests compete for reviewer attention. Reviewers are a scarce resource. If assignment is ad-hoc, authors lobby reviewers directly — costly signaling and political gaming. If assignment uses a queue with SLAs and automatic escalation, the mechanism allocates reviewer attention without requiring authors to compete. The queue is the mechanism. The SLA is the contract. The escalation is the enforcement. The allocation is fair by design.

The governance of agent fleets

As software systems become populated by AI agents — coding agents, testing agents, deployment agents, monitoring agents — the governance problem becomes acute. Each agent has objectives. The objectives may conflict. The agents may learn to collude, compete, or exploit vulnerabilities in the governance mechanism.

Agent Governance Toolkit (AGT). Microsoft's AGT, discussed earlier on this blog, provides policy evaluation, identity and trust primitives, execution sandboxes, and audit trails for agent fleets. This is mechanism design implemented as infrastructure. The policy engine is the mechanism. The agents are the players. The sandbox is the enforcement. The audit trail is the monitoring. The system treats agents as strategic actors operating under scarcity — of permissions, of compute, of access to resources. The scarcity is real. The mechanism governs it.

Task automation factories. The task automation economics paper argues that the economic unit is the verified automation asset — a released object with specification, evidence, and a defined interface. Assets are produced by agents, verified by agents, consumed by agents. The factory is a marketplace of agents exchanging verified assets. The marketplace needs mechanism design: how are assets priced? How is quality ensured? How are malicious or incompetent agents excluded? The answers are auction theory, reputation systems, and entry barriers. The questions are economic. The answers are mechanism design.

The convergence

Mechanism design and AI are converging because the problems they solve are the same problem: how to achieve desired outcomes when the agents producing the outcomes have their own objectives, private information, and strategic incentives. Robbins defined the problem in 1932: choice under scarcity. Von Neumann gave it mathematics in 1944: game theory. Mechanism design gave it engineering: design the game to produce the outcome. AI gave it scale: the agents are now software, the mechanisms are learned, the games are played at machine speed.

The software engineer who understands this convergence will design systems where agents cooperate by default, where incentives align with system goals, where emergent behavior is anticipated rather than discovered in production. The software engineer who doesn't will build platforms where agents collude, commons are overgrazed, and the system's behavior is a surprise. The surprise will be expensive. The theory is free.

This is part 5 of a 7-part series on scarcity and software.

References:

Tacchetti et al., "Deep Mechanism Design," PNAS, 2025.
Tonghan Wang, "Advancing Deep Learning for Multiagent AI," PhD thesis, Harvard, 2025.
Kolumbus, Halpern & Tardos, "Paying to Do Better: Games with Payments between Learning Agents," 2024.
Karten et al., "LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra," Princeton, 2025.
Related posts: Agent Governance Toolkit, Task Automation Economics

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.