arXiv:2507.18229 · econ.GN / cs.AI
Correcting structural and parametric biases in RL simulations of economic models
Applying Reinforcement Learning (RL) to economic modeling reveals a fundamental tension: standard RL agents do not behave as competitive price-takers. While macroeconomic models typically assume agents take aggregate conditions as given, a naive single-agent RL algorithm learns to internalize its impact on the environment, effectively exerting market power. We illustrate this divergence in a search-and-matching model with concave production, showing that a standard RL agent converges to a monopsonistic allocation rather than the competitive equilibrium. Furthermore, we show that RL's approach to time discounting distorts standard intertemporal economic trade-offs. To resolve these issues, we propose a Calibrated Mean-Field Reinforcement Learning framework that restricts the agent to optimize against an exogenous aggregate state to restore price-taking behavior, and recalibrates the RL reward structure to reflect true economic opportunity costs. By iteratively updating the aggregate state, our algorithm converges to a stationary fixed point that recovers the competitive equilibrium.
Canonical macroeconomic models rest on agents so small that they take aggregate conditions as given. A single-agent RL environment is a closed loop: whatever the agent does comes back to it through the state. Put an optimizer in that loop and it will find the feedback and exploit it. This is not a bug in the optimizer — it is the optimizer doing its job in a world that was never meant to be a closed loop.
What the theory says
Each firm is one of a continuum. Its own vacancy posting has no measurable effect on market tightness θ, so it optimizes taking θ — and therefore the wage — as fixed. This tightness-taker assumption is what makes the job-creation condition an equilibrium condition rather than a monopsonist's first-order condition.
What the simulation does
The RL agent's action sets the vacancy count; the environment computes θ = v/u from it and feeds back a wage. So ∂θ/∂v ≠ 0, and the agent learns it can hold vacancies back to depress the wage it pays on every worker it already has. It converges to monopsony.
A unit mass of risk-neutral workers, a continuum of identical firms, both discounting at rate r. Firms produce with labor alone under diminishing returns, post vacancies at cost c, and fill them with probability q(θ). Matches dissolve at rate λ. Everything the rest of the paper does is measured against this model's unique steady state.
Choose vacancies each period to maximize the present value of profits, subject to the law of motion for employment.
Functional forms. Production f(l)=A lα; matching q(θ)=a θ−φ; and a Nash-bargained wage in which the unemployment benefit is endogenous, b = ρw.
The condition everything turns on
From the Bellman equation and the envelope condition, a firm hires until the marginal surplus of a worker equals the expected discounted cost of finding one.
Two things are doing quiet work on the right-hand side. The (r+λ) capitalises the flow cost over the life of a job and charges the firm for the capital it tied up. And the whole condition presumes θ is beyond the firm's control. Each of the paper's two biases attacks one of these.
Table 1
Matched to the Japanese labor market. The Hosios condition fixes η = φ; targeting θ = 0.78 and a quarterly job-finding rate of 0.426 pins down a; and c is then determined residually so that the job-creation condition holds at the target.
| Symbol | Description | Value |
|---|---|---|
| A | Productivity | 1 |
| a | Matching efficiency | 0.471 |
| α | Parameter in the production function | 0.667 |
| λ | Separation rate | 0.0144 |
| η | Worker's bargaining power | 0.6 |
| φ | Matching elasticity | 0.6 |
| c | Vacancy cost | 0.268 |
| r | Interest rate | 0.01 |
| ρ | Unemployment replacement rate | 0.6 |
Table 2
Solving the seven equations of Appendix A at that calibration gives a unique competitive equilibrium. Every number the rest of this page is measured against comes from this column.
| Symbol | Variable | Value |
|---|---|---|
| l | Employment | 0.967 |
| u | Unemployment | 0.033 |
| q | Vacancy fill rate | 0.547 |
| w | Wage | 0.831 |
| v | Vacancies | 0.025 |
| θ | Market tightness | 0.78 |
| R* | Flow profit per period | 0.168 |
The two numbers that matter downstream are θ* = 0.78 and R* = 0.168. A correct simulation has to land on both; the naive one lands on neither.
Seven equations in {l, u, v, q, θ, w, b}: the job-creation condition, the bargained wage, the flow balance λl = qv, the matching function q = aθ−φ, the definition θ = v/u, the adding-up constraint l + u = 1, and the benefit rule b = ρw.
Flow balance and adding-up collapse to θ(l) = [λl / (a(1−l))]1/(1−φ), so the whole system reduces to a single monotone root-find in l. That reduction is what the 2×2 grid and the first-order-condition diagram further down run on: at the calibration above it returns l = 0.9674, u = 0.0326, q = 0.5460, w = 0.8306, v = 0.0255, θ = 0.7818, R* = 0.1678 — Table 2 to three decimals in every variable.
Differentiate the RL agent's steady-state objective with respect to its own vacancy posting and an extra term appears that has no business being there.
Set ∂θ/∂v = 0 and this collapses to the job-creation condition. Leave it non-zero and the numerator acquires a term the firm dislikes — posting one more vacancy raises θ, which raises the wage it must pay on every worker it already employs. The agent responds by holding vacancies back. That is monopsony, learned from scratch.
Economics capitalises the vacancy cost over the life of the job and charges the firm for the capital it tied up. RL discounts a per-period penalty by γ and stops there. The job separation rate enters the RL environment through the transition dynamics but never through the valuation of cost.
Economics
Separation hazard and opportunity cost of capital.
Naive RL
Separation hazard only. The capital is free.
The bias is worst where jobs are long-lived. Push λ toward zero — lifetime employment — and the RL agent's perceived hiring cost falls toward nothing, because the entire cost of a match is the capital locked up in it.
One correction per bias. Freeze the aggregate during training so the agent cannot see its own footprint, and rescale the cost so that a period of vacancy posting means what the economist means by it. Then let the aggregate catch up between training runs.
Structural correction
Reframe the economy as a mean-field game. The representative agent optimizes against a fixed field θ(k), so ∂θ/∂v = 0 by construction and price-taking is restored. Between iterations the field is recomputed from the population the policy implies. Atomism is imposed by the architecture rather than hoped for.
Parametric correction
Replace c in the reward with an effective cost that capitalises the flow over the job's expected life and prices the foregone return.
At the paper's calibration c = 0.268 becomes ceff = 0.454. This is not a numerical patch: it embeds a worldview in which capital has alternative uses.
Line 2 is the parametric correction; freezing the field in line 5 is the structural one. Neither touches the network architecture — the agent is an ordinary DDPG learner throughout.
Theorem 1 · Appendix C
Write Ψ(θ) = Φ(π*θ) for the composite of "best-respond to the field" and "aggregate the population back into a field". If the policy map is Lipschitz with constant L₁ and the aggregator with L₂, and L = L₁L₂ < 1, then Ψ is a contraction: a unique fixed point exists and the iterates converge at rate Lk by Banach.
The paper is careful here — verifying global Lipschitz bounds for a deep network is intractable, so it leans on DDPG's Polyak averaging to make the policy move gradually. The panel below shows how much that gradualism is actually doing.
Each outer iteration moves the field part of the way toward the agent's best response: θk+1 = θk + ω(Ψ(θk) − θk). In the paper's algorithm ω is implicit — it is how far a partially trained DDPG agent moves in one round. (It is a device of this page, not the paper's notation; η is reserved for the bargaining share of Table 1.) Near the fixed point the deviation obeys θk+1 − θ* = s(θk − θ*) with s = 1 + ω(Ψ′ − 1), and Theorem 1 asks for |s| < 1.
Below the stability threshold the loop contracts to θ*. Above it, the field overshoots and oscillates away.
What this panel adds to the paper. Solve the firm's problem exactly at each iteration and
the resulting map is far from a contraction: its slope at the fixed point is about −224 at this
calibration, and for any field below θ* the static best response is a corner — hire everyone,
which sends the implied tightness to infinity. Exact-best-response fictitious play does not merely converge
slowly here; it cannot be started. That is why the trajectory above is the linearised one, and why
the stability threshold is so small: ω ≲ 0.009.
None of this is a defect of the framework. It is a quantitative version of the caveat the paper itself
raises in Section 5.4 — L₁L₂ < 1 is delivered by the learning algorithm's
gradualism, not by the economics. A partially trained DDPG agent with bounded actions and Polyak-averaged
targets never takes the full step, which is precisely what makes the published iteration in Figure 2
converge smoothly where the idealised one would not.
Derived here from the paper's equations; not a figure from the paper.
The paper's central empirical claim is that the two fixes are not substitutes. Below, each of the four configurations is solved directly from its own first-order condition, at the calibration in Table 1.
How these are computed. Each cell root-solves the job-creation condition of its own configuration: the mean-field rows drop the ∂θ/∂v terms from the manipulator's first-order condition, and the calibrated columns use (r+λ)c in place of λc. Market clearing θ(l) = [λl/(a(1−l))]1/(1−φ) closes the system. Derived here from the paper's equations; the published figures below report what the trained agents actually did.
The downward-sloping curve is the marginal surplus of a worker as a function of market tightness. The four flatter curves are the marginal cost of hiring one, as each configuration perceives it. Where a cost curve crosses the surplus curve is where that configuration stops hiring.
The two monopsony curves sit far above the two price-taking ones, because the manipulator adds the wage it would pay on its whole existing workforce to the cost of one more hire. That vertical gap is the structural bias. The smaller gap within each pair — a factor of χ — is the parametric bias. Only the green curve crosses at θ*.
Finding 1
The naive agent behaves as a monopsonist, deviating sharply from the competitive equilibrium. Standard RL agents exploit their environment if not structurally constrained.
Finding 2
With both corrections the simulation converges to the theoretical benchmark, reproducing the target tightness and unemployment rate with high precision.
Finding 3
Structural correction alone over-hires through cost underestimation; parametric correction alone still manipulates the market. Only the joint application is consistent.
@misc{chen2025individual,
author = {Chen, Ruxin and Zhang, Zeqiang},
title = {From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models},
year = {2025},
eprint = {2507.18229},
archivePrefix= {arXiv},
primaryClass = {econ.GN},
doi = {10.48550/arXiv.2507.18229},
url = {https://arxiv.org/abs/2507.18229}
}